# Live Traffic Scoring
Relevant source files The following files were used as context for generating this wiki page: * [frontend/components/settings/SystemPromptsTab.tsx](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/frontend/components/settings/SystemPromptsTab.tsx) * [orchestrator/core/services/futureagi\_service.py](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/orchestrator/core/services/futureagi_service.py) * [services/agent-opt-worker/Dockerfile](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/agent-opt-worker/Dockerfile/README.md) * [services/agent-opt-worker/automatos\_logging.py](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/agent-opt-worker/automatos_logging.py) * [services/agent-opt-worker/automatos\_metrics.py](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/agent-opt-worker/automatos_metrics.py) * [services/agent-opt-worker/main.py](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/agent-opt-worker/main.py) * [services/agent-opt-worker/requirements.txt](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/agent-opt-worker/requirements.txt) * [services/shared/automatos\_logging.py](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/shared/automatos_logging.py) * [services/shared/automatos\_metrics.py](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/shared/automatos_metrics.py) * [services/workspace-worker/automatos\_logging.py](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/workspace-worker/automatos_logging.py) * [services/workspace-worker/automatos\_metrics.py](https://github.com/AutomatosAI/automatos-ai/blob/main/docs/prompt-optimization/services/workspace-worker/automatos_metrics.py)
**Purpose**: This page documents the FutureAGI live traffic scoring system, which automatically evaluates every chat interaction against quality metrics in real-time. This feature builds a dataset of scored input/output pairs that powers prompt optimization and provides continuous quality monitoring. For information about the broader prompt optimization system, see **15.4 Prompt Optimization**. For details on the worker service that executes scoring, see **15.5 Agent-Opt Worker Service**. For prompt management and versioning, see **15.1 System Prompt Management**. *** ## Overview Live traffic scoring is a fire-and-forget evaluation system that runs after each chat response is generated. The `FutureAGIService.eval_live_traffic()` method is invoked asynchronously from the chat pipeline and performs the following steps: 1. Captures user input and assistant output from real conversations. [orchestrator/core/services/futureagi\_service.py:234-245](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) 2. Queries all `SystemPrompt` records where `futureagi_eval_enabled=True`. [orchestrator/core/services/futureagi\_service.py:251-255](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) 3. Sends input/output pairs to the `agent-opt-worker` service `/score` endpoint. [orchestrator/core/services/futureagi\_service.py:265-272](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) 4. Receives scores for quality metrics (completeness, helpfulness, conciseness). [services/agent-opt-worker/main.py:303-331](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) 5. Creates `SystemPromptEvalRun` records with `run_type='live'` for each enabled prompt. [orchestrator/core/services/futureagi\_service.py:277-296](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) 6. Builds a dataset for future prompt optimization via `_collect_optimization_dataset()`. [orchestrator/core/services/futureagi\_service.py:308-344](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) The system is designed to have **zero impact** on user-facing latency—scoring happens asynchronously after the response is already delivered. If the worker is unavailable or scoring fails, errors are logged but never propagated. [orchestrator/core/services/futureagi\_service.py:299-302](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Sources**: [orchestrator/core/services/futureagi\_service.py:234-344](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:303-331](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## System Architecture **Diagram: Live Traffic Scoring Data Flow** ```mermaid graph TB subgraph "Chat Response Pipeline" StreamingChat["StreamingChatService
streaming response"] ResponseComplete["Response delivered
to user"] end subgraph "Orchestrator: FutureAGIService" EvalLive["FutureAGIService.eval_live_traffic()
input_text, output_text, context_text"] CheckAvailable["is_available property
check config.AGENT_OPT_WORKER_URL"] ExtractText["_extract_text()
parse message parts"] QueryPrompts["db.query(SystemPrompt)
.filter(futureagi_eval_enabled==True)"] CallWorker["_call_worker('/score', payload)
timeout=120s"] CreateRuns["SystemPromptEvalRun
run_type='live', status='completed'"] end subgraph "Worker Service: main.py" ScoreEndpoint["@app.post('/score')
ScoreRequest schema"] BuildInputs["_build_inputs()
per template requirements"] ThreadPool["ThreadPoolExecutor
max_workers=len(metrics)"] RunTemplate["_run_single_template()
template, inputs, model"] SDKEval["fi.evals.Evaluator
.evaluate()"] end subgraph "Database: PostgreSQL" SystemPromptTable[("system_prompts
futureagi_eval_enabled BOOLEAN")] EvalRunsTable[("system_prompt_eval_runs
run_type VARCHAR
scores JSONB")] end StreamingChat --> ResponseComplete ResponseComplete --> EvalLive EvalLive --> CheckAvailable CheckAvailable -->|"Available"| ExtractText CheckAvailable -->|"Not available"| Return["return early"] ExtractText --> QueryPrompts QueryPrompts --> SystemPromptTable QueryPrompts -->|"Found prompts"| CallWorker CallWorker --> ScoreEndpoint ScoreEndpoint --> BuildInputs BuildInputs --> ThreadPool ThreadPool --> RunTemplate RunTemplate --> SDKEval SDKEval --> RunTemplate RunTemplate --> ThreadPool ThreadPool --> ScoreEndpoint ScoreEndpoint --> CallWorker CallWorker --> CreateRuns CreateRuns --> EvalRunsTable ``` **Sources**: [orchestrator/core/services/futureagi\_service.py:45-70](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [orchestrator/core/services/futureagi\_service.py:234-301](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:10-16](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:298-331](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## Enabling Live Scoring Live traffic scoring is controlled by the `futureagi_eval_enabled` flag on the `SystemPrompt` model. [frontend/components/settings/SystemPromptsTab.tsx:53](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Database Model | Field | Type | Description | | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `futureagi_eval_enabled` | Boolean | When `true`, every chat interaction scores this prompt [frontend/components/settings/SystemPromptsTab.tsx:53](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | | `slug` | String | Unique identifier (e.g., "chatbot-friendly") [frontend/components/settings/SystemPromptsTab.tsx:47](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | | `category` | String | Grouping (personality, orchestrator, specialized) [frontend/components/settings/SystemPromptsTab.tsx:49](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | ### Frontend Toggle The `SystemPromptsTab` admin UI includes a toggle switch for each prompt to enable or disable FutureAGI evaluation. [frontend/components/settings/SystemPromptsTab.tsx:554-572](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ```mermaid sequenceDiagram participant Admin as "Admin User" participant UI as "SystemPromptsTab.tsx
toggleFutureAGI()" participant API as "/api/admin/prompts/{id}
/futureagi-toggle" participant DB as "PostgreSQL" Admin->>UI: "Click toggle switch" UI->>API: "PATCH request" API->>DB: "UPDATE system_prompts
SET futureagi_eval_enabled = NOT ..." DB-->>API: "Updated record" API-->>UI: "PromptResponse" UI-->>Admin: "Toggle state updated" ``` **Sources**: [frontend/components/settings/SystemPromptsTab.tsx:52-59](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [frontend/components/settings/SystemPromptsTab.tsx:249-259](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [frontend/components/settings/SystemPromptsTab.tsx:554-572](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## Scoring Flow ### Invocation The `eval_live_traffic()` method is called from the chat streaming pipeline after a response completes. [orchestrator/core/services/futureagi\_service.py:234-245](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ```mermaid graph LR subgraph "Trigger Point" ChatComplete["Chat Response
Fully Streamed"] end subgraph "FutureAGIService.eval_live_traffic()" CheckAvailable["Check is_available
property"] ExtractText["Extract plain text
from message parts"] QueryEnabled["Query enabled prompts
futureagi_eval_enabled=true"] BuildPayload["Build scoring payload
input, output, metrics"] CallWorker["HTTP POST to worker
/score endpoint"] ParseScores["Parse scores dict"] StoreRuns["Create SystemPromptEvalRun
per enabled prompt"] end ChatComplete --> CheckAvailable CheckAvailable -->|"Available"| ExtractText CheckAvailable -->|"Not available"| Return["Return early"] ExtractText --> QueryEnabled QueryEnabled -->|"No enabled prompts"| Return QueryEnabled -->|"Found prompts"| BuildPayload BuildPayload --> CallWorker CallWorker --> ParseScores ParseScores --> StoreRuns ``` **Sources**: [orchestrator/core/services/futureagi\_service.py:234-301](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Text Extraction Messages are stored as JSON arrays of parts. The `_extract_text()` helper extracts plain text from these complex structures by iterating through parts and filtering for `type: "text"`. [orchestrator/core/services/futureagi\_service.py:30-42](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Worker Communication The orchestrator's `_call_worker()` method wraps HTTP communication with error handling, connection retries, and timeouts using `httpx.AsyncClient`. [orchestrator/core/services/futureagi\_service.py:79-98](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Request Schema** (`ScoreRequest`): [services/agent-opt-worker/main.py:177-181](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | Field | Type | Required | Description | | -------------- | ------------- | -------- | ---------------------------------------------------------- | | `input_text` | String | Yes | User's input message | | `output_text` | String | Yes | Assistant's response | | `context_text` | String | No | Additional context | | `metrics` | List\[String] | No | Defaults to `["completeness", "is_helpful", "is_concise"]` | **Sources**: [orchestrator/core/services/futureagi\_service.py:79-98](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [orchestrator/core/services/futureagi\_service.py:258-272](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:177-181](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:303-331](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## Metrics and Templates ### Default Metrics Live scoring uses three core quality metrics by default. [orchestrator/core/services/futureagi\_service.py:232](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | Metric | Template | Model | Description | | -------------- | -------------- | -------------- | ---------------------------------------- | | `completeness` | Quality metric | `turing_large` | Does the output fully address the input? | | `is_helpful` | Quality metric | `turing_large` | Is the response genuinely useful? | | `is_concise` | Quality metric | `turing_large` | Is it brief without losing clarity? | **Sources**: [orchestrator/core/services/futureagi\_service.py:232](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:129-141](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Scoring Engine **Diagram: Worker \_run\_single\_template() Execution** ```mermaid graph TD Start["_run_single_template(template, inputs, model)"] GetKeys["_get_keys()
os.getenv('FUTUREAGI_API_KEY')
os.getenv('FUTUREAGI_SECRET_KEY')"] SetEnv["os.environ.setdefault()
FI_API_KEY, FI_SECRET_KEY"] CreateEval["fi.evals.Evaluator(fi_api_key, fi_secret_key)"] CallSDK["evaluator.evaluate(
eval_templates=template,
inputs=inputs,
model_name=model,
timeout=55)"] ParseResult["Parse result.eval_results[0]
output, reason, score attrs"] ExtractScore["Convert output to score:
- 'Passed' → 1.0
- 'Failed' → 0.0
- float → use directly"] Return["Return {score, passed, reason}"] Start --> GetKeys GetKeys --> SetEnv SetEnv --> CreateEval CreateEval --> CallSDK CallSDK --> ParseResult ParseResult --> ExtractScore ExtractScore --> Return ``` The worker's `_run_single_template()` function handles multiple output types from the FutureAGI SDK, converting string results ("Passed"/"Failed") or numeric scores into a unified format. [services/agent-opt-worker/main.py:59-126](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Sources**: [services/agent-opt-worker/main.py:46-56](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:59-126](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Concurrent Execution All requested metrics run **concurrently** using Python's `ThreadPoolExecutor` within the worker service to minimize total scoring latency. [services/agent-opt-worker/main.py:313-316](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## Data Storage ### SystemPromptEvalRun Model Each live scoring event creates one database record **per enabled prompt** with `run_type='live'`. [orchestrator/core/services/futureagi\_service.py:277-296](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | Field | Type | Description | | ---------- | ------ | ----------------------------------------------------------- | | `run_type` | String | `"live"` for live traffic scoring | | `scores` | JSONB | Full scoring results from worker (metrics, scores, reasons) | | `status` | String | Typically `"completed"` for live runs | **Sources**: [orchestrator/core/services/futureagi\_service.py:277-296](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## Integration with Prompt Optimization Live traffic scoring builds the dataset used for prompt optimization by logging real user interactions. [orchestrator/core/services/futureagi\_service.py:308-344](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Dataset Collection The `_collect_optimization_dataset()` method queries historical chat messages to build training data for the Meta-Prompting algorithms. It extracts text from message parts and pairs user messages with the subsequent assistant response. [orchestrator/core/services/futureagi\_service.py:308-344](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Sources**: [orchestrator/core/services/futureagi\_service.py:30-42](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [orchestrator/core/services/futureagi\_service.py:308-344](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## Performance Considerations ### Fire-and-Forget Pattern Live scoring is implemented as a fire-and-forget operation to ensure **zero user-facing latency**. The orchestrator dispatches the task asynchronously after the chat stream is finalized. [orchestrator/core/services/futureagi\_service.py:234-245](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Error Handling The `eval_live_traffic()` method implements multiple layers of error handling to ensure scoring failures (e.g., worker timeout, SDK errors) never impact chat functionality. [orchestrator/core/services/futureagi\_service.py:268-272](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [orchestrator/core/services/futureagi\_service.py:299-302](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Sources**: [orchestrator/core/services/futureagi\_service.py:79-98](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [orchestrator/core/services/futureagi\_service.py:234-245](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [orchestrator/core/services/futureagi\_service.py:268-272](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [orchestrator/core/services/futureagi\_service.py:299-302](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## Frontend Visualization ### Assessment Runs Display The `SystemPromptsTab` admin UI displays live scoring results in the "Assessments" tab, allowing admins to see real-world performance. [frontend/components/settings/SystemPromptsTab.tsx:551-723](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Sources**: [frontend/components/settings/SystemPromptsTab.tsx:551-723](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Score Rendering Each metric displays a color indicator, the metric name, score percentage, and a collapsible reason explanation derived from the worker's SDK response. [frontend/components/settings/SystemPromptsTab.tsx:617-635](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Sources**: [frontend/components/settings/SystemPromptsTab.tsx:617-635](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** ## Configuration ### Environment Variables **Orchestrator** (`config.py`): | Variable | Required | Default | Description | | ---------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AGENT_OPT_WORKER_URL` | Yes | N/A | Worker service base URL [orchestrator/core/services/futureagi\_service.py:25](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | **Worker Service** (`services/agent-opt-worker/main.py`): | Variable | Required | Description | | ---------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `FUTUREAGI_API_KEY` | Yes | FutureAGI platform API key [services/agent-opt-worker/main.py:47](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | | `FUTUREAGI_SECRET_KEY` | Yes | FutureAGI platform secret key [services/agent-opt-worker/main.py:48](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) | **Sources**: [orchestrator/core/services/futureagi\_service.py:21-26](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [orchestrator/core/services/futureagi\_service.py:63-69](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:46-56](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Worker Dependencies The `agent-opt-worker` requires the `agent-opt` and `ai-evaluation` packages to interface with the FutureAGI platform. [services/agent-opt-worker/requirements.txt:3-4](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Sources**: [services/agent-opt-worker/requirements.txt:3-4](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) ### Docker Setup The worker runs as an isolated service with its own container, exposing port 8080 and using `uvicorn` as the server. [services/agent-opt-worker/Dockerfile:1-16](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) **Sources**: [services/agent-opt-worker/Dockerfile:1-16](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md), [services/agent-opt-worker/main.py:32-40](/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md) *** --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.automatos.app/automatos-ai-docs/design-docs/prompt-optimization/live-traffic-scoring.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.