Evaluation scores
Your evaluation tooling or the model provider's safety filter already produces quality and harmful-content scores. Post them against the recorded event and Wytness keeps them beside the action they describe: on the event, rolled up on the session, counted in the Evidence Pack, and — when a score crosses a threshold you set — delivered as an anomaly alert through your connectors. Available on Growth plans and above.
How it fits
- An agent action is recorded; the SDK (or the CloudEvent) gives you its
event_id. - Your scorer evaluates the prompt and response on your infrastructure.
- You post the resulting numbers against the
event_idwith your API key. - Wytness stores them, checks them against your score thresholds and, on a breach, raises an anomaly alert with rule
llm_quality_threshold— once per event per score name. - Connectors deliver the alert unchanged; the event detail, the session header and the Evidence Pack show the scores.
Score one event
POST /v1/events/{event_id}/scores, authenticated with the same API key the SDK ingests with — as X-API-Key or Authorization: Bearer. Several scorers may score the same event; a scorer that posts again for the same event replaces its earlier numbers.
curl -X POST https://api.wytness.ai/v1/events/${EVENT_ID}/scores \ -H "X-API-Key: ${WYTNESS_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "scorer": "azure_content_safety", "scores": { "hate": 0.02, "sexual": 0.00, "violence": 0.11, "self_harm": 0.00 }, "labels": { "violence": "low" }, "observed_at": "2026-11-17T09:14:02Z" }'Response — 200 with the breaches evaluated against your thresholds:
{ "event_id": "4c1f9d2e-…", "scorer": "azure_content_safety", "status": "stored", "breaches": [ { "score_name": "violence", "value": 0.11, "op": "gt", "threshold": 0.10, "severity": "medium" } ], "alerts": ["a7e3…"]}| Status | Meaning |
|---|---|
| 200 | Stored. `breaches` lists every threshold the scores crossed; `alerts` the anomaly alerts raised (empty when the same event and score name already alerted). |
| 404 | The event is not on your organisation's tape — check the event_id. |
| 422 | Malformed body: unknown scorer, a score outside 0..1, a score name that is not lower-case snake_case, or any key that is not scorer / scores / labels / observed_at. |
| 403 | Your plan does not include evaluation scores (Growth and above). |
Body
| Field | Required | Shape |
|---|---|---|
| scorer | yes | `azure_content_safety`, `openai_moderation`, or `custom:<name>` (letters, digits, . _ -) |
| scores | yes | object of `<score name>: <number 0..1>`; one to fifty entries |
| labels | no | object of `<score name>: <string up to 200 characters>` — a category label, never content |
| observed_at | no | ISO 8601 timestamp of the scoring; defaults to the time Wytness received it |
Documented score names
Any lower-case name is accepted so your own metrics fit; these twelve are the vocabulary the dashboard explains and the threshold dialog suggests:
| Quality (higher is better) | Safety (higher is worse) |
|---|---|
| quality | toxicity |
| groundedness | hate |
| relevance | sexual |
| coherence | violence |
| fluency | self_harm |
| jailbreak | |
| pii_leak |
Score many events
POST /v1/events/scores takes up to 500 items, each with its own event_id, and answers for every item separately — an unknown event never fails the batch.
curl -X POST https://api.wytness.ai/v1/events/scores \ -H "X-API-Key: ${WYTNESS_API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "items": [ { "event_id": "4c1f9d2e-…", "scorer": "custom:ragas", "scores": { "groundedness": 0.91, "relevance": 0.87 } }, { "event_id": "9b02aa71-…", "scorer": "openai_moderation", "scores": { "toxicity": 0.74 }, "labels": { "toxicity": "harassment" } } ] }'{ "results": [ { "event_id": "4c1f9d2e-…", "scorer": "custom:ragas", "status": "stored", "breaches": [], "alerts": [] }, { "event_id": "9b02aa71-…", "scorer": "openai_moderation", "status": "not_found" } ], "stored": 1, "not_found": 1}Thresholds and alerts
Thresholds live on the Anomalies page (Score thresholds, owners and admins). Each one names a score, a comparison — below, at or below, above, at or above — a value and a severity. No thresholds means scores are stored and nothing alerts. A breach writes an anomaly alert with rule llm_quality_threshold carrying the agent, the event, the scorer, the score name, the value, the threshold and a link to the event, so Sentinel, Splunk, ServiceNow, Slack and generic webhooks deliver it with no change on your side. Re-sends never alert twice for the same event and score name.
Where the scores show
| Surface | What you see |
|---|---|
| Event detail | An Evaluation section: one row per scorer, each score as a badge, breached scores marked with the threshold's severity. |
| Events list | A small marker on rows that carry scores. |
| Session detail | One line: scored events, breaches, and the average per score name. |
| Evidence Pack | The LLM usage section counts scored events and threshold breaches in the reporting window. |
SDK helpers
The Python and TypeScript SDKs ship a score_event helper and an Azure AI Content Safety adapter that post to these endpoints for you; the raw HTTP contract above is what they call.
Put your evaluation scores beside the actions they judge.
Start recording