Evaluation scores

Your evaluation tooling or the model provider's safety filter already produces quality and harmful-content scores. Post them against the recorded event and Wytness keeps them beside the action they describe: on the event, rolled up on the session, counted in the Evidence Pack, and — when a score crosses a threshold you set — delivered as an anomaly alert through your connectors. Available on Growth plans and above.

Note
Content inspection stays on your side. Wytness accepts numbers between 0 and 1 and short labels — never the prompt, the response, or any text that was scored. A body carrying any other key is refused before it is read.

How it fits

  1. An agent action is recorded; the SDK (or the CloudEvent) gives you its event_id.
  2. Your scorer evaluates the prompt and response on your infrastructure.
  3. You post the resulting numbers against the event_id with your API key.
  4. Wytness stores them, checks them against your score thresholds and, on a breach, raises an anomaly alert with rule llm_quality_threshold — once per event per score name.
  5. Connectors deliver the alert unchanged; the event detail, the session header and the Evidence Pack show the scores.

Score one event

POST /v1/events/{event_id}/scores, authenticated with the same API key the SDK ingests with — as X-API-Key or Authorization: Bearer. Several scorers may score the same event; a scorer that posts again for the same event replaces its earlier numbers.

curl -X POST https://api.wytness.ai/v1/events/${EVENT_ID}/scores \
-H "X-API-Key: ${WYTNESS_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"scorer": "azure_content_safety",
"scores": { "hate": 0.02, "sexual": 0.00, "violence": 0.11, "self_harm": 0.00 },
"labels": { "violence": "low" },
"observed_at": "2026-11-17T09:14:02Z"
}'

Response — 200 with the breaches evaluated against your thresholds:

{
"event_id": "4c1f9d2e-…",
"scorer": "azure_content_safety",
"status": "stored",
"breaches": [
{ "score_name": "violence", "value": 0.11, "op": "gt", "threshold": 0.10, "severity": "medium" }
],
"alerts": ["a7e3…"]
}
StatusMeaning
200Stored. `breaches` lists every threshold the scores crossed; `alerts` the anomaly alerts raised (empty when the same event and score name already alerted).
404The event is not on your organisation's tape — check the event_id.
422Malformed body: unknown scorer, a score outside 0..1, a score name that is not lower-case snake_case, or any key that is not scorer / scores / labels / observed_at.
403Your plan does not include evaluation scores (Growth and above).

Body

FieldRequiredShape
scoreryes`azure_content_safety`, `openai_moderation`, or `custom:<name>` (letters, digits, . _ -)
scoresyesobject of `<score name>: <number 0..1>`; one to fifty entries
labelsnoobject of `<score name>: <string up to 200 characters>` — a category label, never content
observed_atnoISO 8601 timestamp of the scoring; defaults to the time Wytness received it

Documented score names

Any lower-case name is accepted so your own metrics fit; these twelve are the vocabulary the dashboard explains and the threshold dialog suggests:

Quality (higher is better)Safety (higher is worse)
qualitytoxicity
groundednesshate
relevancesexual
coherenceviolence
fluencyself_harm
jailbreak
pii_leak

Score many events

POST /v1/events/scores takes up to 500 items, each with its own event_id, and answers for every item separately — an unknown event never fails the batch.

curl -X POST https://api.wytness.ai/v1/events/scores \
-H "X-API-Key: ${WYTNESS_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"items": [
{ "event_id": "4c1f9d2e-…", "scorer": "custom:ragas",
"scores": { "groundedness": 0.91, "relevance": 0.87 } },
{ "event_id": "9b02aa71-…", "scorer": "openai_moderation",
"scores": { "toxicity": 0.74 }, "labels": { "toxicity": "harassment" } }
]
}'
{
"results": [
{ "event_id": "4c1f9d2e-…", "scorer": "custom:ragas", "status": "stored", "breaches": [], "alerts": [] },
{ "event_id": "9b02aa71-…", "scorer": "openai_moderation", "status": "not_found" }
],
"stored": 1,
"not_found": 1
}

Thresholds and alerts

Thresholds live on the Anomalies page (Score thresholds, owners and admins). Each one names a score, a comparison — below, at or below, above, at or above — a value and a severity. No thresholds means scores are stored and nothing alerts. A breach writes an anomaly alert with rule llm_quality_threshold carrying the agent, the event, the scorer, the score name, the value, the threshold and a link to the event, so Sentinel, Splunk, ServiceNow, Slack and generic webhooks deliver it with no change on your side. Re-sends never alert twice for the same event and score name.

Where the scores show

SurfaceWhat you see
Event detailAn Evaluation section: one row per scorer, each score as a badge, breached scores marked with the threshold's severity.
Events listA small marker on rows that carry scores.
Session detailOne line: scored events, breaches, and the average per score name.
Evidence PackThe LLM usage section counts scored events and threshold breaches in the reporting window.

SDK helpers

The Python and TypeScript SDKs ship a score_event helper and an Azure AI Content Safety adapter that post to these endpoints for you; the raw HTTP contract above is what they call.

Put your evaluation scores beside the actions they judge.

Start recording

We set no cookies. Sign-in and preferences use essential first-party browser storage only — no tracking, advertising, or third-party analytics. Privacy Policy

Evaluation scores
TABLE OF CONTENTS