Model Monitoring

Your ML platform computes accuracy, drift and bias. Wytness accepts those figures through its API per registered agent, judges each one against the threshold rules you set, raises a breach through your existing connectors, and keeps the signed record and trend as evidence. Wytness never computes drift itself.

Note
Model monitoring is included from the Growth plan. The agent must already be in your Registry — metrics for an unknown agent are refused with a 404 rather than creating a register row.

What to send

One request carries up to 500 metric points for one agent. Every point names the metric, its value, when it was observed and, optionally, the evaluation window, a segment label and the tool that computed it. Re-sending the same metric at the same observation time overwrites the point instead of duplicating it, so retries are safe.

FieldRequiredMeaning
metricyesA documented name (table below) or custom:<name>
valueyesA finite number
observed_atyesISO-8601 timestamp with a timezone offset
windowno{ start, end } — the period the figure was computed over
segmentnoA slice label such as a region or cohort
sourcenodataiku · azure_ml · mlflow · custom (default custom)
metanoAny JSON object you want kept with the point

Metric names

FamilyNames
Qualityaccuracy · precision · recall · f1 · auc · rmse · mae · log_loss
Driftdrift_psi · drift_ks · drift_js
Fairnessbias_disparity · bias_equal_opportunity
Datadata_quality
Anything elsecustom:<name> — letters, digits, dot, dash or underscore

Send metrics with curl

curl -X POST https://api.wytness.ai/v1/registry/credit-scorer/metrics \
  -H "X-API-Key: $WYTNESS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "metrics": [
      {
        "metric": "accuracy",
        "value": 0.93,
        "observed_at": "2026-11-17T09:00:00+00:00",
        "window": { "start": "2026-11-16T09:00:00+00:00", "end": "2026-11-17T09:00:00+00:00" },
        "source": "dataiku"
      },
      { "metric": "drift_psi", "value": 0.07, "observed_at": "2026-11-17T09:00:00+00:00", "source": "dataiku" }
    ]
  }'

# → {"status":"ok","metrics_recorded":2,"alerts_raised":0}

Send metrics from Python

Any job that can make an HTTPS request can report — a Dataiku scenario step, an Azure ML pipeline step, an MLflow callback or a plain cron job. No SDK is required.

import os
from datetime import datetime, timezone

import requests

API = "https://api.wytness.ai"
AGENT_ID = "credit-scorer"

now = datetime.now(timezone.utc).isoformat()
payload = {
    "metrics": [
        {"metric": "accuracy", "value": 0.93, "observed_at": now, "source": "azure_ml"},
        {"metric": "drift_psi", "value": 0.07, "observed_at": now, "source": "azure_ml"},
        {"metric": "bias_disparity", "value": 0.03, "observed_at": now, "segment": "EU", "source": "azure_ml"},
    ]
}

response = requests.post(
    f"{API}/v1/registry/{AGENT_ID}/metrics",
    headers={"X-API-Key": os.environ["WYTNESS_API_KEY"]},
    json=payload,
    timeout=30,
)
response.raise_for_status()
print(response.json())  # {"status": "ok", "metrics_recorded": 3, "alerts_raised": 0}

Thresholds and alerts

An owner or admin sets the rules on the agent page under Model monitoring, one metric at a time: alert when the value is below, at or below, above, or at or above a figure, with the severity the alert should carry. On every intake Wytness judges the newest value against the rules. A breach raises an anomaly alert of kind model_metric_threshold that names the agent, the metric, the value, the threshold and the severity, with a link to the agent page — so Microsoft Sentinel, Splunk, ServiceNow, Slack and webhook connectors deliver it exactly as they deliver every other alert. One alert per rule per 24 hours while the breach holds; every change to the rules is written to the audit log.

What lands on the Evidence Pack

Every Evidence Pack carries a Model monitoring section: the latest value of each metric in the period, the threshold rules in force, and the breaches the period raised. The agent page shows the same figures with their trend.

Tip
Send metrics on the cadence your platform evaluates them — daily or per retraining run is typical. The trend on the agent page and the Evidence Pack is only as dense as what you send.

Questions about model monitoring? Ask us.

Contact us

We set no cookies. Sign-in and preferences use essential first-party browser storage only — no tracking, advertising, or third-party analytics. Privacy Policy

Model Monitoring
TABLE OF CONTENTS