Human Signals

Data-driven dashboard for human-in-the-loop metrics — track interventions, escalations, sentiment, and operational patterns across your AI agents.

Why Use Human Signals?

Automated evaluation scores tell you how well your AI performs. Human Signals tells you what happens when humans step in — the overrides, escalations, sentiment shifts, and learnings that only surface through real operator interactions.

Quick Start

Get human signals data loaded in under two minutes:

Page Anatomy

Here's how the Human Signals dashboard is organized, with every major section labeled:

KPI Strip

The KPI strip auto-generates cards from your display_config.kpi_strip configuration. Each card shows a metric icon, a large value, an optional sparkline, and a descriptive label.

KPI card types:

• Aggregate KPIs — total_cases (count). No sparkline.
• Boolean rate KPIs — percentage of cases where a boolean signal is true (e.g., resolution rate, escalation rate). Includes a sparkline and is clickable to expand a trend chart.
• Match value KPIs — percentage of cases where a string signal equals a specific value. Set match_value in the config (e.g., match_value: "success" to count cases where analysis_mode == "success").
• Numeric aggregation KPIs — aggregated numeric signals using aggregation (mean, median, sum, min, max, p95). Supports duration and compact formats.

Filters & Time Range

The filter bar sits at the top of the dashboard with the time range selector inline on the right. Click the bar to expand the filter panel, which shows a responsive grid of dropdown selectors.

Two filter categories:

• Source filters — filter by source_component and environment (single-select). Only appear if your data contains those fields.
• Metric filters — filter by signal values like override type, sentiment, escalation type (multi-select with checkboxes). Only appear for signals with 8 or fewer unique values.

The active filter count badge shows how many filters are currently applied. The Clear button resets all source and metric filters. The time range selector offers presets (7d, 30d, 90d, 6mo, 1yr) and a custom date range picker.

Chart Section

The chart section contains two areas: the Signal Trends time-series chart and the config-driven chart sections with various visualization types. All sections are collapsible and start collapsed by default to keep the page compact.

Signal Trends Over Time

A collapsible multi-line chart plotting signal rates as weekly percentages. Each line represents a KPI that has format: percent in the display config. The chart automatically extracts trend signals from your KPI configuration. Click the section header to expand.

Chart Types

Below the trend chart, chart sections render in configurable grid layouts (full, grid_2, or grid_3). Each section has a collapsible title with a chevron toggle. The available chart types are:

Case Table

The full paginated case table appears below the charts. Each row represents one aggregated case (one dataset_id) with all its flattened signal values.

Table features:

• Column Picker — click the "Columns" button to show/hide columns. Defaults to 7 of the most important columns.
• Sortable columns — click any column header with ↕ to sort ascending, descending, or reset. Uses lexicographic comparison with numeric awareness.
• Color-coded badges — signal values render as colored badges using color_maps from your display config. Control which columns show badges with the table_badge_columns list in signals_metrics.yaml. Omit the list to badge all mapped columns.
• Pagination — configurable page size (10, 25, 50, 100). Page pills with a 5-page sliding window around the current page.
• Eye icon — click to open the Case Detail Modal.

Case Detail Modal

Click the eye icon on any case row to open a full detail view. The modal shows every signal value, metadata, the complete conversation thread, learnings, and feature requests.

Modal sections (top to bottom):

1. Header — Case ID with source badge and close button
2. Slack Link — purple button linking to the original Slack thread (if message_url is present in additional_input)
3. Status Badges — color-coded badges for intervention type, sentiment, resolution status, priority, etc. Colors come from color_maps
4. Metadata Row — timestamp, message count, business name, agent name
5. Metric Sections — grouped by metric name, with simple signals in a 2-column grid and complex/structured signals rendered full-width with nested key-value displays
6. Suggested Action — highlighted panel when actionable feedback is detected
7. Learnings — expandable section with numbered learning items and category tags
8. Feature Requests — expandable section with numbered feature request items
9. Full Conversation — expandable chat view with assistant (left) and user (right) message bubbles, Slack markdown parsing

Data Format

Human Signals uses a long format where each row is one metric observation for one case. Cases are grouped by dataset_id and signals are flattened as {metric_name}__{signal_key}.

Required Columns

dataset_id,metric_name,signals,source_name,timestamp

Full CSV Schema

dataset_id,metric_name,metric_score,metric_category,signals,conversation,conversation_stats,additional_input,source_name,source_component,environment,timestamp

How Flattening Works

The backend groups rows by dataset_id, then for each metric row flattens all signal keys as {metric_name}__{signal_key}. For example:

# Input row:
metric_name: "intervention_type"
signals: {"intervention_type": "correction", "has_intervention": true}

# Becomes flattened case fields:
intervention_type__intervention_type: "correction"
intervention_type__has_intervention: true

Configuration

The display layout is auto-generated from the metric schema but can be customized via config/signals_metrics.yaml. The backend generates a display_config that controls KPIs, charts, filters, table columns, and color maps.

Display Config Structure

{
  "kpi_strip": [
    { "label": "Total Cases", "aggregate": "total_cases", "icon": "zap", "highlight": true },
    { "label": "Intervention Rate", "metric": "intervention_type", "signal": "has_intervention",
      "format": "percent", "icon": "alert-triangle" }
  ],
  "chart_sections": [
    {
      "title": "Outcome Distribution",
      "layout": "grid_2",
      "charts": [
        { "metric": "intervention_type", "signal": "intervention_type",
          "type": "donut", "title": "Intervention Type" }
      ]
    }
  ],
  "filters": [
    { "type": "source", "field": "source_component", "label": "Component" },
    { "type": "metric", "metric": "sentiment_category", "signal": "sentiment",
      "label": "Sentiment", "options": ["positive", "neutral", "negative"] }
  ],
  "table_columns": [
    { "key": "Case_ID", "label": "Case ID", "sortable": true },
    { "key": "intervention_type__intervention_type", "label": "Intervention", "sortable": true }
  ],
  "color_maps": {
    "intervention_type__intervention_type": {
      "correction": "#4f46e5", "escalation": "#ca8a04", "edge_case": "#16a34a"
    }
  }
}

Database Configuration

To auto-import from a PostgreSQL database, create config/human_signals_db.yaml from the example template:

human_signals_db:
  enabled: true
  auto_load: true
  url: "postgresql://user:pass@host:5432/db"
  dataset_query: |
    SELECT dataset_id, conversation, additional_input, timestamp
    FROM hitl_cases WHERE created_at > NOW() - INTERVAL '30 days'
  results_query: |
    SELECT dataset_id, metric_name, signals, source_name, timestamp
    FROM hitl_results WHERE timestamp > NOW() - INTERVAL '30 days'
  query_timeout: 60
  row_limit: 10000
  visible_metrics: []  # Empty = show all

Next Steps