Annotation Studio

Human-in-the-loop annotation for quality assurance — score, tag, and critique LLM outputs with keyboard-driven workflows and CSV export.

Why Use Annotation Studio?

Automated metrics are powerful, but they are not the whole story. The Annotation Studio gives your team a structured interface to manually review and label evaluation data — building ground truth datasets, validating automated scores, and catching failure modes that metrics alone miss.

Quick Start

Start annotating in under a minute:

Page Anatomy

Here is how the Annotation Studio is organized once data is loaded, with every major section labeled:

Data Upload

When you first open Annotation Studio with no data loaded, the page shows a centered upload zone. This is the same drag-and-drop CSV uploader used across AXIS.

Supported data formats:

• Standard evaluation CSVs — must contain at least query and actual_output columns
• Previously annotated CSVs — if columns like judgment, critique, user_tags, or annotation_flagged exist, they are imported automatically
• Records are deduplicated by ID column (auto-detected or configured)

Annotation Interface

The annotation card is the main workspace. It displays each record's content fields vertically, followed by the scoring and tagging controls.

Content Display

By default, two columns are shown:

• User Query (query) — displayed in a neutral gray panel
• AI Response (actual_output) — highlighted with a left accent border (primary color) and a "To Evaluate" badge

You can show additional columns (like expected_output, context, or custom fields) by configuring them in the Configure modal. Content supports Markdown rendering via the built-in ContentRenderer.

Accent Gradient Bar

A thin gradient bar (sage green to gold) appears at the top of the annotation card. This is a visual cue inherited from the AXIS brand palette. It separates the navigation header from the content.

Scoring & Labels

Below the content panels, three annotation controls appear: the score selector, tag selector, and critique field.

Score Modes

AXIS supports three scoring modes, selectable via the Configure modal:

Tags

Tags let you categorize outputs with descriptive labels. AXIS ships with sensible defaults and supports fully custom tag sets.

• Positive tags (Excellent, Cool, Correct, Positive) highlight in green when selected
• Negative/neutral tags (Hallucination, Off-topic, Incomplete, etc.) highlight in red when selected
• Unselected tags appear as neutral gray pills
• Multiple tags can be selected per record

Tag presets are available in the Configure modal:

Critique / Notes

A free-text field below the tags for detailed feedback. This text is exported in the critique column of your CSV.

Navigation & Progress

AXIS is optimized for rapid sequential annotation. All navigation is available from both the UI and the keyboard.

Keyboard Shortcuts

The full shortcut reference is always visible in the sidebar. These work globally unless focus is in a text input.

Progress Sidebar

The right sidebar provides at-a-glance progress tracking:

• Completion percentage — large numeric display with animated progress bar
• Stats grid — Done (green), Pending (gray), Flagged (orange), and Total counts
• Filter tabs — switch the record list between All, Pending, Done, or Flagged views
• Record list — scrollable list with colored status dots. Click any record to jump to it. The current record is highlighted with a left border accent and pulsing dot.

The record list shows up to 100 items at a time. If your dataset has more than 100 records, a "+N more records" indicator appears at the bottom.

Flagging Records

Flagging is useful for marking records that need a second look, are ambiguous, or should be discussed with the team. Flagged records:

• Show an orange flag icon in the navigation bar
• Display an orange dot in the record list
• Can be filtered to view only flagged items
• Export with annotation_flagged = true in the CSV

Configure Modal

Click the Configure button in the page header to open the settings modal. All configuration persists in localStorage.

Configuration sections:

1. ID Column — Select which column uniquely identifies records. Auto-detects id, *_id, and *uuid* columns. Shows a warning if duplicate IDs are found.
2. Display Columns — Checkbox grid to select which data columns appear in the annotation card. Defaults to query and actual_output.
3. Score Mode — Radio buttons for Binary, 1–5 Scale, or Custom Range. Custom range shows min/max number inputs.
4. Tags — Shows current tag set with a "Manage Tags" popover for adding custom tags, applying presets, or resetting to defaults.

Export & Review

Click the Export button in the header to download a CSV with all your annotations merged into the original data.

Exported Columns

The export includes all original columns plus four annotation columns:

dataset_id,query,actual_output,judgment,critique,user_tags,annotation_flagged
01KFX-a1b2,"What is RAG?","RAG retrieves...",accept,"Good explanation","[""Correct""]",
01KFX-b2c3,"Explain fine-tuning","Fine-tuning is...",reject,"Misses key details","[""Incomplete""]",true

Undo System

Every annotation action (scoring, tagging, flagging, critique changes) is tracked in an undo stack. Press Ctrl+Z (or Cmd+Z on macOS) to revert the last action. The stack holds up to 20 actions.

A toast notification appears at the bottom-right of the screen confirming the undo. The undo button in the navigation header is only visible when there is an action to undo.

Data Persistence

Annotation state is managed carefully to balance persistence with performance:

Next Steps