CaliberHQ

Align LLM judges with human judgment through calibration, pattern discovery, and iterative prompt optimization.

Why Use Calibration?

LLM-as-judge evaluation is only as reliable as the alignment between machine and human judgment. CaliberHQ gives you a structured workflow to measure that alignment, discover where your judge fails, and optimize its evaluation prompts — all without writing code.

Quick Start

Calibrate your LLM judge in three steps:

Page Anatomy

CaliberHQ uses a three-step wizard. Here is the full page layout with each major section labeled:

Step 1: Upload Ground Truth

The Upload step accepts CSV files with query-response pairs that you will annotate to create ground truth for calibrating your LLM judge.

CSV Format

The minimum required columns are:

id,query,actual_output
rec-001,"What are the refund policies?","We offer a 30-day refund policy for all purchases..."
rec-002,"How do I reset my password?","You can reset your password by clicking..."

Optionally include pre-existing labels:

id,query,actual_output,score,notes
rec-001,"What are the refund policies?","We offer...",1,"Accurate and complete"
rec-002,"How do I reset my password?","Click the button...",0,"Missing 2FA instructions"

Step 2: Review & Label

The annotation interface presents each record one at a time. Your job: decide whether the AI response passes or fails your quality bar, and optionally explain why.

Keyboard Shortcuts

CaliberHQ supports rapid annotation with keyboard shortcuts:

Step 3: Build Eval

The Build Eval step is where calibration happens. It has three sections: Pattern Insights + Judge Configuration (side by side), Run Evaluation, and Evaluation Results.

Pattern Insights

If you added notes during annotation, CaliberHQ can cluster them into recurring error patterns using AI. Three clustering methods are available:

Each discovered pattern shows a category name, example count, and a representative note excerpt. Click "Use as criteria" on any pattern to inject it directly into your evaluation criteria prompt.

LLM Judge Configuration

The right panel contains three tabs for configuring how the LLM judge evaluates records:

Model Tab

Select the LLM provider and model. Supported providers include OpenAI, Anthropic, and others configured in your AXIS settings. The provider must have an API key set in Settings for the run button to activate.

Prompt Tab

Edit the system prompt and evaluation criteria that the LLM judge uses. The default prompt includes a template variable {evaluation_criteria} that gets replaced with the criteria text. The prompt instructs the judge to output a binary Score (0 or 1) plus reasoning.

Examples Tab

Add few-shot examples to guide the LLM judge. Each example includes a query, response, expected score, and reasoning. More examples generally improve consistency but increase token usage.

Running the Evaluation

The "Run Evaluation" bar shows the current model and record count. Click to start:

1. The LLM evaluates each annotated record using your configured prompt
2. A progress bar shows completion percentage in real-time
3. Results appear in the panel below once the run completes

Comparison Table

The default results tab. Shows every record with both the human label and LLM label side by side, so you can see exactly where they agree and disagree.

• Filter pills — quickly isolate aligned or misaligned results
• Misaligned rows have a subtle red background tint
• Click the chevron on any row to expand and see the full AI response and LLM reasoning side by side
• Export JSON exports the filtered results for external analysis

Alignment Metrics

The Metrics tab provides a comprehensive statistical view of how well your LLM judge aligns with human judgment.

Metric Definitions

Confusion Matrix

The 2x2 confusion matrix visualizes the four possible outcomes of each comparison:

• True Positives (TP) — Both human and LLM accepted (green cells)
• True Negatives (TN) — Both human and LLM rejected (green cells)
• False Positives (FP) — LLM accepted but human rejected. The judge is too lenient here.
• False Negatives (FN) — LLM rejected but human accepted. The judge is too strict here.

The interpretation text below the matrix automatically tells you whether your judge tends to be too lenient (FP > FN) or too strict (FN > FP), guiding your next prompt iteration.

Insights & Prompt Optimization

The Insights tab provides AI-powered analysis of misalignment patterns and automatic prompt optimization to improve judge accuracy.

The iterative calibration loop:

1. Analyze Patterns — get an AI summary of why misalignment occurs
2. Review Recommendations — actionable suggestions to improve your prompt
3. Generate Optimized Prompt — AI rewrites your evaluation criteria
4. Apply & Re-run — one-click applies the new criteria and re-evaluates
5. Repeat until Kappa reaches your target level

The Calibration Workflow

CaliberHQ is designed for iterative improvement. Here is the recommended workflow:

Configuration Reference

CaliberHQ can be fine-tuned through the Configure modal (gear icon in the header) and the Build Eval step panels.

Column Configuration

Accessible from the gear icon on the annotation card. Lets you control:

• ID Column — which column uniquely identifies each record (default: id)
• Display Columns — which columns appear in the annotation card (default: query, actual_output)
• LLM Columns — which columns are sent to the LLM judge for evaluation

Judge Config Persistence

CaliberHQ persists the following to localStorage so you don't lose work between sessions:

• Human annotations (score + notes per record ID)
• Judge configuration (model, provider, system prompt, evaluation criteria, few-shot examples)
• Column configuration and file metadata

Next Steps