Skip to content

Levenshtein Ratio

Calculate character-level string similarity
Heuristic Single Turn Fast

At a Glance

🎯
Score Range
0.0 ──────── 1.0
Character-level similarity
⚡
Default Threshold
0.2
Pass/fail cutoff
📋
Required Inputs
actual_output expected_output
Text comparison

What It Measures

Levenshtein Ratio calculates the character-level similarity between two strings using the SequenceMatcher algorithm. It measures how many edits (insertions, deletions, substitutions) are needed to transform one string into another.

Score Interpretation
1.0 Identical strings
0.8+ Very similar, minor typos
0.5-0.8 Moderate similarity
< 0.5 Significant differences
✅ Use When
  • Checking for typos or small variations
  • Fuzzy string matching needed
  • Comparing names or identifiers
  • Near-match detection
❌ Don't Use When
  • Semantic similarity matters
  • Word-level comparison preferred (use BLEU)
  • Long texts with different structures
  • Exact match required

See Also: Sentence BLEU

Levenshtein Ratio measures character-level edit distance. Sentence BLEU measures word-level n-gram precision.

Use Levenshtein for typo detection; use BLEU for paraphrase comparison.

How It Works

Uses Python's SequenceMatcher to calculate the ratio of matching characters.

Step-by-Step Process

flowchart TD
    subgraph INPUT["📥 Inputs"]
        A[Actual Output]
        B[Expected Output]
    end

    subgraph PROCESS["🔍 Processing"]
        C[Optional: Convert to lowercase]
        D[Find matching subsequences]
        E[Calculate similarity ratio]
    end

    subgraph OUTPUT["📊 Result"]
        F["Score: 0.0 to 1.0"]
    end

    A & B --> C
    C --> D
    D --> E
    E --> F

    style INPUT stroke:#f59e0b,stroke-width:2px
    style PROCESS stroke:#3b82f6,stroke-width:2px
    style OUTPUT stroke:#10b981,stroke-width:2px

The SequenceMatcher ratio is calculated as:

ratio = 2.0 * M / T

where:
M = number of matches (characters in common)
T = total number of characters in both strings

Example: 

String 1: "hello"
String 2: "hallo"

Matches: h, l, l, o = 4 characters match
Total: 5 + 5 = 10 characters
Ratio: 2.0 * 4 / 10 = 0.8

Configuration

Parameter Type Default Description
case_sensitive bool False Whether comparison is case-sensitive

Case Sensitivity

By default, comparison is case-insensitive (both strings converted to lowercase). Set case_sensitive=True for strict character matching.

Code Examples

from axion.metrics import LevenshteinRatio
from axion.dataset import DatasetItem

metric = LevenshteinRatio()

item = DatasetItem(
    actual_output="The quick brown fox",
    expected_output="The quick brown fox",
)

result = await metric.execute(item)
print(result.score)  # 1.0 - identical strings

from axion.metrics import LevenshteinRatio

# Case insensitive (default)
metric = LevenshteinRatio(case_sensitive=False)

item = DatasetItem(
    actual_output="HELLO",
    expected_output="hello",
)

result = await metric.execute(item)
print(result.score)  # 1.0 - case ignored

# Case sensitive
metric_strict = LevenshteinRatio(case_sensitive=True)
result_strict = await metric_strict.execute(item)
print(result_strict.score)  # 0.0 - case matters

from axion.metrics import LevenshteinRatio
from axion.runners import MetricRunner

metric = LevenshteinRatio()
runner = MetricRunner(metrics=[metric])
results = await runner.run(dataset)

avg_similarity = sum(r.score for r in results) / len(results)
print(f"Average similarity: {avg_similarity:.2%}")

Example Scenarios

✅ Scenario 1: High Similarity (Score: 0.95)

Minor Typo

Expected: "accommodation"

Actual: "accomodation" (missing 'm')

Result: ~0.92

Single character difference results in high similarity.

⚠️ Scenario 2: Moderate Similarity (Score: 0.67)

Multiple Differences

Expected: "Hello World"

Actual: "Helo Wrld" (missing letters)

Result: ~0.67

Several missing characters reduce similarity.

❌ Scenario 3: Low Similarity (Score: 0.2)

Very Different Strings

Expected: "The quick brown fox"

Actual: "A lazy dog sleeps"

Result: ~0.2

Completely different content results in low similarity.

✅ Scenario 4: Case Handling

Case Insensitive Match

Expected: "OpenAI"

Actual: "openai"

Result (default): 1.0

Result (case_sensitive=True): ~0.67

Case sensitivity significantly affects scoring.

Why It Matters

⚡ Fast & Deterministic

No LLM calls needed. Instant, reproducible results.

🔤 Typo Detection

Perfect for detecting spelling errors and near-matches.

📊 Gradual Scoring

Unlike binary metrics, provides nuanced similarity scores.

Quick Reference

TL;DR

Levenshtein Ratio = How similar are the strings at the character level?

  • Use it when: Checking for typos or fuzzy matching
  • Score interpretation: Higher = more similar characters
  • Key config: case_sensitive controls case handling