LLM as a Verifier Paper

 The research paper "LLM-as-a-Verifier: A General-Purpose Verification Framework" (published July 6, 2026), which introduces a training-free method to grade LLM outputs more accurately by utilizing continuous rewards based on token probabilities rather than discrete scoring.

Key Takeaways:

Mechanism: Instead of traditional discrete judgments, the framework computes continuous scores from token logits. This approach significantly reduces ties in evaluation.

Performance: The framework demonstrates state-of-the-art results across four benchmarks, including 86.5% on Terminal Bench 2.0 and 78.2% on SWE Bench Verified.

Scaling Dimensions: Verification accuracy is improved through three main axes: scoring granularity, repeated evaluations, and criteria decomposition.

Efficiency: The authors introduce the probabilistic pivot tournament, a cost-efficient five-stage pipeline that reduces search costs from the order of n squared to n*k, making it practical for selecting the best candidate solution.

Impact: By providing fine-grained feedback, the framework bridges the gap between candidate generation and selection, serving as a powerful, training-free scaling axis for agentic AI.

The paper "LLM-as-a-Verifier: A General-Purpose Verification Framework" (published in July 2026 by researchers from Stanford, UC Berkeley, Nvidia, and other institutions) introduces a training-free framework that treats solution verification as a major new scaling axis for AI performance. [1, 2]

Instead of using standard "LLM-as-a-judge" methods that output rigid, discrete scores, this framework calculates a mathematical expectation over the model's entire scoring token logit distribution, generating a highly precise, continuous reward signal. [1, 2]

The 3 Dimensions of Verification Scaling

The paper establishes that verification accuracy scales along three core levers: [1, 2, 3]

• Score Granularity: Fine-grained continuous scores eliminate ties and better separate correct solutions from incorrect ones.
• Repeated Evaluation: Multiple inference passes dramatically reduce variance.
• Criteria Decomposition: Breaking down complex evaluation goals into individual, distinct sub-criteria eliminates rounding and signal loss. [1, 2, 3, 4, 5]

Difference between LLM-as-a-Judge and LLM-as-a-Verifier lies in how they extract data from the model.  

• LLM-as-a-Judge relies on standard text generation (max_tokens=1 or JSON mode) to capture a single, hard-coded discrete score token (e.g., "7" out of 10). [1, 2, 3]
• LLM-as-a-Verifier requests the model's raw log probabilities (logprobs) for a predetermined set of score tokens, converting the entire next-token probability distribution into a highly precise, continuous expected mathematical value. [1, 2]

 

 LLM-as-a-Judge: The Discrete Approach

This traditional method asks the model for a rating, extracts the single highest-probability token, and throws away the rest of the model's underlying distribution data. [1]

import openai

def llm_as_a_judge(prompt, response_to_evaluate):

judge_prompt = f"""

Evaluate the following response based on correctness.

Output exactly one single integer from 1 to 5. Do not write anything else.

Response: {response_to_evaluate}

Score:"""

client = openai.OpenAI()

response = client.chat.completions.create(

model="gpt-4o",

messages=[{"role": "user", "content": judge_prompt}],

max_tokens=1, # Force a single discrete token output

temperature=0.0

)

# Returns a hard string like "4", 

# treating a 4.1 confidence and 4.9 confidence identically

return int(response.choices[0].message.content.strip())

 

 LLM-as-a-Verifier: The Continuous Logprob Approach

The verifier framework forces the model to choose between specific tracking tokens (the paper uses a letter-based scale like A, B, C, D, E to perfectly isolate score tokens) and uses logprobs=True to extract the full confidence distribution. [1, 2, 3] 

import numpy as np

import openai

def llm_as_a_verifier(prompt, response_to_evaluate):

verifier_prompt = f"""

Evaluate the following response based on strict criteria.

Select the single most accurate letter grade from this scale:

A: Perfect (Value 5)

B: Good (Value 4)

C: Average (Value 3)

D: Poor (Value 2)

E: Failing (Value 1)

Output exactly one letter character. Do not write anything else.

Response: {response_to_evaluate}

Grade:"""

# Map token strings to their designated numeric weight

score_mapping = {"A": 5.0, "B": 4.0, "C": 3.0, "D": 2.0, "E": 1.0}

client = openai.OpenAI()

response = client.chat.completions.create(

model="gpt-4o",

messages=[{"role": "user", "content": verifier_prompt}],

max_tokens=1,

logprobs=True, # CRITICAL: Ask the API for the log probabilities

top_logprobs=5 # Get the probabilities for all 5 score tokens

)

# Extract the top token logprob array

top_logprobs = response.choices[0].logprobs.content[0].top_logprobs

# Convert logprobs back to raw probabilities using softmax: e^logprob

raw_probabilities = {}

denominator = 0.0

for logprob_obj in top_logprobs:

token = logprob_obj.token.strip()

if token in score_mapping:

prob = np.exp(logprob_obj.logprob)

raw_probabilities[token] = prob

denominator += prob

# Calculate the continuous mathematical expectation

expected_score = 0.0

if denominator > 0:

for token, prob in raw_probabilities.items():

# Normalize probabilities specifically among the score tokens

normalized_prob = prob / denominator

expected_score += score_mapping[token] * normalized_prob

# Returns a highly granular float (e.g., 4.38 instead of a flat 4)

return expected_score

Feature	LLM-as-a-Judge	LLM-as-a-Verifier
API Parameters	logprobs=False	logprobs=True, top_logprobs=5
Output Type	int or str (Discrete)	float (Continuous Expectation)
Tie Resolution	Low (Many responses get the exact same score)	High (Resolves ties down to decimal confidence)
Parsing Logic	Simple string stripping	Softmax transformation on logits
API Availability	Works on all text generation APIs	Requires engines exposing token logits (OpenAI, Together AI, vLLM)

Rubrics 

In LLM-as-a-Judge, the rubric is a formatting tool used to force the model into making a strict decision. In LLM-as-a-Verifier, the rubric is a mathematical device used to isolate specific tokens, prevent logit pollution, and enable multi-criteria decomposition

 LLM-as-a-Judge (Text-Based Decisions) [1]

• Design: Uses detailed descriptive text explaining what qualifies as a score of 1, 2, 3, 4, or 5.
• Mechanism: The model reads the full prompt, interprets the text, and outputs a single character text block (e.g., "3").
• Flaw: The rubric text often leads to "token bias." LLMs naturally favor certain score tokens (like 7 or 8on a 10-point scale) regardless of the rubric definitions. [1, 2, 3]

import json

import openai

def llm_as_a_judge_monolithic(response_to_evaluate):

# The judge prompt bundles all criteria and rubric text into a single context window

judge_prompt = f"""

You are an expert code reviewer. Evaluate the provided agent code response against the strict 3-pillar rubric below.

### EVALUATION RUBRICS

1. Correctness (Scale 1-5):

- 5: Mathematically flawless, passes all edge cases, functionally perfect.

- 1: High critical failure rate, logic errors, or fails to solve the core problem.

2. Formatting (Scale 1-5):

- 5: Follows all JSON/Markdown constraints perfectly.

- 1: Violated formatting constraints completely, unparsable.

3. Conciseness (Scale 1-5):

- 5: Code is clean, modular, and has zero redundancy.

- 1: Heavy boilerplate, duplicate logic, or extremely bloated.

### TARGET RESPONSE TO EVALUATE

{response_to_evaluate}

### OUTPUT FORMAT

You must output valid JSON matching this schema exactly. Do not include any other text or markdown wrappers.

{{

"correctness_score": <int 1-5>,

"formatting_score": <int 1-5>,

"conciseness_score": <int 1-5>,

"justification": "<brief text explaining the scores>"

}}

JSON:"""

client = openai.OpenAI()

# Standard text generation request (no logprobs needed)

response = client.chat.completions.create(

model="gpt-4o",

messages=[{"role": "user", "content": judge_prompt}],

response_format={"type": "json_object"}, # Enforces JSON structure

temperature=0.0 # Minimize randomness

)

# Parse the discrete textual responses

result = json.loads(response.choices[0].message.content)

# Calculate final score based on the same weights used in the verifier (50% / 30% / 20%)

final_weighted_score = (

(result["correctness_score"] * 0.5) +

(result["formatting_score"] * 0.3) +

(result["conciseness_score"] * 0.2)

)

return {

"raw_scores": result,

"final_score": final_weighted_score

}

# Example usage:

# eval_result = llm_as_a_judge_monolithic(agent_code_string)

# print(eval_result["final_score"]) # Returns a discrete stepped float like 4.1 or 3.5

 LLM-as-a-Verifier (Logit Allocation)

• Design: Uses a highly rigid, mapped categorical rubric (e.g., A, B, C, D, E).
• Mechanism: The rubric is designed to map specific, high-frequency, single-byte tokens to exact numerical weights.
• Feature: It forces the model to distribute its total probability mass (logits) only among those specified rubric characters. This prevents standard text token bias from corrupting the raw math.

 

# The paper splits evaluation into targeted, non-overlapping rubric queries

sub_rubrics = {

"correctness": "A: Mathematically flawless. \nE: Contains math errors.",

"formatting": "A: Follows JSON/Markdown constraints. \nE: Violated constraints.",

"conciseness": "A: Code has no redundancy. \nE: Heavy boilerplate code present."

}

final_weights = {"correctness": 0.5, "formatting": 0.3, "conciseness": 0.2}

total_agent_score = 0.0

# Loop evaluates each rubric independently to calculate distinct logprob expectations

for criteria, rubric_prompt in sub_rubrics.items():

# 1. Inject 'rubric_prompt' into the LLM-as-a-Verifier prompt template

# 2. Call the API with logprobs=True

# 3. Calculate continuous expected score for this specific pillar

expected_sub_score = run_verifier_logprobs(rubric_prompt, response_to_evaluate)

# Accumulate weighted continuous metrics

total_agent_score += expected_sub_score * final_weights[criteria]

Token Allocation Blindness: 

If the judge is torn between giving a Correctness score of 4 or 5 (e.g., 51% confidence for 4, 49% confidence for 5), it will output a hard 4. The verifier code would capture that split and mathematically calculate a score of 4.49.

Context Contamination:

In this Judge code, the explanation for "Conciseness" can accidentally bias the score for "Correctness" because they are processed simultaneously in the same attention window. The Verifier isolates them into completely separate API calls

📊 Performance and Benchmarks

By pairing this probabilistic approach with a novel "probabilistic pivot tournament" ranking algorithm, the framework achieves state-of-the-art results without any additional model training: [1, 2, 3]

• Terminal-Bench V2: Scores 86.5% accuracy, significantly outperforming baseline LLMs.
• SWE-Bench Verified: Achieves 78.2% accuracy on complex software engineering and code-patching tasks.
• RoboRewardBench: Hits 87.4% accuracy, outperforming existing trained reward models on robotics tasks.
• MedAgentBench: Scores 73.3% accuracy on multi-step medical workflow tracking

Terminal-Bench V2

LLM as a Judge

Requirements 

pip install uv

uv tool install harbor

pip install openai evalscope

Evaluator

import re

import json

import openai

from typing import Dict, Any, List

class TerminalJudge:

def __init__(self, client: openai.OpenAI, model: str = "gpt-4o"):

self.client = client

self.model = model

def evaluate_trajectory(self, objective: str, trajectory_logs: str) -> Dict[str, Any]:

"""Judges a terminal execution trajectory and extracts a discrete score."""

system_prompt = (

"You are an expert system administration judge.\n"

"Analyze the given terminal trajectory against the target objective.\n"

"Provide a brief critique of the tool choices, error handling, and efficiency.\n"

"Conclude your response by strictly outputting your final rating in this format:\n"

"FINAL_SCORE: [X]\n"

"Where [X] is an integer between 1 (failed/stalled) and 10 (perfect execution)."

)

prompt = (

f"### TARGET OBJECTIVE:\n{objective}\n\n"

f"### TERMINAL TRAJECTORY LOGS:\n{trajectory_logs}\n\n"

f"### YOUR EVALUATION:"

)

response = self.client.chat.completions.create(

model=self.model,

messages=[

{"role": "system", "content": system_prompt},

{"role": "user", "content": prompt}

],

temperature=0.2, # Lower temperature for stable discrete selection

max_tokens=500

)

critique = response.choices[0].message.content

# Extract the discrete score using regex parsing

match = re.search(r"FINAL_SCORE:\s*\[?(\d+)\]?", critique)

score = int(match.group(1)) if match else 1 # Fallback to minimum if parsing fails

return {

"score": score,

"critique": critique

}

Pair wise Judging , comparing two Agents : 

def judge_pairwise_matchup(self, objective: str, traj_a: str, traj_b: str) -> str:

"""Compares two agent trajectories side-by-side to declare a winner."""

system_prompt = (

"You are an impartial referee evaluating two terminal execution attempts for the same task.\n"

"Determine which trajectory achieved the objective more safely, cleanly, and efficiently.\n"

"At the very end of your response, strictly output your decision as either:\n"

"WINNER: A\n"

"WINNER: B\n"

"WINNER: TIE"

)

prompt = (

f"### TARGET OBJECTIVE:\n{objective}\n\n"

f"### TRAJECTORY A:\n{traj_a}\n\n"

f"### TRAJECTORY B:\n{traj_b}\n\n"

f"### DECISION:"

)

# To mitigate positional bias (the judge favoring whichever trace is labeled 'A'),

# standard judge harnesses run this twice, swapping the positions of A and B,

# and averaging the outcome.

response = self.client.chat.completions.create(

model=self.model,

messages=[

{"role": "system", "content": system_prompt},

{"role": "user", "content": prompt}

],

temperature=0.0

)

decision_text = response.choices[0].message.content

if "WINNER: A" in decision_text:

return "A"

elif "WINNER: B" in decision_text:

return "B"

return "TIE"

Main method calling Evaluator for Terminal Bench Evaluation :

if __name__ == "__main__":

openai_client = openai.OpenAI(api_key="YOUR_API_KEY")

judge = TerminalJudge(client=openai_client)

# Simulated dataset payload from Terminal-Bench V2 tasks

test_case = {

"objective": "Fix the broken python environment dependencies inside the container and ensure pytest runs cleanly.",

"candidates": [

{

"id": "agent_run_alpha",

"logs": "$ pip install -r requirements.txt\nError: Conflict on numpy.\n$ pip install numpy --upgrade\n$ pytest\nStatus: Passed 12 tests."

},

{

"id": "agent_run_beta",

"logs": "$ pytest\nError: ModuleNotFound numpy.\n$ pip install numpy==1.26.0\n$ pytest\nStatus: Passed 12 tests."

}

]

}

# 1. Absolute Scoring Approach

print("--- Running Absolute Scoring ---")

for cand in test_case["candidates"]:

eval_result = judge.evaluate_trajectory(test_case["objective"], cand["logs"])

print(f"ID: {cand['id']} | Discrete Score: {eval_result['score']}/10")

print(f"Reasoning snippet: {eval_result['critique'][:120]}...\n")

# 2. Pairwise Matchup Approach (Removes individual bias)

print("--- Running Pairwise Matchup ---")

winner = judge.judge_pairwise_matchup(

objective=test_case["objective"],

traj_a=test_case["candidates"][0]["logs"],

traj_b=test_case["candidates"][1]["logs"]

)

if winner == "A":

print(f" Winner: {test_case['candidates'][0]['id']}")

elif winner == "B":

print(f" Winner: {test_case['candidates'][1]['id']}")

else:

print(" The matchup ended in a Tie.")

LLM as a Verifier 

Requirements

# 1. Install Harbor engine and Terminal-Bench dependencies

pip install uv

uv tool install harbor

pip install openai evalscope

# 2. Clone the official LLM-as-a-Verifier evaluation harness repo

git clone https://github.com

cd llm-as-a-verifier

pip install -e .

Run Agents to collect Trajectories

# Perform a sanity test on Terminal-Bench v2 tasks using harbor

harbor run -d terminal-bench/terminal-bench-2 -a oracle -l 5

Verifier 

- Tri dimentional

import math

import openai

from llm_verifier import LogprobVerifier, TournamentEvaluator

# Initialize model connection (e.g., GPT-4o or an open-weight engine via vLLM)

client = openai.OpenAI(api_key="YOUR_API_KEY")

# Define decomposed operational axes specifically for Terminal-Bench trajectories

terminal_criteria = {

"command_efficiency": "Did the agent issue safe, purposeful commands without looping or hanging?",

"error_recovery": "Did the agent correctly parse unexpected stderr or missing packages and pivot?",

"state_completion": "Does the final environment file structure and log match the target objective?"

}

# Load the candidate trajectory pool generated via Harbor

# trajectories = json.load(open("terminal_bench_candidates.json"))

Main Method Verification

from llm_verifier.benchmarks import evaluate_trajectory_continuous

results = []

for candidate in trajectories:

# Continuous score extraction averages expected logprob values over the 1-20 token range

scores = evaluate_trajectory_continuous(

client=client,

model="gpt-4o",

trajectory=candidate["terminal_logs"],

criteria=terminal_criteria,

num_samples=8, # Repeated evaluation lever

score_range=(1, 20) # Finetuned granularity scale

)

candidate["verification_scores"] = scores

results.append(candidate)

# Apply the Tournaments mechanism to isolate the top-performing run

best_run = TournamentEvaluator.select_best(results, metric="aggregate_score")

print(f"Optimal Terminal Trajectory: {best_run['id']} with Score: {best_run['verification_scores']['aggregate_score']}")

Performance Gain :

The main metric is Verification Accuracy: the ability of the evaluator to reliably distinguish a successful agent trajectory from a broken, stalled, or hallucinated one

Benchmark Suite	LLM-as-a-Judge (Discrete Baselines)	LLM-as-a-Verifier (Continuous Framework)	Performance Gain
Terminal-Bench V2	~70.2%	86.5%	+16.3%
SWE-Bench Verified	~71.4%	78.2%	+6.8%
RoboRewardBench	~79.1%	87.4%	+8.3%

SWE Bench :

Software Bug Benchmark: SWE-bench evaluates AI models on their ability to resolve genuine, complex issues sourced from real-world open-source GitHub repositories. [1, 2, 3, 4]

Codebase Navigation: Instead of writing short code snippets, agents must navigate massive codebases, understand inter-dependencies, and modify multiple files. [1]

Patch Generation

: Models are required to output an explicit code patch file (diff) that targets and fixes the described system error. [1]

Execution Verification: The framework automatically runs a verification test suite inside an isolated Docker sandbox to evaluate the correctness of the model's patch. [1]

Rigorous Testing Standard: Passing requires the agent's patch to successfully resolve the issue without breaking existing functionalities or causing regressions.

Software Engineering Benchmark

requirements

pip install openai gitpython

 

import os

import re

import math

import shutil

import git

import openai

from typing import Dict, Any, List

# --- BASE CONFIGURATION ---

# Set your active key directly or via environment variable

os.environ["OPENAI_API_KEY"] = "your-api-key-here"

client = openai.OpenAI()

# Real SWE-bench asset replication target (e.g., a localized issue from pallets/flask)

SWE_BENCH_CASE = {

"repo_url": "https://github.com",

"commit": "8064b85ab196ad37707ee7b2c1f9644f19b22a01", # Pin specific test case version

"issue_description": "Flask blueprinted routes fail to parse when trailing slashes are appended dynamically inside a Blueprint group.",

# Genuine candidate patch generated by an engineering agent

"candidate_patch": """

diff --git a/src/flask/blueprints.py b/src/flask/blueprints.py

index a24f81..b35e92 100644

--- a/src/flask/blueprints.py

+++ b/src/flask/blueprints.py

@@ -190,6 +190,8 @@ class Blueprint(_PackageBoundObject):

url_prefix = options.get("url_prefix", self.url_prefix)

if url_prefix is not None:

url_prefix = url_prefix.rstrip("/")

+ if url_prefix == "":

+ url_prefix = None

for rule, endpoint, view_func, options in self.deferred_functions:

options = options.copy()

"""

}

# --- EVALUATION MODULE ---

class SWEEvaluator:

def __init__(self, rubrics: Dict[str, str]):

self.rubrics = rubrics

self.model = "gpt-4o"

def run_judge(self, issue: str, patch: str) -> Dict[str, float]:

"""Discrete Judge: Uses absolute generation text and searches via regex."""

scores = {}

for criterion, rubric_text in self.rubrics.items():

system_prompt = (

"You are an expert repository reviewer for SWE-bench.\n"

f"Evaluate this patch based strictly on: {rubric_text}\n"

"End your analysis with this exact string format:\n"

"FINAL_SCORE: [X]\n"

"Where [X] is an integer between 1 (broken) and 10 (perfect solution)."

)

user_prompt = f"### TARGET ISSUE:\n{issue}\n\n### CANDIDATE PATCH DIFF:\n{patch}"

response = client.chat.completions.create(

model=self.model,

messages=[

{"role": "system", "content": system_prompt},

{"role": "user", "content": user_prompt}

],

temperature=0.0

)

text_output = response.choices[0].message.content

match = re.search(r"FINAL_SCORE:\s*\[?(\d+)\]?", text_output)

scores[criterion] = float(match.group(1)) if match else 1.0

scores["aggregate"] = sum(scores.values()) / len(scores)

return scores

def run_verifier(self, issue: str, patch: str) -> Dict[str, float]:

"""Continuous Verifier: Captures first token logprob weights over the 1-10 distribution."""

scores = {}

score_mapping = {str(i): float(i) for i in range(1, 11)}

for criterion, rubric_text in self.rubrics.items():

system_prompt = (

"You are an expert repository reviewer for SWE-bench.\n"

f"Evaluate this patch based strictly on: {rubric_text}\n"

"Output exactly one integer token from 1 to 10. Do not write text."

)

user_prompt = f"### TARGET ISSUE:\n{issue}\n\n### CANDIDATE PATCH DIFF:\n{patch}"

response = client.chat.completions.create(

model=self.model,

messages=[

{"role": "system", "content": system_prompt},

{"role": "user", "content": user_prompt}

],

max_tokens=1,

logprobs=True,

top_logprobs=15

)

top_logprobs = response.choices[0].logprobs.content[0].top_logprobs

valid_probs = {}

total_raw_prob = 0.0

for item in top_logprobs:

token = item.token.strip()

if token in score_mapping:

prob = math.exp(item.logprob)

valid_probs[token] = prob

total_raw_prob += prob

if total_raw_prob == 0:

scores[criterion] = 1.0

continue

# Continuous expectation extraction

expected_score = 0.0

for token, prob in valid_probs.items():

normalized_prob = prob / total_raw_prob

expected_score += normalized_prob * score_mapping[token]

scores[criterion] = expected_score

scores["aggregate"] = sum(scores.values()) / len(scores)

return scores

# --- PIPELINE IMPLEMENTATION ---

if __name__ == "__main__":

repo_dir = "./swe_bench_sandbox_repo"

# 1. Clean local folders and check out actual codebase files

if os.path.exists(repo_dir):

shutil.rmtree(repo_dir)

print("Fetching actual source repository files...")

repo = git.Repo.clone_from(SWE_BENCH_CASE["repo_url"], repo_dir)

repo.git.checkout(SWE_BENCH_CASE["commit"])

print(f"Repository isolated successfully at commit: {SWE_BENCH_CASE['commit']}")

# 2. Set up code validation rubrics

swe_rubrics = {

"syntactic_correctness": "Does the patch use valid Python syntax without breaking indentation bounds?",

"regression_safety": "Does the solution avoid breaking universal routing functionalities elsewhere in the file?"

}

evaluator = SWEEvaluator(rubrics=swe_rubrics)

print("\nExecuting live API evaluation calls...")

# Run Judge pipeline

judge_scores = evaluator.run_judge(SWE_BENCH_CASE["issue_description"], SWE_BENCH_CASE["candidate_patch"])

# Run Verifier pipeline

verifier_scores = evaluator.run_verifier(SWE_BENCH_CASE["issue_description"], SWE_BENCH_CASE["candidate_patch"])

# Clean up file structures

shutil.rmtree(repo_dir)

# 3. Print side-by-side output metrics

print("\n=== SYSTEM PERFORMANCE RESULTS ===")

print(f"LLM-as-a-Judge Score : {judge_scores['aggregate']:.2f} / 10.0")

print(f"LLM-as-a-Verifier Score : {verifier_scores['aggregate']:.4f} / 10.0")

print("\n=== THE GRAP_GAP ANALYSIS ===")

resolution_loss = abs(judge_scores["aggregate"] - verifier_scores["aggregate"])

print(f"Absolute Resolution Loss: {resolution_loss:.4f} score units.")

print("\nDemonstrated Improvements of the Verifier:")

print("1. Fine-Grained Scaling: The Judge forced a rigid integer bound, ignoring alternative distributions.")

print("2. Uncertainty Tracking: The Verifier captured the model's token distribution split, indicating code execution risk.")

print("3. Deterministic Safety: Verifier avoids faulty regex failures because it samples exact logprob layers.")

Metrics and Observable Differences

Running this pipeline highlights three major differences between the two methods:

• The Resolution Gap: If the model thinks a code patch is between an 8 and a 9, a Judge is forced to pick one token via argmax (e.g., dropping down to exactly 8.00). The Verifier preserves this uncertainty by producing an expected score (e.g., 8.4120) based on the underlying token probabilities.
• Token Tracking Costs: The Judge uses complete sentence string generation to produce a text critique before outputting its final score. The Verifier tracks the exact same mathematical parameters using only max_tokens=1, saving over 90% in token costs.
• Format Fragility: The Judge relies on regular expressions to extract text strings from a model response. If a frontier model tweaks its formatting (e.g., writing SCORE=8 instead of FINAL_SCORE: [8]), the Judge pipeline breaks. The Verifier avoids this issue by mapping logprob indices directly. [1, 2, 3]