JEM Trajectory Evaluation for MCP Tool Calls
Evaluate and benchmark multi-turn Model Context Protocol agent trajectories using Judged Exact Match, step-level invariants, and automated CI regression suites.
Generate & Validate Multi-Client MCP Config
One-click export with environment variables & path locators for Claude Desktop, Cursor, Windsurf, and OpenAI Codex CLI.
JEM Trajectory Evaluation for MCP Tool Calls
Traditional automated software benchmarks evaluate language models using lexical string distance metrics such as BLEU, ROUGE, or naive string exact match. In autonomous agent engineering powered by the Model Context Protocol (MCP), these legacy metrics break down entirely.
When an agent resolves an incident or inspects a database cluster, there is rarely a single "golden" sequence of commands. An agent investigating an application performance anomaly might invoke database_explain_query before checking system_metrics, or it might filter database rows using equivalent SQL expressions with differing whitespace or column projections. Checking strictly whether the final string or intermediate tool calls match a recorded template generates massive false negative rates, paralyzing automated continuous integration pipelines.
To solve this evaluation bottleneck, production AI engineering teams have standardized on Judged Exact Match (JEM) and Joint Execution Verification (JEV).
JEM evaluates whether an agent trajectory preserves semantic correctness, respects tool schema constraints, and achieves verified goal transitions across multi-step MCP execution logs. This guide provides the complete mathematical framework, Pydantic data models, and an automated continuous integration test harness for benchmarking MCP agent trajectories.
1. Problem Statement and Architecture
An autonomous agent operating over MCP executes a multi-turn trajectory $\tau$ composed of an alternating sequence of environment states, model tool actions, and server observations:
$\tau = (s_0, a_0, o_1, a_1, o_2, \dots, a_T, o_T, s_F)$
Where $s_0$ is the initial environment state (such as a database snapshot or git commit), $a_t$ represents the JSON-RPC 2.0 tool invocation issued by the agent, $o_t$ is the structured output returned by the MCP server, and $s_F$ is the final environment state.
Traditional Lexical Matching (Brittle & Flaky):
Agent Output: "SELECT id, name FROM users" != Golden: "SELECT * FROM users" ──> FAIL (False Negative)
Judged Exact Match (JEM) Evaluation Architecture:
┌────────────────────────────────────────────────────────┐
│ RAW MCP EXECUTION TRACE │
│ Step 1: tools/list ──> Step 2: query_db ──> Step 3:... │
└──────────────────────────┬─────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ STAGE 1: DETERMINISTIC INVARIANT VERIFICATION │
│ • JSON-RPC 2.0 Frame Structure │
│ • Registered Tool Catalog Validation │
│ • JSON Schema Type & Constraint Checking │
│ • Destructive Side-Effect Boundary Rules │
└──────────────────────────┬─────────────────────────────┘
│ Passed (Step Validity Score: 1.0)
▼
┌────────────────────────────────────────────────────────┐
│ STAGE 2: SEMANTIC STATE & GOAL VERIFICATION (LLM JUDGE)│
│ • State Transition: Did the environment reach goal? │
│ • Step Efficiency: Were intermediate calls circular? │
│ • Parameter Safety: Were queries non-destructive? │
└──────────────────────────┬─────────────────────────────┘
│
▼
Final JEM Benchmark ScoreThe JEM evaluation engine operates in two decoupled stages:
- ▸Deterministic Invariant Verification ($JEM_{\text{step}}$): Verifies that every intermediate tool call $a_t$ adheres strictly to the target MCP server's published schema, contains required parameters, avoids prohibited side effects, and adheres to JSON-RPC 2.0 protocol specifications without requiring expensive LLM inference.
- ▸Semantic Outcome Verification ($JEM_{\text{goal}}$): Uses a calibrated reasoning model (or deterministic unit tests against a sandbox database) to evaluate whether the final state transition $s_0 \to s_F$ satisfies the user's initial objective with mathematical and logical equivalence.
2. Prerequisites and Environment Matrix
Before configuring the JEM test harness, ensure your testing environment meets the runtime dependency requirements:
| Component | Minimum Version | Recommended | Purpose |
|---|---|---|---|
| Python | 3.11.0 | 3.12.4 | Base evaluation harness runtime |
| Pydantic | 2.8.0 | 2.10.4 | Structured schema validation for trace steps |
| Pytest | 8.2.0 | 8.3.4 | Test runner and regression report generation |
| Anthropic SDK | 0.40.0 | 0.45.0 | Evaluator judge model API client |
| @modelcontextprotocol/sdk | 1.6.0 | 1.7.0 | Core MCP client libraries and type definitions |
Ensure your local or CI execution sandbox has access to test fixtures (such as disposable SQLite databases or ephemeral Git repositories) for verifying environment state transitions.
3. Multi-Host Configuration Snippets
To collect and evaluate execution traces across varied developer environments, register the evaluation harness server in your agent host configurations.
Claude Desktop Configuration
Configure Claude Desktop to emit verbose diagnostic logs that capture standard I/O communication frames:
{
"mcpServers": {
"analytics-database": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-postgres", "postgresql:///test_db"]
},
"git-tools": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-filesystem", "/workspace/test-repo"]
}
}
}Cursor IDE Configuration (.cursor/mcp.json)
Configure Cursor to log tool invocation traces into a dedicated workspace test directory:
{
"mcpServers": {
"eval-harness": {
"command": "python",
"args": ["-m", "mcp_eval_harness"],
"env": {
"TRACE_OUTPUT_DIR": "./tests/fixtures/traces",
"RECORD_JSONRPC_FRAMES": "true"
}
}
}
}OpenAI Codex CLI Configuration (~/.codex/config.toml)
When capturing trajectories using the OpenAI Codex CLI:
[mcp.servers.eval_harness]
command = "python"
args = ["-m", "mcp_eval_harness"]
[mcp.servers.eval_harness.env]
TRACE_LOG_PATH = "./test_traces.jsonl"4. Step-by-Step Test Harness and Trajectory Scoring Engine
Step 1: Trajectory Wire Frame Representation
During live execution, an agent logs each JSON-RPC request and response pair into an execution trace file:
{
"trajectory_id": "traj_deploy_fix_842",
"goal": "Find all users registered before 2024 who have never placed an order and mark their status as archived.",
"steps": [
{
"step_index": 1,
"method": "tools/call",
"tool_name": "query_database",
"arguments": {
"sql": "SELECT u.id FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.created_at < '2024-01-01' AND o.id IS NULL;"
},
"result": {
"rows": [{"id": 104}, {"id": 208}, {"id": 319}],
"row_count": 3
},
"latency_ms": 142
},
{
"step_index": 2,
"method": "tools/call",
"tool_name": "execute_mutation",
"arguments": {
"sql": "UPDATE users SET status = 'archived' WHERE id IN (104, 208, 319);"
},
"result": {
"rows_affected": 3
},
"latency_ms": 89
}
],
"expected_final_state": {
"archived_user_count": 3
}
}Step 2: Python Evaluation Engine Implementation
Below is a complete test harness implementing deterministic schema validation alongside an LLM semantic judge:
# jem_evaluator.py
import json
from typing import List, Dict, Any, Optional
from pydantic import BaseModel, Field
import anthropic
class ToolExecutionStep(BaseModel):
step_index: int
method: str
tool_name: str
arguments: Dict[str, Any]
result: Any
latency_ms: int
class TrajectoryTrace(BaseModel):
trajectory_id: str
goal: str
steps: List[ToolExecutionStep]
expected_final_state: Dict[str, Any]
class JEMEvaluationResult(BaseModel):
trajectory_id: str
step_validity: float = Field(ge=0.0, le=1.0)
outcome_accuracy: float = Field(ge=0.0, le=1.0)
composite_jem_score: float = Field(ge=0.0, le=1.0)
passed: bool
failure_reason: Optional[str] = None
diagnostics: str
class JEMEngine:
def __init__(self, registered_schemas: Dict[str, Any]):
self.schemas = registered_schemas
self.client = anthropic.Anthropic()
def evaluate(self, trace: TrajectoryTrace) -> JEMEvaluationResult:
# Phase 1: Deterministic Invariant Validation
for step in trace.steps:
if step.tool_name not in self.schemas:
return JEMEvaluationResult(
trajectory_id=trace.trajectory_id,
step_validity=0.0,
outcome_accuracy=0.0,
composite_jem_score=0.0,
passed=False,
failure_reason=f"Unknown tool invoked: {step.tool_name}",
diagnostics="Agent attempted to invoke an unregistered MCP tool."
)
# Check required parameters against schema
schema = self.schemas[step.tool_name]
required_keys = schema.get("required", [])
for key in required_keys:
if key not in step.arguments:
return JEMEvaluationResult(
trajectory_id=trace.trajectory_id,
step_validity=0.5,
outcome_accuracy=0.0,
composite_jem_score=0.25,
passed=False,
failure_reason=f"Missing required parameter '{key}' for tool '{step.tool_name}'",
diagnostics=f"Step {step.step_index} violated JSON Schema requirements."
)
# Phase 2: Semantic Trajectory Judge via Frontier Model
judge_prompt = f"""
You are a Judged Exact Match (JEM) evaluator for an autonomous MCP agent.
Goal: {trace.goal}
Recorded Execution Steps:
{json.dumps([s.model_dump() for s in trace.steps], indent=2)}
Expected Environment State:
{json.dumps(trace.expected_final_state, indent=2)}
Score the trajectory on two dimensions:
1. Step Validity (0.0 to 1.0): Were the tool calls efficient, non-redundant, and logically safe?
2. Outcome Accuracy (0.0 to 1.0): Did the actions achieve mathematical and state equivalence with the goal?
Respond strictly in valid JSON format:
{{
"step_validity": 1.0,
"outcome_accuracy": 1.0,
"reasoning": "Explanation of score"
}}
"""
response = self.client.messages.create(
model="claude-3-7-sonnet-20250219",
max_tokens=1024,
temperature=0.0,
messages=[{"role": "user", "content": judge_prompt}]
)
content = response.content[0].text
# Strip potential markdown formatting
cleaned_json = content.replace("```json", "").replace("```", "").strip()
parsed = json.loads(cleaned_json)
step_score = float(parsed["step_validity"])
outcome_score = float(parsed["outcome_accuracy"])
composite = (step_score * 0.4) + (outcome_score * 0.6)
passed = (step_score >= 0.8) and (outcome_score >= 0.9)
return JEMEvaluationResult(
trajectory_id=trace.trajectory_id,
step_validity=step_score,
outcome_accuracy=outcome_score,
composite_jem_score=composite,
passed=passed,
failure_reason=None if passed else "Score fell below acceptance threshold",
diagnostics=parsed.get("reasoning", "")
)5. Common Errors and Production Troubleshooting
Error 1: Flaky Benchmarks Due to Evaluator Temperature Drift
- ▸Symptom: CI test runs pass on initial push but intermittently fail on identical code commits, reporting fluctuating outcome scores (0.75 vs 0.95).
- ▸Root Cause: The LLM judge request was dispatched with default non-zero sampling temperatures (such as 0.7 or 1.0), introducing stochastic evaluation variance.
- ▸Resolution: Hardcode
temperature=0.0on all evaluator inference calls. In enterprise production, employ majority voting across three deterministic evaluations to eliminate tail ambiguity.
Error 2: Latency Bottlenecks During Multi-Step CI Runs
- ▸Symptom: The automated regression test suite takes more than forty minutes to run fifty agent trajectory benchmarks.
- ▸Root Cause: Trajectories are evaluated sequentially with synchronous API requests, causing cumulative network delays.
- ▸Resolution: Refactor the test harness to process trajectory evaluations concurrently using Python's
asyncio.gatherand theAsyncAnthropicclient with bounded semaphore concurrency (asyncio.Semaphore(10)).
Error 3: State Contamination Across Concurrent Test Fixtures
- ▸Symptom: Step 2 passes in local single-threaded runs but fails during parallel CI runs because target database records have already been modified by another test thread.
- ▸Root Cause: Multiple concurrent tests execute against a single persistent PostgreSQL database instance without transaction isolation.
- ▸Resolution: Wrap each test trajectory execution inside a PostgreSQL transaction that executes an explicit
ROLLBACKduring test teardown, or assign each worker an ephemeral SQLite memory instance.
6. External Citations and Security Guardrails
Official Specifications and References
- ▸SWE-bench Evaluation Methodology (opens in a new tab) — Foundational research on evaluating autonomous software engineering agents against verifiable real-world environments.
- ▸Model Context Protocol Specification (opens in a new tab) — Official JSON-RPC 2.0 schema specification for tool definitions, call parameters, and lifecycle events.
- ▸Pydantic Core Documentation (opens in a new tab) — High-performance data validation library used for deterministic JSON Schema enforcement.
Security and Redaction Guardrails
When logging and evaluating MCP trajectories:
- ▸Redact Sensitive Credentials in Traces: Before storing JSON-RPC frames in repository fixtures or transmitting them to an LLM judge, sanitize all Authorization headers, API keys, database connection strings, and personal access tokens.
- ▸Prevent Arbitrary Execution: In sandbox verification environments, restrict tool execution privileges to prevent agents from escaping container boundaries or making unauthenticated external network requests.
Build your full agent toolstack in the Visual Generator
Combine verified MCP servers with databases, search APIs, and memory graphs in a single configuration file.
Did this setup guide work with your AI host?
Help test this configuration across Claude, Cursor, and Codex CLI. Be the first to report:
Written by Rad Tome
Lead AI Systems Architect & Maintainer, MCP Codex
Specializing in Model Context Protocol (MCP) integrations, autonomous AI agent orchestration, and distributed developer toolchains. Researches and benchmarks production MCP client-server architectures across OpenAI Codex, Claude, and Cursor.
Related Guides
OpenCode Free AI Agent Guide
Guide to running OpenCode coding agent completely free without API keys using Zen free tier, local LLMs, and OAuth accounts.
Dev ToolsSelf-Correcting Verification Loops in Agents
Build robust self-correcting /loop verification harnesses for AI agents using Test-Driven Agent Development, dual-agent critics, and MCP test sandboxes.
Dev ToolsAutomated Doc Verification Loops with Claude CLI
Build automated documentation verification loops with Claude CLI and MCP. Validate code snippets, test tool schemas, and sync SDK docs in CI/CD. Verified.