Dev Tools·
advanced
·12 min read·Sep 24, 2026·Updated Nov 12, 2026
·Lead AI Systems Architect·Verification Charter

JEM Trajectory Evaluation for MCP Tool Calls

Evaluate and benchmark multi-turn Model Context Protocol agent trajectories using Judged Exact Match, step-level invariants, and automated CI regression suites.

jemevaluationbenchmarksmcptrajectory-analysistestingci-cd
Interactive Tool
1-Click Export

Generate & Validate Multi-Client MCP Config

One-click export with environment variables & path locators for Claude Desktop, Cursor, Windsurf, and OpenAI Codex CLI.

Open in Generator

JEM Trajectory Evaluation for MCP Tool Calls

Traditional automated software benchmarks evaluate language models using lexical string distance metrics such as BLEU, ROUGE, or naive string exact match. In autonomous agent engineering powered by the Model Context Protocol (MCP), these legacy metrics break down entirely.

When an agent resolves an incident or inspects a database cluster, there is rarely a single "golden" sequence of commands. An agent investigating an application performance anomaly might invoke database_explain_query before checking system_metrics, or it might filter database rows using equivalent SQL expressions with differing whitespace or column projections. Checking strictly whether the final string or intermediate tool calls match a recorded template generates massive false negative rates, paralyzing automated continuous integration pipelines.

To solve this evaluation bottleneck, production AI engineering teams have standardized on Judged Exact Match (JEM) and Joint Execution Verification (JEV).

JEM evaluates whether an agent trajectory preserves semantic correctness, respects tool schema constraints, and achieves verified goal transitions across multi-step MCP execution logs. This guide provides the complete mathematical framework, Pydantic data models, and an automated continuous integration test harness for benchmarking MCP agent trajectories.


1. Problem Statement and Architecture

An autonomous agent operating over MCP executes a multi-turn trajectory $\tau$ composed of an alternating sequence of environment states, model tool actions, and server observations:

$\tau = (s_0, a_0, o_1, a_1, o_2, \dots, a_T, o_T, s_F)$

Where $s_0$ is the initial environment state (such as a database snapshot or git commit), $a_t$ represents the JSON-RPC 2.0 tool invocation issued by the agent, $o_t$ is the structured output returned by the MCP server, and $s_F$ is the final environment state.

code
Traditional Lexical Matching (Brittle & Flaky):
Agent Output: "SELECT id, name FROM users" != Golden: "SELECT * FROM users" ──> FAIL (False Negative)

Judged Exact Match (JEM) Evaluation Architecture:
┌────────────────────────────────────────────────────────┐
│               RAW MCP EXECUTION TRACE                  │
│ Step 1: tools/list ──> Step 2: query_db ──> Step 3:... │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────────┐
│ STAGE 1: DETERMINISTIC INVARIANT VERIFICATION          │
│ • JSON-RPC 2.0 Frame Structure                         │
│ • Registered Tool Catalog Validation                   │
│ • JSON Schema Type & Constraint Checking               │
│ • Destructive Side-Effect Boundary Rules               │
└──────────────────────────┬─────────────────────────────┘
                           │ Passed (Step Validity Score: 1.0)
                           ▼
┌────────────────────────────────────────────────────────┐
│ STAGE 2: SEMANTIC STATE & GOAL VERIFICATION (LLM JUDGE)│
│ • State Transition: Did the environment reach goal?   │
│ • Step Efficiency: Were intermediate calls circular?   │
│ • Parameter Safety: Were queries non-destructive?      │
└──────────────────────────┬─────────────────────────────┘
                           │
                           ▼
               Final JEM Benchmark Score

The JEM evaluation engine operates in two decoupled stages:

  1. ▸Deterministic Invariant Verification ($JEM_{\text{step}}$): Verifies that every intermediate tool call $a_t$ adheres strictly to the target MCP server's published schema, contains required parameters, avoids prohibited side effects, and adheres to JSON-RPC 2.0 protocol specifications without requiring expensive LLM inference.
  2. ▸Semantic Outcome Verification ($JEM_{\text{goal}}$): Uses a calibrated reasoning model (or deterministic unit tests against a sandbox database) to evaluate whether the final state transition $s_0 \to s_F$ satisfies the user's initial objective with mathematical and logical equivalence.

2. Prerequisites and Environment Matrix

Before configuring the JEM test harness, ensure your testing environment meets the runtime dependency requirements:

ComponentMinimum VersionRecommendedPurpose
Python3.11.03.12.4Base evaluation harness runtime
Pydantic2.8.02.10.4Structured schema validation for trace steps
Pytest8.2.08.3.4Test runner and regression report generation
Anthropic SDK0.40.00.45.0Evaluator judge model API client
@modelcontextprotocol/sdk1.6.01.7.0Core MCP client libraries and type definitions

Ensure your local or CI execution sandbox has access to test fixtures (such as disposable SQLite databases or ephemeral Git repositories) for verifying environment state transitions.


3. Multi-Host Configuration Snippets

To collect and evaluate execution traces across varied developer environments, register the evaluation harness server in your agent host configurations.

Claude Desktop Configuration

Configure Claude Desktop to emit verbose diagnostic logs that capture standard I/O communication frames:

json
{
  "mcpServers": {
    "analytics-database": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-postgres", "postgresql:///test_db"]
    },
    "git-tools": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/workspace/test-repo"]
    }
  }
}

Cursor IDE Configuration (.cursor/mcp.json)

Configure Cursor to log tool invocation traces into a dedicated workspace test directory:

json
{
  "mcpServers": {
    "eval-harness": {
      "command": "python",
      "args": ["-m", "mcp_eval_harness"],
      "env": {
        "TRACE_OUTPUT_DIR": "./tests/fixtures/traces",
        "RECORD_JSONRPC_FRAMES": "true"
      }
    }
  }
}

OpenAI Codex CLI Configuration (~/.codex/config.toml)

When capturing trajectories using the OpenAI Codex CLI:

toml
[mcp.servers.eval_harness]
command = "python"
args = ["-m", "mcp_eval_harness"]

[mcp.servers.eval_harness.env]
TRACE_LOG_PATH = "./test_traces.jsonl"

4. Step-by-Step Test Harness and Trajectory Scoring Engine

Step 1: Trajectory Wire Frame Representation

During live execution, an agent logs each JSON-RPC request and response pair into an execution trace file:

json
{
  "trajectory_id": "traj_deploy_fix_842",
  "goal": "Find all users registered before 2024 who have never placed an order and mark their status as archived.",
  "steps": [
    {
      "step_index": 1,
      "method": "tools/call",
      "tool_name": "query_database",
      "arguments": {
        "sql": "SELECT u.id FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.created_at < '2024-01-01' AND o.id IS NULL;"
      },
      "result": {
        "rows": [{"id": 104}, {"id": 208}, {"id": 319}],
        "row_count": 3
      },
      "latency_ms": 142
    },
    {
      "step_index": 2,
      "method": "tools/call",
      "tool_name": "execute_mutation",
      "arguments": {
        "sql": "UPDATE users SET status = 'archived' WHERE id IN (104, 208, 319);"
      },
      "result": {
        "rows_affected": 3
      },
      "latency_ms": 89
    }
  ],
  "expected_final_state": {
    "archived_user_count": 3
  }
}

Step 2: Python Evaluation Engine Implementation

Below is a complete test harness implementing deterministic schema validation alongside an LLM semantic judge:

python
# jem_evaluator.py
import json
from typing import List, Dict, Any, Optional
from pydantic import BaseModel, Field
import anthropic

class ToolExecutionStep(BaseModel):
    step_index: int
    method: str
    tool_name: str
    arguments: Dict[str, Any]
    result: Any
    latency_ms: int

class TrajectoryTrace(BaseModel):
    trajectory_id: str
    goal: str
    steps: List[ToolExecutionStep]
    expected_final_state: Dict[str, Any]

class JEMEvaluationResult(BaseModel):
    trajectory_id: str
    step_validity: float = Field(ge=0.0, le=1.0)
    outcome_accuracy: float = Field(ge=0.0, le=1.0)
    composite_jem_score: float = Field(ge=0.0, le=1.0)
    passed: bool
    failure_reason: Optional[str] = None
    diagnostics: str

class JEMEngine:
    def __init__(self, registered_schemas: Dict[str, Any]):
        self.schemas = registered_schemas
        self.client = anthropic.Anthropic()

    def evaluate(self, trace: TrajectoryTrace) -> JEMEvaluationResult:
        # Phase 1: Deterministic Invariant Validation
        for step in trace.steps:
            if step.tool_name not in self.schemas:
                return JEMEvaluationResult(
                    trajectory_id=trace.trajectory_id,
                    step_validity=0.0,
                    outcome_accuracy=0.0,
                    composite_jem_score=0.0,
                    passed=False,
                    failure_reason=f"Unknown tool invoked: {step.tool_name}",
                    diagnostics="Agent attempted to invoke an unregistered MCP tool."
                )

            # Check required parameters against schema
            schema = self.schemas[step.tool_name]
            required_keys = schema.get("required", [])
            for key in required_keys:
                if key not in step.arguments:
                    return JEMEvaluationResult(
                        trajectory_id=trace.trajectory_id,
                        step_validity=0.5,
                        outcome_accuracy=0.0,
                        composite_jem_score=0.25,
                        passed=False,
                        failure_reason=f"Missing required parameter '{key}' for tool '{step.tool_name}'",
                        diagnostics=f"Step {step.step_index} violated JSON Schema requirements."
                    )

        # Phase 2: Semantic Trajectory Judge via Frontier Model
        judge_prompt = f"""
You are a Judged Exact Match (JEM) evaluator for an autonomous MCP agent.
Goal: {trace.goal}
Recorded Execution Steps:
{json.dumps([s.model_dump() for s in trace.steps], indent=2)}

Expected Environment State:
{json.dumps(trace.expected_final_state, indent=2)}

Score the trajectory on two dimensions:
1. Step Validity (0.0 to 1.0): Were the tool calls efficient, non-redundant, and logically safe?
2. Outcome Accuracy (0.0 to 1.0): Did the actions achieve mathematical and state equivalence with the goal?

Respond strictly in valid JSON format:
{{
  "step_validity": 1.0,
  "outcome_accuracy": 1.0,
  "reasoning": "Explanation of score"
}}
"""
        response = self.client.messages.create(
            model="claude-3-7-sonnet-20250219",
            max_tokens=1024,
            temperature=0.0,
            messages=[{"role": "user", "content": judge_prompt}]
        )

        content = response.content[0].text
        # Strip potential markdown formatting
        cleaned_json = content.replace("```json", "").replace("```", "").strip()
        parsed = json.loads(cleaned_json)

        step_score = float(parsed["step_validity"])
        outcome_score = float(parsed["outcome_accuracy"])
        composite = (step_score * 0.4) + (outcome_score * 0.6)
        passed = (step_score >= 0.8) and (outcome_score >= 0.9)

        return JEMEvaluationResult(
            trajectory_id=trace.trajectory_id,
            step_validity=step_score,
            outcome_accuracy=outcome_score,
            composite_jem_score=composite,
            passed=passed,
            failure_reason=None if passed else "Score fell below acceptance threshold",
            diagnostics=parsed.get("reasoning", "")
        )

5. Common Errors and Production Troubleshooting

Error 1: Flaky Benchmarks Due to Evaluator Temperature Drift

  • ▸Symptom: CI test runs pass on initial push but intermittently fail on identical code commits, reporting fluctuating outcome scores (0.75 vs 0.95).
  • ▸Root Cause: The LLM judge request was dispatched with default non-zero sampling temperatures (such as 0.7 or 1.0), introducing stochastic evaluation variance.
  • ▸Resolution: Hardcode temperature=0.0 on all evaluator inference calls. In enterprise production, employ majority voting across three deterministic evaluations to eliminate tail ambiguity.

Error 2: Latency Bottlenecks During Multi-Step CI Runs

  • ▸Symptom: The automated regression test suite takes more than forty minutes to run fifty agent trajectory benchmarks.
  • ▸Root Cause: Trajectories are evaluated sequentially with synchronous API requests, causing cumulative network delays.
  • ▸Resolution: Refactor the test harness to process trajectory evaluations concurrently using Python's asyncio.gather and the AsyncAnthropic client with bounded semaphore concurrency (asyncio.Semaphore(10)).

Error 3: State Contamination Across Concurrent Test Fixtures

  • ▸Symptom: Step 2 passes in local single-threaded runs but fails during parallel CI runs because target database records have already been modified by another test thread.
  • ▸Root Cause: Multiple concurrent tests execute against a single persistent PostgreSQL database instance without transaction isolation.
  • ▸Resolution: Wrap each test trajectory execution inside a PostgreSQL transaction that executes an explicit ROLLBACK during test teardown, or assign each worker an ephemeral SQLite memory instance.

6. External Citations and Security Guardrails

Official Specifications and References

Security and Redaction Guardrails

When logging and evaluating MCP trajectories:

  1. ▸Redact Sensitive Credentials in Traces: Before storing JSON-RPC frames in repository fixtures or transmitting them to an LLM judge, sanitize all Authorization headers, API keys, database connection strings, and personal access tokens.
  2. ▸Prevent Arbitrary Execution: In sandbox verification environments, restrict tool execution privileges to prevent agents from escaping container boundaries or making unauthenticated external network requests.
Ready to Deploy?

Build your full agent toolstack in the Visual Generator

Combine verified MCP servers with databases, search APIs, and memory graphs in a single configuration file.

Customize in Generator
Developer Verification & Feedback
0 reports

Did this setup guide work with your AI host?

Help test this configuration across Claude, Cursor, and Codex CLI. Be the first to report:

RT✓

Written by Rad Tome

Lead AI Systems Architect & Maintainer, MCP Codex

Specializing in Model Context Protocol (MCP) integrations, autonomous AI agent orchestration, and distributed developer toolchains. Researches and benchmarks production MCP client-server architectures across OpenAI Codex, Claude, and Cursor.

Editorial Integrity: Package names verified via npm registry API. Repositories verified via GitHub REST API.
Verification Charter →

Related Guides