OpenTelemetry and Distributed Tracing in MCP

OpenTelemetry and Distributed Tracing in MCP

As engineering teams deploy autonomous AI agents across production workflows, monitoring system behavior moves from simple log review into full-fidelity distributed observability. In a multi-turn diagnostic or coding session, an agent might coordinate ten or more discrete tool invocations across diverse Model Context Protocol (MCP) servers—querying PostgreSQL, searching GitHub pull requests, inspecting Kubernetes pods, and triggering build pipelines.

When a multi-turn turn stalls or fails with an opaque -32603 Internal Error, diagnosing the root cause without distributed tracing becomes nearly impossible. Engineers cannot tell whether latency was consumed by foundation model token generation, slow database queries, network throttling on external APIs, or serialization bottlenecks within local stdio pipes.

By instrumenting MCP servers with OpenTelemetry (OTel) and propagating W3C Trace Context across JSON-RPC boundaries, developers gain complete end-to-end visibility into agent execution graphs.

This technical guide provides the architectural blueprints for OpenTelemetry instrumentation in Node.js and TypeScript MCP servers, demonstrates W3C traceparent context extraction across stdio and SSE transports, and details production integration with Jaeger and Prometheus.

1. Problem Statement and Architecture

In traditional microservices, incoming HTTP requests carry W3C headers (traceparent and tracestate) that allow APM platforms to correlate spans across network hops. However, MCP servers often communicate over local standard I/O (stdio), which lacks native transport-level headers:

Unmonitored Agent Execution (Opaque Black Box):
User Prompt ──> Agent ──> [Tool: query_database] ──> [Tool: fetch_git] ──> 15s Latency Spike!
(Zero visibility into query times, memory usage, or external API failures)

Instrumented OpenTelemetry Architecture:
┌────────────────────────────────────────────────────────┐
│ HOST AI AGENT (Claude Desktop / Cursor / Codex CLI)    │
│ Generates Root Span: trace_id = 4bf92f3577b34da6...    │
└──────────────────────────┬─────────────────────────────┘
                           │ JSON-RPC 2.0 with _meta.traceparent
                           ▼
┌────────────────────────────────────────────────────────┐
│ INSTRUMENTED MCP SERVER (Node.js / TypeScript)         │
│ • Extracts Trace Context from params._meta             │
│ • Creates Active Child Span: "mcp-tool::query_db"      │
│ • Records Metrics: tool.duration_ms, tool.invocations  │
└─────────────┬────────────────────────────┬─────────────┘
              │ OTLP Protocol (gRPC/HTTP)  │ Database Query / API
              ▼                            ▼
┌──────────────────────────┐  ┌──────────────────────────┐
│ OBSERVABILITY BACKEND    │  │ TARGET INFRASTRUCTURE    │
│ • Jaeger (Trace Graphs)  │  │ • PostgreSQL Analytics   │
│ • Prometheus (Metrics)   │  │ • GitHub API Rate Limits │
│ • Datadog / Honeycomb    │  │ • Kubernetes Cluster     │
└──────────────────────────┘  └──────────────────────────┘

To bridge this gap without breaking protocol specifications, instrumented MCP clients attach W3C Trace Context metadata inside the JSON-RPC request frame's params._meta object:

$\text{traceparent} = \text{00}-\underbrace{\text{4bf92f3577b34da6a3ce929d0e0e4736}}{\text{Trace ID}}-\underbrace{\text{00f067aa0ba902b7}}{\text{Parent Span ID}}-\text{01}$

When the MCP server receives a tools/call request, its OpenTelemetry middleware extracts this metadata, resumes the active distributed trace context, and records fine-grained telemetry attributes for downstream export to Jaeger, Prometheus, or cloud observability vendors.

Beyond distributed tracing, production MCP monitoring requires collecting operational metrics:

• ▸mcp.tool.invocations: A counter tracking call frequency per tool name and host client.
• ▸mcp.tool.duration: A histogram capturing execution latencies, alerting on 95th and 99th percentile degradation.
• ▸mcp.tool.errors: A counter partitioned by JSON-RPC error codes (-32601, -32602, -32603).

Telemetry data is exported via the OpenTelemetry Protocol (OTLP). Teams can choose between gRPC (port 4317) for high-efficiency binary streaming or HTTP/Protobuf (port 4318) for simpler firewall traversal in locked-down enterprise networks.

2. Prerequisites and Environment Matrix

Before instrumenting your custom MCP server, verify that your development and monitoring infrastructure satisfies the version matrix:

Running a Local Jaeger OTLP Collector

To verify your trace spans locally, launch Jaeger with native OTLP ingestion support via Docker:

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Once running, navigate to to inspect live distributed trace waterfalls.

3. Multi-Host Configuration Snippets

Configure your AI assistant host to pass standard OpenTelemetry environment variables to the MCP server process.

Claude Desktop Configuration

Configure Claude Desktop to provide service naming and collector endpoint configuration:

{
  "mcpServers": {
    "analytics-mcp": {
      "command": "node",
      "args": ["./dist/server.js"],
      "env": {
        "OTEL_SERVICE_NAME": "analytics-mcp-service",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "",
        "NODE_ENV": "production"
      }
    }
  }
}

Cursor IDE Configuration (.cursor/mcp.json)

Configure Cursor to enable telemetry instrumentation:

{
  "mcpServers": {
    "analytics-mcp": {
      "command": "node",
      "args": ["./dist/server.js"],
      "env": {
        "OTEL_SERVICE_NAME": "cursor-analytics-mcp",
        "OTEL_EXPORTER_OTLP_ENDPOINT": ""
      }
    }
  }
}

OpenAI Codex CLI Configuration (~/.codex/config.toml)

For terminal workflows using the OpenAI Codex CLI:

[mcp.servers.analytics_mcp]
command = "node"
args = ["./dist/server.js"]

[mcp.servers.analytics_mcp.env]
OTEL_SERVICE_NAME = "codex-analytics-mcp"
OTEL_EXPORTER_OTLP_ENDPOINT = ""

4. Step-by-Step Server Instrumentation and Context Propagation

Step 1: OpenTelemetry SDK Bootstrap (tracing.ts)

OpenTelemetry must initialize before any other application modules are imported to ensure seamless monkey-patching of Node.js core libraries (such as http, https, and net):

// tracing.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const exporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT 
    ? `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces` 
    : '/v1/traces'
});

export const otelSdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME || 'mcp-server'
  }),
  spanProcessor: new BatchSpanProcessor(exporter, {
    maxQueueSize: 2048,
    scheduledDelayMillis: 1000
  })
});

otelSdk.start();

Step 2: Context Extraction and Tool Wrapping Middleware

In your primary MCP server implementation, wrap tool execution handlers in active spans and extract incoming W3C traceparents:

// server.ts
import './tracing.js';
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';
import { trace, context, propagation, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('mcp-instrumentation', '1.0.0');

const server = new Server(
  { name: 'instrumented-mcp-server', version: '1.0.0' },
  { capabilities: { tools: {} } }
);

server.setRequestHandler(ListToolsRequestSchema, async () => {
  return {
    tools: [
      {
        name: 'query_customer_records',
        description: 'Fetch customer analytics by region with execution tracing.',
        inputSchema: {
          type: 'object',
          properties: {
            region: { type: 'string' }
          },
          required: ['region']
        }
      }
    ]
  };
});

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const { name, arguments: args } = request.params;
  const meta = (request.params as any)._meta || {};

  // Extract W3C traceparent from incoming JSON-RPC metadata
  const parentContext = propagation.extract(context.active(), meta, {
    get: (carrier, key) => carrier[key.toLowerCase()]
  });

  return context.with(parentContext, async () => {
    return tracer.startActiveSpan(`mcp-tool::${name}`, async (span) => {
      span.setAttribute('mcp.tool.name', name);
      span.setAttribute('mcp.tool.argument_count', Object.keys(args || {}).length);

      try {
        const startTime = Date.now();

        // Execute core tool logic
        const resultPayload = [
          { customer_id: 'cust_8412', name: 'Acme Corp', status: 'active' }
        ];

        span.setAttribute('mcp.tool.latency_ms', Date.now() - startTime);
        span.setStatus({ code: SpanStatusCode.OK });

        return {
          content: [{ type: 'text', text: JSON.stringify(resultPayload) }]
        };
      } catch (error: any) {
        span.recordException(error);
        span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
        throw error;
      } finally {
        span.end();
      }
    });
  });
});

async function run() {
  const transport = new StdioServerTransport();
  await server.connect(transport);
}

run();

5. Common Errors and Production Troubleshooting

Error 1: Broken Trace Context Propagation Across Standard I/O

• ▸Symptom: Jaeger displays tool invocations as disconnected root traces rather than child spans of the caller's agent session.
• ▸Root Cause: The client host dispatched tools/call over stdio without providing W3C context headers. Because stdio lacks HTTP headers, standard automatic propagation fails.
• ▸Resolution: Pass the traceparent key inside the request frame's _meta parameter object: {"params": {"name": "...", "arguments": {...}, "_meta": {"traceparent": "00-4bf...-01"}}}.

Error 2: Event Loop Latency and Crashes with SimpleSpanProcessor

• ▸Symptom: High tool invocation throughput causes the Node.js event loop to block, leading to intermittent timeout errors.
• ▸Root Cause: Using SimpleSpanProcessor forces synchronous network I/O to the OTLP exporter for every finished span, starving the main event loop.
• ▸Resolution: Always utilize BatchSpanProcessor in production environments. BatchSpanProcessor buffers spans in memory and flushes them asynchronously in batches.

Error 3: Sensitive Data Leaks in Trace Attributes

• ▸Symptom: Confidential database passwords or customer PII are discoverable in the Jaeger web UI.
• ▸Root Cause: Middleware naively serializes the raw request.params.arguments object directly into span attributes.
• ▸Resolution: Enforce attribute masking. Never serialize arguments without checking against an allowlist of non-sensitive parameter keys.

6. External Citations and Security Guardrails

Official Specifications and References

• ▸OpenTelemetry JavaScript SDK Documentation (opens in a new tab) — Official implementation guide for Node.js tracing and metric collection.
• ▸W3C Trace Context Specification (opens in a new tab) — Standard for distributed tracing header formats and context propagation.
• ▸Model Context Protocol Specification (opens in a new tab) — Official JSON-RPC 2.0 schema specification for tools, resources, and clients.

Security and Observability Guardrails

When instrumenting production MCP servers:

1. ▸Sanitize Tool Arguments: Never record raw SQL query strings containing personal information or authentication secrets into telemetry attributes.
2. ▸Configure Secure OTLP Transports: When shipping telemetry across public network boundaries, require mTLS or authenticated HTTPS connections to your OpenTelemetry collector.