Automated Doc Verification Loops with Claude CLI

Automated Doc Verification Loops with Claude CLI

Maintaining accurate developer documentation in fast-evolving software projects is a persistent engineering challenge. As software development kits (SDKs), Model Context Protocol (MCP) tool schemas, and API contracts undergo rapid iteration, developer documentation inevitably suffers from "doc rot." Code snippets in README files, quickstart guides, and developer portals quietly become obsolete when parameter names change, packages are deprecated, or schema requirements shift. When prospective developers copy broken sample code into their IDEs, the resulting developer experience leads to frustration, lost developer trust, and unnecessary support issues.

By integrating headless Claude CLI (claude-code) runs into continuous integration (CI/CD) pipelines alongside the Model Context Protocol (MCP), engineering teams can eliminate manual doc audits. In this verification loop architecture, Claude CLI inspects repository markdown files, extracts code blocks and client configurations, tests them against local MCP servers, asserts JSON-RPC schema parity, and opens pull requests with corrective diffs when breaking changes are caught.

Problem Statement and Architecture

Static documentation linters (such as markdown formatters or dead-link checkers) are fundamentally incapable of validating semantic code correctness. A markdown linter can confirm that a fenced code block has valid syntax, but it cannot determine whether an MCP server configuration actually connects, whether a tool argument matches the server's current Zod schema, or whether an exported TypeScript function still exists in the latest SDK build.

Automated documentation verification loops solve this problem by integrating agentic reasoning with executable tool sandboxes:

+-----------------------------------------------------------------------+
|                    GITHUB ACTIONS / CI-CD RUNNER                      |
|             Trigger: Scheduled Cron / Pull Request Event              |
+-----------------------------------+-+---------------------------------+
                                    |
                                    v
+-----------------------------------------------------------------------+
|                  CLAUDE CLI HEADLESS VERIFICATION AGENT               |
|                                                                       |
|  1. Discovers markdown docs across docs/ and packages/               |
|  2. Extracts code snippets (TypeScript, Python, JSON, TOML)           |
|  3. Connects to local MCP servers via Stdio transports                |
|  4. Tests tool invocations against live "tools/list" schemas          |
|  5. Executes test harnesses in ephemeral sandbox containers          |
+-----------------------------------+-+---------------------------------+
                                    |
                    +---------------+---------------+
                    |                               |
                    v                               v
    [All Snippets Passed]               [Doc Rot Detected]
+-------------------------------+       +-------------------------------+
| Log Clean Audit Pass          |       | 1. Generate git diff fix      |
| Update CI Status Badge        |       | 2. File GitHub PR / Issue     |
+-------------------------------+       +-------------------------------+

The verification loop executes through four distinct pipeline phases:

1. ▸Extraction and AST Parsing: Claude CLI traverses repository markdown files, parsing code fences and extracting configuration snippets for Claude Desktop, Cursor, and Codex CLI.
2. ▸Schema Introspection via MCP: The agent connects to local MCP servers over stdio pipes, issues a standard tools/list discovery request, and validates documented parameter names, types, and required fields against live server capabilities.
3. ▸Sandbox Execution: Code snippets are executed in temporary sandbox environments to confirm that imports resolve, types check cleanly, and commands succeed.
4. ▸Automated Remediation: If an API parameter has been renamed or a package reference updated, Claude CLI writes a corrected markdown file and opens a pull request for human maintainers.

Prerequisites and Environment Matrix

Before configuring automated documentation verification loops in your repository, verify that your local workstation or continuous integration runner satisfies the following environment matrix:

You must also configure your continuous integration secrets with:

• ▸ANTHROPIC_API_KEY: Dedicated API key with rate limits configured for CI/CD runs.
• ▸GITHUB_TOKEN: Repository token with contents: write and pull-requests: write permissions.

Multi-Host Configuration Snippets

When validating documentation across diverse AI client ecosystems, the verification agent must test configuration snippets targeting Claude Desktop, Cursor, and OpenAI Codex CLI. Below are the standard multi-host configuration files tested in the documentation pipeline.

Claude CLI Toolchain Configuration

In ~/.claude/config.json (or ~/.config/claude-code/config.json), configure Claude CLI with specialized verification MCP tools:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-filesystem@2026.8.31",
        "./docs"
      ]
    },
    "doc-inspector": {
      "command": "node",
      "args": ["scripts/mcp-doc-inspector.js"],
      "env": {
        "NODE_ENV": "test"
      }
    }
  }
}

Claude Desktop Configuration Under Test

The documentation test suite verifies that documented claude_desktop_config.json snippets match current upstream requirements:

{
  "mcpServers": {
    "postgres": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-postgres@0.6.2",
        "postgresql://reader:secret@/app"
      ]
    }
  }
}

Cursor IDE Configuration Under Test

In .cursor/mcp.json:

{
  "mcpServers": {
    "postgres": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-postgres@0.6.2",
        "postgresql://reader:secret@/app"
      ]
    }
  }
}

OpenAI Codex CLI Configuration Under Test

In ~/.codex/config.toml:

# ~/.codex/config.toml
[mcp_servers.postgres]
command = "npx"
args = [
  "-y",
  "@modelcontextprotocol/server-postgres@0.6.2",
  "postgresql://reader:secret@/app"
]

Step-by-Step Tool Discovery and Invocation

Here is how to implement an automated documentation verification loop in GitHub Actions using Claude CLI and custom MCP verification tools.

1. GitHub Actions Workflow Configuration

Create .github/workflows/doc-verification-loop.yml in your repository:

name: "Documentation Verification Loop"

on:
  schedule:
    - cron: "0 4 * * 1" # Run weekly on Mondays at 04:00 UTC
  workflow_dispatch:

permissions:
  contents: write
  pull-requests: write
  issues: write

jobs:
  verify-documentation:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Setup Node.js Runtime
        uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: "npm"

      - name: Install Project Dependencies
        run: npm ci

      - name: Execute Claude CLI Verification Loop
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          npx @anthropic-ai/claude-code \
            --non-interactive \
            --prompt "Audit all markdown files in docs/. Extract all MCP server JSON and TOML snippets. Validate that all tool names, arguments, and package version tags match the current @modelcontextprotocol schemas. If discrepancies are found, update the files and report the diff."

2. JSON-RPC Schema Inspection Frame

During the verification cycle, the doc-inspector MCP tool connects to the target server binary over stdio and issues a standard tools/list handshake:

{
  "jsonrpc": "2.0",
  "id": 101,
  "method": "tools/list",
  "params": {}
}

The upstream server returns its canonical schema definition:

{
  "jsonrpc": "2.0",
  "id": 101,
  "result": {
    "tools": [
      {
        "name": "query",
        "description": "Execute a PostgreSQL query",
        "inputSchema": {
          "type": "object",
          "properties": {
            "sql": { "type": "string" }
          },
          "required": ["sql"]
        }
      }
    ]
  }
}

The verification loop compares the extracted markdown code block:

// Documented in docs/postgres.md (Obsolete parameter name)
{"name": "query", "arguments": {"query_string": "SELECT 1;"}}

Against the live schema:

// Live schema requirement
{"name": "query", "arguments": {"sql": "SELECT 1;"}}

The agent detects the schema mismatch: query_string has been replaced with sql.

3. Automated Remediation Diff

Upon detecting the mismatch, Claude CLI generates a unified git diff and writes the corrected markdown file directly to disk:

--- a/docs/postgres.md
+++ b/docs/postgres.md
@@ -42,7 +42,7 @@ To query your database:
 ```json
 {
   "name": "query",
-  "arguments": { "query_string": "SELECT * FROM users LIMIT 10;" }
+  "arguments": { "sql": "SELECT * FROM users LIMIT 10;" }
 }

The GitHub Actions workflow then uses `gh pr create` to submit the fix for review by human engineering maintainers.

---

## Common Errors and Production Troubleshooting

When operating Claude CLI in headless continuous integration runners, developers encounter three primary operational failure modes.

### 1. Error: Interactive Prompt Hang in Headless CI

- **Symptom**: GitHub Actions job times out after 60 minutes with `Terminated after 3600 seconds of inactivity`.
- **Root Cause**: The CLI prompted the terminal for confirmation (such as a tool permission prompt or telemetry consent) without receiving a response.
- **Fix**: Always pass the `--non-interactive` flag when executing in automated CI scripts. In environment configurations, export `CI=true` and `NON_INTERACTIVE=true`:
  ```bash
  npx @anthropic-ai/claude-code --non-interactive --prompt "..."

2. Error: spawn npm ENOENT on Windows or Linux Runners

• ▸Symptom: Subprocess execution fails with Error: spawn npm ENOENT.
• ▸Root Cause: The runner subshell does not inherit the global Node.js binary path configured by the action setup step.
• ▸Platform Fixes:
  • ▸Linux / macOS: Explicitly prefix commands with PATH=$PATH:$(npm bin -g) or use absolute paths to Node.
  • ▸Windows (PowerShell): Invoke npm using npm.cmd explicitly in subprocess execution calls:
    powershellCopy

    $env:PATH = "$env:APPDATA\npm;$env:PATH"
    npx.cmd @anthropic-ai/claude-code --version

3. Contaminated Standard Output Corrupting JSON-RPC Streams

• ▸Symptom: SyntaxError: Unexpected token 'I', "INFO: Conn"... is not valid JSON.
• ▸Root Cause: An extracted code snippet or test harness logged diagnostic text directly to stdout, invalidating the JSON-RPC framing required by MCP transports.
• ▸Fix: Ensure that all logging statements inside MCP server implementations write to process.stderr. Reserve process.stdout exclusively for raw JSON-RPC messages.

External Citations and Security Guardrails

To explore advanced documentation verification patterns and Claude CLI features:

• ▸Claude Code CLI Documentation: Read setup and command reference at the official Claude Code CLI Documentation (opens in a new tab).
• ▸Anthropic Tool Use Specifications: Consult tool definition schemas and function calling guidelines at Anthropic Tool Use Documentation (opens in a new tab).
• ▸Model Context Protocol Specification: Review protocol standards and transport requirements at the MCP Specification (opens in a new tab).
• ▸Model Context Protocol TypeScript SDK: View source code and type definitions in the modelcontextprotocol/typescript-sdk repository (opens in a new tab).

Security Guardrails for Automated CI Verification

1. ▸Isolated Sandbox Execution: Never execute arbitrary documentation code blocks with root permissions. Run all snippet tests inside unprivileged Docker containers or isolated gVisor sandboxes.
2. ▸Read-Only API Scopes: Ensure the verification agent's GitHub token is restricted to doc filepaths (docs/**) and cannot modify production application code or deployment workflows.
3. ▸Mandatory Human Approval: Configure GitHub branch protection rules so that automated documentation pull requests require approval from a human maintainer before merging into the main branch.