DevOps and Cloud SRE Tooling with MCP

DevOps and Cloud SRE Tooling with MCP

During critical production outages, Site Reliability Engineers (SREs) and DevOps responders face immense time pressure while navigating fragmented observability systems. When a primary revenue service degrades, engineers spend valuable minutes correlating distributed telemetry: tailing container logs across Kubernetes clusters, querying AWS CloudWatch log groups for error bursts, inspecting Prometheus metrics dashboards, and reviewing recent Git pull requests or Terraform plan diffs.

Empowering autonomous AI assistants (running in Claude Desktop, Cursor, or the OpenAI Codex CLI) with direct operational context dramatically slashes Mean Time to Resolution (MTTR). An agent can ingest alert webhooks, parse failing pod states, analyze CloudWatch query insights, and identify root cause stack traces within seconds.

However, granting AI models unconstrained shell access or full cloud administrative permissions introduces severe operational risks. A hallucinated parameter or mistyped namespace in an automated shell command could terminate production database instances, delete ingress controllers, or trigger cascading restart loops.

Production DevOps MCP implementations enforce Zero-Trust Scoped Tools, Automated Read-Only Diagnostic Pipelines, and a Two-Man Rule Confirmation Protocol for state-changing remediation actions.

This guide provides the complete architectural framework, least-privilege IAM policies, Kubernetes RBAC specifications, and a production TypeScript MCP server for AWS and Kubernetes SRE workflows.

1. Problem Statement and Architecture

Site Reliability Engineering teams require automated intelligence without sacrificing operational safety. The architecture isolates diagnostic read operations from stateful mutations:

Unsafe Agent Integration (High Operational Hazard):
AI Agent ──> Full Shell Access / Cluster Admin ──> Accidental Production Deletion!

Zero-Trust DevOps MCP Architecture:
┌────────────────────────────────────────────────────────┐
│ HOST AI AGENT (Claude Desktop / Cursor / Codex CLI)    │
│ "Investigate CrashLoopBackOff in the payments cluster" │
└──────────────────────────┬─────────────────────────────┘
                           │ JSON-RPC 2.0 stdio
                           ▼
┌────────────────────────────────────────────────────────┐
│ PRODUCTION SRE MCP SERVER (TypeScript)                 │
└─────────────┬────────────────────────────┬─────────────┘
              │ Read-Only (Auto-Approved)  │ Mutation Gate
              ▼                            ▼
┌──────────────────────────┐  ┌──────────────────────────┐
│ OBSERVABILITY ENCLAVE    │  │ REMEDIATION ENCLAVE      │
│ • k8s_diagnose_pods      │  │ • k8s_restart_deployment │
│ • aws_cloudwatch_query   │  │   [Requires 2-Man Token] │
│ • prometheus_check_alert │  │ • terraform_apply_fix    │
└──────────────────────────┘  └──────────────────────────┘

The system segregates tools into two operational categories:

1. ▸Read-Only Diagnostic Tools: Automatically approved for execution by the AI assistant. These tools query Kubernetes APIs using restricted get, list, and watch permissions, and query CloudWatch Insights across bounded time windows.
2. ▸Mutating Remediation Tools: Actions that alter running infrastructure (such as restarting pods, scaling deployment replicas, or modifying security group rules). These methods refuse automated execution and emit a cryptographic challenge token requiring human confirmation before proceeding.

2. Prerequisites and Environment Matrix

Before deploying the DevOps SRE MCP server, verify that your local development workstation or bastion host satisfies the dependency matrix:

Least-Privilege Kubernetes RBAC (sre-mcp-rbac.yaml)

Never run an SRE MCP server with cluster-admin privileges. Deploy a read-only ClusterRole restricted to inspection verbs:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mcp-sre-diagnostic-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events", "services", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: mcp-sre-binding
subjects:
  - kind: ServiceAccount
    name: mcp-sre-sa
    namespace: monitoring
roleRef:
  kind: ClusterRole
  name: mcp-sre-diagnostic-reader
  apiGroup: rbac.authorization.k8s.io

3. Multi-Host Configuration Snippets

Configure your AI assistant host to connect with the DevOps SRE MCP server using environment variables to supply target cloud regions and namespaces.

Claude Desktop Configuration

Add the server definition to claude_desktop_config.json:

{
  "mcpServers": {
    "devops-sre": {
      "command": "node",
      "args": ["./dist/sre-server.js"],
      "env": {
        "AWS_REGION": "us-east-1",
        "KUBECONFIG": "/home/developer/.kube/config",
        "DEFAULT_NAMESPACE": "production"
      }
    }
  }
}

Cursor IDE Configuration (.cursor/mcp.json)

Inside Cursor, configure the workspace server:

{
  "mcpServers": {
    "devops-sre": {
      "command": "node",
      "args": ["./dist/sre-server.js"],
      "env": {
        "AWS_REGION": "us-east-1"
      }
    }
  }
}

OpenAI Codex CLI Configuration (~/.codex/config.toml)

When managing incidents via the OpenAI Codex CLI:

[mcp.servers.devops_sre]
command = "node"
args = ["./dist/sre-server.js"]

[mcp.servers.devops_sre.env]
AWS_REGION = "us-east-1"

4. Step-by-Step Tool Discovery and Invocation

Step 1: Tool Discovery via JSON-RPC 2.0

Upon connection initialization, the host queries the server for available diagnostic and remediation capabilities:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/list",
  "params": {}
}

The server returns registered tools with clear schemas distinguishing diagnostics from mutations:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "tools": [
      {
        "name": "k8s_diagnose_failing_pods",
        "description": "Inspect a namespace for pods in CrashLoopBackOff, Error, or OOMKilled states.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "namespace": { "type": "string", "description": "Target Kubernetes namespace." }
          },
          "required": ["namespace"]
        }
      },
      {
        "name": "aws_query_cloudwatch_insights",
        "description": "Execute CloudWatch Logs Insights query over a specified time window.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "log_group_name": { "type": "string" },
            "query_string": { "type": "string" },
            "time_window_minutes": { "type": "integer", "default": 15 }
          },
          "required": ["log_group_name", "query_string"]
        }
      },
      {
        "name": "k8s_restart_deployment",
        "description": "Trigger rolling restart of a deployment. Requires human approval token.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "namespace": { "type": "string" },
            "deployment_name": { "type": "string" },
            "confirmation_token": { "type": "string", "description": "2FA approval challenge token." }
          },
          "required": ["namespace", "deployment_name"]
        }
      }
    ]
  }
}

Step 2: Querying Failing Pods

When an incident triggers, the agent queries failing workloads in the target namespace:

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "k8s_diagnose_failing_pods",
    "arguments": {
      "namespace": "production"
    }
  }
}

The server inspects pod status records and returns a concise JSON summary:

{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Found 1 failing pod:\n- Name: payment-gateway-7f89d-42bc\n  Status: CrashLoopBackOff\n  Restarts: 8\n  Last Exit Code: 137 (OOMKilled)\n  Terminated At: 2026-10-09T22:15:00Z"
      }
    ]
  }
}

Step 3: Two-Man Rule Confirmation Protocol for Mutations

When the agent attempts to restart a deployment to remediate memory pressure, the server refuses direct execution and emits an approval challenge:

{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "isError": true,
    "content": [
      {
        "type": "text",
        "text": "ACTION REQUIRED: Restarting deployment 'payment-gateway' in 'production' requires verified human approval.\nChallenge Token: 'CHALLENGE_9a81f3'.\nPlease authorize this token via the internal incident portal or re-invoke this tool providing: confirmation_token."
      }
    ]
  }
}

Once the on-call incident lead confirms the action in the incident dashboard, the agent re-invokes the tool with the signed token, safely executing the rolling restart.

5. Common Errors and Production Troubleshooting

Error 1: 403 Forbidden: User cannot list resource pods

• ▸Symptom: Calling k8s_diagnose_failing_pods fails with HTTP 403 Forbidden.
• ▸Root Cause: The ServiceAccount or kubeconfig credential lacks required RBAC permissions in the requested namespace.
• ▸Resolution: Check the current Kubernetes context using kubectl auth can-i list pods -n <namespace>. Bind the mcp-sre-diagnostic-reader ClusterRole to your service account.

Error 2: CloudWatch Query Rate Limiting (ThrottlingException)

• ▸Symptom: Calling aws_query_cloudwatch_insights returns ThrottlingException: Rate exceeded.
• ▸Root Cause: CloudWatch Logs Insights enforces concurrent query limits (typically 20 to 30 concurrent queries per AWS account). Repeated rapid polling exhausts the quota.
• ▸Resolution: Configure the tool handler with exponential backoff and jitter, and constrain query intervals to a minimum of 2 seconds between poll calls.

Error 3: Kubeconfig Token Expiration in Long-Running Daemons

• ▸Symptom: The MCP server works upon launch but fails after sixty minutes with Unauthorized.
• ▸Root Cause: Cloud provider IAM tokens (such as AWS IAM authenticator for EKS) expire after one hour if the client library does not support dynamic token refreshing.
• ▸Resolution: Configure your kubeconfig to utilize native exec credential plugins (such as aws eks get-token or gke-gcloud-auth-plugin) that refresh OAuth tokens automatically.

6. External Citations and Security Guardrails

Official Specifications and References

• ▸Kubernetes Role-Based Access Control (RBAC) (opens in a new tab) — Official guide for scoping ServiceAccount permissions and ClusterRoles.
• ▸AWS CloudWatch Logs Insights Documentation (opens in a new tab) — Query syntax and concurrency limits for cloud operational analysis.
• ▸Model Context Protocol Specification (opens in a new tab) — Official JSON-RPC 2.0 schema specification for tools, resources, and clients.

Security and Redaction Guardrails

When operating AI agents against production infrastructure:

1. ▸Redact Sensitive Environment Variables and Secrets: Ensure your tool handlers sanitize output strings, stripping database passwords, AWS secret access keys, and bearer tokens from logs and pod environment definitions.
2. ▸Audit Trail Logging: Log every tool invocation, including the calling agent ID, input arguments, and human confirmation tokens, into an append-only S3 bucket or audit log stream.