How to Use the Puppeteer MCP Server for Web Scraping & Automation

Automate web browsers using the Puppeteer MCP server. Navigate websites, take screenshots, extract data, and automate web interactions through AI.

Puppeteerweb scrapingbrowser automationscreenshotstesting

Use the Puppeteer MCP Server for Web Automation

The Puppeteer MCP server allows AI agents to securely control a headless web browser instance. This unlocks complex web automation tasks that basic curl commands cannot handle, such as interacting with Single Page Applications (SPAs), bypassing basic bot protections, rendering charts for screenshots, and filling out multi-step forms.

Prerequisites

▸Node.js 18+ installed on the host running the MCP server.
▸Chrome or Chromium browser installed.
▸Sufficient system memory (running multiple headless Chrome tabs can consume gigabytes of RAM).

Configuration Setup

Add the configuration to your MCP client:

json

{
  "mcpServers": {
    "puppeteer": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-puppeteer"]
    }
  }
}

Available Tools

Once connected, your agent gains these capabilities:

Tool	Description
`puppeteer_navigate`	Navigate the active tab to a URL
`puppeteer_screenshot`	Capture a page screenshot and return base64
`puppeteer_click`	Click an element matching a CSS selector
`puppeteer_fill`	Fill form fields or inputs
`puppeteer_select`	Select dropdown options
`puppeteer_hover`	Hover over elements to trigger CSS states
`puppeteer_evaluate`	Execute custom JavaScript in the browser context

Example Workflows

▸Data Extraction: "Go to the pricing page of competitor.com, wait for the dynamic tables to load, and extract all plan details into a comparison markdown table."
▸Form Automation: "Fill out the contact form on example.com with test data and click the submit button. Check for any validation errors on the page afterward."
▸Visual Testing: "Navigate to my staging environment, take screenshots of the homepage at mobile, tablet, and desktop widths, and save them to my disk."

Advanced: JavaScript Evaluation

The puppeteer_evaluate tool lets your agent run arbitrary JavaScript directly in the browser. This is extremely powerful for bypassing complex DOM structures:

javascript

// Example script the agent might run to extract all product prices
const prices = document.querySelectorAll('.price-tag');
return Array.from(prices).map(p => p.textContent.trim());

Best Practices & Troubleshooting

▸Respect robots.txt: AI agents can act like aggressive scrapers. Instruct your agent to respect robots.txt or add artificial delays between navigation clicks to avoid triggering Web Application Firewalls (WAFs) like Cloudflare.
▸Zombie Processes: If your MCP client crashes, it may leave behind zombie Chromium processes eating up your RAM. Periodically check your task manager or configure the MCP server to gracefully kill browser instances on exit.
▸Linux Shared Libraries: If running inside a Linux Docker container, Puppeteer often fails to launch because it lacks required OS libraries (like libxss1 or libnss3). You must install these dependencies in your Dockerfile.

Using with OpenAI Codex

You can use this MCP server with the OpenAI Codex CLI by adding it to your configuration:

bash

codex mcp add --name puppeteer --command "npx -y @modelcontextprotocol/server-puppeteer"

For a full list of recommended servers, see Best MCP Servers for OpenAI Codex.

Related Guides

Dev Tools