ChromaDB & Vector Search: Connecting Semantic Memory to AI Agents via MCP

Learn how to integrate ChromaDB vector store into your AI agents using an MCP server, establishing persistent semantic memory and custom RAG retrieval logic.

ChromaDBvector searchRAGembeddingsdatabasesarchitecture

ChromaDB & Vector Search: Connecting Semantic Memory to AI Agents

Giving AI agents long-term semantic memory requires a vector database. ChromaDB is a popular, lightweight open-source vector store. By wrapping ChromaDB in an MCP server, your agent can dynamically query, insert, and update embeddings to perform Retrieval-Augmented Generation (RAG) on the fly.

Prerequisites

▸Node.js 18+ installed on your host machine.
▸
A running instance of ChromaDB. The easiest way is via Docker:
bash
```
docker run -p 8000:8000 chromadb/chroma
```
▸An embedding model (like OpenAI's text-embedding-3-small or local HuggingFace models) configured in your Chroma client.

Architecture Flow

mermaid

graph LR
    Agent[AI Agent / LLM] -->|Search query| MCPServer[ChromaDB MCP Server]
    MCPServer -->|Embed text using local model / API| Embeddings[Vector Embeddings]
    MCPServer -->|Query database| Chroma[(ChromaDB Instance)]
    Chroma -->|Relevant documents| MCPServer
    MCPServer -->|Context-enriched text| Agent

Server Configuration Example

To run a ChromaDB MCP server, write a lightweight Node.js wrapper that exposes query_collection as an MCP tool:

typescript

import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';
import { ChromaClient } from 'chromadb';

const chroma = new ChromaClient({ path: process.env.CHROMA_URL || '' });

const server = new Server(
  { name: 'chromadb-mcp', version: '1.0.0' },
  { capabilities: { tools: {} } }
);

server.setRequestHandler(ListToolsRequestSchema, async () => {
  return {
    tools: [
      {
        name: 'query_collection',
        description: 'Query a ChromaDB collection for semantically matching documents.',
        inputSchema: {
          type: 'object',
          properties: {
            collectionName: { type: 'string' },
            queryText: { type: 'string' },
            limit: { type: 'number', default: 5 }
          },
          required: ['collectionName', 'queryText']
        }
      }
    ]
  };
});

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const { name, arguments: args } = request.params;
  if (name === 'query_collection') {
    const colName = String(args.collectionName);
    const query = String(args.queryText);
    const limit = Number(args.limit || 5);
    
    const collection = await chroma.getCollection({ name: colName });
    const results = await collection.query({
      queryTexts: [query],
      nResults: limit
    });
    
    return {
      content: [{ type: 'text', text: JSON.stringify(results, null, 2) }]
    };
  }
  throw new Error(`Unknown tool: ${name}`);
});

const transport = new StdioServerTransport();
await server.connect(transport);

Best Practices

▸Chunking: When allowing the agent to insert documents into ChromaDB via MCP, ensure the server middleware enforces maximum chunk sizes. Dumping a 50-page PDF into a single vector embedding will destroy retrieval accuracy.
▸Metadata Filtering: When designing your MCP tools, expose metadata filters (like author, date, or category) so the LLM can narrow down the vector search space before calculating cosine similarity.

Troubleshooting

▸Dimension Mismatch Error: If the MCP server throws an error when inserting or querying vectors, ensure the embedding model used by your Chroma instance exactly matches the dimensions of the model used to create the collection.
▸Connection Refused: Ensure your CHROMA_URL environment variable is correctly set and that the Docker container is exposing port 8000.

Using with OpenAI Codex

To use this MCP server with the OpenAI Codex CLI, you can add it to your configuration using the codex mcp add command:

bash

codex mcp add --name chromadb-vector --command "npx @modelcontextprotocol/server-chromadb-vector"

(Note: Depending on the server, you may need to append arguments or use --env flags for environment variables as described in the configuration section above)

Related Guides

Databases