Kreuzcrawl is a high-performance web crawling engine developed by Kreuzberg Dev for the Model Context Protocol (MCP) platform. It specializes in structured data extraction, integrating with AI agents and RAG/LLM pipelines. Written in Rust, it offers native bindings for multiple languages, including Python, Node.js, Go, and Java. Kreuzcrawl supports various deployment options, from language-specific package managers to Docker images and precompiled binaries, making it a versatile tool for modern data platforms.

What does Kreuzcrawl connect to?

Kreuzcrawl connects to Model Context Protocol (MCP) servers, functioning as a web crawling engine for AI agents. Its primary role is structured data extraction from websites. The core programming language is Rust, with native bindings available for Python, Node.js (TypeScript), Go, Ruby, Java, C#, PHP, Elixir, WebAssembly (WASM), and C FFI. It also ships a bundled MCP server when distributed as a plugin for AI coding assistants.

Core features and tools

Kreuzcrawl exposes several callable tools for web crawling and data processing. These include batch_crawl and crawl for fetching web content, with streaming variants like batch_crawl_stream and crawl_stream. For data extraction, it offers batch_scrape and scrape. Other tools include create_engine, generate_citations, interact for browser automation, map_urls, and waf_rules_from_path or waf_rules_from_str for WAF rule handling.

Who is it for?

The primary audience for Kreuzcrawl includes developers building web-crawling AI agents and those working on RAG/LLM pipelines and modern data platforms. It is designed for tasks requiring structured data extraction, HTML content conversion to Markdown, and concurrent batch operations. Confirmed compatible AI clients include Claude Desktop, Continue.dev, Cursor, and Copilot CLI. Skip this if your project does not involve web crawling or structured data extraction for AI applications.

How to get started

The authentication model supports HTTP Basic, Bearer token, and custom header authentication, along with persistent cookie jars. Server-Sent Events (SSE) are supported for streaming crawl events, particularly for cloud deployments without WebSocket support.

Use cases

• Scraping and crawling websites to extract structured data such as text, metadata, links, and images, with options for concurrent operations and browser rendering for JavaScript-heavy pages.
• Converting HTML content into clean Markdown output with citations and document structure.
• Performing batch operations to scrape or crawl hundreds of URLs concurrently while handling partial failures.

Kreuzcrawl provides a high-performance web crawling engine for MCP servers, supporting AI agents and data platforms. Its multi-language bindings and diverse distribution methods make it highly adaptable. With tools for structured data extraction, browser automation, and HTML-to-Markdown conversion, it addresses key needs for modern data processing. Try using its batch_crawl tool to extract structured data from multiple URLs concurrently.