Crawlie, developed by Spronta, is a specialized Model Context Protocol (MCP) server designed for technical SEO and GEO crawling. Built with Rust for its core engine, CLI, and MCP server components, Crawlie connects to Large Language Model (LLM) agents, enabling automated SEO tasks. It offers capabilities for site auditing, issue identification, and performance tracking. The program is distributed via npm for its server and CLI, with a signed .dmg available for macOS desktop users. Crawlie supports both human and agent-driven workflows, making it a versatile tool for digital optimization.

What does Crawlie connect to?

Crawlie functions as a Model Context Protocol (MCP) server that connects directly to Large Language Model (LLM) agents. This connection positions it as a key component for automated SEO and GEO tasks. Its local MCP server uses JSON-RPC over stdio for communication, while the hosted Crawlie Cloud MCP uses Streamable HTTP. It is compatible with any MCP-compatible client, specifically mentioning Cursor, Cline, and custom agents, allowing diverse integration into existing AI workflows.

Core features and tools

Crawlie exposes a set of callable tools mirroring those of the hosted Crawlie Cloud MCP. These include crawl_site for full site analysis, audit_url for specific page checks, and top_fixes for issue prioritization. For geographic insights, it offers geo_gaps. affected_urls identifies pages impacted by issues, while diff_reports tracks changes between crawls. To manage ongoing tasks, crawl_status polls long crawls, and get_report along with list_reports access saved cloud reports.

Who is it for?

Crawlie is designed for both human users and LLM agents, making it suitable for automated SEO tasks and detailed analysis. Its capabilities benefit those needing to perform site audits, identify technical SEO issues like broken links or missing metadata, and track changes over time. The program is particularly useful for integrating SEO workflows with AI agents. This is not the right fit if your primary need is a non-technical content management system without SEO or GEO analysis requirements.

How to get started

For the hosted Crawlie Cloud MCP, access is granted through a Crawlie API key, which is created in the dashboard settings. The local Crawlie MCP server supports JSON-RPC over stdio as its transport protocol.

Use cases

• Enabling an LLM agent to run a full audit and act on it without human intervention.
• Crawling an entire site to identify broken links, redirects, missing metadata, and perform over 40 SEO and Generative-Engine checks.
• Performing crawl-over-crawl diffing to highlight changes between two crawls, such as health and GEO score deltas, page additions/removals, and newly appearing or resolved issues.

Crawlie offers a specialized solution for technical SEO and GEO crawling, uniquely integrating with LLM agents through its MCP server. Its Rust-based architecture ensures performance, while its toolset supports detailed site audits and change tracking. The program is suitable for automating complex SEO tasks and providing granular insights for digital properties. Try this by having an LLM agent use the crawl_site tool to perform a full audit.