The MCP Apache Spark History Server, developed by Kubeflow, is a free, Python-based Model Context Protocol server. It functions as an AI Agents skill within the Kubeflow ecosystem, designed to expose data from an Apache Spark History Server. This program transforms Spark application insights into actionable tools for various AI clients, enabling automated analysis and investigation. It is distributed as a Python package via PyPI, installable with "pip" or runnable directly using "uvx", making it accessible for integration into modern development workflows.

What does MCP Apache Spark History Server connect to?

The MCP Apache Spark History Server acts as an MCP server, establishing a connection to an existing Apache Spark History Server. Its core purpose is to extract and present Spark History Server data in a structured format suitable for AI agents. Categorized as an AI Agents skill, it integrates with the Kubeflow ecosystem, allowing AI agents to interact with Spark application metrics and logs. The program is built using Python, ensuring compatibility and ease of deployment within Python environments.

Core features and tools

This server exposes Spark History Server data as tools for AI agents, shipping with prompts that guide multi-step workflows. Notable callable workflows include "/mcp__spark-history__investigate_failure <app_id>" for Claude Code and "/prompts investigate_failure" for Kiro CLI. Other capabilities include listing available Spark applications and retrieving their detailed information. It supports both stdio and streamable-http transport protocols, ensuring broad compatibility with AI clients such as Claude Code, Kiro CLI, ChatGPT, and Amazon Q CLI.

Who is it for?

The MCP Apache Spark History Server is designed for AI agents, coding agents, engineers, shell scripts, and CI/CD pipelines. It is particularly valuable for users who want AI agents to perform multi-step investigations, synthesize findings, or answer natural-language questions about their Spark applications. It enables automated root cause analysis of failures and expert-level performance analysis. This is not the right fit if your primary need is direct, manual interaction with the Spark History Server UI without AI agent integration.

How to get started

The server authenticates using username and password credentials configured in its configuration file, which are then used to access the underlying Spark History Server. It supports stdio and streamable-http transport protocols.

Use cases

• Investigating Spark application failures to provide root cause analysis.
• Listing available Spark applications and retrieving their details.
• Enabling AI agents to perform expert-level performance analysis and optimization recommendations.

The MCP Apache Spark History Server extends the utility of Apache Spark data by making it accessible to AI agents. It provides structured access to application details and supports complex, guided workflows for analysis. This program helps automate the investigation and optimization of Spark applications through various AI clients. It is a valuable component for integrating AI into Spark operations. Try using the "/prompts investigate_failure" workflow with Kiro CLI to analyze a Spark application issue.