Nfrx, from Gaspardpetit, serves as an inference exchange gateway and distributed worker pool. It exposes private AI services securely through a public, OpenAI-compatible API. Built primarily with Go, Nfrx connects various AI systems, including LLM runtimes like Ollama and vLLM. It allows organizations to make their internal AI tools accessible without exposing local infrastructure. Nfrx is distributed via Docker images and as bare binaries, offering deployment flexibility for its mcp category functions.

What does Nfrx connect to?

Nfrx functions as an inference exchange gateway and distributed worker pool, connecting to diverse AI services and systems. It acts as an MCP server tool, sitting in front of existing LLM runtimes such as Ollama, vLLM, or Open-WebUI. Nfrx supports relaying Model Context Protocol calls, enabling users to expose private AI services including LLM runtimes, various tools, and RAG processes. This allows for secure integration of internal AI capabilities.

Core features and tools

Nfrx provides an OpenAI-compatible chat/completions API for client interactions. For Model Context Protocol interactions, clients can invoke MCP methods using "POST /api/mcp/id/{id}". Additionally, Nfrx includes a small plugin interface, allowing new modules to register routes, metrics, and state. This interface extends its capabilities, enabling custom module integration for specific organizational needs within the inference exchange gateway.

Who is it for?

Nfrx targets individuals and organizations needing to expose private AI services, such as LLM runtimes, custom tools, or RAG processes, via a secure, public, OpenAI-compatible API. It is ideal for those wanting to make internal AI capabilities accessible without directly exposing their local machines to the public internet. This includes developers and teams managing distributed AI workloads. Skip this if you do not need to expose private AI services publicly.

How to get started

The server authenticates using an API_KEY passed as an environment variable. Client connectors authenticate with a CLIENT_KEY, also an environment variable, when dialing out to the Nfrx server over HTTP or WebSockets.

Use cases

• Securely exposing local LLM runtimes like Ollama, vLLM, or Open-WebUI to be accessible via a public OpenAI-compatible API.
• Enabling clients to invoke private MCP methods by relaying calls through a public Nfrx server.
• Providing a scalable and load-balanced gateway for various heterogeneous AI machines and custom RAG processes.

Nfrx offers a secure and adaptable solution for exposing internal AI services via a public OpenAI-compatible API. It supports various LLM runtimes and custom RAG processes, acting as a flexible inference exchange gateway. The platform allows for secure access to private AI capabilities without direct machine exposure. Developers can extend its functionality through a plugin interface. Start by setting up Nfrx to expose a local Ollama instance via its OpenAI-compatible API.