nfrx, by Gaspard Petit, is an infrastructure bridge that exposes locally hosted AI services to external clients while preserving on-premises data. The tool provides an OpenAI-compatible API surface and uses encrypted WebSocket tunnels so remote applications can invoke local LLMs, document converters, and ASR without opening inbound ports. Key capabilities include zero-configuration networking and a privacy-first architecture with Model Context Protocol support. It targets software developers and privacy-conscious teams needing controlled remote access to local models.

What tasks can you actually use it for?

nfrx functions as a unified gateway for multiple AI outcomes: text generation, document conversion, and speech-to-text. It integrates out of the box with providers such as Ollama and vLLM for model serving, Docling for document processing, and Whisper for transcription, so existing OpenAI-based clients can call local services and obtain generated text, converted documents, or transcripts without rewriting callers to each backend.

How reliable is routing and worker management for production use?

Request distribution is engineered for mixed clusters using dynamic worker discovery and least-busy routing, which directs traffic to the locally connected worker with the lowest load. That routing model supports heterogeneous scaling across different machines and GPUs, enabling a single public endpoint to balance work across varied hardware while new local workers register automatically as they come online.

How does it handle security, privacy, and operational fit?

Security and privacy are operational priorities: the tool uses separate authentication keys for API clients and private connectors, restricting access to authorized entities and keeping model weights and data inside the local network. The developer maintains the project actively on GitHub and the community rates it positively for a lightweight footprint and straightforward setup, making it suitable for teams that want local control with remote access.

Practical infrastructure for developers who run local AI at scale

nfrx is a practical option for developers and privacy-focused teams that need a single entry point to local AI services, because it mimics familiar APIs and preserves data on-premises. Expect operator involvement: it is designed to run on Docker-capable systems and requires runtime setup, and teams should validate connector keys and load distribution in their environment before wide rollout.