Principle: Rent the mind. Own the house. Everything that calls a model calls the gateway. The day the best model changes, you change a route and nothing downstream notices. See PRINCIPLES.md.
We use LiteLLM as the gateway, in Docker.
2.1 Install Docker
On the box:- Install Docker Engine on Ubuntu (manual, recommended for production): https://docs.docker.com/engine/install/ubuntu/
- Run Docker as a non-root user: https://docs.docker.com/engine/install/linux-postinstall/
2.2 Get your provider API keys
You need at least one. Start with two or three. Put them in.env (copy from .env.example) — never commit .env.
- Anthropic (Claude): https://console.anthropic.com/ → API keys
- Groq (fast, cheap inference): https://console.groq.com/keys
- OpenAI (GPT): https://platform.openai.com/api-keys
2.3 Configure the gateway
The config lives ingateway/litellm.config.yaml. It maps friendly model names to provider/model targets and pulls keys from the environment. It already has Anthropic, Groq, and OpenAI entries — trim to what you have.
- Config reference: https://docs.litellm.ai/docs/proxy/configs
- Proxy quick start: https://docs.litellm.ai/docs/proxy/quick_start
2.4 Run it — bound to the tailnet, not the world
gateway/docker-compose.yml binds the proxy to your Tailscale IP on port 4000. Set TAILSCALE_IP and LITELLM_MASTER_KEY in .env first. Then:
${TAILSCALE_IP}:4000:4000 — never 0.0.0.0 (see the Docker gotcha in Step 1). LITELLM_MASTER_KEY is the key your tools present to the gateway; make it long and random.
- Docker quick start for LiteLLM: https://docs.litellm.ai/docs/proxy/docker_quick_start
✅ Verification (must pass before Step 3)
From the box or your laptop (on the tailnet), list the models the gateway is serving:claude, fast, gpt (whatever you configured). A quick chat round-trip: