Serve large models with vLLM, mlx-lm or llama.cpp on one GPU machine or several, and switch models with one command.
Quickstart | Docs | CLI reference | Changelog
llm-helper manages an OpenAI-compatible inference server on one machine or across several: vLLM by default, mlx-lm on Apple Silicon, or llama.cpp for GGUF models. Each model's working settings are saved as a profile, and switching to another profile restarts the server on it, whether that model runs on this machine alone or spans a group of them. It was built for unified-memory machines like the NVIDIA DGX Spark, where one machine holds one big model and a bigger model needs two or more working together. It works on any Linux machine with an NVIDIA GPU, and on Apple Silicon Macs on the GPU through vllm-metal, mlx-lm or llama.cpp.
llm-helper was called vllm-helper until it served more than vLLM. The old command name, config directory and service names all keep working; see Providers for what changes per server.
- Save a model's serving settings as a profile, and switch between profiles with one command
- Search HuggingFace with every result's weight size checked against the machine's memory, and vLLM flags read from its model card
- Adopt a vLLM server that is already running, without restarting it
- Register other machines once, then launch tested recipes across them: configurations run and measured on named hardware, pinned by image digest
- Pool replicas of one model behind a session-sticky router, or shard a model too big for one machine across several
- Control a machine from a web panel that also serves these docs
- Keep a Coder deployment's model picker in step with whatever is loaded
Note
llm-helper is experimental. It is a useful way to launch local LLMs, but use it on dedicated hardware that runs only your models, on a safe network.
A bug in llm-helper can affect what it controls: the service that setup installs, the server restart when you switch profiles, the web panel, and the model list that it syncs to Coder. A bug cannot break a request in flight, because llm-helper is a control plane and inference traffic goes straight to the inference server on its own port. The one exception is a pool, because its router sits in the request path.
See Install and upgrade for what setup writes and how to stay current, and Running on macOS for Apple Silicon.
llm-helper manages an existing vLLM, mlx-lm or llama.cpp install; it does not install any of them. Download the latest release for Linux amd64/arm64 or macOS arm64, then install it and run setup:
tar xzf llm-helper_*.tar.gz
sudo install llm-helper /usr/local/bin/
sudo ln -sf llm-helper /usr/local/bin/vllm-helper # the old name, for scripts that still call it
llm-helper setup # or: --provider mlx, --provider llamacppSetup detects the hardware and the vLLM install, generates an API key, writes config, and installs a service unit (systemd on Linux, launchd on macOS). It never overwrites existing config or restarts a running server. If vllm isn't on your PATH, set VLLM_BIN in ~/.config/vllm-helper/config.env.
Pull a model and serve it, or take over a server that is already running:
llm-helper new Qwen/Qwen3-32B # save a profile and switch to it
llm-helper adopt # or: manage the vLLM server already running here
llm-helper status # service state, active model, endpoint healthPoint any OpenAI-compatible client at http://<machine>:8001/v1 with the VLLM_API_KEY from ~/.config/vllm-helper/secrets.env. A first switch downloads the weights, so it takes as long as the download does.
To serve across machines, register every machine a recipe will use on the one that will serve the API, that one included, then launch. llm-helper drives the other nodes over SSH and runs the recipe's container on each with Docker:
llm-helper node add spark1 --host http://spark1:8001 --ssh coder@spark1 # this machine
llm-helper node add spark2 --host http://spark2:8001 --ssh coder@spark2
llm-helper recipes list # what is tested, and on what hardware
llm-helper recipes launch <id> --nodes spark1,spark2More than one machine covers recipes, pools and shards.
| Piece | What it is | Where it lives |
|---|---|---|
| Config | Settings for every profile: port, VLLM_BIN, service name, Coder sync |
~/.config/vllm-helper/config.env |
| Secrets | The endpoint's API key, and tokens for Coder and HuggingFace | ~/.config/vllm-helper/secrets.env |
| Profile | One model's vllm serve settings, as a small env file |
~/.config/vllm-helper/models/<name>.env |
| Runner | What serves the active profile: the native service unit, or recipe containers | vllm.service / com.coder.vllm, vllm-cluster.service |
| Node registry | Other machines, by name, for recipes, pools and shards | ~/.config/vllm-helper/nodes.json |
| Endpoint | vLLM's own OpenAI-compatible API | port 8001 by default |
restart, stop, status and logs act on whichever runner the active profile belongs to, and switch stops the outgoing runner before starting the new one, so they work the same for a native model and for a multi-machine recipe.
Everyday commands:
llm-helper list # saved profiles
llm-helper switch <profile> # load a different model
llm-helper search <query> # find a model that fits this machine
llm-helper set <profile> K=V # change a profile setting
llm-helper logs # follow the server log
llm-helper doctor # hardware, running servers, integrations, drift
llm-helper serve # web panel, with these docs under Docs
llm-helper recipes list # tested configurations, and what verified themstatus, list, search, inspect, doctor and recipes list take --json, and switch, new and restart take --wait, so scripts and agents can drive llm-helper without parsing prose.
The docs are compiled into the binary: run llm-helper serve and open the Docs tab. The same pages are on the docs site, and their Markdown source is in docs/:
- Quickstart: From nothing to serving a model
- Profiles: What a profile holds, and how to edit one
- Providers: vLLM, mlx-lm, llama.cpp, and what each one changes
- Models and quantizations: Finding a model that fits, and choosing a quant
- Memory budget: How llm-helper decides what this machine can hold
- Web panel: The browser control plane, and the permissions it needs
- Coder integration: Keep a Coder deployment's model picker truthful
- Recipes: Tested launch configurations, and how local overrides layer on top
- More than one machine: Run recipes, pools and shards across machines
- Troubleshooting: When the endpoint is down or the service crash-loops
- CLI reference: Every command and flag, generated from the CLI
Open an issue for bugs and feature requests. Include the output of llm-helper doctor, which covers the hardware, the vLLM install, running servers and config drift.
go test ./... needs no fixtures or network. CI also runs go test -race ./... on Linux and macOS, go vet, and formatting checks:
gofmt -w . # Go
bun install && bun run fmt # Markdown and YAML
go run ./scripts/gendocs # regenerate docs/reference/cli/ after changing a command or flagNotable changes go in CHANGELOG.md. Maintainers cutting a release follow RELEASING.md.