A queue in front of your local inference server.

One GPU holds one model at a time. hearth stops your callers from thrashing it, and speaks the OpenAI API so nothing you use has to change.

Get started Try it in your browser

The hearth console's routing map: a rule's if and else lines on the left, and the models, providers and peers they send to on the right. — The routing map in the console, drawn from the demo node.

The moment two things want different models from one card they thrash, each evicting the other's weights and paying the load tax over again. hearth is the admission point that stops that.

Three things, and only the first one is mandatory.

Queues

Named lanes with priorities, aging so nothing starves, a per-caller cap, and a preference for whatever model is already loaded.

Speaks OpenAI

/v1/chat/completions and /v1/models, streaming included. Bodies pass through byte for byte, so tool calls, vision parts and whatever gets invented next month keep working.

Federates, if you want it to

Point a model at a peer and those requests run on their machine. They never take a local slot, because they never touch the local GPU.

The hearth console's topology view: callers and routers at the top, dispatch in the middle, and this machine's cards, a second machine, a peer and two cloud providers on the right. — The topology: callers, routers, hardware, a peer and cloud providers.

Point your clients at hearth instead of your backend. That's the whole migration.

shell

# install
npm i -g @vcvzgbzz/hearth

# probes for llama-swap / llama.cpp / ollama, writes hearth.yaml
hearth init

# then point any client at this instead of your backend
hearth serve     # http://127.0.0.1:4141/v1

No GPU to hand? Run the demo node in your browser, or clone the repo and run npm install && npm run demo for a node with one of everything and stand-ins behind it. The pictures on this page are of it.

npm@vcvzgbzz/hearth GitHubVcvzgbzz/hearth

Beyond the queue

Each of these is off until you ask for it in hearth.yaml.

Routing rulesA name a client asks for can be a type, and a short list of if / else lines decides which model answers.Routing
A consoleDraws the node, its hardware, callers and rules, shows the queue, and edits the config with a review before anything is written.The console
KeysEach caller can have a key with its own models, expiry, rate and daily limits, and its usage in tokens and dollars.Configuration
More than one backendSeveral servers, several cards, other machines on your network, and backends that do not speak the OpenAI API.Backends
Cloud providersA provider's OpenAI-compatible or Anthropic API as a backend, so a rule can send some requests out and keep the rest home.A provider's models
A canaryAsks each model a question now and then and refuses a model that has stopped making sense.Canary
SharingLend a model to a friend's node, or borrow one.Sharing capacity

Documentation

Lanespriorities, aging, and a ceiling for a lane Configurationevery setting in hearth.yaml, and api keys Endpointsevery route, and what /healthz checks The consolethe page at /ui, and changing the config from it Routingflows, types and variants, routers per caller, actions Backendsseveral servers, shared cards, other machines Providerscloud backends, their keys, and Anthropic's API Modelsids and aliases, per-id defaults, warm state Canaryrefusing a model that stopped making sense Sharinglending a model, borrowing one Deployingrunning it as a service