Serving live AI on a public site without burning tokens.
A live model on a public page is a small standing invitation: to bots, to scrapers, and to anyone who discovers your endpoint at 3 a.m. This note is the design we use on federal.ai, useful to anyone putting inference behind a public form.
Nobody is "attacking" you. The internet is just expensive.
The realistic risk for a public demo isn't an adversary, it's background radiation. Crawlers submit forms. Someone's script hits an endpoint in a loop. A curious visitor pastes in a novel forty times. None of it is malicious, and all of it spends real inference money. The design goal is not to punish anyone; it's to make the worst day cost about as much as a normal week, without a single legitimate visitor ever seeing a degraded experience.
Four layers, and the order matters.
Layer one: a small, fast model with a hard output ceiling. A public demo doesn't need your best model, it needs a good answer in two seconds with a bounded worst case. Cap the output tokens at the API call, not in the prompt; a prompt is a request, a parameter is a law.
Layer two: per-visitor and per-day request caps. Cheap in-memory counters at the serverless edge: a small burst allowance per visitor, a daily total for the whole site. When either trips, nothing errors, the request simply takes the next layer's path.
Layer three: a deterministic fallback that is genuinely good. This is the load-bearing decision. Behind the live model sits a rules-based reference engine that produces the same shape of output from the same inputs, domain-classified, honest about scope, formatted identically. A visitor over the cap gets a real answer, not an apology. If you can't build a fallback you'd be proud to show, the live model shouldn't be public yet.
Layer four: a spend ceiling at the provider. Whatever your code believes, the account itself carries a monthly cap. Application logic is a seatbelt; the provider-side limit is the airbag. When it's hit, layer three simply becomes the only path, and the site keeps working.
Fail closed, to a working path.
This is the same doctrine we hold governed mission workflows to, scaled down to a marketing page: when authority or budget is ambiguous, the system holds, but "holds" means falling to a path that still serves the user, silently, with the degradation invisible from the outside. An error message is a design failure twice over: it tells a legitimate visitor you're broken, and it tells an abusive script exactly where your limits are.
The result on our site: the Mission Lab answers every visitor, every day, and the monthly worst case is a number we chose in advance. The fancy part isn't the model. It's that the model is optional.
Designing AI that has to hold a budget, or a boundary?
Thirty minutes with an engineer who builds fail-closed AI paths for a living. Bring the constraint that worries you.