Designing a Self-Hosted Patient Chatbot With Server-Enforced Safety
DBB Software designed a chatbot that the client hosts, gates, and audits itself, replacing a third-party widget where patients were typing health information. A fixed check screens each message before the model is called; a server-side check clears each answer before the patient reads it, and the service runs on the platform's existing data layer, search, and booking flows.
Industry
Healthcare & Biotech
Service
AI Development
Team
1 Product Owner, 1 AI Engineer, 1 Solution Architect
Project State
July 2026 - Ongoing
Country
United Kingdom
About the Client
A UK-based healthcare technology company operates a global platform that enables patients to review and rate healthcare providers while accessing reliable information to make informed care decisions. With a presence in markets like London, Germany, Austria, Australia, Dubai, and Ireland, the company aims to improve transparency and trust in healthcare through patient feedback and data-driven insights.
The Client's Initial Request
DBB Software was tasked with replacing the chat widget on its homepage with one the company owns, and establishing what a patient-facing assistant has to do to be defensible in healthcare.
Ownership of the Chat Surface
The only AI on the site was a third-party window embedded in the homepage across four markets, in the most visible place on the site. It had no emergency handling, no control over what it said, and no connection to the platform's own search or booking.
01
Appetite the Prototype Proved
An internal prototype showed there was demand for a patient assistant and got the idea in front of the leadership team. Demand was the question it answered.
02
The Strictest Current Standard
A 2026 BMJ Open audit of AI chatbot health answers found 49.6% problematic and 19.6% highly problematic or potentially harmful. In the first quarter of 2026 alone, 36 US states introduced more than 70 bills aimed at AI chatbots, most requiring that the user be told they are talking to a machine.
03
Enforcement Beyond a System Prompt
The quickest replacement was another model wrapped in a prompt and handed access to the platform, with monitoring that notices a breach after the patient has read it. Our client wanted the safety behavior enforced by the system.
04
Symptom Text as Article 9 Data
A patient describing why they are looking for a specialist has disclosed special-category personal data under GDPR. That shapes what may be logged, how long anything is kept, which suppliers may process it, and what consent has to look like.
05
Solutions We Delivered
DBB Software designed a chatbot that our client operates end to end, built so the safety behavior is enforced by the system, not requested of the model. The checks that matter are fixed code, the platform data comes through a layer that already existed, the conversation hands off into enquiry and booking flows already in production, and the whole service sizes to one small instance.
Deterministic Emergency Gate Ahead of the Model
Every patient message passes a deterministic emergency check before the model is called. It is code, so no phrasing, no tool result and no model behavior routes around it. When it fires, the patient gets emergency wording written by a clinician, the model is never called for that turn, and the exchange is logged without personal details for clinician review.
The gate evaluates the rolling conversation window, not the single message in front of it, because intent can be spread across three turns.
Rule variants are per market and per language, versioned together with the test set that proves them.
The ruleset and its test set have to be authored and signed by a named clinician, and the set has to clear at full recall before a release ships. Appointing that clinician gates any rollout past internal testing.
Over-triggering is acceptable at launch. Precision is tuned afterward and recall is never traded for it.
A deterministic screen adds no token cost and no latency worth measuring, and the turns it catches are turns the platform never pays a model to answer.
The Message Pipeline in Order
One message runs a fixed sequence, and the order is the design:
Consent guard, checking a stored consent record exists
Token budget reservation, taken before the model call
Emergency gate over the rolling window
Model call, able to act only through the shared tool layer
Server-side output gate
Segment-level streamed release
Budget reconciliation
The emergency branch bypasses everything downstream. A gate trip produces the clinician-authored message, makes no model call, and logs the turn redacted.
Server-Enforced Output Gating
The model's answer is validated on the server before any of it reaches the patient. On sensitive output classes, a second model reviews the first one's output against policy.
Release is buffered at segment level, sentence-sized, which keeps gating overhead bounded inside the two-second first-token budget.
A violation cuts the stream, substitutes a safe replacement message, and raises an alert on any occurrence, because a violation reaching the gate means a prompt or model regression upstream.
The instructions the assistant runs on live in a layered prompt registry in the repository, split into identity, safety, scope and tool guidance, version-pinned with a changelog. A guardrail cannot be altered from a settings screen outside version control.
Budgets are p95 under two seconds to first token and p95 under ten seconds to a complete answer, measured at UK launch and monthly after.
Navigational Lane on the Shared Tool Layer
The chatbot helps a patient find, compare and contact a verified specialist, and says plainly at the start of every session that it is an AI. It does not diagnose, triage, assess symptoms or give medical advice.
Platform reads run through the same tool specifications the MCP connector exposes, over a private internal path, so the two surfaces return the same envelope and cannot drift.
An architecture test fails the build if the chat service imports any data access outside that tool layer.
The terminal action is always a link into the client's own profile, enquiry or booking flow, which already owns its validation, spam control and consent.
Consent, Session Identity and the Erasure Map
Before a patient can type free text they tick a real checkbox, and the server confirms that record exists before accepting the message. Consent asserted by a model or an agent on the user's behalf does not count. Session identifiers are server-issued signed tokens in HttpOnly cookies, so a client cannot mint or rotate one to reset a budget.
State lives in three DynamoDB tables, chosen for native TTL and zero cluster operations:
Conversations, holding redacted turn history within the rolling gate window, with a TTL set by the data protection officer's retention decision
Budgets, holding session and IP-band counters under atomic conditional updates for reserve-then-reconcile, on a 24-hour TTL
Consent records, holding timestamp, market and copy version as the lawful-basis evidence
Every store a conversation touches is written down with its own expiry and erasure trigger: the conversation store on TTL plus on-request delete, the consent store on the retention decision, service logs PII-masked at write on a 14-day window, observability traces with inputs and outputs disabled, analytics holding aggregate counters and no message text, and the model vendor under no-training and retention terms fixed before signature.
Layered Injection Defense
Injection is answered in three places.
The deterministic gate evaluates the rolling window, which closes the cross-message concatenation path that defeats a single-message filter.
Tool outputs arrive already constrained by the connector's exposure mappers, so the model reads a bounded envelope.
A tool-argument guard rejects one tool's output replayed as another tool's argument, closing an LLM-to-LLM path in agent chains.
Second-model validation runs on sensitive output classes, and request schemas are closed sets with real UTF-8 byte caps.
Four patterns were explicitly rejected: remotely managed prompts, because whoever controls the prompt store controls the guardrails; agent-attested consent; client-supplied budget keys; and any email or write tool.
Designed Failure States
Each dependency has a designed state, and the sensitive path fails closed.
Emergency gate unavailable, and the turn returns a static message with the site's search link. No model call ever bypasses the gate.
Budget store unavailable, and the turn is refused. No unmetered calls.
Provider failure or an open breaker, and the widget shows a static fallback while staying up. N consecutive infrastructure failures trip the breaker closed, with the threshold held centrally in the store.
Empty or truncated model output, and detection fires, one bounded recovery retry runs, then a graceful fallback.
That last one ships with the first release because DBB measured it on its own production chatbot: close to a quarter of conversations ended with the model returning nothing at all, and roughly two thirds of those occurred on turns where the assistant had just called a tool. The recovery pattern is carried into this design from day one.
Gate trips and output-gate blocks go to dashboards rather than paging on-call. The emergency message is the correct handling.
Cost Ceiling on the Most Expensive Request
A chat turn costs more than any other request on the platform, so controlling load here is controlling spend.
Tool calls are capped per turn, starting at five, tighter than the cap of eight DBB runs in its own production chatbot.
Each conversation carries a token budget, starting at 16,000, reserved before the model call and reconciled after.
Each address carries a daily ceiling, and a monthly alarm is wired to the breaker that closes the path.
The arithmetic gives a computable worst case. Tool calls per turn, multiplied by the turns the session budget permits, multiplied by the per-IP daily ceiling, bounds the read load a single abusive session can place on OpenSearch, and the per-IP cap multiplied by worst-case tokens bounds monthly spend to the alarm threshold.
Staged Rollout and Supplier Retirement
The chatbot ships behind the platform's existing feature-flag system, market by market, UK first, moving from employees to 5% to 50% to full release. Each step is held until the dashboards show no safety or cost anomaly.
Safety is not an experiment surface. The emergency gate, the AI disclosure and the consent step are invariant across every variant, and A/B testing is limited to entry-point copy and placement.
Each tenant's release removes that tenant from the third-party widget's allowlist. Removal, not flag-hiding.
The shared widget component and its consent-framework vendor entry are deleted in the release that opens the last market, leaving the platform running one fewer external supplier.
Deployment, Streaming Chain and Test Regime
The service deploys as its own environment with a dedicated CloudFront behavior, EU region first, with the second market enabled only when its flag opens. Streaming end to end requires proxy buffering disabled at every hop: the CloudFront origin settings, the load balancer idle timeout, and the web server configuration on the instance. That checklist starts as a week-one spike and becomes a permanent smoke test.
The clinician-signed emergency set runs in continuous integration as a release gate. Any missed case fails the build; precision cases are tracked and non-blocking.
An adversarial suite runs a prompt-injection corpus including split-across-turns evasion.
Integration tests run the full pipeline against tool fakes and then the real tool layer, assert that no model call happens without a stored consent record, and exercise fail-closed behavior with each dependency artificially down.
Prompt changes run the full safety suite, because prompts are repository files rather than remote configuration.
Rollback is two independent levers: the flag off, which hides the widget in minutes, and a blue/green swap for the service. A signed vendor data-processing agreement is a deployment precondition for any stage that accepts input from someone who is not an employee.
Results Achieved
Guardrails Owned by the Client
Safety rules live in the repository and on the server, outside any settings screen a supplier controls.
Patient Health Data Held In-House
Patient text stays in systems our client runs, under supplier terms that forbid training on it.
A Deletion Request That Completes
Every store a conversation touches is written down, each with its own expiry and deletion trigger.
Safety Screening at No Token Cost
A deterministic check runs before the model, so screened turns never reach a paid model call.
A Calculable Worst-Case Bill
Per-turn tool caps, a reserved token budget and a daily ceiling per address, wired to a breaker.
One Fewer External Supplier
The third-party widget leaves the platform market by market as the new assistant goes live.
A Failure Mode Fixed Before Launch
Empty model output on tool-calling turns was measured in production first, so recovery ships with release one.
Add AI to Your Product Without Rebuilding It
DBB Software designs efficient AI features that run on the search, the data and the infrastructure you already have, gated on accuracy before they reach users.
Contact Us
"Most of our work starts with a 30-minute call where someone describes a product they're trying to ship and one part of the engineering picture they can't get around.
If that's where you are, let's set one up; I'll tell you straight whether we're the right fit.”
Mina Morkos
Business Development Manager
Want a similar outcome for your team?
Ask our AI assistant — it can pull related case studies, talk through the approach, and put you in touch with the team if you want a deeper conversation.
