Designing a Hallucination-Resistant Search Mode for a Healthcare Platform
DBB Software designed a lean AI search mode on our customer's specialist directory, where a single model call turns a patient's sentence into filters the model cannot invent.
Industry
Healthcare & Biotech
Service
AI Development
Team
1 Solution Architect, 1 AI Engineer, 1 Team Lead
Project State
July 2026 - Ongoing
Country
United Kingdom
About the Client
A UK-based healthcare technology company operates a global platform that enables patients to review and rate healthcare providers while accessing reliable information to make informed care decisions. With a presence in markets like London, Germany, Austria, Australia, Dubai, and Ireland, the company aims to improve transparency and trust in healthcare through patient feedback and data-driven insights.
The Client's Initial Request
DBB Software had to establish whether natural-language search moves the funnel, and to deliver that answer as a buildable design with the experiment that settles it.
Simplified Search
Patients needed to reach the right specialist in their own words, without knowing the clinical term. A painful knee splits across orthopedics, sports medicine, rheumatology and physiotherapy.
01
Matching on Review Content
Between clinicians all rated five stars, the differentiator sits in the review text: good with anxious patients, clear explanations, unhurried appointments. The platform needed a route to it.
02
Evidence Before the Heavy Build
Client wanted proof that natural-language search moves the conversion funnel before funding embeddings, semantic retrieval and theme-aware ranking.
03
Cold-Start Clinician Visibility
Clinicians pay for visibility, so any change to search ranking carried revenue risk. Doctors with few or no reviews had to stay findable.
04
Article 9 Health Data Handling
Every search sentence is GDPR Article 9 health data, and misrouting a patient has clinical consequences. Both accuracy and retention needed a stated position.
05
Solutions We Delivered
The engagement produced a product requirements document, an engineering design, and an experiment specification with the decision rule. Classic filter search stays the default, with the AI input and the classic filter controls side by side in the same viewport.
Single Interpret Call on the Existing Search Path
A patient's sentence posts to a thin endpoint inside the existing search module, POST /:tenant/search/interpret, which invokes a separate stateless interpret service under a hard timeout and returns validated filters. The browser never talks to the model path, and provider credentials live only in the interpret service.
The returned fields are those of the specialist search query the platform already runs:
Specialty and keyword
Location with a distance radius
Insurer, language and gender
Appointment type and age range
Health-service status, whether the clinician sees children, and fee-agreed status
Those fields then execute the unchanged search path: the existing controller, the existing read model, the existing per-tenant OpenSearch indices, and the existing ranking. No new datastore, no embeddings, no schema change to any tenant database or index. Because the ranking is untouched, the requirement that clinicians with few reviews must not be systematically disadvantaged holds by construction, with no ranking change to audit.
The full-ambition alternative, semantic retrieval with theme-aware ranking, was evaluated and rejected for v1 on cost, on the cold-start fairness gate it would reopen, and on the Article 9 surface it would widen.
Closed Filter Schema and Server-Side Validation
The interpret call uses function calling constrained to a strict subset of the existing query type. The model is never shown the fields it must not set: internal display flags, result seeds and tenant identifiers are injected by the server after the call. Whatever comes back is validated against that schema, with bounded string lengths, before anything downstream reads it.
The constraint is server-enforced rather than prompt-enforced. Schema validation and taxonomy mapping run after the model regardless of what the prompt asks for, so the only executable output in the path is a closed, server-validated filter set, and no tool call with side effects exists anywhere in it.
That shape also earned a documented exemption. The wider AI programme requires a second model to validate output on sensitive paths before it reaches a patient. W4 was formally waived from that rule, with the rationale recorded: its output is machine-validated against a closed schema and never surfaced as prose, so a second model would add latency without adding a control. The waiver is scoped to this output shape and sets no precedent for the conversational surfaces.
Taxonomy Mapping at Request Time
Specialty, condition and procedure terms resolve against the platform's live keyword taxonomy, read at request time from the same data the connector's taxonomy tools serve. There is no copied lookup table to drift, and taxonomy growth arrives without a code change.
A term that maps cleanly becomes a filter. A term that does not produces a "did you mean" suggestion drawn from the real taxonomy, never a guessed filter.
The resolved filters render as editable chips above the results, so the patient sees how the sentence was read and can correct it in place.
Unmapped terms increment a content-free counter, which becomes a coverage backlog for the product team.
Conversational Refinement and Recovery
Refinement runs on the same mechanism. The current filter state plus a new sentence produces a new filter state in one further model call, expressed only in existing filter fields and held in the page rather than on the server, so there is no conversation memory to secure, store or purge.
An ambiguous query gets one clarification prompt, or a deliberately broad reading with the chips visible.
An over-constrained query returns a concrete relaxation offer, widening the radius or dropping one constraint, applied only when the patient chooses it. A silently empty results page is forbidden.
Allocated Latency Budget and One Safe Failure State
The sub-two-second requirement is broken down and spent deliberately:
Interpret round trip, 900ms at the 95th percentile
Within it, 700ms for the model call and 200ms for validation, taxonomy mapping and transport
A hard timeout at 1,500ms, which triggers the fallback
Provisioned concurrency sized for the experiment's traffic keeps cold starts out of the 95th percentile. The remaining headroom covers the unchanged search request and render, which classic mode already performs inside its current latency.
Every failure resolves to the same patient outcome. A provider timeout, a provider error, a schema-validation failure and an open cost breaker all fall back to a classic parse, with the patient's sentence passed through as the ordinary free-text search term. Error handling is a funnel into one safe state.
Error Taxonomy and Alert Thresholds
Each failure class emits a distinct content-free event, and three thresholds define when a human hears about it:
Fallback rate above the agreed ceiling over 15 minutes, which also counts toward the experiment's integrity review, because a high fallback rate means the control arm is quietly serving the variant
Schema-validation failures above 1% of interpret calls over 5 minutes, which indicates prompt or provider drift
A step change in the unmapped-term rate, which indicates taxonomy or model drift
No stack traces, provider names or utterance fragments appear in any client-visible error.
Article 9 Handling With No Utterance at Rest
Every search sentence is treated as GDPR Article 9 health data from the first request. Utterances are processed in flight and written to no datastore. The only at-rest trace is content-free telemetry.
A redaction pass strips obvious direct identifiers, email and phone shapes, before the provider call.
Property tests assert that nothing email-shaped or phone-shaped survives into the logs.
A scheduled check audits the log sinks for anything utterance-shaped, so the redaction claim stays monitored.
A redacted sample of production utterances is deliberately not collected in v1. It would require the outstanding legal answer first, and the offline labeled set already covers the guardrail need.
No patient text reaches a model vendor before that vendor's data-processing agreement is signed.
Accuracy Gate and Test Regime
A labeled query set is versioned in the repository and scored in continuous integration against the pinned model at temperature zero. Scoring measures field-level accuracy against thresholds rather than exact string matching, because model output is not byte-stable, and a threshold regression fails the pipeline. Any model change clears the same set before it ships, and model version changes are deliberate events with an eval run attached.
Unit tests cover the schema validator rejecting any field outside the closed subset, the taxonomy mapper including its "did you mean" candidate ranking, and the redaction pass.
Integration tests run the interpret endpoint against a deployed stage across the happy path, ambiguity, an unmapped specialty, injected timeouts and injected provider errors, each asserting the classic-parse fallback fires.
Browser tests cover the toggle, the chips rendering and editing, the disclosure, the zero-result relaxation flow, and unchanged classic-mode screenshots that prove the classic path is untouched on every release.
The same labeled set gates the misrouted-specialty rate before launch and is re-run weekly afterward against the identical version, so drift is visible.
The Experiment and Its Decision Rule
The workstream exists to produce a decision. Assignment runs at user level through the platform's existing feature-flag system, UK first. Control is classic search exactly as today. The variant gets the toggle with classic still defaulted, and AI-mode sessions are analyzed as a segment within the variant, which separates being offered the feature from choosing to use it.
The primary metric is search-to-profile-to-inquiry conversion. Secondary metrics are the zero-result rate, end-to-end p95 latency, and AI-mode usage share among variant users. Four guardrails run alongside:
Misrouted-specialty rate against the labeled query set
Chip-edit rate, the content-free online proxy for misinterpretation
Fallback rate, whose sustained breach pauses the experiment
Cost per thousand AI-mode searches
The verdict is written before the data arrives: graduate on conversion at or above classic with guardrails green, iterate once with named changes on mixed signals, or retire the toggle on flat or negative conversion. Minimum run length is set after a power analysis, since committing to a duration beforehand would be invention. Review-theme ranking is unlocked by two conditions together, positive conversion here and confirmation that review-theme data is fit for purpose.
The kill switch is the flag. Off means the toggle disappears and classic search is bit-identical to today, with no deploy required.
Results Achieved
Search Upgrade Without New Infrastructure
The feature runs on the existing index, ranking and results page. Nothing new to fund or operate.
Revenue Protection for Cold-Start Clinicians
Ranking is untouched, so clinicians with few reviews are not pushed down by a new search mode.
Health Data at the Article 9 Standard
Patient search text is redacted before it leaves the platform and stored nowhere.
AI Spend Released by Evidence
The expensive half of the feature is released by conversion data from the experiment.
A Reversible Bet
The mode switches off in seconds behind an existing flag, with no release and no code change.
Patient Vocabulary as a Product Backlog
Unmapped patient terms are counted content-free, giving the product team a coverage backlog.
Add AI to Your Product Without Rebuilding It
DBB Software designs efficient AI features that run on the search, the data and the infrastructure you already have, gated on accuracy before they reach users.
Contact Us
"Most of our work starts with a 30-minute call where someone describes a product they're trying to ship and one part of the engineering picture they can't get around.
If that's where you are, let's set one up; I'll tell you straight whether we're the right fit.”
Mina Morkos
Business Development Manager
Want a similar outcome for your team?
Ask our AI assistant — it can pull related case studies, talk through the approach, and put you in touch with the team if you want a deeper conversation.
