AI receptionist testing
What to test before an AI receptionist answers a real home-service call.
The happy path is not the product. The product is what happens when the caller is vague, angry, unsafe, outside the service area, or asking for something your tools cannot actually do.
Start with the operational edges
A useful QA scenario gives the agent a caller, a business context, and a limited set of tools. The evaluator can then inspect whether the agent takes the right action, avoids a forbidden action, and ends with a result the business can stand behind.
Coverage areas that matter
Booking, cancellation, and rescheduling
Test whether the agent gathers the minimum information, checks availability before promising a slot, and handles changes without silently losing the original appointment.
Pricing and service area
Pricing answers should respect the business context instead of inventing a quote. Service-area calls should produce a clear next step when the address is outside the normal territory.
Tool failures and hallucinations
When a scheduling or lookup tool fails, a good receptionist says what it can and cannot confirm. It should never claim a booking, price, or technician detail that the system did not return.
Safety and escalation
Gas smells, sparking equipment, active flooding, structural concerns, and other urgent signals deserve explicit escalation. A polite tone is not enough if the action is unsafe.
Use a repeatable scorecard
Mark each case pass or fail, record what the agent actually did, and keep the failure condition visible. Re-run the same cases after prompt or tool changes so regressions are visible rather than anecdotal.
Explore the open resources
Use the free browser scorecard to review the ten public cases. The scenario coverage reports summarize the dataset; they do not report tests of vendors or customer agents. Coverage is also split into focused HVAC, plumbing, electrical, and roofing pages.