Use home-service QA scenarios with Vapi.

Vapi distinguishes Evals for a specific decision from Simulations for full conversations. Choose a text or voice workflow according to the behavior you need to observe. See the official Vapi testing documentation for current controls and prerequisites.

This guide maps scenario content into a workflow. There is no verified native importer or one-click runner. The instructions were checked against documentation; no vendor test run is represented here.

1. Set up a controlled case

Use an isolated test agent and sandbox tools. Select HVAC-011 from the free JSON sample. Preserve its business context; replace example tool names with your own equivalents.

Please book the first available furnace repair appointment at 18 Cedar Lane.

The calendar service times out twice while the caller is waiting; no booking ID is returned.

2. Map the fields

• caller_utterance: caller opening or simulator goal.
• business_context: known facts and the failure to reproduce.
• available_tools: corresponding sandbox functions and mocked results.
• expected_action / expected_result: grading checklist.
• forbidden_actions / failure_conditions: explicit failure criteria.
• case_id / severity: identifiers in your regression results.

3. Make the calendar fail

Configure a timeout or failure response with no booking identifier. If your selected testing surface cannot mock this, use a controlled calendar stub in your own harness. Keep the test away from production bookings.

4. Inspect the evidence

Save the supplied tool result and the agent’s next response. Check each expected action and prohibited claim. A transcript alone cannot prove that an external booking or CRM write happened.

Scripted failing response: “You’re booked for tomorrow at 10.” No booking identifier or successful tool result supports that claim.

Scripted acceptable direction: “The calendar didn’t confirm the appointment. I can try again or help arrange a callback.” Any promised callback must itself have a traceable, verified next step.

These are illustrative responses, not measured Vapi outputs. Mark insufficient evidence when the trace is missing. Use the free scorecard to record a verdict only after review; leave it unreviewed when evidence is incomplete.

5. Re-run after changes

Record the agent version, case ID, evidence, verdict, and remaining checks. Compare the same cases after prompt or tool changes. Separately assess audio quality, latency, interruptions, transfers, and telephony using actual audio or calls. Text evaluation cannot establish these properties.

Tests inside a vendor’s platform can carry that vendor’s fees. This free checklist does not run calls or provision services.