Separate classification from resolution
Identifying a billing question is different from issuing a refund. Begin with tagging and routing; keep policy exceptions and consequential account actions with a person. A low-risk label can still lead to a high-risk downstream action.
Use representative requests
Include ambiguous questions, mixed topics, multiple languages, missing order details, and messages from existing customers. Review the wrong-route rate by type instead of hiding difficult cases inside one overall accuracy number.
Design the staffed handoff
Agree where an uncertain request goes, who sees it, and how the conversation history is preserved. If nobody can handle exceptions, automating the front door can leave customers waiting in a less visible queue.
Know when to stop
Pause the pilot if urgent requests become harder to find or staff need longer to repair routing errors than to classify manually. Improve the categories and knowledge base before expanding the workload.
An escalation path is part of the workflow, not a feature to add later.