A Fortune 100 company ran two major customer-facing chatbot projects on the same stack: Azure Bot Service, source code on the Microsoft Bot Framework SDK, skill bots backed by QnA Maker and LUIS, and Microsoft Bot Dispatch to route utterances to the right skill bot. One was the internal gold standard. My team at Core BTS was brought on to bring the other up to that standard, and in some cases past it. One candidate for a leapfrog was a move from Dispatch to Orchestrator. The gold-standard team had been waiting on us to "blaze the trail" on that migration, and this report is the case I built to decide it.
I authored the report in May 2022, built the local Orchestrator bot and the test set, and ran the analysis and statistics myself.
The architecture, and the single-assistant illusion
The chatbot runs on the Bot Framework SDK in combination with Azure, Bot Dispatch, QnA Maker, and LUIS. Azure deploys the services that run the bot. QnA Maker and LUIS are the content repositories. QnA Maker content is basic question-and-answer mechanics, and LUIS extracts intents from user utterances. The project has two QnA Maker knowledge bases and one LUIS app. Each one is a "skill bot," because the content of each knowledge base or LUIS model focuses on different subject matter.
The user does not know they are talking to three different bots. They appear to be talking to one. That illusion is the whole point of the architecture, and a single component holds it together. When a user sends an utterance, Dispatch evaluates which skill bot is best suited to handle it, the chosen skill bot produces a response, and that response is returned to the user. Dispatch is the router. Everything the user perceives as one coherent assistant is Dispatch making one decision per utterance, and the quality of that one decision is the quality of the bot.
The deprecation trigger
The reason to touch a working router was external. Microsoft announced that Dispatch was being deprecated, and in the same announcement encouraged current Dispatch users to migrate to Orchestrator. Unlike Dispatch, Orchestrator would receive long-term support and future updates. That set the question. Migrating was no longer optional in the long run, so the only thing left to establish was whether the replacement was actually a better router or merely a newer one.
The experimental design
To answer it, I built a fully featured local version of the bot migrated to Orchestrator in VS Code, then compared and tested it against the current Dispatch bot for skill-bot routing accuracy. Both routers were tested against the same dataset.
The dataset is where the design earns its result. I built it as a TSV file from a combination of manual writing and real user utterances copied out of the production environment via Azure Application Insights queries. The two sources do different jobs. The manually written utterances traced back to a question in the FAQ knowledge base but did not exist as a training input in that knowledge base, which probes generalization rather than memorization. The production utterances represent how users actually write to the bot, rather than utterances with perfect grammar and spelling.
Labelling followed one strict rule. A test utterance that did not align with anything in the FAQ knowledge base or the LUIS model was labelled chitchat, even when the nature of the question aligned with the purpose of the FAQ knowledge base. A router cannot be expected to route to content that does not exist, so the expected route for such an utterance is chitchat, not the skill bot a human would intuitively pick. The final dataset contained 251 entries and was approved as correctly labelled by the primary UX expert handling the chatbot.
The two routers ran through different harnesses, by necessity. Orchestrator, being
contained in code, was tested with npm test. Dispatch was tested with a Postman
suite a teammate built, querying the dev-environment Dispatch service. The dev
environment was the same as production at the time of testing, so testing it was
equivalent to testing the production Dispatch service.
The result
On the same 251 entries, Orchestrator was correct on 238 and incorrect on 13, for 94.821%. Dispatch was correct on 161 and incorrect on 90, for 64.143%. An AB (split) test produced a Z-score of 8.51, against the critical values of 1.96 at alpha = 0.05 and 2.576 at alpha = 0.01. The corresponding p-value is approximately 0 out to 30 decimal places. The chance that this gap is random is near zero, and the null hypothesis, that there is no difference between the routing accuracy of Orchestrator and Dispatch, is rejected at the 99% level of confidence.
The utterances Dispatch got wrong that Orchestrator got right are often questions with relatively clear mappings to certain FAQ questions or LUIS intents. In those cases Orchestrator was better able to generalize from the training data than Dispatch.
The honest limitation
The result measures one thing and only that thing. It indicates whether each utterance was routed in the expected manner. It does not indicate whether the utterance was routed where the company might prefer the question to land. The two are different. "How do I make a withdrawal?" reads like an FAQ skill-bot question, but there is no question like it in the knowledge base, so it cannot be expected to route there, and its expected route is chitchat. Both routers only know as much as the content they are given.
Correcting the dataset for what was and was not actually in the knowledge bases and LUIS model forced re-labelling, and the most common re-assigned utterances fell into three categories: withdrawal questions and their derivations, loan payment questions, and general website support questions. Once those were corrected for, the remaining incorrect answers clustered into recognizable failure modes. Spelling mistakes. Keywords absent from the training set, such as "brokerage commissions." Combinations pairing a non-existent keyword with an existing one, where anything using the word "status" was often pulled to the LUIS model by its distribution-status intent. Phrasing vague enough to read as multiple questions. And questions hyper-specific to one user, such as a particular dollar amount or a charge on a specific date, even when they trace back to a general FAQ question. These point at the content of the LUIS model and the FAQ knowledge base, not at the router. A better router does not improve the data. The headroom in this result lives in the training set, and that is where the recommendation pointed next.
From analysis to migration
Orchestrator is a more accurate router than Dispatch, and the gap is statistically significant. Holding the router in code rather than in a maintained Azure resource also reduces cost and makes the bot easier to test, since the Dispatch service no longer has to be stood up to run a comparison.
Leadership approved the migration on this case, the one we had been asked to blaze the trail on. I shipped the full Dispatch-to-Orchestrator rebuild in three days. The work was only worth doing because it changed what got built. The number is what moved the decision.