{"id":493900,"date":"2023-10-02T04:32:49","date_gmt":"2023-10-02T11:32:49","guid":{"rendered":"https:\/\/www.genesys.com\/blog\/post\/measuring-ai-quality-bias-accuracy-and-benchmarking-for-conversational-ai"},"modified":"2023-10-02T05:32:03","modified_gmt":"2023-10-02T12:32:03","slug":"measuring-ai-quality-bias-accuracy-and-benchmarking-for-conversational-ai","status":"publish","type":"blog","link":"https:\/\/www.genesys.com\/en-gb\/blog\/post\/measuring-ai-quality-bias-accuracy-and-benchmarking-for-conversational-ai","title":{"rendered":"Measuring AI Quality: Bias, Accuracy and Benchmarking for Conversational AI"},"content":{"rendered":"

[vc_section full_width=”stretch_row”][vc_row][vc_column][vc_column_text]Artificial intelligence (AI) practitioners are often asked to show their work. They have to prove that their AI technology works and is on par with \u2014 or better than \u2014 an alternative AI solution. It seems like a reasonable request. But measuring AI quality is difficult at best and, in some cases, it\u2019s just impossible. There are measures that are used for testing AI \u2014 error rates, recall, lift, confidence \u2014 but many of them are meaningless without context. And with AI, the real KPI is ROI.<\/span>\u00a0<\/span><\/p>\n

That\u2019s not to say that all AI technology is built the same or that quality is irrelevant. The quality of your AI solution has a material impact on your ability to use AI to achieve ROI. In this blog, I\u2019ll examine AI quality benchmarks and concepts as well as some best practices. This can serve as a reference point for those at any stage of the AI implementation journey.\u00a0<\/span>\u00a0<\/span><\/p>\n

Start with Realistic Expectations<\/span>\u00a0<\/span><\/h2>\n

Some expect AI to be consistently accurate. The perception is that AI will correct human flaws and, since human error is inevitable and expected, AI must be its opposite. Achieving this level of perfection is an impossible standard. Expectations need to be realistic; the best way to measure AI success is business impact.\u00a0<\/span>\u00a0<\/span><\/p>\n

There are some things that AI can in the contact centre that a human can\u2019t. For example, even if a chatbot can only answer one question, it can still answer that one question 24\/7 without ever stopping for a break. If that question is important to a large percentage of customers, or a small but important customer segment, then that chatbot has value well beyond its ability to accurately understand and respond conversationally to a wide set of requests. <\/span>\u00a0<\/span><\/p>\n

For conversational AI, expectations of perfection are sure to disappoint.\u00a0<\/span>\u00a0<\/span><\/p>\n

Conversational AI bots are trained on data and the quality of the underlying natural language understanding (NLU) model depends on the data set used for training and testing. You might have seen some reports that show NLU benchmarks. When reviewing the numbers, make sure you understand what data was used.\u00a0<\/span>\u00a0<\/span><\/p>\n

Let\u2019s say Vendor A used the same training and testing data for the analysis, but vendors B and C had a different training data set. The results for Vendor A will likely outperform Vendor B and C. Vendor A is (essentially) using birth year to predict age, which is a model that is 100% accurate but likely not the best use of AI.<\/span>\u00a0<\/span><\/p>\n

Measuring Quality and the Basics of Benchmarks<\/span>\u00a0<\/span><\/h2>\n

NLU models are measured using these <\/span>dimensions<\/span><\/a>:<\/span>\u00a0<\/span><\/p>\n