MUMBAI, India | 19th November 2020: Jio Haptik Technologies Limited, one of the world’s largest conversational AI companies and a subsidiary of Reliance Jio Platforms, was recently recognised at EMNLP 2020 for its exceptional contribution. EMNLP is a Tier-1 venue for researchers around the world to publish results which push the boundaries of Natural Language Processing. The company introduced three new datasets for the NLP community that has the potential to emerge as a major benchmark for Intent Detection globally. The paper also benchmarks Intent detection accuracy of Haptik which is at par and in some cases better than the likes of Google’s DialogFlow, Microsft LUIS and Rasa.
The paper authors presented was HINT3: Raising the bar for Intent Detection in the Wild alongside other companies like Google Brain, IBM, Open AI at the Insights Workshop EMNLP 2020.
The publication also draws attention towards existing gaps in the performance of chatbots in the real world. Unlike current datasets that contain crowdsourced user queries, HINT3 contains samples of real user queries and the likelihood of this performance translating proportionally to Customer Experience (CSAT) is very high.
While analysing the best-case performance of all the platforms on queries which are in the scope of bots present in training data, Haptik performs exceptionally well. At the same time, Haptik’s capability to fallback to humans on the queries which are out of the scope of bot is also at par or better than other platforms as visible from the peak MCC score averaged across all datasets.
This can be ascertained from the table 1 below –
| Conversational AI Platform | Haptik | Dialogflow | Rasa | LUIS |
| Average In-scope Accuracy | 73 | 69.2 | 67.4 | 59.9 |
| Average MCC score | 0.538 | 0.522 | 0.492 | 0.458 |
Even though chatbots and virtual assistants get trained and learn over time as interactions increase, it’s equally important to provide the best customer experience during the initial days of implementation. Upon analyzing the performance of leading vendors with even lesser data containing sufficient signal to learn, Haptik stands out. This also means that for companies where getting sufficient data is a challenge, Haptik can be cost-effective and efficient. The dataset is open-source, results shown are duly accepted at Insights Workshop co-located with EMNLP 2020 and are fully reproducible with the steps available on GitHub.
| Conversational AI Platform | Haptik | Dialogflow | LUIS | Rasa |
| Average In-scope Accuracy | 67.7 | 64.0 | 55.0 | 58.4 |
| Average MCC score | 0.484 | 0.482 | 0.457 | 0.411 |
Echoing Swapan Rajdev, CTO at Haptik thoughts, “I am extremely proud of the hard work, testing and performance standards that have gone into building our NLU technology. With the HINT-3 dataset we are trying to give back to our developer community by ensuring better benchmarking standards for everyone. Benchmarking Haptik’s performance alongside Dialogflow, LUIS, RASA and other players is just another step towards ensuring best-in-class technology for our customer’s advantage.”
As NLP is the backbone of any Conversational AI engine, it is the most frequently used decision-making factor in choosing a Digital Transformation partner. Apart from the recognition at EMNLP, Haptik NLP has also received multiple other accolades like the Raise Award from Government of India for Industry expertise.
