Thank you to the attendees at ICMI for great conversations at this talk! I presented this on October 24 at the ICMI Contact Center Expo 2024 in Orlando, Florida.
Introduction
Whether you’re a head of support trying to figure out how to cut costs and gain efficiency, or a customer support agent worried that you will be replaced by a robot, chances are that AI is on your mind. Maybe you’re worried about AI outside of your work, wondering whether the article you just read was written by a human or a machine, or unsure what to do about your child using GPT to help with their homework. It all seems too sci-fi, that intelligence can be manufactured. That computers may take on human qualities and become sentient; that computers may become sentient.
All of these are valid thoughts, especially when we translate AI as Artificial Intelligence. I’d like to suggest that we switch our language to Augmented Intelligence instead. When we shift to a mindset of AI as Augmented Intelligence rather than Artificial, suddenly we can do more with the same or smaller workforce. English-speaking agents can do multilingual support. Agents can summarize case notes quickly. They can surface customer insights to product teams. And as subject matter experts, they can become the chatbots’ trainers and managers.
First Forays into AI
At Automattic, we support customers via email and live chat. One of our initial uses of AI in customer support was to add a widget in our help desk where our agents, which we call Happiness Engineers (HEs), could paste the customer’s question for GPT to provide a preliminary reply. GPT’s reply required validation by the agent before sharing with the customer, and did not use any specificity to our documentation. We simply embedded regular old GPT into our help desk to see how it would work and to start playing with AI.
Later, we built our own chatbot that is trained on our support documentation and that we are able to tailor to our needs. We integrated this chatbot into our help desk so that HEs could use it as an assistant when responding to customers. The chatbot provides a reply to the customer’s question, and the Happiness Engineer can review the response for accuracy and tone before sending to the customer.
Results
In some cases, the AI reply saved HEs time, but in others, agents spent as much time reviewing and checking the AI reply as they would have spent coming up with the reply on their own. We did not show measurable efficiency gains as a result of these enhancements.
Multilingual Support
As far back as 2018, prior to the easy access to GPT that we have today, we outsourced multilingual support to freelance agents who spoke the languages natively. Our escalation path was clunky – multilingual contractors had to explain the issue in English for our in-house agents, in-house agents would troubleshoot and send the solution back to multi-lingual agents who then communicated with the end user in their native language.
We next engaged with a third-party translation service for escalations: when English-speaking Happiness Engineers received multilingual escalations, the translation service translated the user’s questions using machine learning. Our HEs then sent a reply in English, which the third party used machine learning to translate, a human editor to review, and their integration to send to the customer. This got responses to customers quicker and streamlined some of our operations, but the service was still slower than a direct reply from a Happiness Engineer, and it was expensive.
At Automattic, we have employees from 95 different countries who speak 119 languages. When GPT arrived, folks around the company tried it for translations and realized it’s actually quite good. We realized we no longer needed the translation service for escalations or even multilingual agents. An employee built a browser extension that harnesses GPT and DeepL for translations, so that all agents can send a reply in another language.
Results
Any English-speaking agent can now answer emails in any language GPT knows. Escalations are no longer clunky and time consuming, and we no longer need specialized teams or managers for each language. This creates faster response times for customers, along with more simplified and scalable multilingual support operations. We saw no complaints or degradation in customer satisfaction (CSAT) with this transition.
Amplifying Excellent Agents
Our customers interact directly with AI chatbots in anywhere from 15% to 65% of interactions, depending on the product. As many as 45% of those interactions are resolved by our chatbot, meaning they don’t result in an escalation to an agent. With AI handling this much volume, we determined that just as we have trainers and leads for Happiness Engineers, we needed trainers and leads for the bots.
In our product with the highest incoming support demand, AI resolved 8.6% of interactions in our first six months of working with it. With AI’s help, we responded more quickly to customers even as our team size shrank by 9% through natural attrition.
Our big question, despite those gains in efficiency, was this: are the bots actually resolving the customers’ issues? Are we happy with the experience they’re giving our customers? And if so, how can we drive that resolution rate higher than 8.6%?
Beyond efficiency gains, the thing I’m most excited about with AI is that it has provided opportunities to amplify the expertise of Happiness Engineers. It has provided a new path of development: chatbots pushed us to build out AI squads who are dedicated to reviewing chatbot interactions to find where it’s not performing well, and to learn more about AI and how we can work with it. We recruited excellent Happiness Engineers across our products to form Quality Assurance squads for our chatbots. These AI squads do things like the following:
- Review chatbot interactions
- Identify gaps or fuzzy areas in documentation
- Expand their own AI expertise and expertise across the organization
- Make the bots better through their feedback to our in-house engineering team
- Come up with new ideas for how to leverage AI in customer support
The AI squads work closely with our documentation specialists and with the in-house engineering team who builds our chatbots. Using tools our chatbot team built, squad members first rate whether the bot referenced relevant sources in order to answer the user’s question. In cases where the bot chose poorly, the squad inspects our documentation for ways to ensure there are relevant docs for the bot to find the correct information.
The AI squads also rate the accuracy or inaccuracy of the bot reply:
- As helpful as a Happiness Engineer (accurate)
- Not as helpful as a Happiness Engineer (inaccurate)
- Not as helpful as an HE and also harmful or destructive (inaccurate with flag)
Results
Through the AI squad program, we now see accuracy ratings of more than 80%, up from less than 30%. Our QA process has led to 161 documentation improvements, which puts more and better public resources in customers’ hands in addition to helping the AI provide more accurate responses. Our AI resolution rate is now 41%, compared with 8.6% in the first six months.
The Future
AI squads’ close work with AI has not only made our bots better, it has also tapped into their creativity for other opportunities to harness AI to help better support our customers.
Triage bots
Our engineers created internal triage bots that analyze incoming support and classifies them according to criteria our AI squads created, defined, tested, and fine-tuned until the classifiers were accurate enough to put into production. We instruct the bot to look for signals on things like the following:
- Fatal errors so we can route to the priority queue.
- Issues AI should not reply to because they require human judgment or empathy.
- Feature usage so we can route to the correct queue or agents.
The triage bots add tags to the tickets, which Zendesk triggers then pick up for routing. This speeds up our triage process and also provides tags for tracking.
Demand analysis
Product managers and product teams often ask, “What are the top three issues that customers contact support about?” This can be a surprisingly difficult question to answer given the volume and nuance of words customers use to describe what they need help with. The strength of an LLM is its ability to process natural language, and members of our AI squads are experimenting with using AI to digest incoming issues and summarize the top drivers of demand.
Case Summaries
Because we are a distributed company with employees around the world, we have global 24/7 support with rolling shifts: at any hour of the day, some Happiness Engineer’s shifts are ending and others are beginning. This enables us to pass customer email or live chat cases from one HE to another throughout the day. When transferring a customer to a new agent, the new agent needs to be brought up to speed quickly. Prior to AI, this required the original agent to summarize the case for the receiving agent, which required time and significant cognition. We are currently exploring using AI to summarize conversations at handoff to reduce this load from Happiness Engineers.
Final thoughts
As artificial intelligence evolves, we must remember that its power lies not in replacing human intelligence, but in augmenting it. The true potential of AI lies in its ability to amplify human creativity and ingenuity.
Ginni Rometty
Executive Chairman at IBM, 2022
Andrea Badgley 