Bleeding-edge AI technology
When a Fortune 500 company with $200 billion in annual revenue came to us with the challenge of building a first-of-its-kind AI assistant for its marketing and e-commerce website, iSoftStone’s experts were ready to help.
Our client’s website showcases hundreds of products and contains a rich knowledge base and information library for customers. Instead of manually scrolling through multiple static webpages, an AI assistant could enable its customers to receive accurate, in-depth answers to natural language questions directly from the storefront. That potentially meant better, faster service for customers and more sales conversions.
iSoftStone wants to empower our customers to harness the potential of business AI transformation. We’ve invested heavily in AI technology and in helping our customers integrate AI seamlessly into their enterprises to unlock unparalleled growth and efficiency.
Shunping Tang
Associate Vice President, Delivery Engagement | iSoftStone
We all know that AI implementation is a fast-evolving world and we’ve all seen companies stumble into bad publicity. While our client was eager to lead the way in AI innovation, it was important to its leadership to model responsible AI in both the customer experience and the execution of the project itself. The front end was also subject to multiple design reviews, changes, and scrutiny at the highest level.
Many of the third-party tools used to build AI assistants and chatbots are still maturing. They update rapidly, and sometimes workflows become defunct overnight. iSoftStone’s experts came to the project nimble and ready to pivot. We adjusted and refined our implementation throughout the project and put in place all the necessary safeguards to ensure our client’s reputation for quality customer experience.
An accurate and performant AI assistant
We partnered with our client’s internal teams and design vendor to build an AI assistant on Microsoft Copilot Studio with Adobe Experience Manager (AEM) as the front-end content management system. We developed an initial proof of concept (POC) and, ultimately, the live solution, scoping the AI to the client’s specific knowledge base, and architecting new AEM components and content fragments.
To help improve the AI assistant’s performance and accuracy, our client also requested thumbs up/thumbs down buttons for users to rate the quality of the assistant’s responses. This feedback mechanism isn’t natively available in Copilot Studio, so iSoftStone’s team built it with Power Automate and Azure Cosmos.
Finally, we optimized the assistant for speed and thoroughly stress-tested it for quality.
Attention to AI quality assurance
A key feature of generative AI is that it may provide different answers to the same topic, depending on what you ask and how you ask it.
“It can be a strength,” says Travis Heller, one of iSoftStone’s technical project managers on the engagement. “It can also be a weakness when we’re trying to determine what’s an acceptable experience for customers. We had to make sure that our client’s AI assistant wasn’t going to come back with something harmful or offensive or damaging to the image of the company.”
Our experts tested the AI assistant for malicious questions, irrelevant questions, profanities, slurs, and output errors. Then we fine-tuned it, tested it, and fine-tuned again to ensure the AI would respond appropriately every time and never provide information outside of the training data.
Solving the “poop prompt”
One unexpected issue our team identified during the QA process: repeating “poop” as a single-word prompt multiple times caused the assistant to reply with random product information. They found the issue replicated with a handful of other single words too, including “Jesus.”
“It was a fun project to tackle,” says Trent Macomber, an iSoftStone software engineer. “We looked through our custom code to see if we’d configured something wrong. We looked through all the documentation. Are there prompts we’re missing? Are we forgetting to filter out certain words? Eventually, we deconstructed the chatbot down to its most basic level and function. It was something in the pure AI. We had to raise that bug to Microsoft’s AI team.”
Responsible AI, carefully launched
To monitor the AI assistant’s performance and its impact on website KPIs, analyze how customers were interacting with it, and collect feedback at a granular level, our client decided on a controlled rollout. Using Adobe Target’s capability of splitting a page into a and b variations, iSoftStone’s team ran and monitored a traffic split for our client that started at 1% and grew incrementally as the quality of the assistant proved out.
Now, when visitors to our client’s website want product information, they can ask questions and receive a clear, accurate natural language response with footnotes and citations. The AI assistant will also provide links to relevant pages on the website for a customer to learn more. It suggests follow-up questions too. All a user has to do is click and the AI assistant responds with an accurate, detailed answer.
If a customer asks about something irrelevant to the company’s products? Thanks to the thorough QA work of iSoftStone’s team, the assistant will consistently offer a polite response reminding them that it can only answer questions about the company and its products.
We’re here to help
As our client discovered, organizations embracing the fast-moving world of AI solutions encounter both unique opportunities and unique challenges. Finding the right partner to design and develop responsible AI solutions that are also flexible, scalable, and deliver value has never been more important. We’re here to help.
