Conversational AI Optimization Playbook: 7 Steps to Better Performance

What is Conversational AI performance optimization?

Optimization is the process of assessing and improving application performance based on observation of conversational usage and data about it. It's the way we improve user experience and overall success. We fill in the meaning gaps and fix mismatched experiences. Issues can occur from changing and incomplete contexts, plus biases and flaws in the design, or holes in the supporting data. The process we outline here focuses on the effectiveness and quality of the application and the experience it provides. It's a necessity given the current states of technology and understanding of human behavior and can set your organization above the rest.

Reasons we plan for and perform optimization

• Ensure that business objectives and key results are met

• Ensure the customer experience starts well and improves over time

• To harvest lessons for future changes and applications

What you need to get started

• Clear and measurable goals for application performance (see typical performance metrics below)

• Educate the client about the process, emphasizing that it involves some amount of trial and error as well as margin of error

• A design crafted to achieve those goals

• Complete and thorough data capture validated in the application

• Minimum session volume (calls, chats, etc.) for statistical significance of metrics

• Shared repository of source and working materials for optimization

Typical performance metrics used as business goals

• Resolution of intents that can be fully automated

• Containment (sessions that stay within the application)

• Customer satisfaction (CSAT)

• Net Promoter Score (NPS)

• Correct routing of escalations

Optimization Checklist

1. While the application is going through quality assurance and acceptance testing

☐ Review data set samples and details

☐ Validate reliable and timely access to data sources

☐ Verification of data sets to be used

☐ Session data and transcript needs

☐ Sortable, unique session identifier

☐ Date/time stamp accuracy

☐ Audio recordings for speech

☐ Turn-by-turn interaction transcripts (stripped of markup)

☐ Intent capture

☐ Node by node turn results, including errors

☐ Raw input plus recognition results

☐ Intent disposition

☐ Session disposition

2. When the application is launched

☐ Double check that all production data is coming in reliably and is accessible

☐ Remind client of process and timing

☐ Check stats for possible immediate problem areas

☐ Begin to pull analysis and validation data sets

☐ Start voice transcription of entire sessions (Voice/IVR only)

3. Post-launch analysis

☐ Begin analyzing high level performance (no judgments until statistical significance is reached)

☐ Re-confirm goal metrics by performing manual calculations

☐ Note signs of potential underperformance from metrics (recognition problems, negative stats, expressions of user dissatisfaction)

☐ Initial data, focus on early problems

4. Read transcripts

☐ Listen to whole sessions (Voice/IVR only)

☐ Look at clustered and aggregated events (e.g., all inputs to a single node)

☐ Annotate and tag good and not-so-good interactions

☐ Double-check for mishandled input in areas of satisfactory performance

5. Discuss and classify observations

☐ Identify desired and undesired behaviors and results

☐ Determine the primary undesirable issues to analyze further based on symptoms seen in analysis and priorities for business and user

6. Determine Solutions

☐ Collaborate across design, data science, and development to drill down into problem areas and identify root causes

☐ Examine flow changes, model changes, wording changes, rule changes, etc.

☐ Collaborate to find best solution possibilities and estimate level of effort (LOE)

☐ Prioritize solutions relative to problem importance, potential upside, and LOE

☐ Present findings and recommendations to client

☐ Get agreement on prioritization, work streams, release target planning, and resource scheduling

7. Deliver solutions

☐ Schedule work

☐ Design, build, test, and launch

☐ Return to Step 1

---

Looking to dive deeper with your team on optimizing voice, chat, or multimodal performance? CCAI offers a 3.5 hour workshop for teams looking to get more from conversational AI.