• CCAI

Conversational AI Optimization Playbook: 7 Steps to Better Performance


What is Conversational AI performance optimization?

Optimization is the process of assessing and improving application performance based on observation of conversational usage and data about it. It's the way we improve user experience and overall success. We fill in the meaning gaps and fix mismatched experiences. Issues can occur from changing and incomplete contexts, plus biases and flaws in the design, or holes in the supporting data. The process we outline here focuses on the effectiveness and quality of the application and the experience it provides. It's a necessity given the current states of technology and understanding of human behavior and can set your organization above the rest.


Reasons we plan for and perform optimization

  • Ensure that business objectives and key results are met

  • Ensure the customer experience starts well and improves over time

  • To harvest lessons for future changes and applications

What you need to get started

  • Clear and measurable goals for application performance (see typical performance metrics below)

  • Educate the client about the process, emphasizing that it involves some amount of trial and error as well as margin of error

  • A design crafted to achieve those goals

  • Complete and thorough data capture validated in the application

  • Minimum session volume (calls, chats, etc.) for statistical significance of metrics

  • Shared repository of source and working materials for optimization


Typical performance metrics used as business goals

  • Resolution of intents that can be fully automated

  • Containment (sessions that stay within the application)

  • Customer satisfaction (CSAT)

  • Net Promoter Score (NPS)

  • Correct routing of escalations


Optimization Checklist


1. While the application is going through quality assurance and acceptance testing

Review data set samples and details

Validate reliable and timely access to data sources

Verification of data sets to be used

Session data and transcript needs

Sortable, unique session identifier

Date/time stamp accuracy

Audio recordings for speech

Turn-by-turn interaction transcripts (stripped of markup)

Intent capture

Node by node turn results, including errors

Raw input plus recognition results

Intent disposition

Session disposition


2. When the application is launched

Double check that all production data is coming in reliably and is accessible

Remind client of process and timing

Check stats for possible immediate problem areas

Begin to pull analysis and validation data sets

Start voice transcription of entire sessions (Voice/IVR only)


3. Post-launch analysis

Begin analyzing high level performance (no judgments until statistical significance is reached)

Re-confirm goal metrics by performing manual calculations

Note signs of potential underperformance from metrics (recognition problems, negative stats, expressions of user dissatisfaction)

Initial data, focus on early problems


4. Read transcripts

Listen to whole sessions (Voice/IVR only)

Look at clustered and aggregated events (e.g., all inputs to a single node)

Annotate and tag good and not-so-good interactions

Double-check for mishandled input in areas of satisfactory performance


5. Discuss and classify observations

Identify desired and undesired behaviors and results

Determine the primary undesirable issues to analyze further based on symptoms seen in analysis and priorities for business and user


6. Determine Solutions

Collaborate across design, data science, and development to drill down into problem areas and identify root causes

Examine flow changes, model changes, wording changes, rule changes, etc.

Collaborate to find best solution possibilities and estimate level of effort (LOE)

Prioritize solutions relative to problem importance, potential upside, and LOE

Present findings and recommendations to client

Get agreement on prioritization, work streams, release target planning, and resource scheduling


7. Deliver solutions

Schedule work

Design, build, test, and launch

Return to Step 1




Looking to dive deeper with your team on optimizing voice, chat, or multimodal performance? CCAI offers a 3.5 hour workshop for teams looking to get more from conversational AI.