XiaoIce's own paper names conversation length as its target metric

What happened

In December 2018, Microsoft researchers Li Zhou, Jianfeng Gao, Di Li and Heung-Yeung Shum posted The Design and Implementation of XiaoIce, an Empathetic Social Chatbot, also released as Microsoft Research technical report MSR-TR-2018-42. XiaoIce itself had launched in China in 2014; the paper is the company's own account, four years on, of what the system was built to do and how it measures success.

What the documents show

Both versions of the paper state that XiaoIce is designed around 'IQ' and 'EQ' in combination, and report that, by the paper's writing, XiaoIce 'has communicated with over 660 million users' and established long-term relationships with many of them. The paper names its central success metric directly: Conversation-turns Per Session, or CPS, and reports XiaoIce achieving an average CPS of 23, which the authors describe as significantly higher than competing chatbots and, they state, than typical human conversations. Neither version reports accuracy, task-completion or factual-correctness rates as a primary metric.

The mechanism

Optimising for CPS optimises for the conversation continuing, not for any particular claim within it being true or useful. The paper's own emphasis on an 'empathetic computing' component, designed to detect a user's emotional state and respond in ways suited to it, is explicitly in service of extending sessions - positioning emotional responsiveness as an engagement mechanism much as a recommendation algorithm treats watch time. A system built to maximise turns per session and a system built to answer a question correctly are not the same system, and a paper that discloses CPS as its headline metric is telling a reader which one it is.

What it leaves open

The paper does not report user wellbeing outcomes, retention over months or years, or how emotional-state detection was validated against ground truth rather than user-reported or inferred signals; those questions sit outside what the authors chose to measure and disclose. Whether a high CPS represents a good outcome for the person on the other end of the conversation is a judgment this document does not make, and this record will not make on its behalf.

• Is a chatbot's headline metric about the user's benefit, or about session length and return visits?
• How was 'empathy' or emotional detection validated, and against what standard?
• Does a large user count describe active relationships, or cumulative lifetime contacts?

A paper that names its own optimisation target plainly is, in that respect, more transparent than most companion products' marketing; the target it names is still worth reading carefully before treating engagement as evidence of benefit.