Two 2020 papers proposed a way to grade how human a chatbot sounds

What happened

In January 2020, Google researchers published a paper on Meena, a 2.6-billion-parameter chatbot trained on social-media conversation. Three months later, in April 2020, a Facebook AI team published a paper on BlenderBot, an open-domain chatbot released with model weights the public could inspect. Both papers targeted the same problem: existing chatbots produced replies that were either generic or nonsensical, and there was no agreed way to measure the difference. Google's Meena paper introduced the Sensibleness and Specificity Average (SSA), a human-rated score combining whether a reply makes sense and whether it is specific to what was just said. Meta's BlenderBot paper adopted a related set of human ratings for engagingness and humanness, releasing three model sizes up to 9.4 billion parameters.

What the documents show

The Meena paper reports that its filtered, tuned model scored 79% on SSA, against an estimated 86% for humans, and that this figure correlated closely with the model's perplexity across models tested. Google's own announcement is explicit that the company was not releasing a public demo, citing unresolved safety and bias concerns. The BlenderBot paper adds a claim the Meena paper does not attempt: that a chatbot can hold a consistent persona while showing empathy and topical knowledge, by training on a blended-skill dataset rather than raw conversation logs. Meta's blog post reports that in its own trial, 67% of evaluators found BlenderBot more human-sounding than Meena, and 75% would rather hold a long conversation with it — a comparison Google did not run.

The mechanism

Both systems predict a plausible next utterance from prior conversation, then are narrowed by a human-scored tuning step; neither paper describes a persistent memory of an individual user across sessions. Persona in the BlenderBot paper means a short block of stated facts a model stays consistent with in one conversation, not a system that recalls a specific person over time. That distinction matters for anyone comparing these 2020 papers to a modern companion app: sensibleness and specificity are single-turn qualities, not measures of attachment, continuity or safety over weeks of use.

What it leaves open

Neither paper is a companion-app safety study. Google's post names bias as an open problem rather than a solved one, and Meta's paper documents failure cases without claiming they are resolved. Reading either paper as evidence that a 2020-era chatbot was safe for sustained emotional use would go beyond what the documents state; that caution is editorial, not a finding either source makes.

• Does a cited human-likeness score measure a single reply, or a sustained relationship?
• Is a stated persona a short prompt for one conversation, or a trained, recalled identity?
• Did the team publishing a safety claim also run the comparison, using its own criteria?

The SSA metric and the blended-skill approach were genuine methodological contributions, and each paper's authors treat the other's approach as a real rival rather than a strawman. Read together, they mark the moment open-domain conversation became a measurable target rather than a demo trick — which is also the moment it became possible to optimise a model for sounding human without that model remembering anyone.