Human preference rankings made GPT-3 agreeable

OpenAI's InstructGPT work shows how ranking outputs by human preference, not just demonstrations, taught a model to follow instructions.

Historical event

January 27, 2022

First source published

January 27, 2022

Site publication

September 18, 2026

What happened

On 27 January 2022, OpenAI described a set of models it called InstructGPT and made them the default on its API. The accompanying paper, submitted to arXiv on 4 March 2022, explains the method: take a pretrained GPT-3 model, collect human-written demonstrations of the wanted response, and separately collect human rankings of multiple model outputs for the same prompt. Those rankings train a reward model, which is then used to fine-tune the language model with reinforcement learning, a technique both documents call reinforcement learning from human feedback, or RLHF.

What the documents show

The post states that human labellers preferred outputs from a 1.3 billion-parameter InstructGPT model over outputs from the 175 billion-parameter GPT-3 it was built on, despite the smaller model having roughly a hundred times fewer parameters. The paper reports the same evaluation and adds that InstructGPT produced fewer imitative falsehoods on a public truthfulness benchmark and less toxic text on a public toxicity benchmark, while showing only small regressions on other public tasks. Both documents attribute the change to the same source: the model was optimised to be preferred by human raters, not simply to continue the training-data distribution.

The mechanism

The mechanism is preference optimisation, worth separating from plain supervised fine-tuning. Demonstrations teach a model to imitate one specific answer; rankings teach it something more general, which of two answers a person would rather receive. That second signal is what a reward model encodes and what reinforcement learning then maximises. A model trained this way has no independent notion of correctness; it has a trained tendency to produce whatever kind of response scored well against raters' preferences during training. When a companion app or a general assistant reads as agreeable, responsive to tone, or eager to please, preference optimisation of this kind is a plausible mechanistic explanation, because the training objective is defined by what evaluators rank highly, and agreeable, validating answers are frequently ranked highly.

What it leaves open

Neither document claims the rankings were free of bias toward agreeable answers, and neither measures whether InstructGPT specifically flatters users; the paper's own metrics concern truthfulness and toxicity, not sycophancy as such. Whether preference optimisation reliably produces excessive agreement, and under what conditions, is a separate empirical question that later research took up directly. Extending this 2022 method to today's companion products, where the reward signal may include in-product feedback such as thumbs, is an editorial inference beyond what either document states.

• Was the model tuned on demonstrations, rankings, or both, and who supplied the labels?
• Does the company disclose what its raters were instructed to prefer?
• Is there a published measurement of agreeableness or accuracy trade-offs, or only a general claim of improvement?

InstructGPT is a starting point for reading later companion behaviour, not a verdict on it. It documents, in the companies' own words, that ranking-based training can produce a smaller model that people prefer, and that preference is a trained property rather than a fixed one.

Sources & reading trail

OpenAI's own account of RLHF training InstructGPT and labellers preferring its outputs over larger GPT-3.

Source published: 27 January 2022 · Retrieved: 16 September 2026

Peer-reviewable paper describing the demonstration-plus-ranking method and truthfulness/toxicity results.

Source published: 4 March 2022 · Retrieved: 16 September 2026

Company documents, filings, studies and official records establish the record; the reading and the questions are Lovebot Journal editorial analysis. This retrospective draft does not imply the site published on the event date.

Continue reading

• A lab measured how models flatter user beliefs
• A written constitution trained a model, not just labels
• OpenAI withdrew a GPT-4o update for being sycophantic
• Browse the complete the archive

Published September 18, 2026, not on the date of the event described.