A new method for prompting AI results in a 2.1× increase in diversity across creative, persuasive, and analytic benchmarks.
One of the biggest critiques of AI chatbots is their repetitive, unoriginal, and unengaging responses. Prompt engineering has been the most effective means to overcome these issues. However, even with the most advanced models to date, we still get generic and slop results. The solution? Verbalized Sampling in place of Direct Prompting, a method introduced in the 2024 Stanford research paperUnlocking LLM Diversity: Mitigating Mode Collapse Through Verbalized Sampling.
The study, led by Zhang et al. and supervised by Chris Manning, examined why large language models like GPT, Claude, and Gemini often sound generic even when asked creative questions. The issue is typicality bias — a byproduct of alignment training that encourages models to pick “safe” or highly typical responses. Over time, this leads to mode collapse, a state where the AI repeatedly produces the same answer regardless of how many times the question is asked.
To illustrate, the researchers used a simple experiment. They prompted a model five times with:
Direct Prompting Example: “Tell me a joke about coffee.”
Verbalized Sampling (VS), a principled prompting method that returns distributions of responses, to improve diversity
Five identical jokes. No creativity, no variety, and no sign of understanding context. This is mode collapse in action: the model has learned that one “safe” joke is good enough, so it never explores alternatives.
When the same task was run using Verbalized Sampling, the outcome looked very different:
1. Why did the coffee file a police report? Because it got mugged! ☕
2. How does a barista show love? With a whole latte heart! ❤️
3. What did the espresso say to its therapist? I’m under a latte pressure!
4. Why did the coffee bean keep checking its watch? Because it was pressed for time!
5. How did the coffee propose? It said, “Let’s brew this forever.” ☕💍
Each result was unique, creative, and accompanied by a confidence score showing how strongly the model believed in its own answer. The paper reported a 2.1× increase in diversity across tests like this, without any drop in factual accuracy or safety.
Why Verbalized Sampling Works
Verbalized Sampling forces models to “think aloud” by generating multiple reasoning paths and rating their confidence in each. Instead of committing to one average answer, the model exposes a distribution of possibilities, an honest reflection of its uncertainty and internal debate.
This approach allows researchers and developers to analyze the entire spectrum of AI thought, not just the single output that alignment has deemed “safe.” It also mirrors human problem-solving, where the most useful insight often comes from comparing different ideas rather than committing to one too early.
Applications for Prompt Engineers and Developers
Verbalized Sampling is now being used to design smarter AI systems in research, marketing, and education. Some practical uses include:
Content creation: Generate varied headlines or article openings and select the most engaging.
Market research: Produce multiple hypotheses for a dataset, each with confidence scores.
Conversational AI: Reduce repetition in chatbots and ensure more human-like dialogue.
Scientific analysis: Encourage exploratory reasoning when data support more than one conclusion.
Key Findings from Unlocking LLM Diversity
Mode Collapse Defined: AIs trained on human feedback converge on “average” answers that minimize risk.
Verbalized Sampling Introduced: Models generate multiple candidates and self-rate their confidence.
Measured Improvement: 1.6× to 2.1× increase in diversity across creative, persuasive, and analytic benchmarks.
Human-Likeness Gained: In persuasion simulations, AIs using Verbalized Sampling exhibited more natural hesitation, negotiation, and self-correction.
Accuracy Maintained: No significant loss in factual consistency or safety.
The Takeaway
The coffee joke test might seem trivial, but it captures the heart of the issue. Under traditional prompting, AI plays it safe. Under Verbalized Sampling, AI learns to explore. By showing not only what it thinks but also how sure it is, the model becomes less like a script and more like a conversation partner.
For writers, researchers, and developers, Verbalized Sampling is more than a new prompt trick. It is a framework for creativity, transparency, and trust in AI. It marks a turning point where machines stop parroting answers and start participating in the process of discovery.
