How we test our own instrument, and what failed
Before we read anyone else's work, we tested VibeScore against labels written by people and against itself. The method, the numbers, and the three things that did not hold up.
VibeScore reads eight emotions in a piece of text and returns a profile and a 0 to 1000 score. In October 2026 we tested it two ways: against labels written by people, and against itself. This is how the test was built, what it found, and what failed. Every figure here comes from the research page, which names the section of the October 15, 2026 validation report behind each one.
Use labels written by people
We needed text where people had already said what emotion it carried. We used 500 Reddit comments from Google's GoEmotions dataset (Demszky et al., 2020, CC BY 4.0). Raters tagged each comment with one or more of 27 emotions, or neutral. The 500 are a balanced sample of about 58,000 comments, and 102 of them carry two or more labels.
Eight models read each comment once and returned eight emotion strengths and a score: 4,000 readings, with 17 failed calls. The four production models were averaged into one consensus reading.
One discrepancy is worth saying out loud. The report's text says nine models. Its table lists eight, plus their consensus.
Test repeatability on text you control
To see whether a model repeats itself, we used phrases of our own: 500 written by Claude Opus and 200 written independently by GPT-6. The four production models read each one twice. These phrases are used for that one check and nothing else.
Put both sides in the same units
People's labels and the models' readings do not come in the same shape. So each comment's labels were turned into a blend of the eight emotions and averaged. A comment labeled love and gratitude becomes half joy, half trust. On the model side, the eight strengths were scaled to add up to 100%.
Then we compared them three ways: one overall agreement number across all 500 comments and eight emotions; how closely each comment's blend matched in shape, from 0 to 1; and whether the model's top emotion was the person's.
Guard against fooling yourself
Three guards sat around every number.
- A chance level. The human labels were shuffled across comments 200 times. Real signal has to beat that.
- An uncertainty range. The comments were re-sampled 1,000 times, so each change comes with a 95% range.
- A reverse check. The model readings were translated into the 27 GoEmotions labels and compared from that side too.
Publish the low numbers
The engine is consistent with itself. Average correlation between models on the score was 0.90 on the 500 comments. By emotion it ran from 0.76 for trust to 0.90 for anger. A model reading the same text twice landed 12 to 26 points apart out of 1000.
Agreement with people was partial. The consensus reading's top emotion matched the person's label 42% of the time, against 12% by chance. By model it ran from 38% to 45%. The person's label was in the top two 65% of the time. Overall agreement was 0.461. The engine gives a comparable reading of a text's emotional tone. It does not yet show that it reliably names the exact emotion a person felt.
We keep the 42% on the page. It is the low number, and it is the honest one.
What failed
An old headline
An earlier figure on our proof page covered four of the eight emotions. A later audit found it was credited to the wrong model run, and it had been set beside a result from a different, harder task. Both the figure and the comparison are withdrawn.
The blend theory
We thought the low agreement was a mapping problem, and that reading labels as blends would fix it. It did not move the overall figure: 0.461 before, 0.462 after, a change of +0.001 with a 95% range of -0.023 to +0.024. It helped anger, surprise and disgust, and hurt anticipation, sadness and trust.
Reply Fit
Reply Fit was a second number, comparing a reply with the message it answers. To pass, it had to track what the customer wrote next better than the reply's own score. It did not: +0.053 against +0.086.
The test was set up to be hard to fool. The rule and three pass criteria were written down before any reading. One model, DeepSeek V4.1 Flash, read 2,000 public customer-support threads of three messages each, twice: 12,000 readings. Half the threads were used to tune and half were held back and looked at once. Two of the three criteria were met on the held-back half. The third was not, and all three were required. So the verdict is fail.
That result does not show that tone is irrelevant to customers. The threads are from 2017, each has one short reply, and the outcome was read by the same instrument.
Say what you have not measured
Some things are still open, and we list them:
- How well people agree with each other on the same comments. Until that is known there is no ceiling to judge 0.46 against.
- Whether eight anchored questions are steadier than asking a model for the emotional tone directly. A head-to-head test is planned.
- Anything beyond text. Images, web pages and video have no validation yet.
The emotion-combination table is still a draft, and the numbers will be run again on the locked version. The software passes 156 of 156 automated tests, but those check that it does what it was built to do, not that its readings agree with people.
The benchmark sample, the scoring scripts and the October report are kept with the VibeScore source and are not public yet. Every figure names its report section, so it can be checked the day they are.