Backtest
Status: Passed. One run per version, with the pass rules written down before we saw the result. The current scoring version, score-2.0, was tested on 2026-10-08. The earlier version, score-1.0, was tested first and is kept below as history.
What it tested (score-2.0)
Does adding package usage to Reality hurt the ranking? We drew 200 projects at random from 3,867 candidates that peaked in 7-day star gain between 2024-01-01 and 2024-06-05. This is a fresh set, not the one used for score-1.0. Each was scored using only data that existed at the time, then checked a year later. The pass rule, written first: rank projects that were abandoned a year later at least as well as score-1.0 does, on the same projects.
The result (score-2.0)
- Projects compared: 196 (4 dropped). Abandoned: 55. With an Adopted score: 10.
- score-1.0 Hype Gap AUROC: 0.666 (95% interval 0.582 to 0.744).
- score-2.0 Hype Gap AUROC: 0.674 (95% interval 0.591 to 0.752). 0.5 is a coin flip.
- score-2.0 minus score-1.0, 95% paired interval: -0.006 to 0.021. The interval includes zero, so read it as no worse, not better.
- Star velocity alone, best direction: 0.509.
- Of the 20 highest by Hype Gap, abandoned: 8 for score-1.0, 10 for score-2.0.
It passed: the score-2.0 AUROC is at least the score-1.0 AUROC.
The earlier result (score-1.0, history)
Does the Hype Gap predict which projects were later abandoned better than raw star growth does? We took 198 projects that peaked in the first half of 2024 and scored them using only data that existed at the time. 53 were abandoned.
- Hype Gap AUROC for abandoned: 0.673 (95% interval 0.593 to 0.753).
- Star velocity alone: 0.521.
- Hype Gap minus star velocity, 95% paired interval: 0.082 to 0.227. It stayed above zero.
- Of the 20 highest by Hype Gap, 8 were abandoned. Of the 20 highest by stars in 7 days, 3.
That pass rule: AUROC of 0.60 or more with the interval above 0.50, and a lower bound above zero against star velocity. Both held.
Where it is weak
- In the score-1.0 test, hype alone scored 0.48. That is no signal. The signal comes from Reality: a project with lots of attention and little evidence behind it.
- The abandoned labels came from a language model (Jev). Where it was unsure, we checked the project against its GitHub history ourselves.
- Each version was tested on one cohort of projects that peaked in the first half of 2024. We have not shown it holds for other periods.
- The score-2.0 gain over score-1.0 is within the noise. Only 10 of the 196 projects compared had usage data from that time, so the test says little about how well usage itself works.
- A score is a rule applied to public numbers. It is not a judgment of the people behind a project.
See also the methodology.