I just submitted three papers to ICLR, and five total if I count two collaborative efforts. As of last night’s abstract deadline, there were over 60,000 submissions registered.

Last year, ICLR had 19,525 valid, format-compliant submissions as per the stats posted on their blog. 779 submissions, or 3.99%, were desk-rejected, and 5,042, or 25.82%, were withdrawn. 13,763 papers — 70.49% of the total number of submissions — ultimately completed the peer review cycle in its entirety, and reached a final decision.

Assuming the trends remain more or less the same, and not controlling for LLM-assisted writing resulting in a larger percentage of desk rejections (due to hallucinations, format violations, bibliography errors, etc.), these stats mean we can expect this year’s cycle to yield approximately 42,300 valid submissions that go through peer review, while the number of reviewers in the eligible pool remains roughly the same. That’s more than 3x the reviewer load from last year.

When I started my research career more than a decade ago, we would spend weeks, sometimes months, trying to understand a problem space before forming a mature point of view on what was worth working on. That process involved reading backwards through the literature, figuring out why successive generations of methods made the design choices they did — often by re-implementing them on your problem and your specific dataset, with much struggle to reproduce hyperparameter choices so the comparison was apples-to-apples — then identifying hidden assumptions you were willing to challenge to try new experimental formulations, accumulating evidence for why it works — or doesn’t work — and changing your mind, abandoning ideas that seemed like a dead end at the time, to eventually develop something that sticks, reproduces nicely in multiple contexts or domains, and generalizes across architectures. And sometimes we would get scooped, but the likelihood of that happening was fairly low because you could always find uncharted territory that others had not thought about yet or had not had time to explore.

Now, with LLM tooling, the research loop has compressed dramatically. With the substantial speed-up in our ability to investigate different research directions, we delegate these tasks to autonomous research agents who do literature surveys for us, generate and implement experiment ideas for us, and do the final writeup for us — leaving us to ramp up to new domains in record time, but never fully develop a deep understanding of the problem landscape. As a result, we often end up producing redundant work under a new name, or recreating concurrent work, unaware that someone else is solving the same problem. We aren’t really assessing novelty truthfully, because we don’t write or read papers the same way we used to, and rarely pay attention to anything published more than a year ago.

We have empirical papers going viral without conducting a single experiment, and paper titles grossly overstating the generalization capabilities of a framework to farm engagement, while openly admitting to doing so. Developing context on how a problem space has evolved — what we like to call “research taste and wisdom” — is now a rare skill. Much of what people claim novelty for is the same idea packaged in different marketing language.

Reminds me of one of those 90s lottery machines: a giant transparent sphere filled with numbered balls, each representing a research idea, all bouncing around at high speed until one gets sucked into the draw tube, and emerges as the winning number. We take existing ideas, throw them into a massive lottery sphere, crank up the speed and ask, “What’s the next number?” And voila! A ball pops up at the slot that looks sufficiently different to receive a new name, and a “novel” paper emerges.

I don’t think this is a problem individual researchers can solve on their own. We obviously have an incentive problem as a field, and it extends far beyond researchers to the whole AI ecosystem, investors, founders, regulators, the media, consumers, and the general public alike. When claims of progress help determine where capital, compute, talent, and influence accumulate, research priorities become increasingly entangled with commercial interests. Our production system rewards generating another marketable finding that can sway incentives “the right way”, rather than consolidating what we know into a deeper understanding of the problem.