Forecasting Like a Pro: Q&A with Lennart Heim

How did you get started with crowdsourced forecasting and how did you get so good at it?

It started with my interest in developing evidence-based policy recommendations. Someone told me about Philip Tetlock and the superforecasters. I realized that crowdsourced forecasting was one tool in the evidence-based policy research toolkit. I went to a calibration workshop—which helps forecasters correct bias and sampling error, for example— and understood better how to apply the method toward finding good answers. We worked through questions like "how many cows are there on the planet?" A well-calibrated group will get pretty close to the right answer. Then I started forecasting with CSET-Foretell (which became INFER-Pub and is now RFI). CSET-Foretell was looking for forecasting ambassadors so in that role I met with a group of people every two weeks to work on developing forecasts together.

What are some of your favorite forecasting topics to work on?

While forecasting has become prominent in AI risk analysis, I like forecasting on semi-conductor compute and trying to project future demand. As one example, I worked on a project that sought to answer what historical trends over the past 70 years might imply about the future of machine learning compute.

More broadly, I like to use my forecasting time as a way to stay updated on the rest of the world; I think it's a healthy way for me to engage with the daily news. I look at daily events within the context of what they indicate about the future for forecasting questions that I'm working on. Sometimes I take notes on signposts to watch for based on a news article that I've read. Other times I decide that a new development warrants an immediate update to my forecast on a topic.

In one current example of using the platform on geopolitical topics, I was part of a group of forecasters that accurately predicted the impending Russian invasion of Ukraine. The outlook for the war in Ukraine remains a big topic for forecasters.

How does forecasting as a capability relate to your current work helping find solutions to public policy problems in technology and security?

Lennart: There are three critical trends for AI: data, algorithms, and compute. In my work, when I look at trends, I ask myself if projected trend lines are linear or exponential. When you see an exponential trend, that's when it's time to alert policymakers. In my work on AI governance—a set of principles, practices, and frameworks that ensure the responsible use of AI—my colleagues and I estimated an exponential doubling in AI compute trendlines. Moreover, I saw indicators of the growing importance of compute in the course of my forecasting work before Nvidia stock revealed this insight.

AI hardware is another area where forecasting will be helpful to public policy. Export control policies, for example, hinge on 1) AI systems will require compute; and 2) AI hardware will improve. My job is to collect the data to update my probabilistic forecasts to incorporate new information or additional insights, helping decisionmakers account for changing circumstances.

What is one piece of advice that you would give to those looking to improve their forecasting skills?

I'm teaching a class at the Pardee RAND Graduate School on AI governance and I've assigned the Ten Commandments for Aspiring Superforecasters as one of the class readings. I recommend that all new forecasters read this because it's good for forecasting as well as general good life principles. Read it, and then get started forecasting on the RFI platform.

—Marie Jones

Originally published December 6, 2024