Forecasting Best Practices: Q&A with Joseph Matveyenko and Chris Adams

What initially drew you to forecasting? Why was it something you wanted to do?

Joseph: I first learned about forecasting after reading Philip Tetlock's book and hearing about RFI's work. What fascinated me was the art of estimation—breaking complex problems into manageable pieces. I loved examples like figuring out how many piano tuners a city might need by reasoning through inputs such as demand and workload. I found myself using that step-by-step approach in everyday decisions. I've also been interested in prediction markets to understand how odds and calibration work, so joining a forecasting program felt like a natural extension of those interests.

Chris: Forecasting is crucial to the work that RAND does. It's about knowing what the future will hold. I work on emerging technology, and we care about what the threat space will look like. Having a rigorous approach to forecasting helps to take some of the guesswork out of futuring.

Can you tell us about your experience forecasting?

Joseph: There was a steep learning curve at first. Forecast questions span so many fields that you're constantly pushed to research new areas and build a baseline before assigning probabilities. Making those first forecasts was intimidating—I worried whether something should be 30% or 40% likely—but you learn to accept uncertainty and being wrong. I realized forecasting is iterative: revisit questions and update estimates as new information appears.

Chris: I didn't read the Superforecasting book, but I was familiar with the process. I was familiar with prediction markets. I thought the training was a really useful way of structuring what RAND analysts just do naturally. After the training, I was eager to jump in. I set aside time on Friday afternoons and steadily worked through the forecasts.

Can you describe your forecasting process? How do you decompose a question?

Joseph: My process depends on whether the question is numeric or event-based. For numeric forecasts—like estimating measles cases—I start with reliable data and structural drivers such as policy or exposure. For event questions, I focus on understanding the system behind the outcome. In forecasting a ranked-choice mayoral race, for example, I studied vote-transfer rounds using historical precedents and polling that modeled redistribution. Whatever the topic, I begin with a historical base rate and then adjust for the current context.

Chris: My approach was data-centric. I would find whatever data existed, build a spreadsheet, and examine trends to create a time series to establish a baseline. RFI often specified a dataset for resolution, which helped, though occasionally I'd misinterpret how a metric was defined. The Space Force budget question was one example where I had to course correct after seeing crowd rationales. Once the baseline was set, I would ask why it might be wrong and look for drivers that could shift it. With the wildfire forecast, I built the July trend first, then factored in NOAA's climate outlook (hotter and drier than usual), which nudged my estimate upward. I looked for correlations between similar warm, dry Julys to refine that. For the question on Putin ceasing to be president, I treated it like an actuary would.

When data sources aren't well specified, how do you approach setting a base rate?

Joseph: It's essentially the same principle: look for how often a certain level or event has occurred historically. For questions with limited data, I might initially look only at recent months or use seasonal averages. Other forecasters pointed out that that approach wasn't a true base rate, and they were right—it led me to refine my process by making the steps explicit. A proper base rate should get you reasonably close to historical precedent before you adjust for current factors.

Chris: My first step was always "what is the historical data?" I tried to build a time series, see if there were trends or if there could be a straight-line projection for what the baseline would be.

What types of sources do you use for background research? How do you balance prediction versus historical precedent?

Joseph: Balancing between historical precedent and prediction is challenging—the base rate is often the hardest step. A mentor once advised that the first forecast should be simply the base rate to then update over time. For numeric questions, I use various statistical tools such as seasonally adjusted moving averages, open-source time series forecasting packages in Python, and research papers on forecasting methods. I often aggregate results from several approaches to form a more stable estimate. For non-numeric questions, the process mostly involves reading news coverage from diverse sources. For example, when analyzing India—Pakistan tensions, many reports emphasized de-escalation efforts, which led me to forecast a lower likelihood of conflict. The best sources depend on context.

Chris: I rely heavily on quantitative and official data sources. For most forecasts, my process starts with historical datasets that I can analyze directly—anything from Space Force budget records to environmental data from NOAA. When I was working on the wildfire question, NOAA's long-term climate outlooks were invaluable; because they projected hotter, drier conditions, those inputs helped me adjust the baseline upward. My goal is to anchor forecasts in measurable precedent first, then update based on new evidence or changes in context.

How frequently did you update your forecasts? What would it take to move the needle?

Joseph: I updated forecasts as often as the data was published since staying current improves accuracy. For other questions, I usually updated weekly or biweekly, adjusting for major news developments. Some topics fade from media attention, so I'd apply a "time decay" approach and update monthly if nothing new emerged. Substantial updates occurred mainly for numeric topics, such as measles cases or wildfire acreage burned. I adjusted incrementally—rather than jumping from 20% to 80%, I might shift one-third of that distance at first, to avoid overreacting. This taught me to be careful about volatility and to respect dynamic processes like public health.

Chris: I updated every two weeks on a Friday. I would see which forecasts were stale, and then I would update those. When I could go back and start seeing swings, I started looking into what the crowd was saying. I would see if there was a news release of something I missed. There were many people who were updating daily, but I didn't have the bandwidth to do that.

What other tools or platforms did you use?

Joseph: I searched across various platforms and aggregators to find local and specialized sources. For the NATO-related question, I looked at local coverage from Spain; for [India—Pakistan], I used Indian outlets for better regional insight. On the technical side, I used RAND Chat to help generate Python code once I'd selected a forecasting method. I studied models submitted by top teams in prior forecasting tournaments and implemented some of their techniques. The main statistical tools I used included SARIMAX, Seasonal Auto Regressive Integrated Moving Average. Questions on inflation or oil production required careful selection and calibration of models since many external factors influence outcomes.

Chris: Just following the direction of the crowd is a good way to get an average forecast. I found there was benefit in being outside the mainstream on some things. Some of those I got right, some I got wrong. I did go read what other people said after my prediction. There were a wide range of predictions. I really liked the AI summaries of the crowd rationales.

If you were recommending best practices for other RAND forecasters, what would they be?

Joseph: Base-rate awareness is essential—even if something feels likely, you need to check how often similar events have occurred and consider the timeframe. Challenge your assumptions and biases; ask, "What am I missing?" I learned that lesson on a question where personal views temporarily clouded my judgment. Numeracy helps, but decomposition matters more—breaking questions into drivers and estimating each part clarifies reasoning. And always build datasets; quantifying patterns makes forecasting far easier.

Chris: The biggest advice I'd offer is to stay data-focused. Many rationales rely on news reports, but those invite gut reactions rather than analysis. In the Trump approval rating question, I concentrated on polling trends instead of headlines. Confidence calibration also matters; I sometimes held the right view but underweighted it, which hurt my Brier score. Accuracy is not just about being correct; it's about how sure you are of that correctness.

Were there particular sites or data aggregators you found helpful?

Joseph: Each question started with a fresh dataset. Most of the data wasn't proprietary, but it was convenient for organizing time series from sources like the Federal Reserve Bank. I mainly used CEIC because I was familiar with it and it streamlined data retrieval for ongoing questions.

Chris: I did not use specific aggregators, but I did use RFI-specified datasets for resolution and added my own public sources, including NOAA and FAA data for topics like drones.

How did you handle the diversity of questions throughout the training period?

Joseph: I enjoyed working outside my domain—I learned about topics such as EV batteries in the EU and hurricane patterns. Each question felt independent, which made it interesting even if it limited time savings from shared context. Overall, I gained broad exposure to multiple fields and know where to start if I need to research similar topics in the future.

Chris: I found the RFI interface robust and easy to use, with a great range of questions and data. I managed bandwidth by sorting forecasts by end date and tackling those closing soonest. The group structure was helpful—it made the workload manageable and created a trusted community to exchange insights. While the average forecast update could be hit-or-miss, having colleagues whose predictions and rationales I valued saved time and improved the process.

To what extent did the mentorship, seminars, and videos aid your forecasting process?

Joseph: The training was helpful for understanding the platform mechanics and how scoring works. It had been a while since I'd read Tetlock, so the refresher helped ground me in the fundamentals. To improve it further, I think there could have been more guidance on different question formats. Much of the training focused on binary questions, but forecasting has evolved to include multi-option problems. Many of us struggled with those, especially five-option questions, because it wasn't clear whether they should be treated as multiple forecasts.

Chris: The RFI mentorship/training helped me structure forecasting responses. I appreciated how the training formalized natural analytic habits and energized my participation.

How do you think forecasting could best support ongoing research efforts?

Joseph: Forecasting's value depends on the project and RAND's role—whether advisory, historical, or deep research. I haven't yet used RFI forecasts directly but have applied similar methods in backcasting studies. Techniques like question decomposition help sharpen research design and reveal underlying assumptions. Even if probabilities aren't perfect, the process strengthens analytic rigor. Three-to-twelve-month time horizons are useful for tactical and operational decisions, especially in defense or intelligence contexts.

Chris: Forecasting is a powerful tool, but questions about extremely low probability events often yield little practical value. Instead of asking "Will there be a drone strike?" it's more useful to model quantitative trends—like, "How many drones will be registered in the U.S.?" The FAA's monthly data makes that kind of question valuable for spotting patterns that RAND projects can use. RFI's shorter timelines are a strength, and questions should lean into that, targeting midrange probabilities rather than binary near-zero ones. The Venezuela—Guyana question, for example, offered limited insights because the likelihood was so low. Rolling questions, on the other hand, work well as analytic seismographs—capturing spikes or shifts over time—which could make forecasting even more actionable for ongoing research.