Evaluation of the 2022 West Nile virus forecasting challenge, United States

Authors: HARP, RYAN - Centers For Disease Control And Prevention (CDC) - United States; Humphreys Jr, John; Cohnstaedt, Lee; SCOGLIO, CATERINA - Kansas State University; HOSSEINI, SAMAN - Kansas State University; HOLCOMB, KAREN - Centers For Disease Control And Prevention (CDC) - United States; RETKUTE, RENATA - Cambridge University; PRUSOKIENE, ALISA - Newcastle University; PRUSOKAS, AUGUSTINAS - University Of London; ERTEM, ZEYNEP - Binghamton University; AJELLI, MARCO - Indiana University; KUMMER, ALLISANDRA - Indiana University; LITVINOVA, MARIA - Indiana University; MERLER, STEFANO - Bruno Kessler Foundation; PIONTTI, ANA PASTORE Y - Northeastern University; POLETTI, PIERO - Bruno Kessler Foundation; VESPIGNANI, ALESSANDRO - Northeastern University; WILKE, ANDRE - Indiana University; ZARDINI, AGNESE - Bruno Kessler Foundation; SMITH, KELLY HELM - University Of Nebraska; ARMSTRONG, PHILIP - Connecticut Agricultural Experiment Station; DEFELICE, NICHOLAS - The Icahn School Of Medicine At Mount Sinai; KEYEL, ALEXANDER - New York State Department Of Health; SHEPHERD, JOHN - Connecticut Agricultural Experiment Station; SMITH, REBECCA - University Of Illinois; TYRE, ANDREW - Bayer Biosciences; GORRIS, MORGAN - Los Alamos National Research Laboratory; BARNARD, MARTHA - Los Alamos National Research Laboratory; MOSER, S. KANE - Los Alamos National Research Laboratory; SPENCER, JULIE - Los Alamos National Research Laboratory; MCCARTER, MAGGIE - University Of South Carolina; LEE, CHRISTOPHER - University Of South Carolina; NOLAN, MELISSA - University Of South Carolina; STAPLES, ERIN - Centers For Disease Control And Prevention (CDC) - United States; NETT, RANDALL - Centers For Disease Control And Prevention (CDC) - United States; JOHANSSON, MICHAEL - Centers For Disease Control And Prevention (CDC) - United States

Submitted to: Parasites & Vectors
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 3/24/2025
Publication Date: 4/23/2025
Citation: Harp, R.D., Humphreys Jr, J.M., Cohnstaedt, L.W., Scoglio, C., Hosseini, S., Holcomb, K., Retkute, R., Prusokiene, A., Prusokas, A., Ertem, Z., Ajelli, M., Kummer, A.G., Litvinova, M., Merler, S., Piontti, A., Poletti, P., Vespignani, A., Wilke, A.B., Zardini, A., Smith, K., Armstrong, P., Defelice, N., Keyel, A., Shepherd, J., Smith, R., Tyre, A., Gorris, M., Barnard, M., Moser, S., Spencer, J., Mccarter, M.S., Lee, C., Nolan, M.S., Staples, E., Nett, R., Johansson, M. 2025. Evaluation of the 2022 West Nile virus forecasting challenge, United States. Parasites & Vectors. https://doi.org/10.1186/s13071-025-06767-2.
DOI: https://doi.org/10.1186/s13071-025-06767-2

Interpretive Summary: West Nile virus (WNV) is the most common mosquito-borne disease in the continental U.S., with around 1,200 severe cases reported annually from 2005 to 2021. Despite this, forecasting WNV outbreaks to aid public health has been difficult. This study analyzed forecasts from the 2022 WNV Forecasting Challenge, where teams predicted the number of severe WNV cases for each county. The estimates were evaluated for accuracy using scoring methods and examined for how different approaches and local factors influenced results. The combined forecasts (ensemble) accurately predicted that most counties would have low to average WNV cases. However, some counties had higher than-expected numbers, and the national total was below the long-term average. The ensemble forecast was the most accurate, but some individual models performed nearly as well. Models using statistical methods were generally more accurate than those incorporating climate, mosquito, demographic, or bird data. Counties with a longer history of WNV cases and more variable past outbreaks were easier to predict. The results suggest progress in WNV forecasting since 2020. However, there is still room to improve how additional data is used, and future research should focus on better matching forecasting methods with specific goals across different locations and times.

Technical Abstract: West Nile virus (WNV) is the most common mosquito-borne disease in the continental U.S., with around 1,200 severe cases reported annually from 2005 to 2021. Despite this, forecasting WNV outbreaks to aid public health has been difficult. This study analyzed forecasts from the 2022 WNV Forecasting Challenge, where teams predicted the number of severe WNV cases for each county. The estimates were evaluated for accuracy using scoring methods and examined for how different approaches and local factors influenced results. The combined forecasts (ensemble) accurately predicted that most counties would have low to average WNV cases. However, some counties had higher than-expected numbers, and the national total was below the long-term average. The ensemble forecast was the most accurate, but some individual models performed nearly as well. Models using statistical methods were generally more accurate than those incorporating climate, mosquito, demographic, or bird data. Counties with a longer history of WNV cases and more variable past outbreaks were easier to predict. The results suggest progress in WNV forecasting since 2020. However, there is still room to improve how additional data is used, and future research should focus on better matching forecasting methods with specific goals across different locations and times. Background: West Nile virus (WNV) is the most common mosquito-borne disease in the continental U.S. with an average of ~1,200 severe, neuroinvasive cases reported annually from 2005-2021. Despite this burden, efforts to forecast WNV to inform public health measures to reduce disease incidence have had limited success. Here, we analyze forecasts submitted to the 2022 WNV Forecasting Challenge, a follow-up to the 2020 WNV Forecasting Challenge. Methods: Forecasting teams submitted probabilistic forecasts of West Nile virus neuroinvasive disease (WNND) at the annual-county level for the 2022 WNV season. We used two proper scoring metrics to assess skill of team-specific forecasts, baseline forecasts, and an ensemble created from team-specific forecasts. We then characterized the impact of model characteristics (i.e., model frameworks, data inputs) and county-specific contextual factors (i.e., environmental, demographic, historical WNV factors) on forecast skill. Results: The ensemble forecasts for 2022 anticipated a season at or below median long-term WNND incidence for most counties. More counties than anticipated reported relatively high case numbers, but national incidence was well below the long-term median. The ensemble forecast was the most skillful forecast, though the historical negative binomial baseline model and several team-submitted forecasts had similar forecast skill. Forecasts based on regression-based frameworks tended to have more skill than those that did not and models using climate, mosquito surveillance, demographic, or avian data had less skill than those that did not. County-contextual analysis showed strong relationships with the number of years that WNND had been reported and permutation entropy (historical variability). Weighted interval score and logarithmic scoring metrics produced similar results. Conclusions: The relative success of the ensemble forecast, the best forecast for 2022, suggests potential gains in community ability to forecast WNV, an improvement from the 2020 Challenge. Similarly to the previous challenge, regression-based forecasts tended to perform better and forecasts incorporating additional data tended to perform worse, indicating that there are still opportunities to improve the incorporation of mechanistic approaches and the integration of relevant data. The findings also elucidate other potential avenues for future study improvement like matching spatiotemporal scale with research goals.