New website, OpenChallenges.io, aggregates biomedical challenges to accelerate citizen science and data benchmarking

Developed by Sage Bionetworks, OpenChallenges.io seeks to become the single online destination for all available biomedical challenges.

SEATTLE, WA – Community-driven benchmarks have emerged as a vital way to identify, and ensure the reproducibility of, state-of-the-art artificial intelligence (AI) datasets and algorithms. The 2010 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) served as a benchmark for the development of neural networks for computer vision. Then, the Critical Assessment of Structure Prediction (CASP) competition anointed Google DeepMind’s revolutionary AlphaFold (and AlphaFold2) a leader in predicting the 3D structure of over 200 million proteins. And more recently, community-driven progress in AI has resulted in BIG-Bench to track the capabilities of Large Language Models (LLMs).

Often, these community benchmarks are organized around competitions or “challenges” to increase reach and diversity, where participants may even be incentivized with cash prizes. But until now, no centralized resource existed to find and access data & AI challenges. A new resource developed by Sage Bionetworks, OpenChallenges.io, provides researchers and citizen scientists alike with a singular, searchable catalog of challenges from across the web that address areas of significant unmet medical need.

Launched officially on November 7, 2023, OpenChallenges.io currently tracks and curates 281 challenges from 333 organizations that contribute as data providers, benchmarking communities or sponsors. These challenges invite participants to submit machine learning models that can address bottlenecks in translational research, such as detecting tumors from brain scans, predicting an individual’s heart disease risk, or even identifying the origins of rare, inherited diseases.

“We’ve organized close to 50 challenges at Sage Bionetworks, but we always struggled to find out if anyone else was doing anything similar, anywhere in the world,” says Luca Foschini, PhD, President and CEO of Sage Bionetworks. “We hope this will become the ‘Google’ of challenge-based benchmarking and improve access for everyone who wishes to contribute intelligent solutions to society’s biggest medical problems.”

The platform provides users with weekly updates about upcoming and existing challenges, all of which can be stratified by data submission type, partnering platform and potential financial incentive. It also provides organizers with a standardized template for hosting new challenge events. Once the events have passed, the challenge benchmarks remain accessible in the catalog, becoming a source of open data for other community members studying similar biomedical problems. Members can gather, chat and share resources via the OpenChallenges Discord, which has additional channels for suggesting new features or challenges to add.

The new resource builds upon Sage Bionetworks’ existing presence in the benchmarking community as a platform host for several biomedical challenges. This includes the notable Digital Mammography DREAM Challenge, which crowd-sourced the development of an algorithm that could radically improve breast cancer detection. This later resulted in the FDA clearance of various AI tools for clinical use from multiple teams with the highest-scoring submissions. Participants from all walks of life compete in these challenges, with many high school students (sometimes incentivized with school credits) ranking among the top teams.

“OpenChallenges.io is just one piece of a bigger vision,” says Thomas Schaffter, PhD, Principal Research Software Engineer at Sage Bionetworks and lead architect for the new website. “Our goal is to continue to connect life scientists, data researchers and expert volunteers into a highly distributed network of collaborators, where ideas from one community are amplified by others, eventually leading to life-changing technologies.”

The OpenChallenges.io team hopes that by removing the digital barriers between competition hosts and public data enthusiasts, more groundbreaking AI systems will soon emerge to transform the landscape of diagnostics, therapeutics and healthcare equity.

“We are always looking for new challenges to add to the catalog,” adds Foschini. “We would love to hear from anyone currently running a challenge who does not see it listed in OpenChallenges.io.”

Explore the website at OpenChallenges.io

In addition to Thomas Schaffter, other contributors to the website include Verena Chung, Rong Chai, Maria Diaz, Gaia Andreoletti and Jake Albrecht.

OpenChallenges.io is supported by the National Cancer Institute’s (NCI) Informatics Technology for Cancer Research (ITCR), part of the National Institutes of Health, under grant number U24CA248265.