Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge

Document Type

Article

Publication Date

7-8-2025

Publication Title

Nat Commun

Keywords

Humans, Benchmarking, Algorithms, Brain Neoplasms, Image Processing, Computer-Assisted, Artificial Intelligence, Magnetic Resonance Imaging

Abstract

Computational competitions are the standard for benchmarking medical image analysis algorithms, but they typically use small curated test datasets acquired at a few centers, leaving a gap to the reality of diverse multicentric patient data. To this end, the Federated Tumor Segmentation (FeTS) Challenge represents the paradigm for real-world algorithmic performance evaluation. The FeTS challenge is a competition to benchmark (i) federated learning aggregation algorithms and (ii) state-of-the-art segmentation algorithms, across multiple international sites. Weight aggregation and client selection techniques were compared using a multicentric brain tumor dataset in realistic federated learning simulations, yielding benefits for adaptive weight aggregation, and efficiency gains through client sampling. Quantitative performance evaluation of state-of-the-art segmentation algorithms on data distributed internationally across 32 institutions yielded good generalization on average, albeit the worst-case performance revealed data-specific modes of failure. Similar multi-site setups can help validate the real-world utility of healthcare AI algorithms in the future.

Medical Subject Headings

Humans; Benchmarking; Algorithms; Brain Neoplasms; Image Processing; Computer-Assisted; Artificial Intelligence; Magnetic Resonance Imaging

PubMed ID

40628696

Volume

16

Issue

1

First Page

6274

Last Page

6274

Find It @ Sladen

Share

COinS