Thursday, November 6, 2025
11:00 - 12:00
Fitting models to data is an important part of the practice of science. Ongoing advances in machine learning have made it possible to fit more — and more complex — models, but have also exacerbated a problem: when multiple models fit the data equally well, which one(s) should we pick? The answer depends entirely on the modelling goal. In the scientific context, the essential goal is replicability: if a model works well to describe one experiment, it should continue to do so when that experiment is replicated tomorrow, or in another laboratory. The selection criterion must therefore be robust to the variations inherent to the replication process. In this work we develop a nonparametric method for estimating uncertainty on a model’s empirical risk when replications are non-stationary, thus ensuring that a model is only rejected when another is reproducibly better.
I will present a solution to this problem which comes in two parts. First I provide an ontological argument, which reframes the model selection problem such as to account for uncertainty in replications. I will provide a brief survey of the most common statistical methods and explain how they fall short in this context. In the second part, I then use some linearity assumptions on the quantile functions of the loss, to construct a practical computation procedure for the desired uncertainty. Along the way, I will discuss the pecularities of defining stochastic processes over quantile functions, and how they led us to propose the “hierarchical beta process”.
I will illustrate the method in two settings. One is a pedagogical example comparing the Rayleigh-Jeans and Planck models for the spectral radiance. The other is closer to the machine learning setting, comparing different model candidates for biological neurons which differ only in the value of their parameters.
Dutch Institute for Emergent Phenomena (DIEP)
IAS second floor library room
Group Seminar
biophysics, complexity, computational physics, emergence, soft matter
Alexandre René