Research Internship Proposal

Keywords: Contextual Multi-armed bandits, Group-sequential, Geospatial statistics.

Investigator: The project is proposed and advised by Odalric-Ambrym Maillard from Inria team-project Scool.

Place: This project will be primarily held at the research center Inria Lille -- Nord Europe, 40 avenue Halley, 59650 Villeneuve d'Ascq, France.

Bridging Experimental Science and Adaptive Decision-Making

Traditional experimental methodologies in medicine and agriculture often rely on long-term monitoring to assess treatment effectiveness. This makes purely sequential experimental designs impractical due to the extended time required before making informed decisions.

Group-sequential methodologies \cite{BartroffLai} address this limitation by allowing interim analyses on cohorts treated in parallel. While both hypothesis testing and bandit-based approaches have been explored within this framework, fundamental optimality properties remain underdeveloped.

This research program aims to extend the scope of group-sequential decision-making by incorporating reinforcement learning (RL) and bandit theory. Specifically, we will explore how contextual information, geospatial correlations, and adversarial robustness can be integrated into group-sequential adaptive sampling strategies. The integration of group-sequential methodologies with reinforcement learning opens exciting new research directions at the intersection of experimental design, adaptive decision-making, and geostatistics. By exploring contextual RL, adversarial robustness, structured bandits, and geospatial optimization, this program aims to develop novel methodologies with applications across medicine, agriculture, and environmental science.

Part I: Contextual Bandits and MDP-Based Reinforcement Learning for Group-Sequential Decision-Making

One of the key advancements in modern bandit theory is the use of linear contextual structures, allowing external covariates (e.g., patient age, weather conditions) to inform decision-making. This motivates a reconsideration of group-sequential methodologies through the lens of contextual bandits. By leveraging this framework, we can design more efficient adaptive experiments where each batch of experiments contributes directly to improving future decisions.

Beyond contextual bandits, a natural extension is to incorporate MDP-based RL algorithms into the group-sequential paradigm. This shift enables decision-making over multiple stages, allowing for a more comprehensive adaptive learning process. Addressing these questions requires developing new methodological tools to balance exploration and exploitation within structured group-sequential setups.

Part II: Hypothesis Testing and Experiment Design in Group-Sequential Bandits

An important theoretical direction is the adaptation of hypothesis testing methodologies to the group-sequential bandit framework. Unlike purely sequential bandit strategies, which operate on a one-step-at-a-time basis, group-sequential approaches allow for batch experimentation at each decision step. Key open questions include:

How can state-of-the-art sequential bandit strategies be extended to group-sequential settings, particularly when batch sizes are fixed, evolving, or adaptively chosen?
What are the theoretical limits of regret minimization and statistical power in these setups?
How can structured bandits, which exploit problem-specific structure (e.g., graph-based or low-rank reward models), be adapted to the group-sequential setting?

In structured settings, the challenge is to rapidly identify the optimal action within specific contexts while minimizing the number of required trials. Exploiting correlations across experimental slots naturally reduces sample complexity, but the characterization of optimal performance guarantees remains an open problem

Part III: Geospatial Context and Group-Sequential Bandits

Many real-world applications involve decision-making in geographically distributed environments. Standard contextual bandit models often rely on linear regression or kernel methods, but in geostatistical settings, more advanced techniques such as Kriging, recently revisited SivieroKriging, offer improved interpolation capabilities including in a dependent context.

However, integrating geostatistical methods into bandit-based decision-making presents several challenges:

1. Sparse Observations: Unlike traditional regression settings, we often observe only a single realization of the spatio-temporal process rather than repeated samples.
2. Spatial Correlations: High correlation among nearby observations introduces dependence structures that standard bandit algorithms do not account for. Yet dedicate tools leveraging the variogram offer promising perspective.
3. Exploration-Exploitation in Geostatistical Bandits: Balancing information gain and sampling cost is crucial, particularly in fields such as hydrogeology (where optimal sensor placement affects map accuracy) and agroecology (where adaptive field experiments inform sustainable practices).

This program will explore how recent advances in geostatistics can be combined with multi-armed bandit theory to improve experimental design and decision-making in geospatial contexts.

Part IV: Robustness Against Adversarial Corruption in Group-Sequential Bandits

One fundamental challenge in real-world experiments is the presence of adversarial corruption where a subset of observations may be unreliable due to systematic biases, sensor failures, or external disruptions. For instance, in agricultural trials, sensor malfunctions could lead to incorrect soil moisture readings, or a pest outbreak in a subset of test fields might skew yield results. In clinical trials, certain patient subgroups might exhibit non-compliance with treatment protocols, leading to misleading outcomes. While corruption models have been studied as a variant of bandits Agrawal et al, 2024, Traditional sequential bandit models assume a single-arm selection per time step, limiting their ability to mitigate adversarial attacks.

A more robust alternative is to consider group-sequential bandits, where multiple experimental slots (e.g., multiple farms, patient cohorts) can be tested in parallel. This setting offers two major advantages:

1. Scalability: Many experimental sciences already operate in batch settings, making group-sequential designs more practical.
2. Resilience: By ensuring a batch size large enough relative to the corruption level, we can guarantee that enough uncorrupted observations remain to draw reliable conclusions.

Additionally, group-sequential bandits introduce new challenges related to group-correlated corruption. When experimental units (e.g., farms, clinical sites) are geographically proximate, external hazards (e.g., weather events, local epidemics) can simultaneously affect multiple slots. For instance, a prolonged drought might impact all test fields in a region, distorting yield comparisons, while a hospital-wide data logging error could corrupt multiple patient records. This motivates the need for diversity-enforcing strategies that avoid over-concentration on a single alternative within geographically correlated environments.

Bibliography

Bartroff & Lai, 2012 Bartroff, J., Lai, T. L., and Shih, M.-C. "Sequential experimentation in clinical trials: design and analysis". volume 298. Springer Science & Business Media. (2012).
Lattimore & Szepesvári, 2020 Lattimore, Tor, and Csaba Szepesvári. Bandit algorithms. Cambridge University Press (2020).
Siviero et al, 2024 Siviero, E., Chautru, E., and Clémençon, S. "A statistical learning view of simple kriging". TEST, 33(1):271–296. (2024).
Agrawal et al, 2024 Agrawal, S., Mathieu, T., Basu, D. and Maillard, O.A. "CRIMED: Lower and Upper Bounds on Regret for Bandits with Unbounded Stochastic Corruption". In International Conference on Algorithmic Learning Theory (pp. 74-124). PMLR (2024).