COMP90051 · Statistical Machine Learning · The University of Melbourne
Ten arms, one log, and feedback that arrives late, or never.
A multi-armed bandit picks one of several options each round and only learns how that one did. For a 2023 project I implemented bandits that cope with delayed and lost feedback, plus two kinds of Thompson sampling, and scored them by replaying 300,000 recorded clicks. Bandit Lab rebuilds that work in the browser: the same algorithms, ported line for line, racing on a log you can reshape.
Watch the tour: three captioned walkthroughs and screenshots of every feature- .110
- .291
- .512
- .133
- .184
- .265
- .016
- .387
- .308
- .069
The brief, paraphrased
Recommend the next news story, without ever seeing the counterfactual.
The project framed bandits as a news recommender: show one of ten stories, count a click as reward. It came with a log in which a uniformly random policy chose the story, so any algorithm can be scored offline by keeping only the rounds where it agrees with the log (Li et al., 2011). Each algorithm ran for 20,000 matched rounds.
- 01
Feedback that arrives late, or never
Adapt replay evaluation so a click can be revealed rounds after the choice that earned it, or be lost entirely. Then implement three successive-elimination bandits from Lancewicki et al. (ICML 2021) and test them under heavy-tailed, lossy and reward-dependent delays.
- 02
Thompson sampling
Give every arm a Gaussian prior over its mean reward, update it in closed form after each observed click, and play the arm whose posterior draw is largest.
- 03
Adaptive inference
Implement Doubly-Adaptive Thompson Sampling (Dimakopoulou, Ren & Zhou, NeurIPS 2021): Thompson sampling on doubly-robust, variance-stabilised estimates with arm elimination.
What the notebook printed
Key results, reproduced to the last digit
Average reward over 20,000 matched rounds on the course log (one run, seed 90051). For scale, always playing the best arm earns 0.509 and picking at random about 0.223.
- SETask 1.2
0.38145
Successive Elimination
75% of the best arm · Pareto delays (arms 2 & 5 heavy-tailed)
- PSETask 1.3
0.30405
Phased Successive Elimination
60% of the best arm · Packet loss (90% on arm 2)
- OPSETask 1.4
0.2184
Optimistic-Pessimistic SE
43% of the best arm · Reward-dependent (1,000-round lag)
- TSTask 2
0.4218
Thompson Sampling
83% of the best arm · No delay
top - DATSTask 3
0.24495
Doubly-Adaptive Thompson Sampling
48% of the best arm · No delay
Added in the 2026 upgrade
The same results, measured properly, and an LLM put to the test
Every result with an interval
Each algorithm replayed on 20 seeded repetitions, with bootstrap bands, paired comparisons against Thompson sampling, effect sizes and delay-sensitivity heatmaps.
See the uncertaintyAn LLM as the policy
Bring your own key and let a language model choose the arms, scored against Thompson sampling and UCB1 on the same seeds. Every call is labelled and written to an audit log you can export.
Run the LLM experimentMethods on the record
Data provenance, assumptions and limitations, a model card for the evaluation set-up, and decision records that report the numbers that did not go my way.
Read the methods
How the scoring works
Replay a random log; keep only the agreements
Because the logging policy was uniformly random, the rounds where an algorithm happens to agree with the log are an unbiased sample of what it would have experienced live. Delays add a queue: the logged click is scheduled for a later round, and only counts if the algorithm picks that arm again when it arrives.
- 1
Read
Take the next logged event: which arm was shown, and whether it was clicked.
- 2
Schedule
Draw a delay for that event and put its click in the queue for a future round (or drop it if lost).
- 3
Play
Ask the bandit for an arm. It sees only the feedback that has arrived so far.
- 4
Match
If a click due this round belongs to the arm it chose, the bandit learns from it and the round counts.
In the lab
Everything the notebook plotted, plus what it couldn't show
A live race
Up to seven algorithms replay the same log side by side; reward and regret curves grow as the worker streams results.
Delays you can shape
Per-arm Pareto tails, packet loss and reward-dependent lags, or the exact 2023 setting for each algorithm.
Inside each algorithm
Scrub through confidence bounds, eliminations, posterior ridges and DATS propensities, snapshot by snapshot.
Where the pulls went
Arm-by-time heatmaps show whether an algorithm locked onto the best arm, and how fast.
Digit-for-digit parity
numpy's PCG64 generator, ziggurat normals and pairwise sums are ported exactly, so the TypeScript reproduces the notebook's printed numbers.
Nothing on a server
Every simulation runs in a Web Worker in your browser, with a 347-byte WebAssembly core for the random numbers.
About this project
A 2023 coursework project, revived in 2026
The original was an individual Jupyter notebook marked on code. This site keeps its algorithms exactly as submitted, quirks included, and adds the interactive tooling it never had. The original notebook is preserved in the repository for reference.
View the repository- Subject
- COMP90051 Statistical Machine Learning
- University
- The University of Melbourne
- Teaching period
- 2023, Semester 1 · Project 2
- Team
- Sunchuangyu “Rin” Huang (individual project)
- Credits
- Delay distributions, skeleton and the logged dataset were supplied by the COMP90051 teaching team and are not redistributed here. Algorithms follow Lancewicki et al. (2021), Agrawal & Goyal (2012) and Dimakopoulou, Ren & Zhou (2021).
- Original stack
- Python 3, NumPy, SciPy, Matplotlib, Jupyter
- Revived stack
- Next.js 16, React 19, TypeScript, Tailwind CSS 4, shadcn/ui on Base UI, Web Workers, WebAssembly, Vitest
- Academic integrity
- Shared as a record of completed work. Please don't reuse it in your own assessment.