Mathematical modelling to inform Ontario’s COVID-19 response 🦠

# Mathematical modelling to inform Ontario’s COVID-19 response 🦠
## Successes, challenges, and lessons learned
### Irena Papst Cornell University

---

The slides for this talk are available at

[papsti.github.io/talks/2021-06-21_CAIMS.html](https:/papsti.github.io/talks/2021-06-21_CAIMS.html)

???
- in case you want to follow along (there are hyperlinks that may be of interest sprinkled throughout the talk)
- also in case i go too quickly and you want to flip back to a previously shown slide

---

I'm giving this talk from Mississauga, Ontario, Canada, and so I would like to begin by acknowledging that the land on which I'm speaking today is part of the **Treaty Lands and Territory of the Mississaugas of the Credit**.

For thousands of years, Indigenous peoples inhabited and cared for this land, and continue to do so today. In particular I acknowledge the territory of the **Anishinabek**, **Huron-Wendat**, **Haudenosaunee** and **Ojibway/Chippewa** peoples; the land that is home to the **Metis**; and most recently, the territory of the **Mississaugas of the Credit First Nation** who are direct descendants of the Mississaugas of the Credit.

I am grateful to have the opportunity to work on this land, and by doing so, give my respect to its first inhabitants.

???
- I want to start by taking a moment to acknowledge that today is National Indigenous People's Day.
- I'm giving this talk from Mississauga, Ontario, Canada and so I would like to begin by acknowledging that the land on which I'm speaking today is part of the Treaty Lands and Territory of the Mississaugas of the Credit. 
- For thousands of years, Indigenous peoples inhabited and cared for this land, and continue to do so today. In particular I acknowledge the territory of the Anishinabek (A-ni-SHE-na-bek), Huron-Wendat (HYOO-ron WUN-dat), Haudenosaunee (ho-den-uh-SHO-nee) and Ojibway/Chippewa (oh-JIB-way CHI-peh-wa) peoples; the land that is home to the Metis; and most recently, the territory of the Mississaugas of the Credit First Nation who are direct descendants of the Mississaugas of the Credit.
- I am grateful to have the opportunity to work on this land, and by doing so, give my respect to its first inhabitants.
- (Indigenous land acknowledgement statement adapted from the [Region of Peel's statement](https://www.peelregion.ca/council/indigenous.asp))

---

class: center
background-image: url("data:image/png;base64,#figs/mactheobio.png")
background-position: center
background-size: contain

modelling **by** the [McMaster University Theoretical Biology lab](https://mac-theobio.github.io)

???
- today i'll talk to you about some modelling done by the McMaster University Theoretical Biology Lab

---

class: center
background-image: url("data:image/png;base64,#figs/collaborators.png")
background-position: center
background-size: contain

modelling **by** the [McMaster University Theoretical Biology lab](https://mac-theobio.github.io)

???
- specifically, i've been working with these four wonderful people on modelling to support Ontario's COVID-19 response
- Michael Li currently works for the Public Health Agency of Canada, and Benjamin Bolker, Jonathan Dushoff, and David Earn are all professors at McMaster University

---

modelling **for**

- [Ontario Modelling Consensus Table](https://covid19-sciencetable.ca/our-partners/)
  
--

- [Ontario Science Advisory Table](https://covid19-sciencetable.ca) 
  
--

???
- but who exactly are we doing this modelling for?
- we provide our projections to the Ontario Modelling Consensus Table, which review our forecasts, as well as those from other independent groups, and they come to a consensus on the most likely set of projections
- the consensus forecasts are then passed on to the Ontario Science Advisory table, which advises to the Government of Ontario, and briefs the public, in press conferences you may have seen clips of on the news
- and i mention the public briefings because at the end of the day, we're really doing this for...

---

# the public

???
- this is a public service, and as i hope you'll see in this talk, i strongly believe we have an obligation to communicate clearly to the public and be forthcoming about how we make projections which may inform public policy
- to that end...

---

📈 **public forecast updates:**

[https://mac-theobio.github.io/covid-19](https://mac-theobio.github.io/covid-19/index.html)

???
- we've started publishing our forecasts online, where we show our work in more detail

---

[`McMasterPandemic`](https://github.com/bbolker/McMasterPandemic/graphs/contributors) R package

???
- the software we use to simulate the model and fit it to data is also available publicly as an R package published to Github

---

# model

???
- so that's the context
- now let's get into the model we're using

---

# epidemiology

???
- let's look at the basic epidemiological structure of our model
- it's a compartmental model, where we partition the population into different compartments based on their epidemiological status

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.001.png")
background-position: center
background-size: contain

# epidemiology

???
- everyone starts in the susceptible class

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.002.png")
background-position: center
background-size: contain

# epidemiology

???
- and then individuals can get exposed to (and therefore infected by) SARS-CoV-2, but they aren't infectious until they enter an infectious class
- for COVID-19 we have four infectious classes
- the first two chronologically are

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.003.png")
background-position: center
background-size: contain

# epidemiology

???
1. asymptomatic, where you don't show symptoms but can still infected others (albeit in a reduced capacity)
        
---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.004.png")
background-position: center
background-size: contain

# epidemiology

???
2. pre-symptomatic, where you will eventually develop symptoms

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.005.png")
background-position: center
background-size: contain

# epidemiology

???
- asymptomatic infections simply recover

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.006.png")
background-position: center
background-size: contain

# epidemiology

???
- while presymptomatic infections turn into either mild infections or
        
---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.007.png")
background-position: center
background-size: contain

# epidemiology

???
- or severe infection

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.008.png")
background-position: center
background-size: contain

# epidemiology

???
- mild infections recover and never result in hospitalization or death

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.009.png")
background-position: center
background-size: contain

# epidemiology

???
- while severe infections always result in either hospitalization

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.010.png")
background-position: center
background-size: contain

# epidemiology

???
or death before hospitalization

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.011.png")
background-position: center
background-size: contain

# epidemiology

???
- for hospitalized individuals, they may recover having only received acute care

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.012.png")
background-position: center
background-size: contain

# epidemiology

???
- or they may need ICU care, from which they can either

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.013.png")
background-position: center
background-size: contain

# epidemiology

???
- recover

---

class: right
background-image: url("data:image/png;base64,#figs/flowchart/flowchart.014.png")
background-position: center
background-size: contain

# epidemiology

???
- or they might die
- so that's the epidemiological structure
- each of these arrows represents a flow rate specified with various parameters
- we take this model, and fit (or "calibrate") its parameters to data

---

# calibration

fitting `\(\beta(t)\)`, the **time-varying** transmission rate

`\(\beta(t)\)` used to calculate number of **new infections** per day
 
(not observable)

use daily **infection reports** as a proxy
 
(delayed, underestimate)

???
- in calibration, one of the parameters we consistently fit is a time-varying transmission rate
- the transmission rate is used to calculate the number of new infections per day, which we cannot observe
- instead, as a proxy for the number of infections, we use daily infection reports
- here i'm showing you the first part of the ontario epidemic, starting from february 2020 and into late november
- infection reports are delayed by about 1-2 weeks from when infections actually occur, and they underestimate the true number of infections, so we adjust for both of these issues

---

# successes

???
- in the interest of time, i'm only going to cover one successful forecast, but you can see the accuracy of our other forecasts in our public blog posts, where we continue to update the forecast plots with the latest infection reports

---

### steady increase through the fall

???
- let me take you back to late fall in the province of Ontario
- we had been seeing a slow and steady rise of infection reports AND hospitalizations/ICU admissions through the fall

---

???
- 23 nov: lockdown in toronto/peel, two of the most populous regions in the province (indoor gatherings prohibited), in response to the steadily increasing infection reports

---

### local lockdown did not stop growth

???
- infection reports continued to rise

---

???
- 26 dec: province-wide lockdown
  
---

### province-wide lockdown did not stop growth

???
- two weeks later, which is approximately what it takes for interventions to start having an effect on infection reports, the picture still didn't look good, and so about three weeks after the province-wide lockdown...

---

???
- 14 jan: provincial stay-at-home order issued (first for COVID-19 in the province)

---

### strong decline following stay-at-home order

???
- and it worked really well!! cases declined nicely, and hospitalizations/ICU occupancy began to fall, albeit more slowly

---

???
- it was at this point that we were preparing our routine forecasts for the modelling table
---

# is it safe to start reopening?

???
- the big question was, "is it safe to reopening schools, retail spaces, indoor dining, etc.?"
- infection reports have decreased dramatically, we have a pretty low number of cases... seems like a good time to start opening things back up!

---

???
- so here is what the data looked like (solid points) when the forecast was made

---

???
- and here was our model fit up to the forecast date
- the line is the maximum likelihood estimate ("best fit") and the ribbon is the 95% confidence interval around that fit

---

???
- we assumed the province would start reopening in early March, which we incorporated into our projections as an increase to the effective transmission rate
- we explored two scenarios for the forecast, but this is the one we flagged as more likely

---

### "reopening is not safe right now"

???
- and our message was "no, reopening is not safe right now", could cause third wave
- so what actually happened?

---

???
- the province actually started reopening on 8 march, coincidentally on the same date we chose as the reopening date in our forecasts
- and here's what happened with infection reports

---

### reopening was not safe then

???
- they started taking off, as predicted by our forecast, despite the relatively low number of infection reports at the time...
- reopening indeed was not safe at the time
- so what happened? why were we so convinced that this takeoff was actually likely given the amount of spread we were seeing

---

class: center
background-image: url("data:image/png;base64,#figs/alpha/alpha.001.png")
background-position: center
background-size: contain

???
- let's take a closer look at infection reports up to that forecast date, because there was something brewing below the surface
- this plot courtesy of the ontario science advisory table dashboard shows infection reports in bars and a seven-day average curve overtop

---

class: center
background-image: url("data:image/png;base64,#figs/alpha/alpha.002.png")
background-position: center
background-size: contain

???
- here is the date of the forecast we were just considering

---

class: center
background-image: url("data:image/png;base64,#figs/alpha/alpha.003.png")
background-position: center
background-size: contain

???
- and here are the interventions in response to the the steady growth in infection reports through the fall
- it turns out that another key event happened the day of the province-wide lockdown, purely coincidentally

---

class: center
background-image: url("data:image/png;base64,#figs/alpha/alpha.004.png")
background-position: center
background-size: contain

### Alpha variant of concern
(B.1.1.7)

**50% more transmissible** than previous strains

???
- the Alpha variant of concern was first confirmed in the province
- previously referred to as B.1.1.7, this variant was first detected in the UK
- crucially, it is 50% more transmissible than strains we were previously seeing in the province

---

class: center
background-image: url("data:image/png;base64,#figs/alpha/alpha.005.png")
background-position: center
background-size: contain

### Alpha variant of concern
(B.1.1.7)

**50% more transmissible** than previous strains

???
- the orange curve gives a seven-day average of infection reports attributed to Alpha
- it may not look like much is happening with Alpha at first, but keep in mind: infection reports were falling quickly thanks to the public health measures we were adhering to, and yet Alpha was managing to find enough of a foothold to defy the pressure to decline and was actually managing to increase over the stay-at-home period

---

class: center
background-image: url("data:image/png;base64,#figs/alpha/alpha.005.png")
background-position: center
background-size: contain

### Alpha took over previous strains 
### and drove the third wave

???
- so the public health interventions were working very well on the old strains, but not well enough on Alpha
- so Alpha took over the previous strains and drove the third wave
- want to point out that it wasn't as if there were a lot of Alpha infection reports at the time of forecast... it wasn't the number of them in ontario that made us pay attention

---

benefit of **analysis from the UK**

???
- we were really paying attention to Alpha in Ontario thanks to 
  1. seeing what was happening with Alpha quickly taking over old strains in the UK
- you can see in the lead up to the forecast date, the UK had a much larger surge---this is infection reports normalized per 10,000 population in each location, except that it's on a log scale
- the UK already had the same number of per capita infection reports as we did at our peak in january about a month before that, in early dec

---

benefit of **analysis from the UK**

early **Public Health Ontario** surveillance

???
- we also benefited from early surveillence done by public health ontario
- the first mass screening for this variant was performed on 20 jan, a month ahead of our forecast, which gave us a good amount of variant data to work with
- we were really lucky to have both of these pieces of the puzzle

- so, going back to the forecast, recall that the motivating question was:

---

# is it safe to start reopening?

???
- we knew we'd need to factor in Alpha to ensure our forecasts for reopening were accurate
- so we took what we knew from the UK and public health ontario, as well as evolutionary epidemiology theory on the competition between two strains of the same pathogen and

---

### share of infection reports due to Alpha (model)

<img src="data:image/png;base64,#2021-06-21_CAIMS_files/figure-html/alpha-prop-1.png" width="936" style="display: block; margin: auto;" />
???
- calibrated the share of infection reports due to Alpha over time
- we then feed this percentage into our model and adjust the transmission rate based on the increased transmissibility of Alpha, explicitly accounting for the variant's effect on trasmission

---

???
- and that's what led to this successful forecast

---

???
- what if we hadn't explicitly modelled the Alpha takeover in our projections? what if we just let the signal from infection reports calibrate the transmission rate, as usual?

---

### estimating Alpha's effect purely from infection reports failed to predict the take-off

???
- well we actually did that forecast at the same time, just to have a basis for comparison
- we let compute whatever effect of Alpha already exists in infection reports and simply propagate that forward without any further increases to Alpha
- without this explicit Alpha layer, our model would have completely missed that we were actually in a precarious position with reopening

- success in terms of producing a reliable forecast, but it was also a failure in that this forecast (along with other modelling and evidence) was not convincing enough for the province to reconsider reopening at that moment in time.

---

# challenges

???
- so i promised to tell you about some of the challenges we've been facing over this process as well

---

## real-time model development

adding **variants of concern**
 
& **vaccination**

**documenting** the work
  
???
- one of the biggest challenges has been the pace, trying to keep up with the situation on the ground and developing the model in real time 
- here is a snapshot of the ["intervention scan"](https://www.cihi.ca/en/covid-19-intervention-timeline-in-canada) compiled by CIHI, which painstakingly compiled the timing and detail of case management decisions, closures/openings, distancing measures, healthcare changes, state of emergency declarations, public info, vaccination effort for provinces across the country
- here we have the current state of the scan for ontario (which goes to mid-march), and we can see the massive number of changes juxtaposed with the number of infection reports over time 
- as modellers, we always have to make choices of what are mechanisms that must be included in the model given the questions we're trying to answer with our forecast, and what is unnecessary detail... what do we include? what do we ignore?
- adding new mechanisms (like variants of concern and two-dose vaccination)... adding them on the fly, but still carefully
- testing the code sufficiently, given the little time you have, fixing any bugs you found
- documenting the work so that you can explain it, defend it, and ultimately build on it

---

## communicating the work

conveying model results & implications for policy
 
**clearly** to the public

"this **rocketship** forecast is ridiculous"
 
"this would never happen **here**"

**"all else equal"**
  
???
- another major challenge we've faced is in communicating the work
- conveying both model results and their implications for public health policy clearly to the public, and even in some cases to those advising the government

- it turns out that people don't really like projections of infection report takeoff
- problem is, this absolutely can happen here, and in fact has happened on a smaller scale...
- this forecast curve continues growing, because at this point we still had such a large susceptible pool, thanks to all the interventions that kept spread relatively low up to this point

- however, people misunderstand that these forecasts are made under an "all else equal" assumption
  - everything else remains fixed outside of the scenarios we're modelling
  - a "rocketship" takeoff would not go unchecked... our governments and public health authorities *would* react
  - that doesn't mean the forecast is any less valid, it's just showing what would happen, under the given assumptions, if there are no further changes going forward
- when the forecast is taken out of context of its assumptions, it is easily misunderstood
---

## communicating the work

what does *"a 40% decrease in transmission"* mean
 
to someone's **lived experience**?

what is driving changes in transmission rate?
 
(**contact rate, weather, variants**, ...)
 
???

- here is an example of a forecast that was particularly difficult to explain, that highlights a challenge we keep coming up against
- for context, in this forecast, the 8 april stay at home order had come into effect 11 days before the forecast date
- it was too early to estimate the effect of the stay at home order (we need 2-3 weeks worth of infection reports to estimate its effect with any certainty)
- instead we created projections based on scenarios for what effect the stay at home order *could* have, represented by the different colours in this plot
- but how do you translate these scenarios to someone's lived experience? what does "a 40% reduction in transmission" mean for someone's every day life?
- it would have been great if we could have said something like "if we do what we did in january, we will follow the green curve"
- but it's really difficult to make a statement like that with confidence because it's hard to tease apart what's actually driving changes in the transmission rate... could be changes in the contact rate, weather making the virus transmit more or less easily because of the temperature or humidity, more transmissible variants arising
  
---

# lessons learned

---

{{content}}
]

**redundancy** is important

independent **code walk-throughs** by multiple people
{{content}}

making sure several people can step in and **run fit & forecast**
{{content}}

set up automated **pipelines**
{{content}}

**document** any manual steps
{{content}}

**version control**

{{content}}
]

shorten **the gap**

make graphics as **self-contained** as possible

title plots with the **punchline**

treat forecasts more as **qualitative** than quantitative

but **don't be opaque**! simplify up front but make sure detail is **accessible**

be open about your model's **limitations**, as well as your own

???
- automate the process as much as possible using pipelines (e.g. a script or a Make rule that bring together all the pieces going from the initial inputs to the final forecast---viz and all)
- but inevitable there will be manual steps and it's important to document them well too
- for communicating the work, we're thinking specifically about communicating to the public or higher-level policy-makers. there is some overlap with scientific communication, but a scientific audience will generally buy in more readily to your work, and so you can take the time to be a bit more nuanced. 
- the question you should ask yourself is: how much time will the audience be willing to spend on digesting the work? does this align with the time it would take to cross the gap between what they know and what you're trying to convey?
- it's easy to perceive accuracy where there is _precision_
- it's maybe better to refer to low/medium/high transmission scenarios as opposed to the percise percentage increase, which is hard to interpret
- but don't be opaque! the public deserves **transparency** and **good communication**  
- don't put all the detail up front, but make sure it can be found easily (*e.g.* our [blog posts](https://mac-theobio.github.io/forecasts/outputs/McMasterOntarioForecastsBlog2021-06-05))
- don't overstep your own expertise

---

# [`McMasterPandemic`](https://github.com/bbolker/McMasterPandemic/graphs/contributors) team

Benjamin Bolker

Jonathan Dushoff

David Earn

Morgan Kain

Zachary Levine

Michael Li

Matthew So

Steven Walker

???
- to wrap up, i'd like to thank the other members of the `McMasterPandemic` development team for all of their hard work

---

## thank you for listening!

✉️ [ip98@cornell.edu](mailto:ip98@cornell.edu)

🐦 [@irenapapst](https://twitter.com/irenapapst)

🌐 [papsti.github.io](https://papsti.github.io)

📈 **public forecast updates:**

[https://mac-theobio.github.io/covid-19](https://mac-theobio.github.io/covid-19/index.html)

???
- and thank you all for listening
- email me if you'd like to be notified when we post new forecasts at this website