\documentclass[shortnames,nojss]{jss}
\usepackage{thumbpdf}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amscd}
\usepackage{epsfig}

\newcommand{\bm}[1]{\mbox{\boldmath $#1$}}
\newcommand{\mb}[1]{\mathbf{#1}}

%\VignetteIndexEntry{amei R package tutorial}
%\VignetteKeywords{amei}
%\VignetteDepends{amei}
%\VignettePackage{amei}
%% need no \usepackage{Sweave.sty}

%\doublespacing

\setkeys{Gin}{width=0.85\textwidth}


\title{\pkg{amei}: an \proglang{R} package for the Adaptive\\ 
  Management of Epidemiological Interventions}
\author{Daniel Merl\\
  Lawrence Livermore National Lab
 \And Leah R.~Johnson\\
 University of California, Santa Barbara
  \AND Robert B.~Gramacy\\
  University of Cambridge
  \And Marc Mangel\\
 University of California, Santa Cruz}
%\maketitle

\Plainauthor{Daniel Merl, Leah R.Johnson, Robert B. Gramacy and Marc Mangel}
\Plaintitle{amei: An R package for the Adaptive Management of 
  Epidemiological Interventions}
\Shorttitle{\pkg{amei}: An \proglang{R} package for the Adaptive
  Management of Epidemiological Interventions}

\Abstract{The \pkg{amei} package for \proglang{R} is a tool that
  provides a flexible statistical framework for generating optimal
  epidemiological interventions that are designed to minimize the
  total expected cost of an emerging epidemic. Uncertainty regarding
  the underlying disease parameters is propagated through to the
  decision process via Bayesian posterior inference.  The strategies
  produced through this framework are adaptive: vaccination schedules
  are iteratively adjusted to reflect the anticipated trajectory of
  the epidemic given the current population state and updated
  parameter estimates.  This document briefly covers the background
  and methodology underpinning the implementation provided by the
  package and contains extensive examples showing the functions and
  methods in action.}

\Keywords{SIR model, Bayesian inference, optimal decisions, Markov chain
Monte Carlo}
\Plainkeywords{SIR model, Bayesian inference, optimal decisions,
  Markov chain Monte Carlo}

\Submitdate{0000-00-00}
\Acceptdate{0000-00-00}

\Address{
Daniel Merl\\
Systems and Intelligence Analysis\\
Lawrence Livermore National Laboratory\\
Livermore, CA 94551\\
USA\\
E-mail: \email{merl1@llnl.gov}\\
\\
Leah R. Johnson\\
Ecology, Evolution, and Marine Biology\\
University of California, Santa Barbara\\
Santa Barbara, CA 93106\\
USA\\
E-mail: \email{lrjohnson@lifesci.ucsb.edu}\\
URL: \url{http://www.lifesci.ucsb.edu/~lrjohnson/}\\
\\
Robert B. Gramacy\\
Statistical Laboratory\\
University of Cambridge\\
Wilberforce Road\\
Cambridge, CB3 0WB\\
United Kingdom\\
E-mail: \email{bobby@statslab.cam.ac.uk}\\
URL: \url{http://www.statslab.cam.ac.uk/~bobby}\\
\\
Marc Mangel\\
Department Applied Mathematics and Statistics\\
The Jack Baskin School of Engineering\\
University of California, Santa Cruz\\
Santa Cruz, CA 95064\\
USA\\
E-mail: \email{msmangel@ucsc.edu}\\
URL: \url{http://users.soe.ucsc.edu/~msmangel/}
}

\begin{document}

% \subsection*{Intended audience}
% \label{sec:discaimer}

% This document is intended to familiarize a (potential) user of
% \pkg{amei} with the models implemented and analyses available in
% the package.  After a brief overview, the bulk of this document
% consists of a detailed example illustrating the various functions and
% methodologies implemented. This document has been authored in
% \pkg{Sweave} (try \code{help(Sweave)}).  This means that the code
% quoted throughout is certified by \proglang{R}, and the \code{Stangle}
% command can be used to extract it.  The demo available in this package
% will run the same code via \verb!demo("amei")!.

% Note that this tutorial was not meant to serve as an instruction
% manual.  For more detailed documentation of the functions contained in
% the package, see the package help--manuals. At an \proglang{R} prompt, type
% \code{help(package=amei)}. PDF documentation is also available on the
% world-wide-web.
% \begin{center}
% \tt http://www.cran.r-project.org/web/packages/amei/index.html
% \end{center}


\section{Introduction}
\label{sec:motivate}


The goal of the study of infectious diseases is to better understand
how infections are spread and maintained, and ultimately to find ways
to control the spread of a disease. The most common methods for
intervening in the spread of an infectious disease are to either remove
susceptible individuals or apply treatment to infected
individuals. For instance, the susceptible population may be culled,
as in the case of foot-and-mouth disease
\citep{tildesley06,enserink01}, or the infected population may be
quarantined, as in the case of SARS \citep{lloyd-smith03}. Most
commonly, susceptibles may be vaccinated, as in the case with
influenza or smallpox \citep{ferguson03,halloran02}.

Each of these actions incurs a quantifiable epidemiological cost.  For
culling, the cost is an additional number of deaths; for quarantine,
the cost is likely to be measured in monetary units rather than lost
lives; for vaccination, the cost may be measured in both monetary
units as well the number of additional vaccine--induced infections;
for medical treatment, the cost is again monetary. Furthermore, the
costs associated with each action can depend upon the state of the
disease within the population of interest. This raises the question of
how to find optimal epidemiological interventions in a manner that
adaptively depends on the state of the epidemic.

Most existing methods for finding optimal intervention strategies are
concerned with the situation of pre-emptive intervention which is
assumed to be completed before the onset of the epidemic (for instance
see \citet{ball02,patel05,tildesley06}).  In this kind of situation,
there is no reason to consider sequentially updated (i.e. adaptive)
interventions: as soon as the intervention policy is triggered, the
epidemic threat will be eradicated.  However, in most scenarios, total
and instantaneous intervention will not be an implementable strategy.
Moreover, these methods usually involve calculations that assume no
uncertainty in key model parameters, including transmission rate,
recovery rate, and others (however, see \citet{elderd06}
and our brief discussion in Section~\ref{sec:discuss}).

Here we introduce \pkg{amei}, a software package that implements a
statistical framework introduced by \citet{merl09}, that
allows one to respond to an emerging epidemic while simultaneously
learning about it.  We consider vaccination strategies defined by a
fraction of the current susceptible population to be targeted for
vaccination, and a threshold number of susceptibles below which the
vaccination campaign is called off. We couple the evaluation of
optimal, adaptive, intervention strategies with Bayesian procedures
for performing on-line estimation of the parameters of the underlying
epidemic model, thereby propagating parameter uncertainty through to
policy decisions. We demonstrate the advantages of adaptive
intervention via the functions provided by the package using
simulations modeled after an influenza outbreak at a British boarding
school described by \citet{murray02}. We compare the
distribution of costs arising from epidemiological intervention under
the adaptive policies to those arising from non-adaptive policies
(i.e. policies not dependent on the state of the epidemic and/or not
reflecting parameter uncertainty), and find that the adaptive policies
result in low total costs, efficient use of available resources,
and are robust to model misspecification.

This tutorial is comprised of three main sections.  The mathematical
specification of the Bayesian models used for inference and the Monte
Carlo methods for constructing optimal vaccination strategies (both
static or adaptive and on--line) are contained in
Section~\ref{sec:methods}.  In Section~\ref{sec:pnas}, the functions
and methods implemented in the package are illustrated by following a
single, detailed, example, whose results were first reported by
\citet{merl09}.  Section \ref{sec:alt} contains a second example with
an alternative transmission model, thereby illustrating the additional
benefits of the approach and features of the package.  The paper
concludes in Section \ref{sec:discuss} with a discussion of the
methodology and related work highlighting other freely available
software with comparisons and contrasts.  Miscellaneous details on
implementation, etc., are provided in an appendix.

% {\bf Maybe we should say somewhere that by calculating the stopping
%   number, we put a cap on the total number of vaccine doses that we
%   would need to most efficiently intervene, thus possibly further
%   reducing the costs of intervening in the epidemic.  }

\section{Methods}
\label{sec:methods}

\subsection{SIR Model}
\label{sec:sirmodel}

In \pkg{amei}, we consider a stochastic version of a standard
Susceptible--Infected--Removed (SIR) model
\citep{anderson91,hetchote00} with permanent immunity and with
mortality. In this model, the dynamic variables at time $t$ are the
number of susceptible individuals, $S(t)$; the number of infected
individuals, $I(t)$; the number of recovered individuals, $R(t)$; and
the number of removed/dead individuals, $D(t)$. In a standard SIR
model we would typically lump all of the Recovered and Dead
individuals into a single removed class. However, in controlling an
epidemic we expect there to be differential costs associated with
deaths versus other types of removal (see below), so we count the
numbers of deaths separately. We assume the population is closed to
immigration or emigration so that $S(t)+I(t)+R(t)+D(t)=N$, where $N$
is constant.

Models of this type can include transmission dynamics ranging from the
very simple to the very complex. We adopt a flexible negative binomial
form for the transmission function
\citep{godfray89,mangel06,mccallum01}. Under this assumption, the SIR
model is described by the following system of differential equations
\citep{hetchote00,mccallum01}:
\begin{align}
  \frac{\mathrm{d}S}{\mathrm{d}t} &=
-kS \ln\left(1+\frac{bI}{k}\right) \label{dS} \\
  \frac{\mathrm{d}I}{\mathrm{d}t} &= 
kS \ln\left(1+\frac{bI}{k}\right) - (\nu+\mu) I \nonumber \\
  \frac{\mathrm{d}R}{\mathrm{d}t} &= \nu I \label{dR}\\
  \frac{\mathrm{d}D}{\mathrm{d}t} &= \mu I \label{dD}
\end{align}
The model parameters are: the transmission rate $b$; the
overdispersion (or ``clumpiness'') parameter $k$; the death rate
$\mu$; and the rate of recovery to the immune class $\nu$. The
negative binomial distribution can be interpreted as a compound
stochastic process in which encounters between infected and
susceptible individuals occur randomly (i.e., according to a Poisson
process) such that the encounter rate varies according to a gamma
distribution with coefficient of variation $k^{-1/2}$.  Thus, via $k$,
the negative binomial transmission can account for social interactions
and/or network factors in disease transmission, without requiring
explicit characterization of the population structure.

This SIR formulation leads to a natural discrete time approximation
for the numbers of infections ($\tilde{I}$), recoveries ($\tilde{R})$,
and deaths ($\tilde{D}$) arising in the unit time interval from $t$ to
$t+1$. Assuming the total number of infected individuals, $I$, is
approximately constant and integrating Eq.~(\ref{dS}) over a unit time
interval gives
\begin{equation}
  S(t+1) = S(t) \left[ \frac{k}{k+bI(t)} \right]^k, \label{eq:meanS}
\end{equation}
so that the fraction of susceptible individuals surviving throughout a
unit time interval is $\left[ \frac{k}{k+bI(t)}\right]^k$.  Viewed as
a discrete time stochastic process, where the mean number of remaining
susceptible individuals is given by Eq.~(\ref{eq:meanS}), the number
of new infections occurring between time $t$ and $t+1$ when $S(t)=s$
and $I(t)=i$ with mean
\[
  S(t)\left(1 - \left[ \frac{k}{k+bI(t)} \right]^k\right)
\]
can be described as a binomial process
\begin{align}
  \tilde{I}|s,i &\sim \mbox{Bin}(s, p_i(i,b,k)), \label{bini}
  \intertext{where}
  p_i(i,b,k) &= 1-\left(\frac{k}{k+bi}\right)^k \nonumber
\end{align}
and Bin$(n,\pi)$ is the standard binomial distribution. Similarly, by
integrating Eqs.~(\ref{dR}--\ref{dD}), the numbers of recoveries
and deaths occurring between time $t$ and $t+1$ can be described by
\begin{eqnarray}
\tilde{R}|i & \sim & \mbox{Bin}(i,p_r) \label{rtilde}\\
\tilde{D}|i,\tilde{r} & \sim & \mbox{Bin}(i-\tilde{r},p_d) \label{dtilde}
\end{eqnarray}
where $p_r = 1-e^{-\nu}$ and $p_d = 1-e^{-\mu}$.  The forward dynamics
for the total numbers of susceptible and infected individuals are
therefore
\begin{align*}
S(t+1) = & S(t) - \tilde{I}|s,i \\
I(t+1) = & I(t)  +  (\tilde{I}|s,i) - (\tilde{R}|i) - (\tilde{D}|i,\tilde{r} + \tilde{i}). 
\end{align*}
Here the lower case symbols $\{s,i,r\}$ denote the realized value of
the associated capital letter random variable.  In this discrete time
approximation we have assumed a particular ordering of events, namely
that recoveries occur first, followed by deaths from among those
infected individuals who did not recover, followed by new infections.
Simulation studies indicated that these assumptions, as well as other
possible orderings, resulted in system dynamics that were equal in
expectation to the deterministic solutions to the continuous time SIR
model.

\subsection{Online Parameter Estimation}
\label{sec:estimate}

A primary task of \pkg{amei} is to perform inference on the SIR model
parameters, conditional on sequential observations of the numbers of
susceptible, infected, and recovered individuals in the population.
Given the discrete time approximation in the previous section, it is
possible to do this online (i.e., as the epidemic progresses) via
straightforward parametric Bayesian methods.  In particular, we use
Markov Chain Monte Carlo (MCMC) \citep[e.g.,][]{gamerman06} to learn
about the posterior distributions of $b$, $k$, $\nu$, and $\mu$
conditioned on the evolution of the epidemic observed so far.  The
likelihood is given recursively in
Eq.~(\ref{bini}--\ref{dtilde}). % Assume, at first, that no intervention
%strategy is implemented.  
Let $\tilde{i}_t = S(t-1) - S(t)$ be the number of new infecteds at
time $T$, and similarly for the newly recovered and dead individuals
$\tilde{r}_t$ and $\tilde{d}_t$ so that $\tilde{r}_t + \tilde{d}_t
\leq I(t-1)$.  Then, the likelihood of the data observed up to time
$T$ is given by
\begin{align*}
&\prod_{t=1}^T \mbox{Bin}(\tilde{i}_t \,| S(t-1), p_i(I(t-1),b,k))\\
&\times \prod_{t=1}^T \mbox{Bin}(\tilde{r}_t | p_r)
\times \prod_{t=1}^T \mbox{Bin}(\tilde{d}_t | I(t)-\tilde{r}_t, p_d).
\end{align*}
Observe that it consists of three mutually independent components when
conditioning on the course of the epidemic.

Conditional conjugacy can be exploited for $\nu$ and $\mu$ via beta
priors for $p_r$ and $p_d$. A Beta$(\alpha_r, \beta_r)$ prior for
$p_r$ implies that
\begin{equation}
p(\nu) = (1-e^{-v})^{\alpha_r-1} e^{-v\beta_r}. \label{nup}
\end{equation}
Conjugate updating leads to the posterior conditional
\begin{equation}
p_r|\dots \sim \mbox{Beta}(\alpha_r + 
\textstyle \sum_{t=1}^T \tilde{r}_t, 
\beta_r + \sum_{t=1}^T I(t)- \tilde{r}_t). \label{prp}
\end{equation}
The form of the conditional posterior for $\nu$ is similar to
Eq.~(\ref{nup}) and can be simulated by first drawing $p_r$ via
Eq.~(\ref{prp}) and then applying the inverse transformation $\nu =
-\log(1-p_d)$.  Sampling for $\mu$ proceeds similarly with
\begin{equation}
p_d|\dots \sim \mbox{Beta}(\alpha_d + 
\textstyle \sum_{t=1}^T \tilde{d}_t, 
\beta_d + \sum_{t=1}^T I(t) - \tilde{r}_t - \tilde{d}_t).
\end{equation}
Thus it is possible to take Gibbs samples for $\nu$ and $\mu$ so long
as appropriate hyperparameters $\alpha_r, \beta_r, \alpha_d, \beta_d$
can be found to represent our prior beliefs.  In ignorance we simply
set these to unity by default, leading to a uniform prior on $p_r$ and
$p_d$.  The user is free to specify his/her own prior parameterization
in the package.

Obtaining samples for $b$ and $k$ requires the Metropolis--Hastings
algorithm.  Our prior beliefs can be encoded with gamma distributions,
and conditional on a previous sample $(b,k)$ the next sample $(b',k')$
can be obtained by Metropolis--within--Gibbs steps using:
\begin{align}
  p(b'&| k,\dots) \\
& \propto \Gamma[b' | \alpha_b,\beta_b] \prod_{t=1}^T
  \mbox{Bin}(\tilde{i}_t \,|\,
  S(t-1), p_i(I(t-1),b',k)), && \mbox{and} \nonumber \\
  p(k'&| b',\dots) \\
&\propto \Gamma[k' | \alpha_k,\beta_k] \prod_{t=1}^T
  \mbox{Bin}(\tilde{i}_t \,|\, S(t-1), p_i(I(t-1),b',k')). \nonumber
\end{align}
The default values for the prior parameterization set in the package
are $(\alpha_b, \beta_b) = (\alpha_k, \beta_k) = (1, 3)$ which (though
seemingly informative at first glance) turns out to be uninformative
on the scale of the support of the posterior.  As before, these can
easily be changed by the user.  We use random walk uniform proposals
on the positive real line, i.e., $b' \sim U[3b/4,4b/3]$, which gives
reasonably good mixing in the Markov chain.

The presence of a vaccination strategy (described in the next
section) necessitates a simple change to the above equations.  If $0
\leq v_t \leq S(t-1)$ is the number of susceptibles which have been
vaccinated, then we simply replace $S(t-1)$ with $S(t-1) - v_t$
so that $\tilde{i}_t=S(t-1)-v_t-S(t)$.

\subsection{Optimal Vaccination Strategies: Fixed and Adaptive}
\label{sec:optvac}

Having obtained estimates of the SIR model parameters at some time
during the epidemic, we next want find the best way to intervene in
the spread of the epidemic. The first step is to formalize the notion
of the ``best'' intervention strategy. This requires a specification
of the costs of various actions, such as vaccination, versus the cost
of allowing the epidemic to spread in an uncontrolled fashion. In
\pkg{amei}, we formulate the total expected cost of the epidemic in
terms of the underlying costs associated with maintaining infected
individuals until recovery, suffering death, and administering
vaccinations. We assume costs associated with each action can be
specified in some common currency, such as monetary costs, or
probabilities of mortality (such that the cost to be minimized is the
expected numbers of deaths).

We formulate our vaccination strategies in terms of a policy where a
fraction, $\alpha$, of susceptibles are prevented from risk of
infection by moving them directly into an immune/recovered class, such
as by perfect vaccination, until the number of individuals that are
still susceptible drop below a threshold, $\gamma$, and vaccination is
discontinued. Let $c_1(\alpha,\gamma, s)$ denote the cost associated
with the vaccination strategy $(\alpha,\gamma)$ when $S(t)=s$.  If
$c_v$ denotes the cost per vaccine unit, then
\[
 c_1(\alpha,\gamma,s) = \left\{ \begin{array}{ll}c_v \alpha s &\mbox{ if }
  s>\gamma\\ 0 &\mbox{ if } s\leq \gamma. \end{array}\right.
\]
Thus the number if individuals vaccinated at time $t$ is $v_t=\alpha
s$. Let $c_2(i)$ denote the cost component that depends on the number
of infections in the population, $I(t)=i$.  This component includes
the costs associated with maintaining the non-recovered infected
individuals and costs associated with deaths, as in
\[
c_2(i) = c_t i + c_d \tilde{d},
\]
where $c_t$
is the cost per treatment/maintenance of a non-removed infected
individual, and $c_d$ is the cost per death.  

Assuming the initial epidemiological state is $S(0)=s_0$, $I(0)=i_0$,
the expected total cost of the epidemic under intervention strategy
$(\alpha, \gamma)$ can be expressed recursively as
\begin{equation}
\E\{C_0\} = c_1(\alpha,\gamma, s_0) + c_2(i_0)+ \E\{ C_{1} \},\label{sdp}
\end{equation}
where $\E\{ C_t\}$ denotes the expected cost accumulated from time $t$
onwards, terminating at some arbitrary time horizon.  The optimal
intervention strategy $(\alpha,\gamma)$ is the one that minimizes the
total accumulated cost over the course of the epidemic.  Two methods
for calculating such strategies are as follows.
 
The first case we are interested in is when the parameters of the SIR
model are known, and we wish to calculate the single best intervention
strategy $(\alpha,\gamma)$.  The total expected cost depends on the
parameter values and the initial epidemiological state $(s_0,i_0)$.
Thus, conditional on the parameter values, Monte Carlo simulation of
the disease dynamics specified in Section~\ref{sec:sirmodel} 
can be used to search over values of $\alpha$ and $\gamma$ in order to
find the combination that minimizes $\E\{C_0\}$.  For each combination
of $\alpha$ and $\gamma$ considered, we conduct $n$ stochastic
simulations of the outbreak in order to estimate the mean cost
associated with the intervention $(\alpha,\gamma)$. The strategy
producing the lowest mean cost is defined to be the optimal
intervention. Typically we discretize and create a grid of admissible
$\alpha$ and $\gamma$ settings.  In the examples in
Section~\ref{sec:pnas} we allow $\alpha$ to range from 0 to 1 in
increments of $0.1$, and $\gamma$ to range from from 2 to $s_0-s_0/10$
in increments of $s_0/10$, i.e., taking 10 steps.\footnote{We
  generally do not include $\gamma=s_0$ in the grid since this policy
  (coupled with any $\alpha$) is equivalent to $\alpha=0$ for any
  $\gamma$.}

In the second case, we want to calculate an adaptive strategy that
sequentially updates the best strategy $(\alpha,\gamma)$ as we learn
more about the epidemic model parameters.  As above, the expected cost
surface associated with a given set of parameter values (as obtained
by MCMC, described above), can be explored using standard Monte Carlo
methods.  At each time step, MCMC is used to produce samples from the
current posterior distribution on model parameters conditional on all
data observed up to that point.  These samples are used to calculate
the optimal vaccination strategy from that time point onward, as
outlined as above.  After calculating the optimal strategy associated
with each set of posterior samples, the adaptive strategy to be
implemented at that time step is defined to be the strategy that most
frequently minimizes the expected cost over all samples.

\section{An illustrative example}
\label{sec:pnas}

In this section we demonstrate the advantages of adaptive intervention
using simulations modeled after an influenza outbreak at a British
boarding school described by \citet{murray02}.  We shall compare
the distribution of costs arising from epidemiological intervention
under the adaptive policies to those arising from non-adaptive
policies.  The epidemic conforms to many standard assumptions of SIR
models: a population essentially closed to immigration and emigration;
includes recovery and immunity; and has near homogeneous mixing of
susceptibles and infecteds.  The epidemic was traced back to a single
infected student out of a population of 763 individuals.

For reproducibility of the results in this section we have set
the random seed as follows.
\begin{Schunk}
\begin{Sinput}
> seed <- 12345
> set.seed(seed)
\end{Sinput}
\end{Schunk}

We begin by exploring the behavior of the SIR model, described in
Section~\ref{sec:sirmodel}, without any intervention. The first step
is to set the relevant parameters that are necessary for simulating
the epidemic.  These consist of: the ``true'' underlying parameters
for the SIR model upon which our costs will be evaluated; the initial
condition of the population at the beginning of the epidemic; and the
relative costs of infections, deaths, and vaccinations.
\begin{Schunk}
\begin{Sinput}
> true <- list(b = 0.00218, k = 10, nu = 0.4, mu = 0)
> init <- list(S0 = 762, I0 = 1, R0 = 0, D0 = 0)
> costs <- list(vac = 2, death = 4, infect = 1)
\end{Sinput}
\end{Schunk}
\citet{murray02} provides estimates of the transmission rate
($b$) and recovery rate ($\nu$), which we use in the 
\code{true} parameterization set above.  We set the death rate
($\mu$) to zero since there are no deaths in this epidemic.  Finally,
we set the negative binomial dispersion parameter ($k$) 
to be large to reflect the
homogeneous mixing of the population.  The \code{costs}, chosen purely for 
illustrative purposes,  describe the unit cost (or loss) 
for a single vaccination or death, relative to the
daily cost of maintaining an infected individual.

As a baseline, we are interested in the costs of a no--vaccination
policy on epidemic trajectories with the above parameters. To explore
this, we set the vaccination policy used in the simulation to zero.
\begin{Schunk}
\begin{Sinput}
> vac <- list(frac = 0, stop = 0)
\end{Sinput}
\end{Schunk}

We are now ready to run the Monte Carlo experiment. The function
\code{MCepi} can be used to simulate the population and cost
trajectories for the experiment. This function simulates the
stochastic SIR model under a given vaccination strategy. For each
realization of the epidemic progression the function calculates the
cost over time for the epidemic. This is repeated many
times\footnote{The number of MC simulations can be varied with the
  \code{MCreps} argument to \code{MCepi}. The default value of
  \code{MCreps} is 1000.}, and the mean trajectories for both the
populations and the costs, as well as the 5th and 95th quantiles, are
recorded.
\begin{Schunk}
\begin{Sinput}
> init.MCepi <- MCepi(init, true, vac, costs)
\end{Sinput}
\end{Schunk}

Now we can simply plot the results to look at the distribution of the
susceptible, infected, and recovered individuals in the population as
the epidemic progresses ({\em top} panel of Figure~\ref{f:epis}), as well as
the distribution of costs over time ({\em bottom} panel of Figure
\ref{f:epis}).
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(init.MCepi)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics{amei-epis}
\begin{Schunk}
\begin{Sinput}
> plot(init.MCepi, type = "costs")
\end{Sinput}
\end{Schunk}
\includegraphics{amei-costs}
\caption{Monte Carlo simulated epidemic trajectories ({\em top}) for the numbers
  of susceptible, infected, and recovered individuals, and the associated
  cost(s) ({\em bottom}) with a {\em null} vaccination strategy. 
  (2.5,50,97.5\%) quantiles are shown. }
\label{f:epis}
\end{figure}

Next we examine the dynamics of the system when we have perfect
information and use a fixed vaccination policy. First construct a grid
of admissible policies. In this case, we shall consider strategies
that: vaccinate a fixed proportion of the susceptible population, from
0 to 100\%, in steps of 10\%; and that stop vaccinating susceptibles
when the remaining susceptible population falls below some threshold
between 2 and the initial susceptible population minus 75, in
increments of 75, as explained in Section \ref{sec:optvac}.
\begin{Schunk}
\begin{Sinput}
> vacgrid <- list(fracs = seq(0, 1, 0.1), stops = seq(2, 
+     init$S0 - 75, 75))
\end{Sinput}
\end{Schunk}
Once this grid has been initialized, we
can run the Monte Carlo experiment, using the function \code{optvac},
which finds the vaccination policy that minimizes the total cost of
the (stochastic) epidemic.  This is done by simulating the epidemic
forward under the known, \code{true}, parameterization and calculating
the cost of each vaccination strategy.  
\begin{Schunk}
\begin{Sinput}
> out.optvac <- optvac(init, true, vacgrid, costs)
\end{Sinput}
\end{Schunk}
This function outputs the costs
for each of the possible vaccination strategies. The best and worst
policies can be obtained as follows: 
\begin{Schunk}
\begin{Sinput}
> best <- getpolicy(out.optvac)
> worst <- getpolicy(out.optvac, which = "worst")
> rbind(best, worst)
\end{Sinput}
\end{Schunk}
The same information can be obtained via the generic \code{print}
and \code{summary} commands, which will be shown later.  We can also
plot the cost surface over the space of possible vaccination
strategies. This takes the form of a heat plot, where lower cost
policies appear in deep red, and high cost policies appear in white
(Figure \ref{f:optvac}).

\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(out.optvac)
\end{Sinput}
\end{Schunk}
\centering \includegraphics{amei-optvac}
\caption{Heatmap depicting the expected cost surface associated with
  variable stop time vaccination strategies based on the true
  parameter values.  The minimum expected cost (1657) is
  achieved by a strategy of vaccinating 90\% of
  susceptibles at each time step, until the number of susceptibles
  falls below 152.  As expected, the maximum expected
  cost (2297 cost units) is realized through inaction
  ({\em top} row and left column policies are never implemented)}
\label{f:optvac} 
\end{figure} 
Given the calculated optimal policy, we can explore the effects of the
vaccination strategy on the progression of the epidemic together with
the trajectory of costs under this strategy. We do this by again
simulating the epidemic dynamics using the function \code{MCepi},
however this time we include the best fixed vaccination policy.

\begin{Schunk}
\begin{Sinput}
> vac.opt <- best[3:4]
> opt.MCepi <- MCepi(init, true, vac.opt, costs)
\end{Sinput}
\end{Schunk}
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(opt.MCepi)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics{amei-episov}
\begin{Schunk}
\begin{Sinput}
> plot(opt.MCepi, type = "costs")
\end{Sinput}
\end{Schunk}
\centering
\includegraphics{amei-costsov}
\caption{Monte Carlo simulated trajectories ({\em top}) for the numbers of
susceptible, infected, and recovered individuals and cost(s) 
({\em bottom}) trajectories under optimal (fixed) vaccination strategy.}
\label{f:epis:ov}
\end{figure}
Figure~\ref{f:epis:ov} summarizes the trajectories ({\em top}) of the
epidemic under the optimal vaccination strategy over time, and the
corresponding costs ({\em bottom}). By default we assume a fixed lag of 7
time steps from when the first infection appears to when the first
intervention can take place\footnote{This can be varied with
  the \code{start} argument to \code{MCepi}.}. This is apparent in Figure
\ref{f:epis:ov}, where there is a sharp transition from day 7 when the
vaccinations begin, and the susceptible population drops dramatically.
The number of vaccinated individuals can be added to the plot by
specifying the argument \code{showv = TRUE}, however we omit this here
to reduce clutter in the figure.  Information on the distribution of
the number of vaccine  units dispensed can be extracted as follows.
\begin{Schunk}
\begin{Sinput}
> getvac(opt.MCepi)
\end{Sinput}
\end{Schunk}

We can compare these results to the case without vaccination (Figure
\ref{f:epis:ov}, {\em top}), and see that the optimal vaccination
strategy effectively suppresses the spread of the infection. The costs
also spike around time 7 as the vaccination policy is
implemented. These costs then stabilize at a lower level than that
observed under the no--vaccination strategy in Figure~\ref{f:epis}
({\em bottom}). We can easily extract information on the distribution
of final costs of the no--vaccination and optimal (fixed) vaccination
policies for comparison as follows.
\begin{Schunk}
\begin{Sinput}
> T <- length(opt.MCepi$Median$C)
> optC <- getcost(opt.MCepi)
> initC <- getcost(init.MCepi)
> data.frame(rbind(initC, optC), row.names = c("init", 
+     "opt"))
\end{Sinput}
\end{Schunk}
The optimal (fixed/static) strategy gives a (mean) savings of
approximately 613.004 units, or
27\% compared to the cost 
without an intervention.  The same 
information is available through the generic \code{print} and
\code{summary} commands, e.g.,
\begin{Schunk}
\begin{Sinput}
> opt.MCepi
\end{Sinput}
\end{Schunk}

Now we consider adaptive management strategies using the \code{manage}
function. In this case we assume that we do not have perfect
information on the underlying epidemic.  As such, we want to
simultaneously estimate the epidemic model parameters as well as find
an optimal management strategy.

In order to proceed, we must define the \code{epistep} function, which
will provide the ``observed" SIR count data upon which our inferences
will be based.  This function can be regarded as the ``true'' disease
process that is being approximated by the SIR model we have described.
The default \code{epistep} function provides simulations from the same
disease model (Eqns.~\ref{bini}--\ref{dtilde}) that is being used for
assumed for inference, however the user can specify custom
\code{epistep} functions.  By customizing this function, a user can
provide custom data to \pkg{amei}, or explore the effects of model
misspecification, as we shall describe in Section~\ref{sec:alt}.

We also need to start with an initial guess, {\em i.e.}, priors, for the
epidemic parameters that we want to estimate. The default option is to
do this by choosing appropriate hyperparameters for the priors
explained earlier.  The defaults used by \code{manage} are those
given in Section~\ref{sec:estimate}. 

Here, we run the \code{manage} function with default values for
hyperparameters and \code{epistep} to adaptively design a vaccination
strategy to manage the epidemic.  At each time step in the evolution
of the epidemic the \code{manage} function uses MCMC to sample from
the posterior distribution of the parameters $(b,k,\nu,\mu)$ given the
available history of the epidemic and any already--implemented
intervention.  Then, a thinned subset of these samples are used
propagate uncertainty in the parameter estimates through to the costs
of the vaccination strategies.  These costs are obtained by performing
Monte Carlo forward simulations of the epidemic from the current time
point into the future with those parameters.  As explained in
Section~\ref{sec:optvac} we choose to implement the strategy that
most frequently minimizes the cost. After the intervention is
implemented, the state in the next time step is determined by
\code{epistep}, and the process is repeated.\footnote{The function
  \code{manage} has default time horizons specified for both the last
  time an intervention can be implemented and the time of the final
  observation of the epidemic. These can be varied with the arguments
  \code{Tstop} and \code{T}, respectively. By default we specify
  \code{T=40} and \code{Tstop=T}.}
\begin{Schunk}
\begin{Sinput}
> out.man <- manage(init, epistep, vacgrid, costs)
\end{Sinput}
\end{Schunk}
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(out.man)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics{amei-epi}
\begin{Schunk}
\begin{Sinput}
> plot(out.man, type = "cost")
\end{Sinput}
\end{Schunk}
\centering
\includegraphics{amei-cost}
\caption{Trajectory ({\em top}) in terms of the numbers of susceptible,
  infected, recovered, and vaccinated individuals, and the
  corresponding cost ({\em bottom}) of the epidemic under adaptive
  management.}
\label{f:epi}
\end{figure}
To explore the results of the simulation we again plot the evolution
of the epidemic (Figure~\ref{f:epi}, {\em top}) as well as the cost
trajectory under the optimal vaccination (Figure~\ref{f:epi}, {\em bottom}),
both for a single run. We can now compare the case with adaptive
management to the case without vaccination (Figure~\ref{f:epis},
{\em bottom}), as well as the case of optimal vaccination with perfect
information (Figure~\ref{f:epis:ov}, {\em bottom}).  As before, we can 
extract information on the final cost of the epidemic as follows.
\begin{Schunk}
\begin{Sinput}
> getcost(out.man)
\end{Sinput}
\end{Schunk}
Since the \code{manage} function is not performing a Monte Carlo
experiment, the number returned is a scalar.  Notice that the adaptive
strategy is comparable to the best fixed strategy obtained when the
true parameterization is known.  Later, we shall perform a Monte Carlo
version of the adaptive management strategy to make a more meaningful
comparison.

\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> true <- as.list(formals(epistep)$true)
> plot(out.man, type = "params", true = true)
\end{Sinput}
\end{Schunk}
\centering \includegraphics[trim=0 0 0 20]{amei-params}
\caption{Final posterior distributions of estimated parameters 
(clockwise from the left: transmission rate, overdispersion parameter, 
mortality rate, recovery rate).  ``True'' parameter values are 
indicated by a dot; mean posterior values are indicated by an `x'; 
and the central 95\% region of the distribution is shaded.}
\label{f:params} 
\end{figure} 
We can also see the final distribution of the SIR model parameters
contained in \code{out.man$samps}.
This is shown in Figure~\ref{f:params}.  A summary is provided by
the generic \code{print} and \code{summary} commands:
\begin{Schunk}
\begin{Sinput}
> out.man
\end{Sinput}
\end{Schunk}
%{\bf Perhaps we should also show the evolution of the parameter 
%distribution over time. }
% \begin{figure}[ht!]
% <<label=fracs,fig=TRUE,echo=TRUE,width=8,height=5,include=FALSE,cache=TRUE>>=
% plot(out.man, type="fracs")
% @
% \centering
% \includegraphics[trim=0 0 0 40]{amei-fracs}
% <<label=stops,fig=TRUE,echo=TRUE,width=8,height=5,include=FALSE,cache=TRUE>>=
% plot(out.man, type="stops")
% @
% \includegraphics[trim=0 0 0 40]{amei-stops}
% \caption{The fraction vaccinated ({\em top}) and the stop time ({\em bottom})
% at each time step in the evolution of the epidemic under management
% when vaccinations start at time 8, by default.}
% \label{f:evol}
% \end{figure}
% Finally we can explore how the adaptive vaccination strategy evolves
% over time. Figure~\ref{f:evol} shows the fraction vaccinated ({\em top}) and
% the stop time ({\em bottom}) at each time step after vaccinations are
% allowed to start (the default is time 8).  {\bf Can we say something
%   more about these plots, generally --- not the specific ones
%   we got this time?}

We also note that it is possible to utilize the \code{manage} function
in a piecemeal way, or in ``epochs''.  This is accomplished by passing
the output of the \code{manage} function, an \code{"epiman"}-class
object, as the first (\code{init}) argument.  The management of the
epidemic will re-start where it left off.  It will stop at the newly
specified (later) time horizon \code{Tstop} specified by that argument.
%<<echo=false,results=hide>>=
%seed <- 12345; set.seed(seed)
%@ 

Next, we illustrate a Monte Carlo experiment where epidemics are
initialized and proceed randomly through the adaptive management
strategy illustrated above so that we can see the average behavior,
costs, and associated variability.  The function \code{MCmanage}
facilitates this experiment, and it essentially calls the 
\code{manage} function many times (which can be controlled by the 
\code{MCreps} argument).  In order to provide a (relatively) quick
demonstration we have set a low default of \code{MCreps = 30} and have
used low defaults for the other Monte Carlo parameters to 
\code{management}.

\begin{Schunk}
\begin{Sinput}
> out.MCmanage <- MCmanage(init, epistep, vacgrid, 
+     costs)
\end{Sinput}
\end{Schunk}
To reproduce the results in \citet{merl09}, use 
\begin{verbatim}
MCvits = 100, MCMCpits = 10000, vacsamps = 100, MCreps = 100
\end{verbatim}
and otherwise use the defaults.  The object that is returned is of
class \verb!"MCepi"! with fields similar to those that are output from
the \code{MCepi} function which implements a static (fixed) vaccination 
strategy.  Thus, the same generic plot commands can be used.
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(out.MCmanage)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics{amei-MCmanepis}
\begin{Schunk}
\begin{Sinput}
> plot(out.MCmanage, type = "costs")
\end{Sinput}
\end{Schunk}
\includegraphics{amei-MCmancosts}
\caption{Monte Carlo simulated epidemic trajectories ({\em top}) for the
  numbers of susceptible, infected, and vaccinated individuals, and
  the associated cost(s) ({\em bottom}) under adaptive management}
\label{f:epismanage}
\end{figure}
Figure~\ref{f:epismanage} shows plots summarizing the distribution of
epidemic trajectories ({\em top}) and costs ({\em bottom}) under the adaptive
management.  Distributional information on number of vaccine units
dispensed can be extracted as follows.
\begin{Schunk}
\begin{Sinput}
> getvac(out.MCmanage)
\end{Sinput}
\end{Schunk}
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(out.MCmanage, type = "fracs")
\end{Sinput}
\end{Schunk}
\centering
\includegraphics{amei-MCmanfracs}
\begin{Schunk}
\begin{Sinput}
> plot(out.MCmanage, type = "stops")
\end{Sinput}
\end{Schunk}
\includegraphics{amei-MCmanstops}
\caption{Fractions vaccinated ({\em top}) and stopping levels ({\em bottom}) for  
  the simulated epidemic trajectories shown in Figure~\ref{f:epismanage}
  that were under adaptive management.  In both cases, the $(2.5,50,97.5)$-\%
  quantiles are shown.}
\label{f:MCfracs}
\end{figure}
Figure~\ref{f:MCfracs} shows the distribution of fractions of
individuals vaccinated and the stopping level for the epidemic
trajectories under adaptive management in the Monte Carlo experiment.
We can compare the costs (and quantile bounds) to that of the best
fixed vaccination strategy calculated assuming that the true
parameterization is known.
\begin{Schunk}
\begin{Sinput}
> cinit <- getcost(init.MCepi)
> copt <- getcost(opt.MCepi)
> cman <- getcost(out.MCmanage)
> data.frame(rbind(cinit, copt, cman), row.names = c("init", 
+     "opt", "man"))
\end{Sinput}
\end{Schunk}
The cost of the non-intervention strategy is shown again for
calibration purposes.  As before, the generic \code{print} and 
\code{summary} commands can be used to obtain the same information.
Notice that, in this short Monte Carlo experiment, the adaptive
management scheme approximates the best static vaccination scheme
obtained then the true parameterization is known in advance.  

It is interesting to compare to what the cost of managing the epidemic
would have been {\em without} estimating the parameters as the
epidemic progressed, but rather by guessing what the appropriate
parameters might be.  Suppose our best guess at the parameters
underestimated the true transmission probability $b$ and overestimated
the true recovery probability $\nu$.
\begin{Schunk}
\begin{Sinput}
> bad <- list(b = 0.001, k = 10, nu = 0.9, mu = 0)
\end{Sinput}
\end{Schunk}
The optimal (static) policy under this parameterization can
be constructed, as demonstrated above.
\begin{Schunk}
\begin{Sinput}
> costs.bad <- optvac(init, bad, vacgrid, costs)
> pol.bad <- getpolicy(costs.bad)
> pol.bad
\end{Sinput}
\end{Schunk}
Then, we can calculate the distribution of costs of managing
the true epidemic with a policy developed under our best guess
of the parameterization.
\begin{Schunk}
\begin{Sinput}
> bad.MCepi <- MCepi(init, true, pol.bad[3:4], costs)
> cbad <- getcost(bad.MCepi)
> cbad
\end{Sinput}
\end{Schunk}
Comparing these costs with the ones obtained above, under adaptive
management, we can see that a poor guess can lead to a significantly
worse strategy---nearly 
57\% larger on average. 
Clearly, good quality (online)
estimates of the SIR model are crucial to ensuring a cost--effective
approach to the management of an epidemic.
\section{Alternative Transmission Model}
\label{sec:alt}

For reproducibility of the results in this section we have, again, set
the random seed as follows.
\begin{Schunk}
\begin{Sinput}
> seed <- 12345
> set.seed(seed)
\end{Sinput}
\end{Schunk}

We would like to see how well the fixed and adaptive strategies can do
when faced not only with parameter mis-specification, but when there
may be some component of the underlying transmission model that is not
accounted for in the implemented SIR model upon which its vaccination
strategies are based.  Towards this end, we chose a fairly simple
extension of the SIR model where infection does not pass directly from
individual to individual. Instead, they become infected by
encountering a reservoir of the infectious agent (for instance
bacteria or fungi in water or soil) contributing infected individuals.
The continuous time dynamics are:
\begin{align}
\frac{\mathrm{d}S}{\mathrm{d}t} 
&= -aS\left(\frac{C}{C+C_0}\right) \nonumber \\
\frac{\mathrm{d}I}{\mathrm{d}t} 
&= aS\left(\frac{C}{C+C_0}\right) - (\nu+\mu) I  \nonumber \\
\frac{\mathrm{d}R}{\mathrm{d}t} &= \nu I \label{eq:alt} \\
\frac{\mathrm{d}D}{\mathrm{d}t} &= \mu I \nonumber \\
\frac{\mathrm{d}C}{\mathrm{d}t} &= \rho I - m C \nonumber %\\
%S + I + R + D &=& N. \nonumber
\end{align}
where the concentration of the infective agent in the reservoir is
given by $C$; the transmission rate is modeled by a saturating
function of $C$, so that as $C\rightarrow\infty$ the transmission rate
approaches the constant $a$ at a rate determined by $C_0$; infective
agents die or are removed from the reservoir at rate $mC$; and the
{\it per capita} rate at which new infected agents are added to the
reservoir is $\rho$.

This system can be discretized in a similar manner to the system
discussed earlier, so that the single step transmission dynamics are
given by:
\begin{align}
\tilde{I}|s,C &\sim \mbox{Bin}(s, p_i(a,C,C_0)), 
\intertext{where}
p_i(a,C,C_0) &= 1-\exp\left(\frac{-aC}{C+C_0} \right) \label{itilde}
\end{align}
and Bin$(n,\pi)$ is the standard binomial distribution.  The dynamics
for $\tilde{R}$ and $\tilde{D}$ are exactly as before
(Eqs.~(\ref{rtilde}--\ref{dtilde})).  The single step dynamics for the
reservoir are expected to be more smooth than the epidemic in the
population at large. For instance, for bacteria the concentration
could be in the thousands or millions of individuals. In this case the
evolution of the reservoir is approximately
\begin{equation}
C(t+1)-C(t) = [\rho I(t)] - d_c.
\end{equation}
Here $d_c$ is the number (per unit reservoir) of infectious agents
(stochastically) removed from the reservoir in a unit time, which has
distribution
\begin{equation}
d_c \sim \mbox{Bin}(C, p_{d_c}),
\end{equation}
where $p_{d_z} = 1-e^{-m}$.

In \proglang{R}, we may implement the above transition model by encoding it
in an alternative \code{epistep} function for use with the \pkg{amei}
package methods as follows.
\begin{Schunk}
\begin{Sinput}
> alt.epistep <- 
+ function(SIR, last=list(rem=0, rec=0, infect=0, dead=0, Z=0),
+          true=list(a = 0.05, mu = 0.05, nu = 0.1, m = 0.4, 
+ 	 rho = 200, C = 500))
+ {
+   ## calculate the infection probability based on the
+   ## reservoir, and randomly infect susceptibles
+   Z <- last$Z
+   fz <- Z/(Z+true$C)
+   pi <- 1 - exp(-true$a * fz)
+   infect <- rbinom(1, SIR$S, pi)
+ 
+   ## update recovereds and deaths
+   pr <- 1 - exp(-true$nu)
+   rec <- rbinom(1,SIR$I,pr)
+   pd <- 1 - exp(-true$mu)
+   dead <- rbinom(1, SIR$I-rec, pd)
+ 
+   ## reservoir dynamics
+   pz <- 1 - exp(-true$m)
+   dz <- rbinom(1, Z, pz)
+   bz <- round(SIR$I*true$rho)
+   Z <- Z - dz + bz
+ 
+   ## the returned list is passed in as "last" in a
+   ## subsequent call to this "epistep" function
+   return(list(rem=(rec+dead), rec=rec, infect=infect, 
+               dead=dead, Z=Z))
+ }
\end{Sinput}
\end{Schunk}

Here we first use the \code{manage} function with a \code{NULL}
vaccination strategy (and also, optionally, a \code{NULL} cost
structure), in order to see how the behavior of this system compares
to the default \code{epistep} model implemented within \pkg{amei}. We
can then also look at the estimated ``effective'' SIR parameters.

This system can resemble the standard SIR model, especially when $m$
is large, so infective agents do not remain in the reservoir for long.
% old parameters: a=0.065, m=(0.99, 0.005)
\begin{Schunk}
\begin{Sinput}
> init1 <- list(S0 = 150, I0 = 1, R0 = 0, D0 = 0)
> true <- list(a = 0.1, mu = 0, nu = 0.3, m = 50, 
+     rho = 500, C = 500)
> alt.epistep1 <- alt.epistep
> formals(alt.epistep1)$true <- true
> out.alt <- manage(init1, alt.epistep1, NULL, NULL, 
+     T = 80)
\end{Sinput}
\end{Schunk}
The {\em top} of Figure~\ref{f:alt} shows the resulting dynamics
under the default parameterization offered by the formals of the {\tt
  alt.epistep} function. To illustrate how this new system can
(significantly) differ from the SIR dynamics consider the case 
of very small $m$.  Small $m$ enables new 
infections to occur even if there had been no infected individuals in
a previous time step since the infective agent may persist in the
reservoir for a long time without infected individuals being present.
\begin{Schunk}
\begin{Sinput}
> true <- list(a = 0.1, mu = 0, nu = 0.3, m = 0.001, 
+     rho = 500, C = 500)
> alt.epistep2 <- alt.epistep
> formals(alt.epistep2)$true <- true
> out.alt2 <- manage(init1, alt.epistep2, NULL, 
+     NULL, T = 80)
\end{Sinput}
\end{Schunk}
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(out.alt, showv = FALSE)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt}
\begin{Schunk}
\begin{Sinput}
> plot(out.alt2, showv = FALSE)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt2}
\caption{Trajectories in terms of the numbers of susceptibles,
  infecteds, recovereds, and dead individuals under the alternative
  model in Eq.~(\ref{eq:alt}) with parameters $\{(a=0.1,
  \mu=0.0, \nu=0.3, \rho=500, C_0=500\}$ and with ({\em top}) 
  $m=50$ ({\em bottom}) $m=0.001$. }
\label{f:alt}
\end{figure} 
The effect of these new dynamics may be readily seen in the {\em bottom}
panel of Figure \ref{f:alt} as the epidemic progresses. In this second
case (with small $m$), it is also very unlikely that any susceptible
individuals will be left at the end of an epidemic. This contrasts
with both the stochastic SIR model and these alternative dynamics with
$m$ large, as it is possible that, due to stochastic effects, the
infection will die out before all susceptibles have been exposed.

\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(out.alt, type = "params", showd = TRUE)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt-params}
\begin{Schunk}
\begin{Sinput}
> plot(out.alt2, type = "params", showd = TRUE)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt2-params}
\caption{Posterior parameter distributions for $b$, $k$, $\mu$, and
  $\nu$ when the simulated source epidemic follows the alternative
  transmission model with parameters $\{(a=0.1, \mu=0.0, \nu=0.3,
  \rho=500, C_0=500\}$ and with ({\em top}) $m=50$ ({\em bottom}) $m=0.001$. }
\label{f:alt-params}
\end{figure} 

It is interesting to examine the posterior distributions of the
parameters for these two cases (Figure~\ref{f:alt-params}) if we
assume that these underlying alternative dynamics are well
approximated by the simpler SIR model. As one might expect, the
estimates of the recovery and death rates in both cases are quite
similar, especially since these dynamics are the same as those
implemented within \pkg{amei}. However, the estimates of $b$ and $k$
are quite different in the two cases.  The 95\% credible intervals
hardly overlap, indicating that the two cases result in dynamics that
are quantitatively different.

We now move on to our task of examining how well the fixed and
adaptive strategies can do when faced with an epidemic which is
evolving according to a transmission function outside of the class of
SIR models (\ref{dS}--\ref{dD}) used to calculate the (optimal)
vaccination strategy. The first step here is to let the fixed strategy
``cheat''.  That is, we allow the fixed strategy to see a full
epidemic spread according to the true model and parameterization
without intervention, and estimate the best SIR model
approximation. To do this we use the \code{manage} function with a
\code{NULL} vaccination strategy and a \code{NULL} cost structure, and
default ``true'' parameters.
%
% this one with the tricky cost structure give interesting cost
% surface but isn't necessarily good for comparison
% costs <- list(vac=3.25, death=25, infect=1) 
% true<- list(a=0.035, mu=0.0075, nu=0.35, m=0.01, rho=1000, C=500) 
%
%
% the next is similar to above, but the adaptive one does quite a bit
% worse than the optimal fixed policy, and the epidemic takes longer
% (T=120) but the surface is pretty cool :-) The reason that the
% adaptive strategy does poorly is interesting -- in this case it's
% very important to vaccinate early, since the longer you wait, the
% larger the reservoir, and this reservoir sticks around to keep
% infecting individuals
% costs <- list(vac=3.25, death=40, infect=1)
% true<- list(a=0.08, mu=0.0075, nu=0.35, m=0.01, rho=20, C=5000)
%true<- list(a=0.06, mu=0.01, nu=0.3, m=0.15, rho=1000)
%formals(alt.epistep)$true <- true
%
\begin{Schunk}
\begin{Sinput}
> init <- list(S0 = 600, I0 = 1, R0 = 0, D0 = 0)
> time <- 80
> posterior <- manage(init, alt.epistep, NULL, NULL, 
+     T = time, bkrate = 100)
\end{Sinput}
\end{Schunk}
%Since $b$ and $k$ are anti-correlated, it helps to obtain more sample 
%these at a higher rate to ensure good mixing in the Markov chain.
\begin{figure}[ht!]
 \centering
\begin{Schunk}
\begin{Sinput}
> plot(log(posterior$samp$b), type = "l", main = "", 
+     ylab = expression(b))
\end{Sinput}
\end{Schunk}
\includegraphics[trim=10 20 10 50,scale=0.2]{amei-alt-mixing-b}
\begin{Schunk}
\begin{Sinput}
> plot(posterior$samp$k, type = "l", main = "", 
+     ylab = expression(k))
\end{Sinput}
\end{Schunk}
\includegraphics[trim=10 20 10 50,scale=0.2]{amei-alt-mixing-k}
\begin{Schunk}
\begin{Sinput}
> plot(posterior$samp$nu, type = "l", main = "", 
+     ylab = expression(nu))
\end{Sinput}
\end{Schunk}
\includegraphics[trim=10 20 10 50,scale=0.2]{amei-alt-mixing-nu}
\begin{Schunk}
\begin{Sinput}
> plot(posterior$samp$mu, type = "l", main = "", 
+     ylab = expression(mu))
\end{Sinput}
\end{Schunk}
\includegraphics[trim=10 20 10 50,scale=0.2]{amei-alt-mixing-mu}
\caption{Traces of samples from the posterior distribution of the parameters
  for the (mis-specified) SIR model.}
\label{f:alt-mixing}
\end{figure}
Figure~\ref{f:alt-mixing} shows trace plots of the samples obtained.
From the output we can extract the mean parameterization obtained
at the final time point of the epidemic.
\begin{Schunk}
\begin{Sinput}
> mean.params <- as.list(apply(posterior$samp, 2, 
+     mean))
\end{Sinput}
\end{Schunk}

Based on these parameters, and thus assuming an SIR model, we can
calculate the optimal static vaccination policy.
\begin{Schunk}
\begin{Sinput}
> costs <- list(vac = 2.5, death = 4, infect = 1)
> vacgrid <- list(fracs = seq(0, 1, 0.1), stops = seq(2, 
+     init$S0 - 50, 50))
> alt.optvac <- optvac(init, mean.params, vacgrid, 
+     costs, T = time)
> alt.best <- getpolicy(alt.optvac)
\end{Sinput}
\end{Schunk}
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(alt.optvac)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt-optvac}
\caption{The cost surface calculated for fixed strategies using parameters estimated from a single run of the alternative epidemic model.}
\label{f:alt-optvac}
\end{figure} 
The cost surface is shown in Figure~\ref{f:alt-optvac}.

Then, based on the extracted policy, we can use \code{MCepi} to
simulate the true evolution of the epidemic (via \code{alt.epistep})
many times in order to build up an understanding of the distribution
of costs of the optimal static policy calculated under the simplified
SIR parametrization.
\begin{Schunk}
\begin{Sinput}
> alt.vac.opt <- alt.best[3:4]
> alt.MCepi <- MCepi(init, alt.epistep, alt.vac.opt, 
+     costs, T = time)
\end{Sinput}
\end{Schunk}
\begin{Schunk}
\begin{Sinput}
> getcost(alt.MCepi)
\end{Sinput}
\begin{Soutput}
  q0.025     mean median   q0.975
1   1854 2027.563   2026 2216.088
\end{Soutput}
\end{Schunk}

Now for the comparison. We provide the \code{alt.epistep} function to
\code{MCmanage} in order to build up an understanding of how the
optimal adaptive strategy fares under a restricted SIR model. Here we
use the default, non-informative priors.
\begin{Schunk}
\begin{Sinput}
> alt.MCmanage <- MCmanage(init, alt.epistep, vacgrid, 
+     costs, T = time)
\end{Sinput}
\end{Schunk}
\begin{Schunk}
\begin{Sinput}
> getcost(alt.MCmanage)
\end{Sinput}
\begin{Soutput}
    q0.025     mean  median   q0.975
1 1829.112 2026.867 2039.25 2185.362
\end{Soutput}
\end{Schunk}
An alternative (and possibly fairer) approach would be to allow the
adaptive strategy to cheat as well by choosing the prior to be tightly
concentrated around around the \code{mean.params} estimated above.
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(alt.MCepi, showd = TRUE)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt-MCepi-t} 
\begin{Schunk}
\begin{Sinput}
> plot(alt.MCepi, type = "costs")
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt-MCepi-c}
\caption{Trajectories in terms of {\em top} the numbers of susceptibles,
  infecteds, recovereds, and dead individuals under the alternative
  model in Eq.~(\ref{eq:alt}) and {\em bottom} costs under the default 
parameterization under the optimal static vaccination policy.}
\label{f:alt-MCepi}
\end{figure} 
\begin{figure}[ht!]
\begin{Schunk}
\begin{Sinput}
> plot(alt.MCmanage, showd = TRUE)
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt-MCmanage-t} 
\begin{Schunk}
\begin{Sinput}
> plot(alt.MCmanage, type = "costs")
\end{Sinput}
\end{Schunk}
\centering
\includegraphics[trim=0 20 0 20]{amei-alt-MCmanage-c}
\caption{Trajectories in terms of {\em top} the numbers of susceptibles,
  infecteds, recovereds, and dead individuals under the alternative
  model in Eq.~(\ref{eq:alt}) and {\em bottom} costs under the default parameterization under the adaptive vaccination policy.}
\label{f:alt-MCmanage}
\end{figure} 
We can see by looking at the full trajectories in Figures
\ref{f:alt-MCepi} and~\ref{f:alt-MCmanage} that the strategies
implemented by both the fixed and adaptive methods are very
similar. The adaptive strategy can do better than the
fixed strategy, both in extreme and average cases. The adaptive framework
has the added benefit that prior information on similar epidemics
may be used, which typically results in a narrower spread of
(often lower) costs, leading to even further improvements over
fixed strategies---even ones which can see into the future.

We can also look back to Figure~\ref{f:alt-optvac} to see the kind of
improvement that the optimal strategy can have over the case when no
one is (or very few individuals are) vaccinated. We can extract the
expected cost under this worst fixed strategy and compare it to the
best fixed strategy directly.
\begin{Schunk}
\begin{Sinput}
> alt.worst <- getpolicy(alt.optvac, which = "worst")
> rbind(alt.best, alt.worst)
\end{Sinput}
\begin{Soutput}
      row col frac stop cost
best   11   1  1.0    2 2298
worst   4  11  0.3  502 4611
\end{Soutput}
\end{Schunk}
Here it is easy to see how much better using an optimal fixed
strategy with good parameter settings is compared to not
vaccinating. Since the adaptive strategy has comparable (or lower)
costs to the fixed strategy over many trials of the epidemic, we can
conclude that the adaptive strategy gives similar reductions in costs
over the \code{NULL} vaccination policy as the fixed policy does, even
though the adaptive algorithm must begin planning and implementing the
intervention with considerably less information than the fixed
strategy it is compared with here.

We have shown here that it is straightforward to build functions with
fairly simple extensions to the SIR model to use with the \pkg{amei}
package. Using one example, we can see that the model implemented
within \pkg{amei} to plan the interventions is flexible enough to
allow effective vaccination strategies, even when this model does not
match the one which is known to govern the true underlying epidemic
dynamics. We have also shown how one can estimate ``effective'' SIR
parameters from an alternative dynamic epidemic model. These estimates
can be used to build a fixed policy, or the posterior samples could
even be used as parameters to enhance the adaptive strategies.




\section{Discussion and related work}
\label{sec:discuss}

% {\bf This discussion is much different from what is in the PLOS paper
%   -- we could probably incorporate some of the points from that paper
%   here }

Our \pkg{amei} package for \proglang{R} implements a statistical framework
that enables concurrent estimation of epidemic parameters and optimal
intervention strategies. In particular, it allows parameter
uncertainty to be taken into consideration when planning an
intervention.

In the current implementation, we look for adaptive strategies of only
one type---vaccination of proportions of the population until
susceptibles fall below some threshold. However, if we were to instead
allow the fraction of the population targeted for vaccination to be a
function of future disease states we could regard Eq.~(\ref{sdp}) as a
stochastic iteration equation and use stochastic dynamic programming
\citep{clark00} to calculate the optimal intervention associated with a
set of parameter values. Such an approach may be useful for situations
in which knowledge of the disease state is available, but for whatever
reason sequential inference is not possible.  In the situation
considered here, in which the static strategy is sequentially updated
based on the current disease state and parameter estimates, the
adaptive strategy that emerges is flexible in that it consists of a
state--dependent sequence of target fractions, but does not involve the
additional computational burden associated with stochastic dynamic
programming.

As in many decision problems, it can be difficult to specify cost
functions that truly represent all aspects of the actions.  Although
\pkg{amei} is based on a very simple cost evaluation scheme, it is
straightforward to extend our approach to more complex cost functions,
including state-dependent costs.  We refer the reader to \cite{merl09} for
more detailed discussion of more complex cost assignments.  

If there is not much data observed during an epidemic (for instance if
the epidemic is occurring in a small population) then estimating
disease parameters within any framework could, of course, be
difficult. One could use data about previous epidemics to estimate
parameters, for instance by maximum likelihood methods, to plan an
intervention. However, as we showed in Section~\ref{sec:pnas}, should
these estimates be poor then the cost of intervening in the epidemic can
be greatly increased. By instead using previous epidemic data to
inform the priors within a Bayesian framework, such as the one we
describe here, we can take into account other research and
observations without risking the high costs inherent in using
incorrect parameter estimates for planning an intervention.

Recently other authors \citep[e.g.,][]{elderd06} have also proposed
using Bayesian methods to estimate disease parameters and propagate
uncertainty in parameter estimates through to an optimal of
vaccination strategy. For instance, \citet{elderd06},
considered a Susceptible--Exposed--Infectious--Recovered (SEIR) model
for a small pox epidemic with mass action infection dynamics, and
either mass or trace vaccination. However, in their approach
the likelihood requires numerical solutions of the system of DEs (which
can be computationally intensive).
%, and imposes a tenuous normality assumption.
\citet{ball02,ball07} have also written technical papers
on the optimal vaccination strategies in epidemics, but do not
consider adapting the policy over time. %{\bf Need to check this.}

Compared to these approaches, our proposed method has several
advantages.  Our negative binomial discretization of the SIR model is
much less computationally intensive, and could easily be expanded to
include an exposed class. Additionally, this model is comparatively
simple since it connects intuitively with a straightforward SIR model,
and is thus likely to be much more approachable to policy makers and
practitioners outside of statistics. Our approach is also adaptive,
allows a much more flexible cost framework, and is generalizable to
other types of interventions.  Since the intervention strategy
implemented by \pkg{amei} is iteratively updated upon the arrival of
new data, the vaccination schedule is inherently adaptive to the state
of the epidemic.  This allows significant changes in the vaccination
strategy mid-intervention.  This may be helpful, for example, in a
scenario where an intervention may be discontinued during a lull, a
subsequent refinement of parameter estimates or a surge of new
infections may dictate that the vaccination campaign should be
re-initiated.

Not only can more complicated infection dynamics be used for simulating
the dynamics of a disease, but in addition the method implemented in
\pkg{amei} can be modified to include more complicated disease
dynamics such as latent states or vector-communicated diseases, as
well as more complicated intervention strategies that allow
combinations of vaccination, quarantine, and culling.  Also note the
possibility of calculating policies based on a minimization of some
quantile of the realized cost rather than the mean cost.  This would
lead to a minimization of worst-case-scenarios, which may be useful in
practice.

There are other \proglang{R} packages that involve the simulation of
epidemics, and inference for models like SIR.  As far as we can tell,
ours is the only one which considers (on--line) adaptive management of
interventions.  However, two packages that take approaches similar to
ours for inferring the parameters governing an epidemic are worth
mentioning here.  \pkg{stochasticGEM} \citep{stochasticGEM:R} provides
Bayesian inference for partially observed stochastic epidemics.  The
implementation in this package also allows for estimating parameters
governing the infectious and incubation period length.  Several
variants of the general epidemic model are considered
\citep[e.g.,][]{gibson98,hohle05,oneill99,oneill01,oneill02,streftaris04},
including the stochastic SIR with Markovian and non--Markovian
infectious periods, and SEIR models. As with \pkg{amei}, estimation is
via MCMC.  H\"ohle, et al., whose methods are implemented in the
\pkg{stochasticGEM} package, have themselves released an \proglang{R}
package called \pkg{RLadyBug} \citep{RLadyBug:R}, which requires {\sf
  JAVA}. In \pkg{RLadyBug} maximimum likelihood and Bayesian inference
can be performed to estimate the parameters and provide
confidence/credible intervals and the ability to test hypotheses. The
package can also estimate parameters for epidemics which include
population heterogeneity. The authors of \pkg{RLadyBug} provide a nice
paper outlining the usage \citep{hohle07} with examples.  Both
\pkg{RLadyBug} and \pkg{stochasticGEM} contain interesting data sets
on epidemics and visualization tools.

There are several other \proglang{R} packages that allow for the
manipulation and analysis of epidemic and disease data.  They include
a few packages which are predominantly used for teaching purposes,
e.g.: \pkg{Epi} \citep{Epi:R} with methods for multiscale and censored
data; \pkg{epiR} \citep{epiR:R} focusing on veterinary epidemiology,
and \pkg{epibasix} \citep{epibasix:R}.  Two other packages that include
a range of statistical functionality for data manipulation and
inference are \pkg{epicalc}\citep{epicalc:R} and \pkg{epitools}
\citep{epitools:R}.

\appendix

\section{Implementation details}

At a high level, most of the functions and routines in the \pkg{amei}
package are written in \proglang{R}.  However, most of the sub--routines
implementing the Monte Carlo evaluation of costs for the various
vaccination strategies, which are obtained by repeatedly simulating
the epidemic forward in time given the current samples of the
parameter estimates, are written in \proglang{C} for speed
considerations.  There is significant scope for parallelizations of
these Monte Carlo routines, since each forward simulation is
independent of the next.  For this reason future versions of this
package may leverage \pkg{Pthreads} or \pkg{MPI} to obtain significant
speedups.

This document has been authored in \pkg{Sweave} (try
\code{help(Sweave)}).  This means that the code quoted throughout is
certified by \proglang{R}, and the \code{Stangle} command can be used
to extract it.  The demo available in this package will run the same
code via \verb!demo("amei")!.

% Note that this tutorial was not meant to serve as an instruction
% manual.  For more detailed documentation of the functions contained in
% the package, see the package help--manuals. At an \proglang{R} prompt, type
% \code{help(package=amei)}. PDF documentation is also available on the
% world-wide-web.
% \begin{center}
% \tt http://www.cran.r-project.org/web/packages/amei/index.html
% \end{center}

\subsection*{Acknowledgments}
This work was partially supported by NSF grant DMS 0310542 to MM, and
EPSRC grant \#EP/D065704/1 to RBG.  We thank two referees and 
an editor for thoughtful comments that lead to an improved paper.

%{\bf Can we think of anything else to put here? }

\bibliography{amei}
%\bibliographystyle{plain}

\end{document}
