Likelihood theory

Introduction¶

There are many different strategies for estimating parameters when given a sample $(X_{1},X_{2}, X_{3}, \cdots, X_{n})$, and a single relation of all the random variables in this sample---what we call a dataset ($\mathcal{D}$). Each strategy for estimating parameters has advantages and disadvantages. Up until this point we learned a single strategy called The Method of Moments. Lets explore a second, popular strategy for estimating parameters given a sample and a model called The likelihood approach.

The likelihood approach has as advantages that estimators are often simpler to compute than the method of moments, we can construct a straightforward method to compute the variance of estimators which is based on the curvature of a likelihood functions, and choosing parameters that "best fit" the data is an intuitive approach.

Probability of a dataset¶

Suppose that we are provided a single dataset $\mathcal{D} = (y_{1},y_{2},\cdots,y_{n})$, and we assume that this dataset was generated from a sample $(Y_{1},Y_{2},Y_{3},\cdots,Y_{n})$.

The Probability of our dataset---$P(\mathcal{D})$---is defined as

where we use a comma to denote the $\cap$ symbol. The probability of our dataset is the probability that random variable $Y_{1}$ generates the realized value $y_{1}$, $Y_{2}$ generates $y_{2}$ , and so on, all simultaneously.

If we assume that our sample is independent and identically distributed (i.i.d.) then we can simplify $P(\mathcal{D})$.

We need one fact first.

Conditional probability chains¶

Consider computing the probability of a sample of size three $(X_{1},X_{2},X_{3})$.

We can imagine that inside the event $X_{1}=x_{1} \cap X_{2}=x_{2} \cap X_{3}=x_{3}$ there exists two events.

Here, we partitioned the above event that contained three random variables into two events: one event that contains a single random variable and a second event that contains two random variables.

Now we can use an identity that we learned in the past to break this probability into two. For two events $A$ and $B$, recall

Then

We could apply the same technique above to the event $X_{2}=x_{2} \cap X_{3}=x_{3}$ and find that

The process of breaking a single probability into several probabilities in this way is sometimes called producing a conditional chain of probabilities.

Probability of an i.i.d. sample¶

Given an i.i.d. sample $(Y_{1},Y_{2},\cdots,Y_{n})$ the probability of this sample is

We can break this probability into several conditional probabilities.

But because we assume our sample is (mutually) independent then

To recap, the probability of a dataset generated by an i.i.d. sample is

Suppose an i.i.d. sample $(Y_{1},Y_{2},\cdots, Y_{n})$ generated a dataset $\mathcal{D} = (y_{1},y_{2},\cdots,y_{n})$. Further assume $Y_{i} \sim Bern(\theta)$ for $i=1$ to $N$. Then

The likelihood function¶

The probability of our dataset $P(\mathcal{D} )$ depends on a set of parameters $(\theta)$, and we can write $P(\mathcal{D} | \theta)$. Different choices for our parameter values will result in different values of $P(\mathcal{D} | \theta)$.

(Bernoulli example continued). Suppose that we are given the dataset $\mathcal{D} = (1,0,1,0,0,1,1)$ and we assume that these 7 datapoints were generated from an i.i.d. Bernoulli model. Then, as we saw above,

. For our dataset, we can fill in the $y_{i}$ values to find

We can now compute the probability of our data for different choices of $\theta$

The likelihood function, $\mathcal{L}$, is the probability of our observed data as a function of a set of parameter values

The difference between $P(\mathcal{D} | \theta)$ and $\mathcal{L}(\theta | \mathcal{D})$ is the argument. For $P(\mathcal{D} | \theta)$ we expect, given a set of fixed parameter values, to compute the probability of different datasets. For $\mathcal{L}(\theta | \mathcal{D})$ we expect, given a fixed, observed dataset $\mathcal{D}$, to compute the probability of $\mathcal{D}$ for different values of $\theta$.

The likelihood function allows us to investigate the plausability of different parameter values after we observed a dataset. The likelihood need not be a probability distribution, and often is not.

Suppose we observe for 7 days the number of patients who arrive at a specific hospital and are diagnosed with influenza. The data we collect is $\mathcal{D} = (10,13,4,17,34,11,3)$. We choose to consider our data a random sample $(X_{1},X_{2},\cdots,X_{7})$ where $X_{i} \sim Pois(\lambda)$. Then

If we consider the above as a function of $\lambda$---$\mathcal{L}(\lambda | \mathcal{D})$ then if $\mathcal{L}(\lambda_{1} | \mathcal{D})$ > $\mathcal{L}(\lambda_{2} | \mathcal{D})$ then the parameter value $\lambda_{1}$ may be a more plausible parameter value compared to $\lambda_{2}$.

One way to use the likelihood function is to attempt to find the most plausible parameter value, the value that maximizes the likelihood. However this value may be difficult to find. Lets explore one strategy for simplifying the search for this optimal parameter value and a second strategy that can help us compute this optimal value.

The logarithm and the log likelihood¶

With a likelihood function in hand, one aim may be to find the set of parameter values that maximizes this function---the set of parameter values that are more plausible given an observed dataset. However, the likelihood function $\mathcal{L}$ can in many cases be difficult to compute because it is a sequence of multiplications. Our aims is to find a related function $\ell \ell$ such that parameter values that maximize $\mathcal{L}$ also maximize this "easier to work with" $\ell \ell$ function.

A function $g$ is monotone increasing if for two values $x>y$ then $g(x) \geq g(y)$. If a monotone function $g$ is applied to a second function $f$, the values that maximize $f$ also maximize this new function.

Hypothesis: Suppose the value $x^{*}$ maximizes the function $f$. That is, for all $x$ the value $f(x^{*}) \geq f(x)$. Then if $g$ is a monotone increasing function then $x^{*}$ also maximizes $g(f(x))$.

Evidence: We know that $f(x^{*}) \geq f(x)$. $g$ is a monotone increasing function and so by definition for $x \geq y$, $g(x)>g(y)$. We can use as an input to $g$ the values $f(x^{*})$ and $f(x)$ to find that because $f(x^{*}) \geq f(x)$ then $g( f(x^{*}) ) \geq g( f(x) )$.

The above property of a monotone increasing function is useful if we can find such a function that simplifies our likelihood---a sequence of multiplications. The logarithm is a monotone increasing function (i.e. if $x \geq y$ then $\log(x) \geq \log(y)$ ), and the logarithm transforms multiplications (hard) into sums (easy).

The log likelihood function is the natural logarithm of the likelihood function

The loglikelihood simplifies the likelihood by transforming a large sequence of multiplications into sums.

There is one more principle about maximizing functions that will be useful. Suppose we find the value $(x^{*})$ that maximizes the function

then that value $x^{*}$ also maximizes $f(x)$. Also, suppose we find the value $(x^{*})$ that maximizes the function

then that value $x^{*}$ also maximizes $f(x)$.

The above two principles state that if we are interested in finding the maximum value of a function $g$ then we can consider a simpler function $f$ which does not contain unrelated constants.

The value $x^{*}$ that maximizes the function $g(x) = -x^{2} + 3$ is the same value that maximizes the function $f(x) = -x^{2}$.

The value $x^{*}$ that maximizes the function $g(x) = \frac{e^{-x^{2}}}{4}$ is the same value that maximizes the function $f(x) = e^{-x^{2}}$.

We found in the previous section that the likelihood for the number of admitted patients with influenza was

The log likelihood of the above is, the simpler,

We see that the last term of factorials does not involve our parameter value $\lambda$. This term is then a constant and we could instead aim to find the maximium of the simpler function

In some cases, we may be able to find the parameter value, or set of parameter values, that maximize the log likelihood function. To explore this topic we will need to briefly learn about the derivative of a continuous function.

Just in time derivatives¶

The derivative describes the tangential slope of a function at a point and can serve as a linear approximation to a more complicated function. We will see how we can use the derivative to setup and solve an equation that will return a parameter value that maximizes the log likelihood. This estimate of our parameter will be called a maximum likelihood estimate.

Sequences and limits¶

A sequence is defined as a function whose domain is the natural numbers $\mathbb{N} = \{1,2,3,4,5,6, \cdots \}$. Often as annotate the value associated with the 1, $f(1)$ as $a_{1}$, the value associated with 2, $f(2)$, as $a_{2}$, and so on.

The following is a sequence $f(x) = 2^{-x}$ for $x \in \mathbb{N}$.The first few values of this sequence are $f(1) = a_{1} = 1/2$, $f(2) = a_{2} = 1/4$, $f(3) = a_{3} = 1/8$, and so on. We can also describe the above sequence using this notation $\{ 2^{-n} \}_{n=1}^{\infty}$.

Intuitively, we may wish to know how a sequence behaves as it is enumerated. A sequence may increase to infinity, decrease to negative infinity, oscillate or have some other periodic behavior, or a sequence may move closer and closer to a single value. If a sequence ${a_{n}}_{n=1}^{\infty}$ gets closer and closer to a single value $l$ then we say that the sequence has a limit and that limit is the value $l$.

The more formal definition of a limit is the following: For a sequence $\{a_{n}\}_{n=1}^{\infty}$, given any value $\epsilon >0$ the value $l$ is a limit if you can find an integer $N$ such that $|a_{N} - l| < \epsilon$ for integers greater than $N$ (i.e. $N+1$, $N+2$, $\cdots$).

We may also define a limit $L$ of a function at the value $a$ if given a sequence of values $a_{1}, a_{2}, \cdots a_{n}$ that approach the value $a$, the sequence $f(a_{1}), f(a_{2}), f(a_{3}), \cdots$ approaches the value $L$. We call $L$ the limit of $f$ at the value $a$ and write

Intuitively, the above says that as the input of our function approaches the value $a$, the function values approach the value $L$.

To solve our problem of finding the parameter value that maximizes the log likelihood, we need to learn about a special type of limit called the derivative.

Derivative of a function¶

The derivative of a function at the value $a$, or $f'(a)$, is

For intuition, we can imagine building a sequence

that approaches the value $a$ and computing the associated sequence

One common sequence that approaches $a$ is the sequence $a + \{h_{n}\}_{n=1}^{\infty}$ and where the sequence $h$ approaches 0. Then the definition of the derivative would be

This is because as the sequence $\{h_{n}\}$ approaches zero, the sequence $\{a+h_{n}\}$ approaches the value $a$.

Let $g(x) = x^{2}$. The derivative of $g$ at the value 1 is by definition

For any value $c$ we find that the derivative of $g$ is

In some sense, the derivative $g'(c)$ too is a function, the function $g'(c) = 2c$.

When the derivative equals zero¶

When the derivative of a function at the point $x^{*}$ is 0 the point $x^{*}$ is either a minimum or a maximum value. This is the key to finding a parameter value that maximizes our log likelihood. If we can compute the derivative of our log likelihood $\ell \ell'(\theta)$ and find the value $\theta$ such that $\ell \ell' = 0$ then we know we have found either a parameter value that minimizes or maximizes our log likelhood, which then minimizes or maximizes our likelihood, which suggests that this parameter value is the most plausible value given our observed data. Phew.

Lets prove that when the derivative equal zero then we are at a maximum. Hypothesis: If $f(x^{*}) > f(x)$ for any possible input $x$ then $f'(x^{*})=0$.
Let us build a two sequences, one sequence $\{x_{1}, x_{2}, \cdots \}$ that approaches $x^{*}$ such that all the values are smaller than $x^{*}$ and a second sequence $\{y_{1}, y_{2}, \cdots \}$ such that all values are larger than $x^{*}$.

If we choose any $x_{n}$ in the sequence, the following must be true:

Can't use function '$' in math mode at position 79: …q 0 & \;\; &bc $̲f(x^{*)$ is a m…

\begin{aligned}
    \frac{f(x_{n}) - f(x^{*})}{x_{n}-x^{*}} \geq 0 & \;\; &bc $f(x^{*)$ is a max and } x_{n}< x^{*} \\ 
\end{aligned}

If we choose any $y_{n}$ in the sequence, the following must also be true:

Can't use function '$' in math mode at position 79: …q 0 & \;\; &bc $̲f(x^{*)$ is a m…

\begin{aligned}
    \frac{f(y_{n}) - f(y^{*})}{y_{n}-x^{*}} \leq 0 & \;\; &bc $f(x^{*)$ is a max and } y_{n}>y^{*} \\ 
\end{aligned}

If we assume that there exists a derivative value $f'(x^{*})$ then that value must be greater than or equal to zero and also less than or equal to zero. The only options then is $f'(x^{*}) = 0$.

derivative "rules"¶

There are several rules that can make computing derivatives of complicated functions easier.

• Constant Rule: if $f(x) = c$ where $c$ is a constant then $f'(x) = 0$

  • Example: if $f(x) = 72.67$ then $$\begin{aligned} f’(x) &= 0

    \end{aligned}$$

• Power Rule: if $f(x) = x^{n}$ then $f'(x) = nx^{(n-1)}$

  • Example: if $f(x) = x^{3/2}$ then $f'(x) = \frac{3}{2}x^{3/2-1} = \frac{3}{2}x^{1/2}$

• Exp Rule: if $f(x) = e^{x}$ then $f'(x) = e^{x}$

• log Rule: if $f(x) = \log(x)$ then $f'(x) = \frac{1}{x}$

• Addition Rule: if $f(x) = g(x) + h(x)$ then $f'(x) = g'(x) + h'(x)$

  • Example: if $f(x) = x^{2} + \frac{1}{3} x^{-1}$ then $$\begin{aligned} f’(x) &= \left(x^{2}\right)’ + \left( \frac{1}{3} x^{-1} \right)‘\ f’(x) &= 2x^{1} + \frac{-1}{3} x^{-2}

    \end{aligned}$$

• "Chain" rule: if $f(x) = g( h(x) )$ then $f'(x) = g'(h(x)) \cdot h'(x)$

  • Example:  Suppose $f(x) = (x+2)^{2}$ then g(h(x)) = $h(x)^{2}$ and $h(x) = x+2$. $$\begin{aligned} h’(x) &= 1\ g’(h(x)) &= 2 h(x)

    \end{aligned}

    $$Therefore$$

    (32)

    \begin{aligned} f’(x) &= g’(h(x)) \cdot h’(x)\ &= 2 h(x) \cdot 1\ &= 2(x+2)

    \end{aligned}$$

  • Example:  Suppose $f(x) = \sqrt{ \log(x)}$ then g(h(x)) = $h(x)^{1/2}$ and $h(x) = \log(x)$. $$\begin{aligned} h’(x) &= \frac{1}{x}\ g’(h(x)) &= \frac{1}{2}h(x)^{-1/2}

    \end{aligned}

    $$Therefore$$

    (33)

    \begin{aligned} f’(x) &= g’(h(x)) \cdot h’(x)\ &= \frac{1}{2}h(x)^{-1/2} \cdot \frac{1}{x}\ &= \frac{\log(x)^{-1/2}}{2x} \ &= \frac{1}{2x \sqrt{\log(x)}}

    \end{aligned}$$

Computing the maximum likelihood estimator¶

Given a i.i.d sample $(X_{1},X_{2},\cdots,X_{n})$ such that $X \sim f(x | \theta)$ we can compute a maximum likelihood estimator of the parameter $\theta$ by following these steps: (i) form the likelihood function (ii) convert the likelihood to a log likelihood (iii) compute the derivative of the log likelihood $\ell \ell ' (\theta)$ (iv) find the parameter value $\theta$ such that $\ell \ell ' (\theta) = 0$.

Given $(X_{1},X_{2},X_{3}, \cdots, X_{n})$, the distribution for each $X_{i} \sim Bern(\theta)$, and the dataset $\mathcal{D} = (0,1,1,0,1)$ we can compute the maximum likelihood estimate for $\theta$.

1. The likelihood of $\mathcal{D}$ is $$\begin{aligned} \ell(\theta) &= f(0) \cdot f(1) \cdot f(1) \cdot f(0) \cdot f(1)\ &= \left(\theta^{x_{1}} (1-\theta)^{1-x_{1}}\right) \left(\theta^{x_{2}} (1-\theta)^{1-x_{2}}\right) \left(\theta^{x_{3}} (1-\theta)^{1-x_{3}}\right) \left(\theta^{x_{4}} (1-\theta)^{1-x_{4}}\right) \left(\theta^{x_{5}} (1-\theta)^{1-x_{5}}\right)\ &= \left(\theta^{0} (1-\theta)^{1-0}\right) \left(\theta^{1} (1-\theta)^{1-1}\right) \left(\theta^{1} (1-\theta)^{1-1}\right) \left(\theta^{0} (1-\theta)^{1-0}\right) \left(\theta^{1} (1-\theta)^{1-1}\right)\ &= \left( 1-\theta \right) \left(\theta \right) \left(\theta \right) \left( 1-\theta\right) \left(\theta \right)\ &= \theta^{3} (1-\theta)^{2}

   \end{aligned}$$

2. Compute the log likelihood $$\begin{aligned} \ell \ell (\theta) &= \log \left[ \ell(\theta) \right] \ &= \log \left[ \theta^{3} (1-\theta)^{2} \right] \ &= \log \left[ \theta^{3}] + \log \left[ (1-\theta)^{2} \right] \ &= 3 \log \left[\theta] + 2 \log \left[ (1-\theta)\right] \

   \end{aligned}$$

3. Compute the derivative of the log likelihood $$\begin{aligned} \ell \ell (\theta)’ &= \left( 3 \log [\theta] + 2 \log [ (1-\theta)] \right)’ \ &= \left( 3 \log [\theta] \right)’ + \left(2 \log [ (1-\theta)] \right)’ & Addition rule\ &= 3 \left( \log [\theta] \right)’ + 2 \left( \log [ (1-\theta)] \right)’ & Constants rule\ &= 3 \frac{1}{\theta} + 2 \left( \log [ (1-\theta)] \right)’ & Derivative of log\ &= 3 \frac{1}{\theta} + 2 \left( \log [ z ] \right)’ & let z=(1-\theta)\ &= 3 \frac{1}{\theta} + 2 \frac{1}{z} (z)’ & Chain rule \ &= 3 \frac{1}{\theta} + 2 \frac{1}{1-\theta} (1-\theta)’ & Replace z \ &= 3 \frac{1}{\theta} + 2 \frac{1}{1-\theta} (1)’ - (\theta)’ & Addition rule \ &= 3 \frac{1}{\theta} + 2 \frac{1}{1-\theta} 0 - (\theta)’ & Derivative of a constant \ &= 3 \frac{1}{\theta} + 2 \frac{1}{1-\theta} (-1) & Derivative of \theta \ &= 3 \frac{1}{\theta} - 2 \frac{1}{1-\theta}\ &= \frac{3}{\theta} - \frac{2}{1-\theta}

   \end{aligned}$$

4. Set $\ell \ell(\theta)' = 0$ and solve for $\theta$ $$\begin{aligned} \ell \ell ’ (\theta) &= \frac{3}{\theta} - \frac{2}{1-\theta} = 0\ \frac{3}{\theta} &= \frac{2}{1-\theta}\ 3 - 3\theta &= 2 \theta \ 3 &= 5 \theta \ \theta &= 3/5\ \hat {\theta} &= 3/5

   \end{aligned}$$

   For our observed data $\mathcal{D}$ and our assumed model that the random variables generating each data point are i.i.d and distributed $Bern(\theta)$ our maximum likelihood estimate for $\theta$ is $\hat{\theta}_{mle} = 3/5$.

Given $(Y_{1},Y_{2},Y_{3}, \cdots, Y_{n})$, the distribution for each $Y_{i} \sim Geom(p)$, and the dataset $\mathcal{D} = (7, 19, 1, 11)$ we can compute the maximum likelihood estimate for the parameter $p$.

1. The likelihood of $\mathcal{D}$ is $$\begin{aligned} \ell(p) &= f(9) \cdot f(19) \cdot f(1) \cdot f(11)\ &= p(1-p)^{8} \cdot p(1-p)^{18} \cdot p(1-p)^{0} \cdot p(1-p)^{10}\ &= p^{4} (1-p)^{8+18+0+10}\ &= p^{4} (1-p)^{36}\

   \end{aligned}$$

2. Form the log likelihood $$\begin{aligned} \ell \ell(p) &= \log\left[ \ell(p) \right] \ &= \log \left[ p^{4} (1-p)^{36} \right] \ &= \log \left[ p^{4} \right] + \log \left[(1-p)^{36} \right] \ &= 4\log \left[ p \right] + 36\log \left[(1-p) \right]

   \end{aligned}$$

3. Compute the derivative of the log likelihood $$\begin{aligned} \ell \ell’(p) &= 4\frac{1}{p} + 36 \log( h(x) ) where h(x) = 1-p\ &= \frac{4}{p} + 36\frac{1}{h(x)} \cdot -1\ &= \frac{4}{p} - 36\frac{1}{1-p} \ &= \frac{4}{p} - \frac{36}{1-p}

   \end{aligned}$$

4. Set the derivative equal to zero and solve for the parameter $$\begin{aligned} \ell \ell’(p) &= \frac{4}{p} - \frac{36}{1-p} = 0 \ &= \frac{ 4p(1-p) }{p} - \frac{36 p(1-p)}{1-p} = 0 \ &= 4(1-p) - 36 p = 0 \ &= 4 - 4p - 36 p = 0 \ &= 4 = 40p \ p &= 4/40 = 1/10

   \end{aligned}$$

   For our observed data $\mathcal{D}$ and our assumed model that the random variables generating each data point are i.i.d and distributed $Geom(p)$, our maximum likelihood estimate for $p$ is $\hat{p}_{mle} = \frac{1}{10}$.

Fisher information¶

A maximum likelihood estimate for a parameter describes what single parameter value most probably generated our observed dataset. However, intuitively, if we had a collected a different dataset of the same size then we would likely have arrived at a different maximum likelihood estimate. Repeating the steps: (i) generate dataset of same size, (ii) compute maximum likelihood estimate, (iii) store estimate would lead to many estimates of the same model. To better characterize our model, we will need in addition to a single maximum likelihood estimate a measure of how that estimate may vary.

The Fisher Information $\mathcal{I}(\theta)$ about a parameter $\theta$ is defined as

the intuition behind the Fisher information is that log likelihood function that are very peaked around their maximum likelihood estimate will have large Fisher information values. On the contrary, if a log-likelihood is very flat near it’s maximum likelihood estimate then the Fisher information will be small.

In some sense, the Fisher information describes the curvature of the log likelihood around the maximum likelihood estimate and that curvature is also related to the variability of our estimate.

We will not explore this topic in class, but it is often easier to compute an equivalent definition of the Fisher information

This definition says that in order to compute the Fisher information we will need to compute the second derivative of the log likelihood, compute the expedctation, and then multiply that second derivative by negative one.

Lets look at an example of using this alternative definition to compute the Fisher information for a Bernoulli model.

Suppose a sample $(X_{1},X_{2},X_{3},\cdots,X_{n})$ that we assume $X_{i} \sim Bernoulli(\theta)$. Then the log likelihood for a single data point is

and the derivative of the log likelihood is

To compute the second derivative, we take the derivative of $\ell'(\theta)$.

The above if the second derivative of the log likelihood for a single datapoint. However that data point was generated from a random variable $X_{1}$ and so we can then treat $\ell''(\theta)$ as a transformation of the random variable $X_{1}$.

The above transformation is a complicated function, but the fisher information only requires that we understand as single quantity from this transformation---the expected value.

The final steps asks that we take the negative of the quantity above

Fisher for a sample of size n¶

The Fisher information for a sample of size n $(X_{1},X_{2},\cdots, X_{n})$ is equal to

Intuitively, a sample of size 8 is twice as informative as a sample of size 4.

CLT-like thm that involves the Fisher information¶

An important theorem that will allow us to construct confidence intervals and hypothesis tests about our parameters states the following:

where $\hat{\theta}$ is our maximum likelihood estimate, $\theta$ is our true, and unknown, parameter, and $\mathcal{I}$ is the information about $\theta$.

This theorem

Homework¶

1. Suppose that we collected a dataset with a single observation $\mathcal{D} = \{ 12 \}$. Please compute the probability of $\mathcal{D}$ if we assume our data was generated by a random variable $X$

   1. Such that $X \sim Geom(p)$. Assume we know the parameter value $p$.

   2. Such that $X \sim Pois(\lambda)$. Assume we know the parameter value $\lambda$.

   3. Such that $X \sim \mathcal{N}(\mu,\sigma^{2})$. Assume we know the parameter values $(\mu, \sigma^{2})$.

2. Suppose that we collected a dataset with a two observations $\mathcal{D} = \{ 12, 2 \}$. Please compute the likelihood function for $\mathcal{D}$ if we assume our data was generated by a random variable $X$

   1. Such that $X \sim Geom(p)$. Assume we know the parameter value $p$.

   2. Such that $X \sim Pois(\lambda)$ Assume we know the parameter value $\lambda$.

   3. Such that $X \sim Binom(20, \theta)$. Assume we know the parameter value $\theta$.

3. Suppose that we collected a dataset with a $n$ observations $\mathcal{D} = \{ x_{1}, x_{2}, \cdots, x_{n} \}$. Please compute the log likelihood if we assume our data was generated by a random variable $X$

   1. Such that $X \sim Geom(p)$. Assume we know the parameter value $p$.

   2. Such that $X \sim Pois(\lambda)$ Assume we know the parameter value $\lambda$.

   3. Such that $X \sim Binom(N, \theta)$. Assume we know the parameter value $\theta$.

   4. Such that $X \sim \mathcal{N}(\mu,\sigma^{2})$. Assume we know the parameter values

4. Suppose you collect a dataset $\mathcal{D} = (x_{1},x_{2},\cdots,x_{n})$ that you assume is generated from an i.i.d. sample $X_{1}, X_{2},\cdots, X_{n}$, where $X_{i} \sim f(x | \theta)$ and $\theta$ is a parameter. You decide to compute the loglikelihood $$\begin{aligned} \ell \ell (\theta | \mathcal{D}) = \sum_{i=1}^{n} \log \left [ f(x_{i} | \theta) \right]

   \end{aligned}

   Can't use function '$' in math mode at position 48: …ding the value $̲\theta^{*}$ tha…
   
   Because we are interested in finding the value $\theta^{*}$ that maximizes $\mathcal{\ell \ell}$ then multiplying the log likelihood by a constant will not change the optimal $\theta^{*}$

   \begin{aligned} \frac{1}{n} \ell \ell (\theta | x_{1}, x_{2}, \cdots, x_{n}) = \frac{1}{n} \sum_{i=1}^{n} \log \left [ f(x_{i} | \theta) \right]

   \end{aligned}$$

   1. The law of large numbers states that the above quantity will approach?

5. Suppose that we are asked to help better understand the burden of influenza circulating in in the state of Pennsylvania. Over the course of 6 weeks we collect the following number of cases of reported influenza $\mathcal{D} = (143, 12, 124, 56, 66)$.

   1. Please propose a model for this data and a short description for why you chose this model.

   2. Please compute the likelihood for your model parameters given the data

   3. Please compute the log likelihood.

6. Consider the sequence $a_{n} = 2^{-n}$ for $n=1$ to $\infty$.

   1. Write down the first 3 items in this sequence

   2. Please compute

      $$\lim_{n \to \infty} a_{n}$$

      (45)

7. Does the infinite sequence $\{-1,1,-1,1,-1,1, \cdots \}$ have a limit? Why or why not?

8. Consider the finite sequence $a_{n} = 2^{-n}$ for $n=1$ to 100, a sequence that ends at 2^-100.

   1. Does this sequence have a limit point? Why or why not?

9. Compute the derivative of

   1. $f(x) = e^{2x}$

   2. $f(x) = \log(\frac{1}{2x})$

   3. $f(x) = x^{2} + e^{-x}$

   4. $f(x) = 10$

   5. $f(x) = -3x$

   6. $f(x) = \frac{1}{2}x^{2}$

   7. $f(x) = \log(x)$

   8. $f(x) = \log(x^{2})$

   9. $f(x) = \log(10)$

10. Consider the function $g(x) = -x^{2} + 3$.

    1. Compute the derivative of $g$

    2. Does the $x$ value that maximizes $g$ not depend on the value of $c$? Why or why not?

11. Consider the function $g(x) = f(x) + c$ where $c$ is a constant.

    1. Compute the derivative of $g$

    2. Does the $x$ value that maximizes $g$ depend on the value of $c$? Why or why not?

12. Consider the function $g(x) = cf(x)$ where $c$ is a constant.

    1. Compute the derivative of $g$

    2. Does the $x$ value that maximizes $g$ depend on the value of $c$? Why or why not?

13. Assume a set of random variables $Z_{1}, Z_{2}, \cdots, Z_{n}$ such that $Z_{i} \sim exp(\beta)$ generated an observed dataset. An exponential distribution assign a density over continuous numbers from 0 to infinity. The probability density function is $$\begin{aligned} f(z) = \beta e^{-\beta z}

    \end{aligned}$$Suppose we collect a dataset$\mathcal{D} = (2,4,0.5,3)$.

    1. Please compute the log likelihood of an arbitrary dataset $\mathcal{D} = (z_{1},z_{2},z_{3}\cdots,z_{n})$

    2. Compute the derivative of the log likelihood above

    3. Set the derivative of the log likelihood equal to zero and solve for $\beta$ (ie find the parameter value that maximizes the log likelhiood).

    4. Apply the maximum likelihood estimate you found to the dataset $\mathcal{D} = (2,4,0.5,3)$.

14. Assume a set of random variables $Y_{1}, Y_{2}, \cdots, Y_{n}$ such that $Y_{i} \sim Geom(p)$ generated an observed dataset $\mathcal{D}$. Please compute the maximum likelihood estimate for $p$. Include the log likelihood, derivative, and solution when setting the derivative to zero:

    1. Given a dataset $\mathcal{D} = \{ 12, 1, 0, 3, 8 \}$

    2. Given a dataset $\mathcal{D} = \{y_{1}, y_{2}, \dots, y_{n}\}$

15. Assume a set of random variables $A_{1}, A_{2}, \cdots, A_{n}$ such that $A_{i} \sim Bern(\theta)$ generated an observed dataset. Please compute the maximum likelihood estimate for $p$. Include the log likelihood, derivative, and solution when setting the derivative to zero:

    1. Given a dataset $\mathcal{D} = \{ 0,0,0,0 \}$

    2. Do you think the estimate you computed above is reasonable? Why or why not?

    3. Given a dataset $\mathcal{D} = \{a_{1}, a_{2}, \dots, a_{n}\}$

16. Assume a set of random variables $A_{1}, A_{2}, \cdots, A_{n}$ such that $A_{i} \sim Pois(\lambda)$ generated an observed dataset. Please compute the maximum likelihood estimate for $p$. Include the log likelihood, derivative, and solution when setting the derivative to zero:

    1. Given a dataset $\mathcal{D} = \{ 11,12,2,10 \}$

    2. Given a dataset $\mathcal{D} = \{a_{1}, a_{2}, \dots, a_{n}\}$

17. Assume that we model a sample $(X_{1},X_{2},\cdots,X_{n})$ with a Poisson distribution ($X \sim Pois(\lambda)$). Please derive the Fisher information $\mathcal{I}(\lambda)$

18. The number of influenza A and influenza B cases each week is recorded by the Pennsylvania Department of health. For the year 2022 the number of cases in Northampton county, PA is

    Flu A Flu B

    ---

    100      0
    89       0
    10      10
    45      12
    12      98

    Please pose a model for the number of influenza A cases, and compute the maximum likelihood estimate for any parameters in this model. Explain why you chose your particular model and explain parameter estimates. In addition, use the Fisher information about your parameter to build a 95CI around this estimated parameter value.