Sets, Outcomes, Experiments, Probability and other perfect things

Probability Lingo¶

We define a set as a collection of items, sometimes called elements. A set is typically given a capital letter (for example $A$) and the elements are included inside curly braces.

Here, we define three sets:

The elements inside a set are unordered and do not necessarily need to be numbers. A set can contain any object.
The set $A$ could have been defined as $\{a,b,c\}$ or $\{c,b,a\}$.
We use the symbol $\in$ to express that an element “is a member of” a set.
For example, $a \in A$ can be read “the element $a$ is a member of the set $A$” or “the element $a$ is in the set $A$.”
We can also communicate when an element is not a member of a set with the symbol $\notin$.
For example, $c \notin C$ is read “the element $c$ is not a member of (not in) the set $C$.”

Generating sets:
We can define a set by enclosing in brackets each individual element. $S = \{a,b,c,10\}$.
However, some sets may be easier to build with properties.
A property is a function from elements to either true or false.
For example, we could define the property:

To build a set that contains elements with a specific property, we write:

This is read “the elements $x$ such that $x$ is an even integer.”
The vertical bar replaces the words “such that.”

Two sets are equal if they contain the same elements.
In other words, if for an element $x$, $x \in A$ implies that $x \in B$ and if for an element $y$, $y \in B$ implies that $y \in A$, then the set $A$ and set $B$ are equal.
If a set $A$ and set $B$ are equal, we write $A = B$.
If two sets do not contain the same elements, then they are unequal and we write $A \neq B$.

Let’s look at an example:

Above, $A = B$ but $A \neq C$ and $B \neq C$.

A set $A$ is a subset of a second set $B$ if all of the elements in $A$ are also elements in $B$.
We can say that $A$ is a subset of $B$ if every element in $A$ implies that element is in $B$, or $x \in A$ implies $x \in B$.
We write $A \subset B$ to denote that $A$ is a subset of $B$.
To denote that a set $A$ is not a subset of $B$, we write $A \not\subset B$.
In the example above, the set $C$ is a subset of $A$ $(C \subset A)$, but $A$ is not a subset of $C$ $(A \not\subset C)$.

We can build new sets by operating on two or more existing sets.

Set intersection takes as input two or more sets, $A$ and $B$, and returns a set that contains all the elements that are in both $A$ and $B$.
We use the symbol “cap” to denote set intersection: $A \cap B$ and often say “A intersect B.”

For the above sets $A$ and $C$, their intersection is:

because the element $c$ belongs to both set $A$ and set $C$ and because the element $b$ belongs to $A$ and $C$.

We can intersect more than two sets.

Let $A = \{a,b,10,12\}$, $Z = \{b,10,-1\}$, $U = \{a,b,c,d,10\}$.
Then:

and

If $A$ is a subset of $B$ $(A \subset B)$, then the only elements both sets have in common are those in $A$.
The intersection of $A$ and $B$ is then:

Set union takes as input two or more sets and outputs a set that contains all the elements that belong to the first set or the second set or the third set, and so on.
We use the “cup” symbol to denote set union.

As an example, consider the sets $A$, $Z$, and $U$ to be the same as above.
Then $A$ union $Z$ is:

This is because each of the above elements belongs to at least one of the sets $A$ and $Z$.

As another example:

Intersection and union are the most common set operations.

Before we define the set operation complement, we need to discuss two special sets.

The universal set is the set of all elements.
We denote the universal set with a $\mathcal{G}$.
The set which contains no elements is called the empty set, and we denote the empty set as $\emptyset$.

If a set $A$ and set $B$ have no elements in common, then $A \cap B = \emptyset$.
We often say that the sets $A$ and $B$ are disjoint.

For example:

Because the intersection between $A$ and $Q$ is empty, these sets are disjoint.

Now for our final set operation — complement.
The set complement takes as input a single set, $A$, and outputs a set with elements that are members of $\mathcal{G}$ but are not members of $A$.
We denote set complement in one of two ways: $A^{c}$ or $A'$.

Set operations example¶

Let’s look at an example of how these sets and set operations work.
Define the universal set, $\mathcal{G}$, to be the set of all positive integers (i.e., $1,2,3,4,\ldots$).
Let’s define the sets:

$A_{1} = \{1\}, A_{2} = \{1,2\}, A_{3} = \{1,2,3\}, \ldots, A_{n} = \{1,2,3,4,\ldots,n\}$.

Note that $A_{3} \cap A_{8}$ is not the value 3, but the set $\{3\}$.

The examples above suggest a pattern for any set $A_{i}$ and $A_{j}$ where $i \leq j$:

Lets look at set union in this example

and we see the following pattern for $i \leq j$

Because we defined a universal set, we can look at set complement

Applying set theory to probability¶

Foundation¶

The ideas about sets and set operations are the foundations for how we think about probability, about experiments, and hypotheses. We only need to recast the above set ideas to results from an experiment.

We use $\mathcal{G}$ to define the set of all possible outcomes from an experiment and call this set the sample space. The term “experiment” has a broad meaning. An experiment can mean everything from a randomized controlled trial to an observational study. Here an experiment is the process that generates outcomes.

An outcome is defined as an element of the sample space. An outcome is a single observation from an experiment, and we define an event as a set of outcomes. Most often we use $o_{i}$ to denote an outcome and $E_{i}$ to denote an event.

The probability of an event $E$ is a mapping from the set $E$ to a number between, or equal to, the values 0 and 1. We use $P(E)$ to denote the probability of event $E$. We require that the probabilities assigned to individual sets consisting of a single element in $E$ add up to the probability of $E$. Lets suppose there are $n$ outcomes in the event $E$. Then

We further require $P(\mathcal{G}) = 1$. In other words, the probability that something happens is certain. Note that we do not need to describe how we assign probabilities to events, we only describe what values we expect those numbers to be.

Lets further detail relationships between probabilities of sets that we would expect.

1. If $A \subset B$ then $P(A) \leq P(B)$ \label{subsetrule}.

2. $P(A \cup B) \leq P(A) + P(B)$ (why?)

3. $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

4. If $A$ and $B$ are disjoint then $P(A \cup B) = P(A) + P(B)$

The following three axioms (knowledge we assume true without proof) are called Kolmogorov’s Axioms.

1. For any event $E$, $P(E) \geq 0$

2. $P(\mathcal{G}) = 1$

3. If $E_{1}$ and $E_{2}$ are disjoint then $P(E_{1} \cup E_{2}) = P(E_{1}) + P(E_{2})$

Example: When we want to compute the probability that an outcome fall in any one of the events, say $E_{1}, E_{2},E_{3}$, knowing that these events have no outcomes in common, that they are disjoint, makes the computation easy. An intuitive way to think about events that are disjoint is if they cannot all occur at the same time. Suppose we wish to study the prevalence of stroke among patients who are older than sixety-five. Strokes can be categorized into transient ischemia attacks (TIA), ischemic stroke (IS), and hemorrhagic stroke (HS). Further lets assume the following probabilities for each event: $P(\{\text{TIA}\}) = 0.15$, $P(\{\text{IS}\}) = 0.75$, $P(\{\text{HS}\}) = 0.10$, and that these events cannot occur simultaneously. Then the probability of the event $E=\{TIA,HS\}$ can be broken into the union of two disjoint events $E = \{TIA\} \cup \{HS\}$ and we can use what we know about these disjoint events to compute the probability of $E$

The principle of equally likely outcomes---one way to assign probabilities¶

There are many different ways to assign probabilities to events. In this text, we will only consider using the principle of equally likely outcomes to assign probabilities. The principle of equally likely outcomes (PELO) states that the probability we assign to every event which contains a single outcome, $E_{i} = \{o_{i}\}$, in a sample space $\mathcal{G}$ is equal.

We can use this principle to assign probabilities first to every individual outcome and then to arbitrary events. Assume the PELO is true, then

The PELO tells us that the probability of an event with a single outcome is equal to one divided by the total number of outcomes in the sample space. The PELO also tells us that for a sample space with $n$ elements the probability of an event $E = \{o_{1},o_{20},o_{4}\}$ that contains three outcomes is $P(E) = \frac{3}{20}$.

An example when, and when not, the PELO applies.

PELO only works when we expect each outcome in our sample space to be equally likely.\

Example 1 Suppose our experiment is a coin toss and we will observe whether or not the coin lands heads up (H) or tails up (T). If we assume the coin has not been altered in any way, then PELO applies. Our sample space is $\mathcal{G} = {H,T}$. Because there are two outcomes in our sample space, $P(\{H\}) = 1/2$ and $P(\{T\}) = 1/2$.

Example 2 Suppose for our experiment we observe for one year a single patient who was admitted to the hospital because of an influenza infection and we plan to observe if that patient returns to the hospital because of a second infection. Our sample space contains one outcome $R$ if the patient is re-admitted to the hospital within one year and a second outcome if they are no re-admitted ($N$). Our sample space is $\mathcal{G} = {R,N}$. If we applied the PELO to this problem we would have to assume the probability this patient returns, or does not return, are equal. Intuitively,it feels unreasonable to assume these two events have equal probabilities.

We need an additional way to assign probabilities.

A frequentest approach to assigning probabilities¶

An empirical approach to assign probabilities to an event $A$ with sample space $\mathcal{G}$ is to do the following:

1. Generate $1, 2, 3, 4, \cdots, N$ outcomes from $\mathcal{G}$

2. Define a variable $N(A)$ and set that variable to zero.

3. If the 1st outcome is a member of $A$ than add one $N(A)$, otherwise move on

4. If the 2nd outcome is a member of $A$ than add one to $N(A)$\ \vdots

5. If the Nth outcome is a member of $A$ than add one to $N(A)$

The above algorithm defined a variable $N(A)$ that counts the number of outcomes that belong to $A$ out of the $N$ generated outcomes. We can assign to $A$ the probability

that is, the number of times we observed an outcome that belonged to A divided by the number of outcomes we observed.

As an example, suppose we want to understand the probability that patients with non-medically treated type II diabetes transition to needing treatment within one year of their diagnosis. Our outcomes are $\mathcal{G}$ = {“need treatment” and “no treatment”}. To assign a probability to the event $E = \{\text{need treatment}\}$ using the frequentest approach we decide to request anonymized patient records from a set of hospitals, asking that each hospital only send those patients who had an original diagnosis of type II diabetes without medication and who have had a physician visit one year more later. After our data request, we received 5,000 patient records and find that 1,585 of these patients were asked to start medication for their type II diabetes within one year. A frequentest approach would assign

and also assign to the event $F=\{\text{no treatment}\}$

Conditional probabilities¶

Assume a sample space $\mathcal{G}$. We define the conditional probability of event $A$ given event $B$ as

Example: Lets use the definition above to compute the probability of an event $A$ given $\mathcal{G}$, the sample space (remember that the sample space is a set and so technically an event). By the definition of conditional probability

We know that $P(\mathcal{G}) = 1$ (Kolmogorov axioms) and so

The set $A$ must be a subset of $\mathcal{G}$ and so

The above example shows that we can think of the event that we condition on as a new sample space. Lets return to our example of vaccination and the incidence of infection.

To compute the probability of infection given a patient was vaccinated we need to compute

where the event “inf” contains those outcomes in $\mathcal{G}_{\text{experiment}}$ which have “infected” in the fist position of their tuple $(\text{infected}, \cdot)$ and the event “vacc” contains those outcomes which have “vaccinated” in the second position of their tuple $(\cdot, \text{vaccinated} )$. Events which hold one or more positions in a tuple constant are often called marginal events and the associated probability is called a marginal probability. Then $P(\text{inf} \cap \text{vacc})$ are those outcomes with “infected” in the first position and “vaccinated” in the second position. We have a single outcome where this happens: {(infected, vaccinated)} and this outcome has a probability of $P(\text{inf} \cap \text{vacc}) = 0.1$.

Computing $P(\text{vacc})$ is only slightly more difficult. We can use what we learned about unions and disjoint events for help.

We arrive at our final result.

and we can do the same for

We report back to our experimental team that our estimated probability of infection from someone who received a vaccine is 0.2 and the probability of infection from someone who did not received a vaccine is 0.4, double the probability of a vaccinated individual.

Conditional probabilities give us, for free, an alternative way to compute the probability of the intersection of two sets. All we need to do is rearrange the definition of conditional probability

The above equation is called the **multiplication rule. Lets explore how the multiplication rule may make computing probabilities easier.

Example: We want to compute the probability that it rains and we remember to leave the house with an umbrella. We can imagine our sample space having the following four outcomes: $\mathcal{G}$ = {(rain,remember),(rain,forgot), (no rain, remember), (no rain, forgot)}. We try to remember all the times that the outcome (rain,remember) occurred in our past---but we cannot. Instead, we can certainly estimate the probability we forgot our umbrella given that it rained because it was an unpleasant experience (for some). We estimate $P(\text{forgot}|\text{rain}) = 0.95$. To compute $P(\text{forgot} \cap \text{rain})$ all we need now is the probability it rains. We take a look at the weather station reports over the past year and use the frequentest approach to estimate the probability it rains:$P(\text{rain}) = 0.25$. So then the probability it rains and we forget our umbrella is

The multiplication rule makes computations easier because we can think of events occurring in sequence. First it rains and second we forget our umbrella.

Conditional probabilities also allow us to express what it means for two events to be independent. An event $A$ is independent from an event $B$ if

We know the event $B$ occurred, but this has not changed the probability that $A$ has occurred. When two events are independent we can compute their intersection more easily.

If two events are independent than the probability they occur together is the product of their probabilities. Statistical independence can be difficult to see.

Example We are recruited to track the evolution of an infectious agent for a team of public health officials (PHOs). To support future strategic planning the PHOs want to know the impact of intervention $X$ on increases or decreases in the incidence of this infectious agent. The PHO team collected for each county in their state whether the intervention was enacted, and whether the incidence of case counts of this infectious agent increased or decreased 60 days after the intervention was in place data.

Table 1: Probabilities estimated from our data using a frequentest approach.

When we look at this table, our first thought may be that when the intervention is enacted there is a high probability that the infection rate decreases (0.525), evidence that this is an important intervention for preventing the spread of this agent.

But be careful. Let’s compute the probability of intervention $P(\text{intervention})$ where we define the event intervention as $\{ (\text{intervention},\text{rise}), (\text{intervention}, \text{no rise})\}$ and the probability of a raise in infection $P(\text{raise})$ where the event raise is defined as $\{(\text{intervention},\text{rise}),(\text{no intervention}, \text{rise})\}$. To do this we should define our sample space to be clear about what outcomes we can observe from our experiment.

Our sample space is the set of four pairs

and we want to compute $P(\text{intervention}) = 0.225 + 0.525 = 0.75$ (why?) and $P(\text{raise}) = 0.225+0.075 = 0.30$ (why?).

One way that can test whether intervention is independent of a rise or fall in incidence, is compare the probabilities we estimated from our data (in table \ref{tab.datafreq}) to the product of individual events we computed above. Lets look at an example. The probability of an intervention and rise in incidence was estimated from our data to be $P(\text{intervention},\text{raise}) = 0.225$. Lets compare this probability to the product $P(\text{intervention}) \cdot P(\text{raise})$.

We have a match. Our estimated probability of an intervention and a rise occurring together is equal to the probability that an intervention occurs and a rise in incidence occurs. We can do the same procedure above for the remaining three scenario.

Table 1: Probabilities of the co-occurrence of an intervention and the rise or fall of an infectious agent, and the probability of each of these four events assuming they are independent.

We see that the probabilities we collected form the data match the product of the probabilities of each individual event. Intervention is independent, does not change the probability, of a rise or fall in incidence of the infectious agent we are tracking. We will need to report these results to the PHO team and recommend they try alternative interventions to curb the spread of this agent.

Random Variables¶

Introduction¶

The foundations of probability are built on sets, yet data is more naturally stored and more easily computed on if it is represented numerically.

Random variables match each outcome in our sample space to a value on the number line.

In addition to computational advantages, random variables help us extract from our data the most important characteristics, and they serve as building blocks which we can use to create powerful models. Random variables are also a language we can use to communicate our modeling efforts to other mathematicians, statisticians, and data scientists.

Suppose we hypothesize that the frequency of social media posts on some popular outlet are related to influenza-like illness~(ILI)---a syndromic diagnosis suggesting a patient may have influenza. A patient is diagnosed with ILI if their temperature is measured to be at or above 38C and symptoms resembling the flu. Because influenza is most active in winter and spring, we collect a random sample, each day, of social media posts from September to May and in addition we collect the proportion of patients who are admitted to the hospital and are diagnosed with influenza-like illness at the US national level.

The above hypothesis, data collection, and future inference has numerous details. However, we will see shortly that we can simplify our hypothesis by using random variables.

Maps from the sample space to the number line¶

Given a sample space $\mathcal{G}$, a \textbf{random variable}, (e.g. $X$), is a function from each element in $\mathcal{G}$---from each outcome---to a value on the real number line. The real number line contains all numbers: integer and decimal, from negative to positive infinity.

\textbf{Example:} Suppose our sample space contains two elements $\mathcal{G} = \{ a,b \}$. We may decide to define a random variable $X$ that maps the outcome $a$ to the value -1 and the outcome $b$ to the value 1. In otherwords, $X(a)=-1$ and $X(b)=1$. We could as well define a random variable $Y$ on the same sample space such that $Y(a)=0$ and $Y(b)=1$.

\textbf{Example:} Suppose our sample space contains all integers from 0 to 1000 $\mathcal{G} = \{0,1,2,3\cdots,1000 \}$. We may be most interested in when an integer is even or odd, and so we can define a random variable $Y(y)=0$ when $y$, our outcome, is an odd integer and $Y(y)=1$ when $y$ is even. This is an example of how a random variable can distill down a sample space with many outcomes into a random variable with two.

\textbf{Example:} Suppose we decide to study the relationship between the cumulative total number of cigarettes smoked by a person form the date that they started smoking and the presence of lung cancer. We define our sample space to be $\mathcal{G} = \{ (x,y) | x \in \mathbb{Z}, y \in \{0,1\} \}$. We define two random variables, a random variable $X$ that maps the outcome $(x,y)$ to the value in the first position $x$, and a random variable $Y$ that maps the $outcome (x,y)$ to the value in the second position $y$. Though our outcomes are linked, we can use random variables to think about two separate outcomes---cigarettes smoked and lung cancer---and how they interact.

How to assign probabilities to a random variable¶

Random variable themselves do not require that we include the probability of each of their values. Random variables are a function from outcomes to the real numbers---nothing more. That said, in practice we build random variables expecting that the probabilities we assign to outcomes in our sample space will correspond to probabilities assigned to values of our random variable.

We assign a probability to the value $x$, which belongs in the support of random variable $X$, the sum of the probabilities of all the outcomes that $X$ maps to $x$.

where each outcome $o_{1},o_{2},\cdots,o_{n}$ is mapped by $X$ to the value $x$. In other words, $X(o)=x$ for each of $o_{1},o_{2},\cdots,o_{n}$.

\ex Define a $\mathcal{S} = \{a,b,c,d,e\}$ and a random variable $X$ that maps the outcomes to the following values

We assign the probability that $X=0$ as the sum of the probabilities assigned to outcome $a$ and outcome $e$, or

We can run the same procedure for all the elements in the support of $X$,

and organize our work in a table

A \textbf{probability distribution} for a random variable $X$ is a set of tuples where the first position in each tuple is a value in the support of $X$ and the second position in the tuple is the corresponding probability assigned to that value.

\ex A probability distribution for the random variable $X$ above is $\{(0,0.30),(1,0.40),(2,0.30)\}$.

\ex Imagine we run an experiment that collects data on marathon runners. We decide to collect the number of elapsed minutes until they finish the race. Our sample space is defined as all positive integers $\mathcal{S} = \{1,2,3,\cdots,\}$. We may decide to build a random variable $X$ that maps outcomes less than 60 to the value 1, outcomes from 61 to 120 to the value 2, and outcomes greater than 120 to the value 3. One potential probability distribution for $X$ is $\{(1,0.10),(2,0.50),(3,0.40)\}$. For this probability distribution, $P(X=1) = 0.10$, $P(X=2) = 0.50$, and $P(X=3) = 0.40$.

Functionals of a random variable¶

There are times that we may wish to summarize the behavior of a random variable. One common way to describe how probability is distributed among values in the support of a random variable is by computing some function of that random variable.

Expectation¶

Suppose we build a random variable $X$ with a corresponding probability mass function $f_{X}$.

The \textbf{expected value} of a random variable $X$ is computed as

where $x_{1},x_{2},\cdots,x_{n}$ are all values in the $supp(X)$.

An intuitive definition of the expected value is that $\mathbb{E}(X)$ is a weighted average of all values in the support of $X$ where the weight for $x_{i}$ is the probability of $x_{i}$. The expected value of $X$ will be close to values in $supp(X)$ with high probability.

Example Build a random variable $Y$ with support $supp(Y) = \{-1,0,1\}$ and $f_{Y} = \{(-1,0.2),(0,0.5),(1,0.3)\}$. The expected value of $Y$ is $\mathbb{E}(Y) = 0.2 (-1) + 0.5 (0) + 0.3 (1) = 0.1$.

Homework¶

1. Increasing your plot (earnings) in life:Suppose you decide to play a game of chance. The game costs 2 dollars, and when you spend 2 dollars “the house” will roll a single die. Before they roll the die, you get to choose from one of three options for how to determine a win. Choice A assigns you a win if the die lands on a 4,5, or 6. The payout you get is 2 dollars. Choice B assigns you a win if the die lands on a 6. The payout is 10 dollars. Choice C assigns you a win if the die lands on a 2,3,4,5,6. The payout is 2.50 dollars. (a) Please construct three random variables: $W_{a}$, $W_{b}$, and $W_{c}$. The random varable $W_{a}$ will map a winning die roll to the net winnings defined as the amount of money you would win minus the cost of playing the game for game choice A. $W_{b}$ maps a due roll to net winnings for choice B, and $W_{c}$ for choice c. (b) Please compute the expected value for $W_{a}$, $W_{b}$, and $W_{c}$. (c) Please discuss briefly what choice you should make: A, B, or C.

2. Plotting Crime: Please read the article titled “Robbing banks: Crime does pay – but not very much” linked here Reilly et al. (2012). In this article on page 3/5 there is an orange box with an equation that estimates the Penalty that would make a criminal indifferent to robbing a bank. They find that the expected next benefit is $vqR - (1-q)P$. Using “And” and “OR” rules, can you please describe this mathemtical equation in words to someone who we assume hasn’t learned about probability.

3. Plotting and Natural Language Processing: Please read the poem by Emily Dickinson titled Luck is not Chance https://poets.org/poem/luck-not-chance-1350. Suppose we number the lines from top to bottom as 1,2,3, ... 7. Our experiment will select at random a single word from this poem. Please estimate the probability of selecting line 3. Please estimate the probability of selecting a word with more than 5 letters.

4. Conditional probability - NLP part two: Suppose we tell you that the when we chose at random, we chose a word in the first three lines of the poem. Please compute the probability of selecting a word with more than 5 letters given that word appeared in the first three lines. Why is this conditional probability different than the probability of selecting a word with 5 letters over the whole poem?