Experiments Outcomes and Datapoints

Introduction¶

When we decide to study a topic, we often have in mind a population, a protocol to observe members of our population, a hypothesis that posits some aspects of our observed members. We collect data, we compile results. At the close of our study we want to know how the results (“what happened”) contribute to our hypotheses (“what we thought may happen”) and, importantly, the chances that if we reproduced the same study that the results would lead us towards the same conclusions.

Foundation¶

The ideas about sets and set operations are the foundations for how we think about probability, about experiments, and hypotheses. We only need to recast the above set ideas to results from an experiment.

We use $\mathcal{G}$ to define the set of all possible outcomes from an experiment and call this set the sample space. The term “experiment” has a broad meaning. An experiment can mean everything from a randomized controlled trial to an observational study. Here an experiment is the process that generates outcomes.

An outcome is defined as an element of the sample space. An outcome is a single observation from an experiment, and we define an event as a set of outcomes. Most often we use $o_{i}$ to denote an outcome and $E_{i}$ to denote an event.

The datapoint, dataset, and dataframe¶

The sample space, event, and outcome are all potential results from an experiment. When we conduct an experiment we will generate an outcome from our sample space and call this realized outcome a data point.

Example Consider the experiment of flipping a coin and recording whether the coin lands heads or tails side up. We can define a sample space $\mathcal{S} = \{H,T\}$ where $H$ is an outcome that represents the coin landing heads up and $T$ represents tails up. Up until this point we have structured our experiment, but we have generated no data. We flip the coin and the coin lands tails side up. Now we have performed an experiment and generated the data point $T$.

Now suppose that we conduct an experiment with the same sample space $( \mathcal{S})$ a number $N$ times and with each experiment we record a data point. A tuple of data points $d$ is called a data set $\mathcal{D} = (d_{1}, d_{2}, d_{3}, \cdots, d_{N})$ where $d_{i}$ is the data point generated from the $i^\text{th}$ experiment. We say that we have drawn or that we have sampled a data set $\mathcal{D}$. Further, data points $(d)$ are often called realized outcomes because they are no longer in a set of potential possibilities but are now determined items.

A data set $\mathcal{D}$ can be unwieldy depending on the number of data points, the complexity of the sample space, or both. A data frame is one way to organize a data set. A data frame $\mathcal{F}$ is a table where each data point $d$ in a dataset $\mathcal{D}$ is represented as a row in the table and if the data point is a tuple then a separate column is created for each position in the tuple.

Nomenclature around experiments¶

Given the ability to generate observations, most (if not all) experiments have at their core the goal to understand a causal relationship between an exposure and the subsequent effect it has on the experimental unit. Experiments try to elucidate cause and effect and that effect that the researcher thinks will be modified by the exposure is called the endpoint. In some way, the researcher is trying to learn about how observations collected from the experimental unit change when the context or environment in which they are sampled is manipulated. The experimental unit is the smallest, irreducible, entity that is needed to generate observations to learn about the impact of the exposure.

For example, 1. 2. 3.

Randomized controlled trials¶

The traditional experiment is a randomized controlled trial (RCT). In a RCT the researcher has the ability to distribute an exposure to experimental units. The “randomized” part of the RCT refers to the fact that the exposure is randomly delivered to experimental units. By randomizing an exposure delivered to a sample of experimental units, the researcher automatically accounts for other potential factors that could modify what they observe other than the exposure.

For example, suppose you choose to run an experiment and collect observations from N experimental units $u_{1}, u_{2}, \cdots, u_{N}$. With probability (<-we will get to a good definition later) 1/2 you assign an exposure and with probability 1/2 you do not. The observations that you collect from each unit are placed in a data frame like this

We have set this experiment up purposely so that units that recieve the exposure have a higher probability of the endpoint equalling one. We also set this up so that variable 1 also creates a higher probability of the endpoint equalling one.

Lets estimate (<--we will talk about this) the probability (<-we will get to this too) that the endpoint equals one given the exposure and not given the exposure AND also when an experimental unit just happens to have a high amount of variable 1 versus not.

Uh oh. We’re stuck. Our endpoint has a high probability of a one when we exposure experimental units but it also has that same probability when variable 1 is high. So how do figure out if its variable one or if its our exposure that had an impact?

This is the importance of randomization. Because we randomly assigned exposures, the distribution of “high” and “low” among variable 1 is equal across the units that were exposed and not exposed.

If variable one was the main contender for having an impact on our endpoint then we would see that among those units with (or without) the exposure, the probability of the endpoint changes when we change variable one.

But we do not see this. Instead, we see that the exposure continues to have an effect when we group together variable 1. Take a look below at the table sorted so that the variable 1 values are next to one another.

When variable 1 is “high” the exposure increases the probability of the endpoint equalling one by 0.50. When variable 1 is “low” the exposure does the same thing, increases the probability of the endpoint equalling one by 0.50. Nice. This is the power of randomization. By choosing to apply the exposure to experimental units at random, we build a sample that will have characteristics that are: (1) representative of our target population and (2) evenly distributed across the exposed and unexposed group.

Natural experiments¶

Randomized controlled experiments have alot of hype, so much so that you will often hear of RCTs called “the gold standard” of science. But, science isnt that easy. There are many times where a RCT is not feasible, not ethical, too costly. When a RCT isnt possible, an alternative is a natural experiment.

The goal is the same: we want to understand the impact of an exposure on an endpoint among a target population. We’ll use some measurements of experimental units to confirm to support or deny the impact of an exposure on an endpoint.

However, the main difference between a RCT and natural experiment is that we are unable or not allowed to assign the exposure to experimental units. Also, for a natural experiment to be valid when comparing exposure versus non-exposure, that exposure should be as if it was randomized (even though not by the researcher) over the experimental units. In other words, the exposure (and non-exposure) should not be linked or depend on another set of characteristics in the population. If this was the case then those characteristics would be unevenly distributed between exposure and non-exposure groups, making it much more difficult to tell if those characteristics or the exposure was the cause of a change in endpoint.

Non-experimental studies¶

What about if we are not necessarily interested in studying the impact of an exposure (these include any intervention by the way) on some downstream endpoint? Then we can conduct a non-experimental study. Non-experimental studies have many different goals, and are most often used to characterize a population, characterize a phenomena, or otherwise study experimental units.

Examples

Parks and recreation: https://www.imdb.com/video/embed/vi3468016921/
Duncan Principle: https://www.youtube.com/watch?v=60HEgeZvhFI
The 5-minute rule: https://www.facebook.com/watch/?v=2177147879222455

M&M experiment¶

You will conduct a simple experiment using one M&M.

Before selecting your M&M, do not look inside the bag. Reach into the bag and randomly select one candy.

Place the M&M flat on your desk without intentionally changing the way it lands.

Record two things about your candy:

1. Its color: red, blue, green, yellow, orange, or brown.

2. Whether the printed m is facing up or facing down.

Do not eat your M&M yet: but soon.

Once everyone has recorded their result, we will collect the results from the entire class. Be prepared to report your M&M’s color and whether the m was facing up or down.

We will use the results from this experiment to build a class dataset and to think about the difference between what could have happened before you selected your M&M and what actually happened when you performed the experiment.

Homework¶

1. You (i hope) completed the previous homework that asked you to run your own experiment and collect data. Please spend a few paragraphs disucssing the experimental units in your past study, the exposure (if there was one), the potential endpoint (if there was one), the type of study that you ran (experimental, natural, non-experimental). Can you briefly reflect on what youve learned, looking back at your experiment with this new context?

2. Please find a human who is not in this class, and describe/discuss your experiment, what you found (your results), the type of experiment you ran etc. In addition, pleae teach them about the difference between outcomes, data points, data frames. Were they able to identify the differnce between potential outcomes and data points in your experiment? Why or why not?

3. Lets revist the article here Leon et al. (2010). What was the exposure in this study? What endpoint were the researchers interested in and what effect did they think their exxposure would have on this endpoint? Can you give an example of an outcome versus a datapoint in this article? Behind this study, there must exist a dataframe. What do you expect would be (some of) the columns in this dataframe and what do the rows likely represent?

4. Please read the short article here about cats https://www.the-independent.com/life-style/cats-children-mental-health-study-pets-b3037053.html and describe whether: (1) this study is experimental or non-experimental and why; (2) the experimental unit, (3) the exposure (if one was presented), (4) the potential endpoint the researcher felt could be modified by the expoure, (5) the main results. Reflect on this study and results, and assess whether you feel like the study was conducted well or could use improvement.