{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "80f01878", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Warning message:\n", "\"package 'pwr' was built under R version 3.6.3\"" ] } ], "source": [ "library(pwr)\n", "library(purrr)\n", "\n", "inv.logit <- function(x) {\n", " 1 / (1 + exp(-x))\n", "}\n", "\n", "data_causal_inf <- function(n, tau = 1, b = 2, base = 100) {\n", " ## DGP\n", " age <- rnbinom(n, 10, 0.3)\n", " ps <- inv.logit(0.2*(age- mean(age)))\n", " Y0s <- round(base + b * age + rnorm(n, 0, 3))\n", " Y1s <- round(Y0s + tau)\n", " Z <- map_dbl(ps, ~ sample(c(0,1), 1, F, c(1-., .)))\n", " Y <- Z * Y1s + (1-Z) * Y0s\n", " return(list(age = age, Y = Y, Z = Z, ps = ps))\n", "}\n", "\n", "data_causal_inf_rand_exp <- function(n, tau = 1, b = 2, base = 100) {\n", " ## DGP\n", " age <- rnbinom(n, 10, 0.3)\n", " ps <- inv.logit(0.2*(age- mean(age)))\n", " Y0s <- round(base + b * age + rnorm(n, 0, 3))\n", " Y1s <- round(Y0s + tau)\n", " Z <- rbinom(n, 1, 0.5)\n", " Y <- Z * Y1s + (1-Z) * Y0s\n", " return(list(age = age, Y = Y, Z = Z, ps = ps))\n", "}\n", "\n", "set.seed(7)\n", "n <- 500\n", "\n", "causal_int_obs <- data_causal_inf(n, tau = 0.9123, b = - 1.0314)\n", "\n", "Age <- causal_int_obs$age; StreamingMinutes <- causal_int_obs$Y; AccountType <- ifelse(causal_int_obs$Z, \"Premium\", \"Free\")\n", "musicfi <- data.frame(Age, AccountType, StreamingMinutes)\n", "\n", "causal_int_obsrand <- data_causal_inf_rand_exp(n=500, tau = 0.9123, b = - 0.5314)\n", "\n", "Age <- causal_int_obsrand$age; StreamingMinutes <- causal_int_obsrand$Y; AccountType <- ifelse(causal_int_obsrand$Z, \"Premium\", \"Free\")\n", "musicfiExp <- data.frame(Age, AccountType, StreamingMinutes)" ] }, { "cell_type": "markdown", "id": "09922b5b", "metadata": {}, "source": [ "# Randomized Experiments\n", "\n", "Randomized experiments remove the selection problem and ensure that there are no confounding variables (observed or unobserved). They do this by removing the individual’s opportunity to select whether or not they receive the treatment. In the Musicfi example, suppose we ran an experiment with 500 participants where we randomly upgraded some free accounts to premium accounts. Now, we no longer have to adjust for age as (on average) there will be no age difference between the treated and control subjects. Suppose the data from our experiment is stored in a data frame called `musicfiExp`:" ] }, { "cell_type": "code", "execution_count": 2, "id": "5d3cdcbb", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\n", "
AgeAccountTypeStreamingMinutes
20 Premium92
28 Free 88
20 Premium89
24 Premium86
22 Premium91
15 Premium93
\n" ], "text/latex": [ "\\begin{tabular}{r|lll}\n", " Age & AccountType & StreamingMinutes\\\\\n", "\\hline\n", "\t 20 & Premium & 92 \\\\\n", "\t 28 & Free & 88 \\\\\n", "\t 20 & Premium & 89 \\\\\n", "\t 24 & Premium & 86 \\\\\n", "\t 22 & Premium & 91 \\\\\n", "\t 15 & Premium & 93 \\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "| Age | AccountType | StreamingMinutes |\n", "|---|---|---|\n", "| 20 | Premium | 92 |\n", "| 28 | Free | 88 |\n", "| 20 | Premium | 89 |\n", "| 24 | Premium | 86 |\n", "| 22 | Premium | 91 |\n", "| 15 | Premium | 93 |\n", "\n" ], "text/plain": [ " Age AccountType StreamingMinutes\n", "1 20 Premium 92 \n", "2 28 Free 88 \n", "3 20 Premium 89 \n", "4 24 Premium 86 \n", "5 22 Premium 91 \n", "6 15 Premium 93 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "head(musicfiExp)" ] }, { "cell_type": "markdown", "id": "b2ab4cf0", "metadata": {}, "source": [ "Unlike the observational data from the previous section, there is now no significant difference in age between the treatment and control groups:" ] }, { "cell_type": "code", "execution_count": 3, "id": "c6a14995", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAMFBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD////QFLu4AAAACXBIWXMAABJ0AAAS\ndAHeZh94AAAY30lEQVR4nO3d7ULa2haG0QQQFRHu/26PoK3Y7dHa9ZKsGcb40bKVNItZn4P5\n6HE4As2GuRcASyAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBA\nSBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIE\nCAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKC\nACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBI\nECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQI\nCQKEBAFCggAhQYCQIEBIEDBBSAMU8w9f5flwZtgFJAkJAoQEAUKCACFBgJAgQEgQICQIEBIE\nCAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKERD/+6V9s90FI9OJcUdWUhEQvhotfyxESnRj+\n+L0WIdEJIeXVnCVNhJRXc5a0cYwUV3SYNHHWLq7oMGnkOlJY2XFyq4QEAUKCACFBgJAgQEgQ\nICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJ\nAoQEAUKCACFBgJAgQEgQICQIEBIECAkCJg3p6X4znGy2T9faBcxiwpAOq+Hd+iq7gJlMGNJ2\nGB/350fPu3HYXmMXMJMJQxqH/e/H+2G8xi5gJhOG9OEnVn/946uFRDHekSBg2mOk3fP5kWMk\nlmbK09/ri7N2q8NVdgHzmPY60vZ8HWnc3LuOxLK4swEC+glpuHSdXcC1THn6e/zmG7r2XcBM\nJr2ONGy+PMXQvguYyaQhnc56/1VKQqKYae9sOGyG4W53vV3ATKa+RWh/OgG+edh//cYkJIqZ\n/l67/Xb89sSckChmlptW9w+blZBYkrnu/r7OLmAmQoKAfu5smHgXkCQkCBASBAgJAoQEAUKC\nACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBI\nECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQI\nCQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBk4b0\ndL8ZTjbbp2vtAmYxYUiH1fBufZVdwEwmDGk7jI/786Pn3Thsr7ELmMmEIY3D/vfj/TBeYxcw\nkwlDGob/9x+xXcBMvCNBwLTHSLvn8yPHSCzNlKe/1xdn7VaHq+wC5jHtdaTt+TrSuLl3HYll\ncWcDBPQT0nDpOruAa5kypMP2dKrufjUM68cr7QLmMWFIz+PLO81hdIsQCzRhSHfD5vDyy93z\nS1N3Tn+zKJPe2XB4++XluzwXZFmUqW8RGoeL/4jvAmYy6bd2++Px/vU+ocPXB0lCopgJQ9oP\n43Z/3IwvJe1Ww+4au4CZTHn6eze+Xyi6v84uYB7TXpB9vDv/K9nN/fPVdgFz6OfOhol3AUlC\nggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBA\nSBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIE\nCAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQkoah6sppJKScc0VSuk1Cyhku\nfuXGCClm+ON3bomQYoR0y4QUI6RbJqQcx0g3TEg5ztrdMCEluY7UpvD8hJTjHalN6flNGtLT\n/WY42WyfrrWLOTlGalN6fhOGdFgN79ZX2cWsnLVrU3t+E4a0HcbH/fnR824cttfYxaxqfyHM\nr/b8JgxpHPa/H++H8Rq7mFXtL4T51Z7fhCF9OIz8+piy8ixrrr0HpefnHSmn9FmnDpSe37TH\nSLvn86NlHiMdS18H6ULh+U15+nt9cdZudbjKLmZV+n9RaTPtdaTt+TrSuLl3HYllcWdDTO2z\nTrTpJ6Th0nV2cV1CumXTh/SwGobN7qq7mIeQbtnk15Hezjh8edKu6NeiY6QbNnVI22F7OB6f\nt8PDNXYxL2ftbtjUIY3D+bz3YVhdYxdzK3p0143C85s6pF+TWuItQrQp/Y4+dUh3v0Ja4C1C\ntCl9jDlpSJv7h93w+PLwsF3mLUI0qH3Wc9KQfl8jGoZxgbcI0URIf2u/f3jYbM6nHLZfdlR0\nljQRUl7NWdLGMVJc0WHSxFm7uKLDpJHrSGFlx0kD70hxRYdJE8dIcUWHSQtn7fJqzpImQsqr\nOUuaCCmv5ixp4xgprugwaeKsXVzRYdLIdaSwsuOkgXekuKLDpIljpLiiw6SFs3Z5NWdJEyHl\n1ZwlTYSUV3OWtHGMFFd0mDRx1i6u6DBp5DpSWNlxcquEBAFCggAhfdzx3OZ64TQSUlTZhdNI\nSFFlF04jIUWVXTiNhBRVduE0EhIECAkChAQBQooqu3AaCSmq7MJpJKSosgunkZCiyi6cRkKK\nKrtwGgkJAoQEAUKCACFFlV14yNz/nGu++QspquzCO1F3fkKKKrvwTtSdn5Ciyi68E3XnJ6So\nsgvvRN35CQkChAQBQoIAIUWVXXgn6s5PSFFlF96JuvMTUlTZhXei7vyEFFV24Z2oO7/mkHab\n0w1Om+fQej7bRSFlF96JuvNrDWn9eqfgMEZLqjtPblRjSA/D+nAK6WG4iy3pKCTKaQxpHA7X\n+NGfQqKYxpDO39YJ6beyC+9E3fk1hrR6e0faD6vYko6F51l24Z2oO7/MMdJuHB5iSzoWnmfZ\nhXei7vxaz9pt3v6J7zq1oP/uopKyC+9E3flFriMNm8fQcj7dRSFlF96JuvNzZwMECAkCmk9/\n/7befrvl0/3rIdVm+xRfFcwpF9IwjF9vd1hdZpdeVR/KLrwTdefX+q3d3bh7+XU3Dk/HzfD1\ne9J2GB/350fPL8//8rll51l24Z2oO7/GkLbDaxr7l7eYwzcXZce3574+/8u3r7LzLLvwTtSd\nX+AWod8PvrlNaPh0w9Sq+lB24Z2oO7/mm1Z/vSON34bkHYnv1J1f87d2v46RtsfHb84gnJ77\n+q+WlnuMxK2K/MO+17Nww3f3260vztqtDuFVwZxCtwid3paG+++2fNqeryONm3vXkVgWdzZE\nlV14J+rOLxTSfvvN1di/+GO7+HlRjcouvBN155cI6fl+9e1tDR//hO/2WnaeZRfeibrzaw7p\n8Hi682e9+9GfICQ+VXd+jSE9vp6J+6v/L64f/LTPsvMsu/BO1J1fS0i7u9MpuO3+Lw9pnsbl\nh8StaghpPFV0Oo/9t+cGDpthfX7vWu63dtyqhpCGX3cn/P1JtsdhePybDYREMVO+I714Xg+b\nw4JDKrvwTtSdX+AY6elHl33uh3EnJD5Xd34TnrV7s199f8G17DzLLrwTdecXuo60+cl1pDsh\n8bm685vlzoaf76KKsgvvRN359XOv3Xe7gI65+xsChAQBQooqu/BO1J2fkKLKLrwTdecnpKiy\nC+9E3fkJKarswjtRd35Ciiq78E7UnZ+QIEBIECAkCBBSVNmFd6Lu/IQUVXbhnag7PyFFlV14\nJ+rOT0hRZRfeibrzE1JU2YV3ou78hAQBQoIAIUGAkKLKLrwTdecnpKiyC+9E3fkJKarswjtR\nd35Ciiq78E7UnZ+QosouvBN15yckCBASBAgJAoQUVXbhnag7PyFFlV14J+rOT0hRZRfeibrz\nE1JU2YV3ou78hBRVduGdqDs/IUGAkCBASBAgpKiyC+9E3fkJKarswjtRd35Ciiq78E7UnZ+Q\nosouvBN15yekqLIL70Td+QkJAoQEAUKCACFFlV14J+rOT0hRZRfeibrzE1JU2YV3ou78hBRV\nduGdqDs/IUWVXXgn6s5PSBAgJAgQEgQIKarswjtRd35Ciiq78E7UnZ+QosouvBN15yekqLIL\n70Td+QkpquzCO1F3fkKCACFBwKQhPd1vhpPN9ulau4BZTBjSYTW8W19lF7Mru/BO1J3fhCFt\nh/Fxf370vBuH7TV2MbuyC+9E3flNGNI47H8/3g/jNXYxu7IL70Td+U0Y0jD8v/+I7WJ2ZRfe\nibrz844UVXbhnag7v2mPkXbP50fLPUbiVk15+nt9cdZudbjKLmAe015H2p6vI42be9eRWBZ3\nNkSVXXgn6s6vn5CGS9fZxfWVXXgn6s5vhpAexmH1cN1dzKbswjtRd35ThrTfDOPD8d4tQvw/\ndec3YUj7c0Hb4e5wfN4MX74nlZ1n2YV3ou78Jgzp7nTtaPt6JfYwrK6xC5jJ5LcIDZuL/0jv\nAmYyeUiPr9/TLfQWIW7VpN/a3f26neFwt9BbhMouvBN15zflP+wbf38/N3z9hvTv8xxu3b8O\nrhN11z/pdaTtr3zGL9+PWkL61w0Xovrrr7v+fu5siOyi7l9ERvXXX3f9QlqU6q+/7vqFtCi3\n/vrnI6RFufXXPx8hLcqtv/75CGlRqr/+uusX0qK0vv65L4PNbsrJC6lfzSFFVlGXkFo3XAgh\ntRFS64YLIaQ2QmrdcCGE1EZIrRsuhJDaCKl1w4UQUhshtW64EEJqI6TWDRdCSG2E1LrhQgip\njZBaN1wIIbURUuuGCyGkNkJq3XAhhNRGSK0bLoSQ2gipdcOFEFIbIbVuuBBCaiOk1g0XQkht\nhNS64UIIqY2QWjdcCCG1EVLrhgshpDZCat1wIYTURkitGy6EkNoIqXXDhRBSGyG1brgQQmoj\npNYNF0JIbYTUuuFCCKmNkFo3XAghtRFS64YLIaQ2QmrdcCGE1EZIrRsuhJDaCKl1w4UQUhsh\ntW64EEJqI6TWDRdCSG2E1LrhQgipjZBaN1wIIbURUuuGCyGkNkJq3XAhhNRGSK0bLoSQ2gip\ndcOFEFIbIbVuuBBCaiOk1g0XQkhthNS64UIIqY2QWjdcCCG1EVLrhgshpDZCat1wIYTURkit\nGy6EkNoIqXXDhRBSGyG1brgQQmojpF8b3rrWwd+6f5/cJJtMtou5/x5m1zr4W/fvk5tkk8l2\nMfffw+xaB3/r/n1yk2wy2S58jz/v9tUJqXXDhRBSGyG1brgQQmojpNYNF0JIbYqE9HS/OR/R\nbbZPV9qFL4R5t6+uREiH1cXZkfVVduELYebtqysR0nYYH/fnR8+7cdheYxe+EGbevroSIY3D\n/vfj/TBeYxe+EGbevroSIX242vX1pS8h/SMhtSkRknek6xNSmxIhvRwj7Z7PjxwjXYuQ2pQI\n6bi+OGu3OlxlF74Q5t2+uhohHZ+25+tI4+bedaTrEFKbIiFdfxe+EObdvrolhBS5m90Xwrzb\nV1cjpMPdMKx3b3+I09/XIKQ2JUI6jK832r3+IUK6BiG1KRHSdnh4qelhPN9mJ6SrEFKbEiGN\nrxs+j6tnIV2JkNqUCOlXO4f1WkhXIqQ2JUJaDb8uwq7WQroOIbUpEdLDcPf26HlYC+kqhNSm\nREjH7e96dt9cKhLSPxJSmxohHfebX4+e74R0DUJqUySk6+/CF8K821cnpNYNF0JIbYTUuuFC\nCKmNkFo3XAghtRFS64YLIaQ2QmrdcCGE1EZIrRsuhJDaCKl1w4UQUhshtW64EEJqI6TWDRdC\nSG2E1LrhQgipjZBaN1wIIbURUuuGCyGkNkJq3XAhhNRGSK0bLkRzSLduyskLqV/VX3/d9Qtp\nUW799c9HSIty669/PkJalFt//fMR0qJUf/111y+kRan++uuuX0iLUv31113/0kK6dcm/hRnU\nXf/CQppb2YV3ou78hAQBQoIAIUGAkKLKLrwTdecnpKiyC+9E3fkJKarswjtRd35Ciiq78E7U\nnZ+QosouvBN15yckCBASBAgJAoQUVXbhnag7PyFFlV14J+rOT0hRZRfeibrzE1JU2YV3ou78\nhBRVduGdqDs/IUGAkCBASBAgpKiyC+9E3fkJKarswjtRd35Ciiq78E7UnZ+QosouvBN15yek\nqLIL70Td+QkJAoQEAUKCACFFlV14J+rOT0hRZRfeibrzE1JU2YV3ou78hBRVduGdqDs/IUWV\nXXgn6s5PSBAgJAgQEgQIKarswjtRd35Ciiq78E7Und+kIT3db84/w36zfbrWLmZWduGdqDu/\nCUM6rIZ366vsYnZlF96JuvObMKTtMD7uz4+ed+OwvcYuZld24Z2oO78JQxqH/e/H+2G8xi5g\nJhOGNAz/7z9iu4CZeEeCgGmPkXbP50eOkfhc3flNefp7fXHWbnW4yi7mVnbhnag7v2mvI23P\n15HGzb3rSHym7vzc2RBVduGdqDu/fkIaLl1nF9dXduGdqDu/fkKaeBeQJCQIEBIETHpnw18f\nBpUNqezCO1F3fhOG9CAkvlF3flN+a7cfv/7HE4FdzK3swjtRd36THiPtv74xKLGLmZVdeCfq\nzm/akw0PF/etXmkX8yq78E7UnZ+zdhAgJAgQEgQIKarswjtRd35Ciiq78E7UnZ+QosouvBN1\n5yekqLIL70Td+QkpquzCO1F3fkKCACFBgJAgQEhRZRfeibrzE1JU2YV3ou78hBRVduGdqDs/\nIUWVXXgn6s5PSFFlF96JuvMTEgQICQKEBAFCiiq78E7UnZ+QosouvBN15yekqLIL70Td+Qkp\nquzCO1F3fkL6uOO5zfXCO1H39QsJAoQEAUKCACHRkbp/8UKiI3X/4oVER+r+xQuJjtT9ixcS\nQXNfhpvvC0dIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBA\nSBAgJAgQEgQICQKEBAFCgoBOQ4Ji/uGrPB/OYphNm5ua30292B8ymzY3Nb+berE/ZDZtbmp+\nN/Vif8hs2tzU/G7qxf6Q2bS5qfnd1Iv9IbNpc1Pzu6kX+0Nm0+am5ndTL/aHzKbNTc3vpl7s\nD5lNm5ua30292B8ymzY3Nb+berE/ZDZtbmp+N/Vif8hs2tzU/G7qxcK1CAkChAQBQoIAIUGA\nkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBDSm4b///Qb9Dqq8e75L5995eV04AZe\n4t8R0k/8Gtb4VyXdwlBv4CX+nVv4y855ndZhPWznXkkvfPm8EdJPvE3rMIwzL6QbvnzeXIQ0\nDIfVsHl58LAaxofXj1085H1ap9//O66Xj94P4/3xuB3Ob1mvz3p//sfPLoOQ3nwIaXP+K96c\nDwPWpw9dPOT48R3pv+MahvvTw9369Ov2vyF9/OwyCOnNxbmGl6+Gw8tvu9NvL4cBuw8POXnN\n4nn9msmf4zp/5OHt1/G/IX387DII6c2HkJ5OH9kMp6+Pw+m7louHnPw+a3f4bFyvHxmG5+N7\nRJchffzsMizmhbT68K3d22+/23Ju/A+X15E+G9fx+DGdP//r469LsJgX0kpIP3E5CSGdLOaF\ntPospM8+ycmnIf3xESHdpP9ms3k/t7BxmuGjT0K6mNH/DelJSIv335Aeh3F/PD6cjp4vHnLy\nSUgXM/ospNXwcDqnJ6Sl++QbufX77WQXDzl+GtLFjD4L6eH0yY2QFu+zI6KH1TC83eB88ZDP\nQ3qf0WchHe/H4c4xEvAVIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJ\nAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAh\ndWW48M1Tf/Djof0k6esTUlf+PqTV3//N/eCp/Csz7s9f/WDVH/z01eX8oNaOmXF/hFSQGffn\n1xf+MBxWw+Z4/nHh48P5Q7vNMIzb49v3gOen3g/j/fG4HYbt+Rm/nzsMz5vzp/7mgItWJtyf\n95A25zw25xTWLx+5fz162l6EdP7Qbv324YvnvhR3engvpEmYcH/eQ1ofXn7bnX47rIfd6SOP\nx+Pj+fOvTzo/5eHt1/GP554/uPKt3STMuD/vIT2dftsMp5wO52/y3j//K6Sn86/Pbx+5eO6v\nTwlpEmbcn/eQ3n67OB/+vLtffwjp468Xz7384NSv4AaZcX++CGn9+5GQ+mLG/flPSL8/czes\nHnbPX4b0x58ipImYcX/+CGkz7D585quQ/nyukKZixv35I6THYdwfjw+/TiDsfx0jPR//G9KH\n5/764OtTuSoh9eePkN4OjMbn82XXs6fTDXSn093/eV96f+77B1+fylUJqT9/hnS6W2G4O7+r\n3A3D+ml3esN5Wn0a0vtz3z/4+lSuSkgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQI\nCQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIA\nIUHA/wBIjIqZa79jrAAAAABJRU5ErkJggg==", "text/plain": [ "plot without title" ] }, "metadata": { "filenames": { "image/png": "C:\\Users\\phamilton\\Documents\\Projects\\dsm\\dsm_website\\_build\\jupyter_execute\\04_causal_inference\\rand_experiments_4_0.png" } }, "output_type": "display_data" } ], "source": [ "boxplot(musicfiExp$Age ~ musicfiExp$AccountType, ylab = \"Age\", xlab = \"Treatment\")" ] }, { "cell_type": "markdown", "id": "591c06e3", "metadata": {}, "source": [ "Age is not a confounding variable as it is independent of the treatment assignment. We can now directly attribute any differences in the outcome due to the intervention. This allows us to conclude that giving people a premium account increases streaming minutes." ] }, { "cell_type": "code", "execution_count": 4, "id": "71d8dac4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\n", "\tWelch Two Sample t-test\n", "\n", "data: musicfiExp$StreamingMinutes by musicfiExp$AccountType\n", "t = -2.5057, df = 497.95, p-value = 0.01254\n", "alternative hypothesis: true difference in means is not equal to 0\n", "95 percent confidence interval:\n", " -2.2728058 -0.2750465\n", "sample estimates:\n", " mean in group Free mean in group Premium \n", " 87.55285 88.82677 \n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAMFBMVEUAAABNTU1oaGh8fHyM\njIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD////QFLu4AAAACXBIWXMAABJ0AAAS\ndAHeZh94AAAaIklEQVR4nO3d6XrayBaGUTHEdvDA/d/tCdjp4MTH0/5AtcVaPxK3Y7WKKt4H\nEEKe9kDZNPcAYAmEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIE\nCAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKC\nACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBI\nECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQI\nCQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIA\nIUGAkCBASBAgJAgQEgQICQIuENIEzXzjXp4PZ4ZdQJKQIEBIECAkCBASBAgJAoQEAUKCACFB\ngJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKERNDc18Sa744jJAbSd+GF\nxED6LryQGEjfhRcSA+m78EKCACFBgJAgQEgMpO/CC4mB9F14ITGQvgsvJAbSd+GFxED6LryQ\nIEBIECAkCBASA+m78EJiIH0XXkgMpO/CC4mB9F14ITGQvgsvJAgQEgQICQKExED6LryQGEjf\nhRcSA+m78BcN6f52e7yu7Pbm/ly7oLW+C3/BkJ7WJ9do3pxlFzTXd+EvGNLNtPr5cPzqcbea\nbs6xC5jJBUNaTQ//ff0wrc6xC5jJBUN69Ts33v8FHEKiGY9IDKTvwl/2NdLu8fiV10i8re/C\nX/Lw9+bkqN366Sy7oLe+C3/Z95Fuju8jrba33kfiLX0X3pkNDKTvwgsJApwiBAFOEYIApwgx\nkL4L7w1ZBtJ34cc5RWg69c1d0FzfhfeIxED6LrxThBhI34V3ihAEOEUIApzZAAFCYiB9F15I\nDKTvwguJgfRdeCExkL4Lf9EzGz598kLf+aSk78JfMKQ7IbFYl3xq97B6/8MTgV3APC76Gunh\n/RODEruAWVz2YMPdyXmrZ9oFnfVdeEftGEjfhRcSA+m78EJiIH0XXkgMpO/CCwkChAQBQoIA\nITGQvgsvJAbSd+GFxED6LryQGEjfhRcSA+m78EKCACG93vHc5rrhFAkpqu3AKRJSVNuBD6Lv\n/Akpqu3AB9F3/oQU1Xbgg+g7f0JiIH0XXkgMpO/CCwkChBTVduAUCSmq7cApElJU24EPou/8\nCSmq7cAH0Xf+hBTVduCD6Dt/QmIgfRdeSAyk78ILCQKEFNV24BQJKartwCkSUlTbgQ+i7/wJ\nKartwAfRd/6EFNV24IPoO39CYiB9F15IDKTvwgsJAoQU1XbgFAkpqu3AKRJSVNuBD6Lv/Akp\nqu3AB9F3/oQU1Xbgg+g7f0JiIH0XXkgMpO/CCwkChBTVduAUCSmq7cApElJU24EPou/8CSmq\n7cAH0Xf+hBTVduCD6Dt/QmIgfRdeSAyk78ILCQKEFNV24BQJKartwCkSUlTbgQ+i7/wJKart\nwAfRd/6EFNV24IPoO39CYiB9F15IDKTvwgsJAoQU1XbgFAkpqu3AKRJSVNuBD6Lv/Akpqu3A\nB9F3/oQU1Xbgg+g7f0JiIH0XXkgMpO/CCwkChBTVduAUCSmq7cApElJU24EPou/8CSmq7cAH\n0Xf+hBTVduCD6Dt/QmIgfRdeSAyk78ILCQKEFNV24BQJKartwCkSUlTbgQ+i7/wJKartwAfR\nd/6EFNV24IPoO39CYiB9F15InJiu3fdn7iKbDLgL3nLtEy+kQbQd+Ivu468S0iDaDvxF9/FX\nCWkQbQf+ovv4q4Q0iLYDf9F9/FVCGkTbgb/oPv4qIRFx7RMvJCKufeKFRMS1T7yQBtF24C+6\nj79KSINoO/AX3cdfJaRBtB34i+7jrxLSINoO/EX38VcJ6feG126uiV8IIVU3XAgh1QipuuFC\nCKlGSNUNF0JINUKqbrgQQqoRUnXDhRBSjZCqGy6EkGqEVN1wIYRUI6TqhgshpBohVTdcCCHV\nCKm64UIIqUZI1Q0XQkg1QqpuuBBCqhFSdcOFEFKNkKobLoSQaoRU3XAhhFQjpOqGCyGkGiFV\nN1wIIdU0Cen+dnv8GOf25v5Mu3BHmHf77lqE9LQ++Uj05iy7cEeYefvuWoR0M61+Phy/etyt\npptz7MIdYebtu2sR0mp6+O/rh2l1jl24I8y8fXctQnp1iZv3r3cjpG8SUk2LkDwinZ+QalqE\n9Os10u7x+JXXSOcipJoWIe03J0ft1k9n2YU7wrzbd9cjpP39zfF9pNX21vtI5yGkmiYhnX8X\n7gjzbt/dEkKKXMLaHWHe7btrEtLDzfPLpPX255l24Y4w7/bd9Qjp9uQhZ3ueXbgjzLt9dy1C\n2k0/Hvf7+812/3C3nnbn2IU7wszbd9cipM10POT9MN3+yun9hyQhfZOQalqE9PsQwvGkBqcI\nnYWQalqEtHp+RHo6NiSksxBSTYuQbqbN/X7/uJ1+7J9+/PrjDLtwR5h5++5ahPT7FKHV06/H\no9XjWXbhjjDv9t31CGl/9yul9e2vL1Y3755qJ6TvElJNk5DOvwt3hHm3705I1Q0XQkg1Qqpu\nuBBCqhFSdcOFKId07S4580Ial5CKLjnzQhqXkIouOfNCGpfXSDVCqm64EEKqEVJ1w4UQUo2Q\nqhsuhJBqhFTdcCGEVCOk6oYLIaSa+UK6W+/3j+tp/cGF6iq7uMiGCyGkmtlC2h2OvK8OB+Cj\nJQnpm4RUM1tIm+nn/mFa739+8JvDCru4zIYLIaSa2UI6PCA9HC6IX3hP+INdXGbDhRBSzawh\nbQ+X1hLSEIRUM+NTu4fd4apAwzy1u3ZzTfxCzHmwYTpcp256/4KPlV18bcNrN9fEL8SMh7+f\nf2XY+oOLeVd28aUNr91cE78Q3pCtbrgQQqoRUnXDhRBSzYwh7bbHI3fvXqauuItLbLgQQqqZ\nL6TN8xPzDy74WNrFRTZcCCHVzBbS3bQ5Xsv77v1LEFd2cZkNF0JINbOFdLgw/icuil/ZxWU2\nXAgh1cx6ZoOQxiGkmtlCWr88Ih1OXA0S0jcJqWbu10i71XT37SF8sIvLbLgQQqqZ76jd9uUd\n9eipdkL6LiHVzPw+0rTNniEkpO8SUo0zG6obLoSQaoRU3XAhhFQz6+Hvo9Xq20P4YBeX2XAh\nhFQze0iP3kcagpBqZglp9+qDMN5HGoGQauZ5RFqfduRyXCMQUs3sT+3ChPRNQqpx1K664UII\nqUZI1Q0XQkg1Mz61S1124//u4jIbLoSQaoRU3XAhhFQz91O7+8322yP45C7OvOFCCKlm7pD2\nTz5qPgQh1cwekk/IjkFINbOHdDc5124EQqoZ4GDD7beH8MEuLrPhQgipZvaQ1tFPmgvpu4RU\nM/tTuzAhfZOQaoRU3XAhhFQjpOqGCyGkmvlCul07s2EgQqqZLaRbpwgNRUg1s4UUvjDkW7u4\nzIYLIaQaH+yrbrgQQqqZLaTt9PTtXX9yF5fZcCGEVDNbSI+rTfRiDW/s4jIbLoSQamY/s8HB\nhjEIqUZI1Q0XQkg13pCtbrgQQqoRUnXDhSiHdO0uOfO/N3n+tZf1IWRGVdxwIa799s9HSIty\n7bd/Pp7aLUr32993/EJalO63v+/4hbQo3W9/3/EXQlqFjndkRlXccCG63/6+4y+EtBXScLrf\n/r7jL4R0N61vfj5GR/P3Li634UJc++2fTyGkxx+HJ3erH2eISUjfdO23fz61gw0Pd8fnd/GY\nhPRN137751M/and/uznGlBnPm7u4wIYL0f329x1/5PD3042DDWPofvv7jt8j0qJ0v/19x7+0\n10jXLrkKM+g7/vJRu7McAu87n5T0Xfji+0i7s1z7pPF8cqUWdmbD3NoOnKKFnWs3t7YDp2hh\nZ3/Pre3AB9F3/oQU1Xbgg+g7f0KKajvwQfSdPyExkL4LLyQG0nfhhQQBQopqO3CKhBTVduAU\nCSmq7cAH0Xf+iiGdnNmwuYkNqu98th34IPrOXy6k5EeS2s5n24EPou/8VZ/a/Vjtfv25W033\n++0Ue0zqO5+U9F34Ykg308Px74dps3+a1pkxdZ5PSvoufPmp3ckXuTPA+84nV6oY0uq/R6SV\nkPaNB05R+and79dIN/ufv57ezTeqMbQdOEXVgw2b3we/Dw9IdzOOagxtBz6IvvNXfkN2d/jE\n+fbwsDTdZob0zy4aaTvwQfSdP2c2RLUd+CD6zp+QGEjfhRcSA+m78NWQbteuIgTVkG5djuuV\ntgOnqPyGbOyQ9//bRSttB05R6hShrLb3x7YDH0Tf+SuGtJ3OcvHvtvPZduCD6Dt/xZAeV5v7\n2Fje3kUrbQc+iL7zl/tgX2xI+87zSUnfhRcSA+m78N6QhQAhRbUdOEWFkJ4/FOup3am2A6dI\nSFFtBz6IvvPnqV1U24EPou/8CSmq7cAH0Xf+hMRA+i58NaSblddIxPRd+PJVhBxsgMCZDT5G\ncartwCnyMYqotgOnqPzU7isfo7i/3R6fBW5vPjhlvO39se3AB9F3/soXiNw8fna7p/XJ27fv\nX5S17Xy2Hfgg+s5fNaTd5w823Eyrn89XCn88XuI4O6oxtB34IPrO3wUvfvL7gvsHD+//VrK+\n80lJ34W/4MVPps8fpeg7n5T0XfgLHrXziMRylZ/aff6o3eFXwDwfmfAaiaUpX2n1Cxc/2Zwc\ntVu/G2Db+2PbgVN00Ws23N8c30dabW+9j8Rb+s6fi59EtR34IPrO3zgfo5imM1V5SW0HPoi+\n83fJkB5/TKvb/f5uPa3ePdTQeT4p6bvwqZDutx9u93T86NLd7ZJPEaKm78KXP9j3lVOEbo4f\nBPzxtH+6Wejhb65V7IN9uw+3Wx03nJ7feVroG7JtB05R+RShn/vN9Pi4mT5+O2ma/vy51FOE\n2g6cosApQre/Ho0ePnjRc7A6CenJIxJv6Dt/gZB2hxNXv/Aa6ebp5evoqMbQduCD6Dt/xZC2\nv57aPU7r/f0nQrqGo3ZtBz6IvvNXDGl3COh4Dt2Pjzf0PhIf6Lvw5ZNWD//1Y3r/mVptF1yP\nvgs/zilCF94FJAkpqu3AKSqHtNseXiZtP30toW/sopG2A6eofDmu57ODplW0pLb3x7YDH0Tf\n+SuGdDdtng4h3X3mqN33dtFK24EPou/8lU8Renp+L9YH+47aDnwQfecvcGaDkEjpu/DFkNYv\nj0gP0zo2pH3n+aSk78JnXiPtvnKhyC/uAhqoHrXbfuqi+KVddNJ24BRF3keatj9Dw3lzF420\nHThFzmyIajvwQfSdv+rHKLInq761i1baDnwQfecvcPj7DNrOZ9uBD6Lv/AUOf59B3/mkpO/C\nF0N62n7hIvrf2wVXpO/Cl5/aufY3CCms7cApcvg7qu3AKRJSVNuBD6Lv/KUOf6/eveBjZRet\ntB34IPrOXyikR6+RjtoOfBB9568Q0u7VrwbzMQrq+i585RFpfdpR9O2kvvNJSd+Fd4oQBDhq\nF9V24BRVQnq6OX55v55W0c/HNr4/th04RZWQVscndjufkP2j7cAH0Xf+CiEdrtewP7yD9LB/\n2kzRz8i2nc+2Ax9E3/krhLSZDldXvZ9uj39GH5LazmfbgQ+i7/wVQno+Ynfz/NtjvSFLQN+F\nL4e0nk7+I6XvfFLSd+ELIa0PT+0eny/6/cEvV/7uLqCJQkg3h4MNP6bd4WsX0X/WduAUFUJ6\n/uXKx4MMd9P0EBxU3/tj24FTVHpD9vfvjp38DtkXbQc+iL7zFzlFaNqGr4DSdj7bDnwQfefP\nuXZRbQc+iL7zJyQG0nfhhcRA+i68kCBASFFtB06RkKLaDpwiIUW1Hfgg+s6fkKLaDnwQfedP\nSFFtBz6IvvMnJAbSd+GFxED6LryQIEBIUW0HTpGQotoOnCIhRbUd+CD6zp+QotoOfBB9509I\nUW0HPoi+8yckBtJ34YXEQPouvJAgQEhRbQdOkZCi2g6cIiFFtR34IPrOn5Ci2g58EH3nT0hR\nbQc+iL7zJyQG0nfhhcRA+i68kCBASFFtB06RkKLaDpwiIUW1Hfgg+s6fkKLaDnwQfedPSFFt\nBz6IvvMnJAbSd+GFxED6LryQIEBIUW0HTpGQotoOnCIhRbUd+CD6zp+QotoOfBB9509IUW0H\nPoi+8yckBtJ34YXEQPouvJAgQEhRbQdOkZCi2g6cIiFFtR34IPrOn5Ci2g58EH3nT0hRbQc+\niL7zJyQG0nfhhUTQNLf5bvlFNhlwF5AkpKi2A6dISFFtB06RkKLaDpwiIUW1HThFQopqO3CK\nhAQBQoIAIUGAkKLaDpwiIUW1HThFQopqO3CKhBTVduAUCen1juc21w2nSEgQICQIEBIECAkC\nhAQBQoIAIUGAkCBASBAgpCSnJlwtIeUcK5LSdRJSznTyJ1dGSDHTX39zTYQUI6RrJqQYIV0z\nIeV4jXTFhJTjqN0Vu2hI97fb48dAtzf359rFvLyPdLUuGNLT+uQj1Zuz7AJmcsGQbqbVz4fj\nV4+71XRzjl3ATC4Y0mp6+O/rh2l1jl3ATC4Y0qvXD++/mBASzXhEgoDLvkbaPR6/8hqJpbnk\n4e/NyVG79dNZdgHzuOz7SDfH95FW29uFvo/E1XJmAwSME5JLYNPYJQ9/rz54QlffBczkou8j\nTdt3DzHUdwEzuWhIh6Pen0pJSDRz2TMbnrbT9GN3vl3ATC59itDD4QD49u7h/QcmIdHM5c+1\ne7hZfXhgTkg0M8tJqw9327WQWJK5zv4+zy5gJkKCgHHObLjwLiBJSBAgJAgQEgQICQKEBAFC\nggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBA\nSBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIE\nCAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKC\nACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBI\nECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJMYxTW1X\nXkiM4lhR15SExCimkz/bERKDmP76uxchMQgh5fWcS0qElNdzLqnxGimu6WRS4qhdXNPJpMj7\nSGFtp5NrJSQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAk\nCBg0JGjmG/fyfDiLYW5qrmr+rurGfpG5qbmq+buqG/tF5qbmqubvqm7sF5mbmquav6u6sV9k\nbmquav6u6sZ+kbmpuar5u6ob+0Xmpuaq5u+qbuwXmZuaq5q/q7qxX2Ruaq5q/q7qxn6Ruam5\nqvm7qhv7Ream5qrm76pu7BeZm5qrmr+rurFwLkKCACFBgJAgQEgQICQIEBIECAkChAQBQoIA\nIUGAkCBASBAgJAgQEgQICQKE9KJw/fQr9DxVqx+Pn/zpMw9nAFdwEz9HSF/xe7JWnyrpGib1\nCm7i51zDYuc8z9bTZrqZeySjcPd5IaSveJmtp2k180CG4e7z4iSkaXpaT9tfX9ytp9Xd8/dO\nvuTPbB3+/ne6fn33dlrd7vc30/Eh6/mn/vz8639dBiG9eBXS9rjE2+PLgM3hWydfsn/9iPTv\ndE3T7eHL3ebw582/Ib3+12UQ0ouTYw2/7g1Pv/7aHf769TJg9+pLDp6zeNw8Z/L3dB2/c/fy\n5+rfkF7/6zII6cWrkO4P39lOh/vH0+FZy8mXHPx31O7prel6/s40Pe7/RHQa0ut/XYbF3JCq\nV0/tXv76ry3Hxv9y+j7SW9O1379O5+//ev3nEizmhlQJ6StOZ0JIB4u5IVVvhfTWP3LwZkh/\nfUdIV+nfbLZ/ji1sHWZ47Y2QTubo/4Z0L6TF+zekn9PqYb+/O7x6PvmSgzdCOpmjt0JaT3eH\nY3pCWro3nsht/pxOdvIl+zdDOpmjt0K6O/zjVkiL99Yrorv1NL2c4HzyJW+H9GeO3gppf7ua\nfniNBLxHSBAgJAgQEgQICQKEBAFCggAhQYCQIEBIECAkCBASBAgJAoQEAUKCACFBgJAgQEgQ\nICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBIQ5lOfPCjX/j1\n0H6T9PkJaSifD2n9+ZX7wo/yXeZ4PJ/6xapf+O2ry/lFrQMzx+MRUkPmeDy/7/jT9LSetvvj\nrwtf3R2/tdtO0+pm//Ic8Pijt9Pqdr+/maab40/897PT9Lg9/tNnXnBRZYbH8yek7TGP7TGF\nza/v3D6/ero5Cen4rd3m5dsnP/uruMOXt0K6CDM8nj8hbZ5+/bU7/PW0mXaH7/zc738e//35\nh44/cvfy5+qvnz1+c+2p3UWY4/H8Cen+8Nd2OuT0dHyS9+fff4d0f/zz8eU7Jz/7+5+EdBHm\neDx/Qnr56+R4+OPudvMqpNd/nvzs6TcvfQuukDkezzshbf77SkhjMcfj+Sek//7lx7S+2z2+\nG9Jf/xchXYg5Hs9fIW2n3at/eS+kv39WSJdijsfzV0g/p9XDfn/3+wDCw+/XSI/7f0N69bO/\nv/n8o5yVkMbzV0gvL4xWj8e3XY/uDyfQHQ53//O49Odn/3zz+Uc5KyGN5++QDmcrTD+Ojyo/\npmlzvzs84Nyv3wzpz8/++ebzj3JWQoIAIUGAkCBASBAgJAgQEgQICQKEBAFCggAhQYCQIEBI\nECAkCBASBAgJAoQEAUKCACFBgJAgQEgQICQIEBIECAkChAQBQoIAIUGAkCBASBAgJAgQEgQI\nCQL+B7Qzdw9/W2KbAAAAAElFTkSuQmCC", "text/plain": [ "plot without title" ] }, "metadata": { "filenames": { "image/png": "C:\\Users\\phamilton\\Documents\\Projects\\dsm\\dsm_website\\_build\\jupyter_execute\\04_causal_inference\\rand_experiments_6_1.png" } }, "output_type": "display_data" } ], "source": [ "boxplot(StreamingMinutes ~ AccountType, ylab = \"Streaming Minutes\", xlab = \"Treatment\")\n", "t.test(musicfiExp$StreamingMinutes ~ musicfiExp$AccountType)" ] }, { "cell_type": "markdown", "id": "5e826c99", "metadata": {}, "source": [ "The difference may be relatively small, but the low p-value indicates that the difference is statistically significant. \n", "\n", "\n", "## Designing an Experiment\n", "\n", "Now that we know why experimentation is necessary, let’s review the steps that businesses use to design their experiments.\n", "\n", "#### 1. Generate a Hypothesis\n", "\n", "The starting point to any experiment is to generate a hypothesis. For companies these hypotheses take the following form:\n", "\n", "> If we [do this] then [this outcome] will increase/decrease by [this much].\n", "\n", "+ [do this]: describes the treatment; most of the time, we implicitly assume that the control is the current approach, and hence it is not explicitly mentioned.\n", "+ [this outcome]: describes the outcome we expect the treatment will affect.\n", "+ [this much]: describes our best guess at the likely effect of the treatment on the outcome. It is also essential to make a reasonable guess at what the effect will be; this guess will form the basis of a calculation that will determine the number of people (sample size) necessary to detect an effect of such a magnitude in an experiment.\n", "\n", "Some examples:\n", "\n", "+ If we enlarge our “Buy now” button by 20% then total sales will increase by +5%.\n", "+ If we launch a simpler user interface on our home page then total sessions will increase by 10%.\n", "+ If we remove fake accounts then complaints will decrease by 2%.\n", "\n", "Although the above examples focus on a single outcome, it is common to look at one to five primary outcomes and possibly another ten to fifteen secondary (less important) outcomes. No matter the number, however, it is crucial that these are specified before data collection.\n", "\n", "#### 2. Select the Study Population\n", "\n", "After generating the hypothesis, we have to identify the group(s) of people who will take part in the study; this is often called determining the inclusion/exclusion criteria. Choosing these criteria depends on the context.\n", "\n", "For the above examples we could consider the following populations:\n", "\n", "+ All U.S. cell phone users that visit the website.\n", "+ All customers using the company’s iOS mobile application with version XXX or above.\n", "+ All worldwide customers.\n", "\n", "This step will provide us with an estimate of the overall population size. From here, we need to determine how we are going to recruit people into our study. The next step explains how to assign people to either treatment or control.\n", "\n", "#### 3. Define the Assignment Mechanism\n", "\n", "The assignment mechanism is a probability distribution that specifies the likelihood of each person in our study receiving a treatment. There are, of course, many possible assignment mechanisms for an experiment on a population. Below, we introduce the two most popular mechanisms:\n", "\n", "+ **Bernoulli Design**: Each person is independently assigned with probability $p$ to receive the treatment and a probability $1 − p$ to receive the control. In other words, for every person, we toss an independent coin with probability of heads = $p$, if we get heads, we assign them to treatment, and if we get tails, we assign them to control. This can be implemented using the `sample()` command with `replace = TRUE`.\n", "+ **Completely Randomized Design:** In this design, we specify how many people will be placed in each condition. For example, we might specify that $n_{0}$ will get the control and $n_{1}$ will get the treatment. Then, we randomly pick $n_{0}$ of our participants to receive the control and $n_{1}$ to receive the treatment. This again can be implemented using the `sample()` command.\n", "\n", "The main difference between a Bernoulli and a completely randomized design is that we guarantee how many people receive the treatment in the latter. On the other hand, the Bernoulli is much easier to implement if we do not know exactly how many people will be in our study (*e.g.*, people “arrive” one after another)." ] } ], "metadata": { "jupytext": { "cell_metadata_filter": "-all", "formats": "md:myst", "text_representation": { "extension": ".md", "format_name": "myst", "format_version": 0.13, "jupytext_version": "1.11.5" } }, "kernelspec": { "display_name": "R", "language": "R", "name": "ir" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "3.6.1" }, "source_map": [ 16, 59, 65, 67, 71, 73, 77, 80 ] }, "nbformat": 4, "nbformat_minor": 5 }