Hal Varian (Google Chief Economist) – From Possibilities to Responsibilities (Mar 2017)

Chapters

Abstract

Unraveling the Complexity of Causal Inference in the Social Sciences and Beyond

In the field of social sciences and beyond, understanding the intricate relationship between cause and effect is paramount. The article delves into various methodologies and challenges associated with causal inference, highlighting the significance of distinguishing between mere correlations and actual causal relationships. It underscores the complexity of causal analysis in fields like marketing, economics, and public policy, examining techniques like regression, randomized controlled trials (RCTs), and counterfactual models. The discussion extends to the application of machine learning in counterfactual estimation, addressing inherent challenges such as selection bias and unobserved confounding. Emphasizing the importance of domain expertise and methodological rigor, the article also explores the role of data-driven decision-making and the increasing relevance of human capital in leveraging AI and machine learning for causal analysis.

The Complexities of Causal Inference in Marketing and Social Sciences

Causal inference is critical in establishing cause-and-effect relationships, a task that goes beyond simple correlation analysis. In marketing, for instance, understanding the causal impact of advertising expenditure on donations demands meticulous scrutiny. A naive approach, such as using regression, may lead to biased results due to the non-random selection of advertising expenses. Similarly, confounding variables, like varying interests in specific products across regions, can simultaneously affect both the outcome (e.g., movie revenue) and the variables under study (e.g., advertising expenditure), leading to skewed estimates. This challenge is not unique to marketing but pervades social sciences, where variables are often non-randomly selected, necessitating a nuanced approach to data analysis.

Challenges in Causal Inference:

Establishing cause-and-effect relationships is a primary objective of causal inference, as mere correlations can be misleading. In economics, for instance, understanding the true impact of advertising expenditure on donations requires careful analysis. A straightforward regression approach assumes independence between advertising expenditure and other factors, which is often invalid. Individuals choose advertising expenditure based on observed factors, like varying product interests, that also influence the outcome. Thus, a confounding variable, such as regional product preferences, may simultaneously affect both advertising expenditure and movie revenue, leading to biased estimates.

Confounding Variables:

Confounding variables pose a significant challenge in causal inference, as they can lead to spurious relationships between variables. In the marketing context, factors like regional product interests can confound the relationship between advertising expenditure and movie revenue. The presence of confounding variables highlights the need for rigorous methods to disentangle the true causal effect from confounding influences.

Methodological Approaches in Causal Inference

In observational social science, the use of non-experimental data, where predictors may be interrelated and influenced by unseen factors, is common. Though traditional regression methods are valid under stable data-generating processes, they fall short in causal inference, which requires understanding the impact of altering these processes. This is evident in examples from economics, such as the relationships between fertilizer use and crop yields, education and income, and healthcare access and income. The crux of causal inference lies in the concept of the “treatment effect,” which varies depending on whether the treatment is imposed or offered. Random assignment in experiments helps eliminate selection bias, equating the treatment’s impact on the treated group to its effect on the entire population.

Challenges of Estimating Causal Effects:

Causal inference often requires experimentation to measure the impact of a change. Randomized experiments are ideal but may be infeasible. Observational data, though more accessible, contains confounding factors that bias causal inferences.

Methods for Estimating Causal Effects:

Several methods can estimate causal effects from observational data:

1. Randomized Experiments:

Randomly assigning treatment to subjects allows for direct comparison between treated and control groups, providing an accurate estimate of the causal effect.

2. Natural Experiments:

Naturally occurring events that randomly assign treatment can serve as quasi-experiments, enabling causal analysis.

3. Instrumental Variables:

Using a variable that affects the treatment but is independent of the confounding factors can help estimate causal effects through statistical techniques.

Experimental and Quasi-Experimental Techniques

Observational data alone is typically inadequate for causal inference, necessitating experimental methods for accurate impact measurement. Randomized experiments, like A/B testing, offer causal estimates by contrasting treated and control groups. Counterfactual models estimate potential outcomes in the absence of treatment, essential for calculating causal effects. Natural and quasi-experiments, such as the increased viewership during major events like the Super Bowl in host cities, provide opportunities for causal analysis. Instrumental variables, particularly in economics, help identify factors influencing the treatment but remain independent of other confounding elements. These methods, however, require careful consideration of confounding variables that can distort causal relationships.

Examples of Causal Inference Methods:

1. Randomized Experiments:

A/B testing is a common randomized experiment in which subjects are randomly assigned to treatment or control groups, allowing for direct comparison of outcomes to estimate the causal effect.

2. Natural Experiments:

Major events like the Super Bowl can serve as natural experiments, as the increased viewership in host cities can be attributed to the event itself rather than other factors, providing insights into the causal impact of the event.

3. Instrumental Variables:

In economics, instrumental variables can be used to estimate the causal effect of ticket prices on air travel revenue. Factors like fuel costs or unionization can serve as instrumental variables as they affect ticket prices but are independent of confounding factors like economic conditions.

Advanced Methodologies in Causal Analysis

Regression Discontinuity Design (RDD) and Difference-in-Differences (DID) are pivotal in estimating causal impacts. RDD leverages sharp changes in treatment assignment at specific thresholds, exemplified by the impact of class sizes on student performance. DID, on the other hand, compares outcomes before and after treatment, assuming parallel trends in the absence of treatment. Both techniques illustrate the nuanced nature of causal analysis in policy evaluation and intervention effectiveness.

Advanced Causal Analysis Methods:

1. Regression Discontinuity Design (RDD):

RDD exploits sharp changes in treatment assignment at specific thresholds to estimate causal effects. For instance, studying the impact of class size on student performance by comparing students just above and below a class size threshold provides a natural experiment-like setting.

2. Difference-in-Differences (DID):

DID compares outcomes before and after treatment, assuming parallel trends in the absence of treatment. For example, evaluating the effectiveness of a job training program by comparing the outcomes of participants with those of a control group before and after the program provides insights into the causal impact of the program.

Counterfactuals and Machine Learning

Counterfactual models are crucial for estimating intervention impacts, with DID providing a straightforward approach by comparing treated and control groups’ outcome changes. Machine learning enhances this by predicting counterfactual outcomes, aiding in causal effect estimation. However, challenges like selection bias, unobserved confounding, and measurement errors persist. Domain knowledge becomes vital in selecting appropriate variables and interpreting counterfactual analyses.

Counterfactual Models and Machine Learning:

1. Counterfactual Models:

Counterfactual models estimate potential outcomes in the absence of treatment, allowing for the estimation of causal effects. DID is a common counterfactual method that compares outcomes before and after treatment, assuming parallel trends in the absence of treatment.

2. Machine Learning in Counterfactual Estimation:

Machine learning algorithms can be used to predict counterfactual outcomes, improving the accuracy of causal effect estimation. However, challenges like selection bias, unobserved confounding, and measurement errors remain, requiring careful consideration and domain expertise.

Practical Applications and Future Directions

The article concludes with an exploration of practical applications and future directions in causal inference. It highlights the underutilization of experimentation in organizations and the public sector, advocating for a cultural shift towards data-driven decision-making. The increasing importance of human capital in AI and machine learning, coupled with the availability of open-source tools and online learning resources, signifies a new era in causal analysis. The article advocates for prioritizing the testing of confounding variables and leveraging experimentation for more effective policy-making.

In summary, the article provides a comprehensive overview of the complexities and methodologies of causal inference across various domains, emphasizing the necessity of methodological expertise, domain knowledge, and a data-centric approach in unraveling the intricacies of cause and effect.

Regression Discontinuity Design (RDD) and Difference-in-Differences (DID):

– RDD statistically estimates causal impacts by utilizing abrupt changes in treatment assignment at specific thresholds. An example is examining the impact of class size on student performance by comparing students just above and below a class size threshold.

– DID compares outcomes before and after treatment, assuming parallel trends in the absence of treatment. For instance, comparing the health outcomes of individuals who received Medicaid with those who didn’t, before and after the Medicaid program, can estimate the causal impact of Medicaid on health.

Impact of ISP Speed on Housing Values:

– A study compared housing values on either side of ISP boundaries. Very similar cases allowed for comparison of housing values.

– Faster ISP speeds were associated with higher housing values.

Impact of Minimum Legal Drinking Age on Mortality:

– Comparing 20-year-olds to 21-year-olds before and after the legal drinking age was raised, a significant jump in mortality at age 21 was attributed to increased drinking, indicating the potential impact of changing the drinking age.

Impact of Fellowships or Student Aid:

– Examining the impact of student aid awarded based on test scores and comparing outcomes of those just above and below the breakout point can assess educational program impacts.

Oregon Medicaid Lottery:

– A true randomized experiment due to Medicaid funds shortage, where a lottery determined who received Medicaid, enabled the study of health outcomes of those who received Medicaid and those who didn’t.

Causal Inference Using Counterfactuals and Machine Learning:

– Counterfactuals are models that estimate what would have happened if a different treatment or intervention had been applied.

– Difference-in-Differences (DID) is a simple counterfactual model that compares the difference in outcomes between a treated and control group before and after the treatment, assuming that the difference in outcomes between the groups would have remained the same without the treatment.

– Machine learning algorithms can be used to build predictive models that explain or describe historical data and extrapolate to new populations or into the future, creating counterfactuals.

– The difference between the observed and the counterfactual outcomes provides a true causal estimate of the treatment’s impact.

Challenges in Causal Inference:

– Hidden or unobserved factors can influence outcomes and confound causal relationships. Adding more variables to models may not eliminate this error term, especially in complex situations with human choices.

– Propensity score matching models estimate the likelihood of an individual receiving treatment based on their characteristics. Comparing individuals with similar propensity scores who received and did not receive treatment can estimate causal effects.

Opportunities for Machine Learning in Causal Inference:

– Machine learning techniques, such as regression discontinuity, can be used to estimate counterfactuals and causal effects. They can evaluate the impact of programs or interventions where random assignment is not feasible, such as online education.

– Collaboration between domain experts and methodologists is essential, as domain knowledge is crucial for identifying plausible causal relationships and factors that influence treatment effects, while methodological experts provide statistical and quantitative skills to analyze data and estimate causal effects.

Causality and the Benefits of Experimentation:

– Experimentation is crucial for evaluating product improvements, advertising effectiveness, and establishing causality. Organizations often overlook the value of smaller, ongoing experiments that can provide valuable insights.

– Simple experimental techniques, such as rolling out interventions gradually, encourage an experimental culture and rigorous methods for answering questions, avoiding reliance on opinions or intuition.

– Causal inference techniques like odds ratios and logistic regression are commonly used. Clinical studies with random assignment and placebos are the gold standard for causality in some domains, but not always practical in social sciences. Case-control studies, though retrospective, can provide insight into causality by examining relationships between variables.

Challenges in Public Sector Experimentation:

– Fear of failure, short-term focus, and denying treatment to individuals in need are barriers to experimentation in the public sector.

– Using lotteries for treatment assignment can be effective when demand exceeds supply. Leveraging experiments to push for evidence-based policies can improve public sector decision-making. External pressure and advocacy may be necessary to drive experimentation and innovation in the public sector.

Notes by: Flaneur