Educational policy-makers around the world constantly make decisions about how to use scarce resources to improve the education of children. Unfortunately, their decisions are rarely informed by evidence on the consequences of these initiatives in other settings. Nor are decisions typically accompanied by well-formulated plans to evaluate their causal impacts. As a result, knowledge about what works in different situations has been very slow to accumulate. Over the last several decades, advances in research methodology, administrative record keeping, and statistical software have dramatically increased the potential for researchers to conduct compelling evaluations of the causal impacts of educational interventions, and the number of well-designed studies is growing. Written in clear, concise prose, Methods Matter: Improving Causal Inference in Educational and Social Science Research offers essential guidance for those who evaluate educational policies. Using numerous examples of high-quality studies that have evaluated the causal impacts of important educational interventions, the authors go beyond the simple presentation of new analytical methods to discuss the controversies surrounding each study, and provide heuristic explanations that are also broadly accessible. Murnane and Willett offer strong methodological insights on causal inference, while also examining the consequences of a wide variety of educational policies implemented in the U.S. and abroad. Representing a unique contribution to the literature surrounding educational research, this landmark text will be invaluable for students and researchers in education and public policy, as well as those interested in social science.
An ambitious new model of experimentation that will reorient our understanding of the key features of experimental practice. What is experimental knowledge, and how do we get it? While there is general agreement that experiment is a crucial source of scientific knowledge, how experiment generates that knowledge is far more contentious. In this book, philosopher of science James Mattingly explains how experiments function. Specifically, he discusses what it is about experimental practice that transforms observations of what may be very localized, particular, isolated systems into what may be global, general, integrated empirical knowledge. Mattingly argues that the purpose of experimentation is the same as the purpose of any other knowledge-generating enterprise—to change the state of information of the knower. This trivial-seeming point has a non-trivial consequence: to understand a knowledge-generating enterprise, we should follow the flow of information. Therefore, the account of experimental knowledge Mattingly provides is based on understanding how information flows in experiments: what facilitates that flow, what hinders it, and what characteristics allow it to flow from system to system, into the heads of researchers, and finally into our store of scientific knowledge.
Sections include: experiments and generalised causal inference; statistical conclusion validity and internal validity; construct validity and external validity; quasi-experimental designs that either lack a control group or lack pretest observations on the outcome; quasi-experimental designs that use both control groups and pretests; quasi-experiments: interrupted time-series designs; regresssion discontinuity designs; randomised experiments: rationale, designs, and conditions conducive to doing them; practical problems 1: ethics, participation recruitment and random assignment; practical problems 2: treatment implementation and attrition; generalised causal inference: a grounded theory; generalised causal inference: methods for single studies; generalised causal inference: methods for multiple studies; a critical assessment of our assumptions.
In this Element and its accompanying second Element, A Practical Introduction to Regression Discontinuity Designs: Extensions, Matias Cattaneo, Nicolás Idrobo, and Rocıìo Titiunik provide an accessible and practical guide for the analysis and interpretation of regression discontinuity (RD) designs that encourages the use of a common set of practices and facilitates the accumulation of RD-based empirical evidence. In this Element, the authors discuss the foundations of the canonical Sharp RD design, which has the following features: (i) the score is continuously distributed and has only one dimension, (ii) there is only one cutoff, and (iii) compliance with the treatment assignment is perfect. In the second Element, the authors discuss practical and conceptual extensions to this basic RD setup.
Some central questions in the natural and social sciences can't be answered by controlled laboratory experiments, often considered to be the hallmark of the scientific method. This impossibility holds for any science concerned with the past. In addition, many manipulative experiments, while possible, would be considered immoral or illegal. One has to devise other methods of observing, describing, and explaining the world. In the historical disciplines, a fruitful approach has been to use natural experiments or the comparative method. This book consists of eight comparative studies drawn from history, archeology, economics, economic history, geography, and political science. The studies cover a spectrum of approaches, ranging from a non-quantitative narrative style in the early chapters to quantitative statistical analyses in the later chapters. The studies range from a simple two-way comparison of Haiti and the Dominican Republic, which share the island of Hispaniola, to comparisons of 81 Pacific islands and 233 areas of India. The societies discussed are contemporary ones, literate societies of recent centuries, and non-literate past societies. Geographically, they include the United States, Mexico, Brazil, western Europe, tropical Africa, India, Siberia, Australia, New Zealand, and other Pacific islands. In an Afterword, the editors discuss how to cope with methodological problems common to these and other natural experiments of history.
We shall examine the validity of 16 experimental designs against 12 common threats to valid inference. By experiment we refer to that portion of research in which variables are manipulated and their effects upon other variables observed. It is well to distinguish the particular role of this chapter. It is not a chapter on experimental design in the Fisher (1925, 1935) tradition, in which an experimenter having complete mastery can schedule treatments and measurements for optimal statistical efficiency, with complexity of design emerging only from that goal of efficiency. Insofar as the designs discussed in the present chapter become complex, it is because of the intransigency of the environment: because, that is, of the experimenter’s lack of complete control.
Thoroughly updated to reflect changes in both research and methods, this Third Edition of Remler and Van Ryzin’s innovative, standard-setting text is imbued with a deep commitment to making social and policy research methods accessible and meaningful. Research Methods in Practice: Strategies for Description and Causation motivates readers to examine the logic and limits of social science research from academic journals and government reports. A central theme of causation versus description runs through the text, emphasizing the idea that causal research is essential to understanding the origins of social problems and their potential solutions. Readers will find excitement in the research experience as the best hope for improving the world in which we live, while also acknowledging the trade-offs and uncertainties in real-world research.