A thorough and comprehensive guide to the theoretical, practical, and methodological approaches used in survey experiments across disciplines such as political science, health sciences, sociology, economics, psychology, and marketing
This book explores and explains the broad range of experimental designs embedded in surveys that use both probability and non-probability samples. It approaches the usage of survey-based experiments with a Total Survey Error (TSE) perspective, which provides insight on the strengths and weaknesses of the techniques used.
Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment addresses experiments on within-unit coverage, reducing nonresponse, question and questionnaire design, minimizing interview measurement bias, using adaptive design, trend data, vignettes, the analysis of data from survey experiments, and other topics, across social, behavioral, and marketing science domains.
Each chapter begins with a description of the experimental method or application and its importance, followed by reference to relevant literature. At least one detailed original experimental case study then follows to illustrate the experimental method’s deployment, implementation, and analysis from a TSE perspective. The chapters conclude with theoretical and practical implications on the usage of the experimental method addressed. In summary, this book:
- Fills a gap in the current literature by successfully combining the subjects of survey methodology and experimental methodology in an effort to maximize both internal validity and external validity
- Offers a wide range of types of experimentation in survey research with in-depth attention to their various methodologies and applications
- Is edited by internationally recognized experts in the field of survey research/methodology and in the usage of survey-based experimentation - featuring contributions from across a variety of disciplines in the social and behavioral sciences
- Presents advances in the field of survey experiments, as well as relevant references in each chapter for further study
- Includes more than 20 types of original experiments carried out within probability sample surveys
- Addresses myriad practical and operational aspects for designing, implementing, and analyzing survey-based experiments by using a Total Survey Error perspective to address the strengths and weaknesses of each experimental technique and method
Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment is an ideal reference for survey researchers and practitioners in areas such political science, health sciences, sociology, economics, psychology, public policy, data collection, data science, and marketing. It is also a very useful textbook for graduate-level courses on survey experiments and survey methodology.
Table of Contents
List of Contributors xix
Preface by Dr. Judith Tanur xxv
About the Companion Website xxix
1 Probability Survey-Based Experimentation and the Balancing of Internal and External Validity Concerns 1
Paul J. Lavrakas, Courtney Kennedy, Edith D. de Leeuw, Brady T. West, Allyson L. Holbrook, and Michael W. Traugott
1.1 Validity Concerns in Survey Research 3
1.2 Survey Validity and Survey Error 5
1.3 Internal Validity 6
1.4 Threats to Internal Validity 8
1.5 External Validity 11
1.6 Pairing Experimental Designs with Probability Sampling 12
1.7 Some Thoughts on Conducting Experiments with Online Convenience Samples 12
1.8 The Contents of this Book 15
References 15
Part I Introduction to Section on Within-Unit Coverage 19
Paul J. Lavrakas and Edith D. de Leeuw
2 Within-Household Selection Methods: A Critical Review and Experimental Examination 23
Jolene D. Smyth, Kristen Olson, and Mathew Stange
2.1 Introduction 23
2.2 Within-Household Selection and Total Survey Error 24
2.3 Types of within-Household Selection Techniques 24
2.4 Within-Household Selection in Telephone Surveys 25
2.5 Within-Household Selection in Self-Administered Surveys 26
2.6 Methodological Requirements of Experimentally Studying Within-Household Selection Methods 27
2.7 Empirical Example 30
2.8 Data and Methods 31
2.9 Analysis Plan 34
2.10 Results 35
2.11 Discussion and Conclusions 40
References 42
3 Measuring within-Household Contamination: The Challenge of Interviewing More Than One Member of a Household 47
Colm O’Muircheartaigh, Stephen Smith, and Jaclyn S.Wong
3.1 Literature Review 47
3.2 Data and Methods 50
Investigators 53
Field/Project Directors 53
3.3 The Sequence of Analyses 55
3.4 Results 55
3.5 Effect on Standard Errors of the Estimates 57
3.6 Effect on Response Rates 58
3.7 Effect on Responses 61
3.8 Substantive Results 64
References 64
Part II Survey Experiments with Techniques to Reduce Nonresponse 67
Edith D. de Leeuw and Paul J. Lavrakas
4 Survey Experiments on Interactions and Nonresponse: A Case Study of Incentives and Modes 69
A. Bianchi and S. Biffignandi
4.1 Introduction 69
4.2 Literature Overview 70
4.3 Case Study: Examining the Interaction between Incentives and Mode 73
4.4 Concluding Remarks 83
Acknowledgments 85
References 86
5 Experiments on the Effects of Advance Letters in Surveys 89
Susanne Vogl, Jennifer A. Parsons, Linda K. Owens, and Paul J. Lavrakas
5.1 Introduction 89
5.2 State of the Art on Experimentation on the Effect of Advance Letters 93
5.3 Case Studies: Experimental Research on the Effect of Advance Letters 95
5.4 Case Study I: Violence against Men in Intimate Relationships 96
5.5 Case Study II: The Neighborhood Crime and Justice Study 100
5.6 Discussion 106
5.7 Research Agenda for the Future 107
References 108
Part III Overview of the Section on the Questionnaire 111
Allyson Holbrook and Michael W. Traugott
6 Experiments on the Design and Evaluation of Complex Survey Questions 113
Paul Beatty, Carol Cosenza, and Floyd J. Fowler Jr.
6.1 Question Construction: Dangling Qualifiers 115
6.2 Overall Meanings of Question Can Be Obscured by Detailed Words 117
6.3 Are Two Questions Better than One? 119
6.4 The Use of Multiple Questions to Simplify Response Judgments 121
6.5 The Effect of Context or Framing on Answers 122
6.6 Do Questionnaire Effects Vary Across Sub-groups of Respondents? 124
6.7 Discussion 126
References 128
7 Impact of Response Scale Features on Survey Responses to Behavioral Questions 131
Florian Keusch and Ting Yan
7.1 Introduction 131
7.2 Previous Work on Scale Design Features 132
7.3 Methods 134
7.4 Results 136
7.5 Discussion 141
Acknowledgment 143
7.A Question Wording 143
7.A.1 Experimental Questions (One Question Per Screen) 143
7.A.2 Validation Questions (One Per Screen) 144
7.A.3 GfK Profile Questions (Not Part of the Questionnaire) 145
7.B Test of Interaction Effects 145
References 146
8 Mode Effects Versus Question Format Effects: An Experimental Investigation of Measurement Error Implemented in a Probability-Based Online Panel 151
Edith D. de Leeuw, Joop Hox, and Annette Scherpenzeel
8.1 Introduction 151
8.2 Experiments and Probability-Based Online Panels 153
8.3 Mixed-Mode Question Format Experiments 154
8.4 Summary and Discussion 161
Acknowledgments 162
References 162
9 Conflicting Cues: Item Nonresponse and Experimental Mortality 167
David J. Ciuk and Berwood A. Yost
9.1 Introduction 167
9.2 Survey Experiments and Item Nonresponse 167
9.3 Case Study: Conflicting Cues and Item Nonresponse 170
9.4 Methods 170
9.5 Issue Selection 171
9.6 Experimental Conditions and Measures 172
9.7 Results 173
9.8 Addressing Item Nonresponse in Survey Experiments 174
9.9 Summary 178
References 179
10 Application of a List Experiment at the Population Level: The Case of Opposition to Immigration in the Netherlands 181
Mathew J. Creighton, Philip S. Brenner, Peter Schmidt, and Diana Zavala-Rojas
10.1 Fielding the Item Count Technique (ICT) 183
10.2 Analyzing the Item Count Technique (ICT) 185
10.3 An Application of ICT: Attitudes toward Immigrants in the Netherlands 186
10.4 Limitations of ICT 190
References 192
Part IV Introduction to Section on Interviewers 195
Brady T. West and Edith D. de Leeuw
11 Race- and Ethnicity-of-Interviewer Effects 197
Allyson L. Holbrook, Timothy P. Johnson, and Maria Krysan
11.1 Introduction 197
11.2 The Current Research 205
11.3 Respondents and Procedures 207
11.4 Measures 207
11.5 Analysis 210
11.6 Results 211
11.7 Discussion and Conclusion 219
References 221
12 Investigating Interviewer Effects and Confounds in Survey-Based Experimentation 225
Paul J. Lavrakas, Jenny Kelly, and Colleen McClain
12.1 Studying Interviewer Effects Using a Post hoc Experimental Design 226
12.2 Studying Interviewer Effects Using A Priori Experimental Designs 230
12.3 An Original Experiment on the Effects of Interviewers Administering Only One Treatment vs. Interviewers Administrating Multiple Treatments 232
12.4 Discussion 239
References 242
Part V Introduction to Section on Adaptive Design 245
Courtney Kennedy and Brady T. West
13 Using Experiments to Assess Interactive Feedback That Improves Response Quality in Web Surveys 247
Tanja Kunz and Marek Fuchs
13.1 Introduction 247
13.2 Case Studies - Interactive Feedback in Web Surveys 251
13.3 Methodological Issues in Experimental Visual Design Studies 258
References 269
14 Randomized Experiments for Web-Mail Surveys Conducted Using Address-Based Samples of the General Population 275
Z. Tuba Suzer-Gurtekin, Mahmoud Elkasabi, James M. Lepkowski, Mingnan Liu, and Richard Curtin
14.1 Introduction 275
14.2 Study Design and Methods 278
14.3 Results 281
14.4 Discussion 285
References 287
Part VI Introduction to Section on Special Surveys 291
Michael W. Traugott and Edith D. de Leeuw
15 Mounting Multiple Experiments on Longitudinal Social Surveys: Design and Implementation Considerations 293
Peter Lynn and Annette Jäckle
15.1 Introduction and Overview 293
15.2 Types of Experiments that Can Be Mounted in a Longitudinal Survey 294
15.3 Longitudinal Experiments and Experiments in Longitudinal Surveys 295
15.4 Longitudinal Surveys that Serve as Platforms for Experimentation 296
15.5 The Understanding Society Innovation Panel 298
15.6 Avoiding Confounding of Experiments 299
15.7 Allocation Procedures 301
15.8 Refreshment Samples 304
15.9 Discussion 305
15.A Appendix: Stata Syntax to Produce Table 15.3 Treatment Allocations 306
References 306
16 Obstacles and Opportunities for Experiments in Establishment Surveys Supporting Official Statistics 309
Diane K. Willimack and Jaki S. McCarthy
16.1 Introduction 309
16.2 Some Key Differences between Household and Establishment Surveys 310
16.3 Existing Literature Featuring Establishment Survey Experiments 312
16.4 Key Considerations for Experimentation in Establishment Surveys 314
16.5 Examples of Experimentation in Establishment Surveys 318
16.6 Discussion and Concluding Remarks 323
Acknowledgments 324
References 324
Part VII Introduction to Section on Trend Data 327
Michael W. Traugott and Paul J. Lavrakas
17 Tracking Question-Wording Experiments across Time in the General Social Survey, 1984-2014 329
Tom W. Smith and Jaesok Son
17.1 Introduction 329
17.2 GSS Question-Wording Experiment on Spending Priorities 330
17.3 Experimental Analysis 330
17.4 Summary and Conclusion 338
17.A National Spending Priority Items 339
References 340
18 Survey Experiments and Changes in Question Wording in Repeated Cross-Sectional Surveys 343
Allyson L. Holbrook, David Sterrett, Andrew W. Crosby, Marina Stavrakantonaki, Xiaoheng Wang, Tianshu Zhao, and Timothy P. Johnson
18.1 Introduction 343
18.2 Background 344
18.3 Two Case Studies 347
18.4 Implications and Conclusions 362
Acknowledgments 364
References 364
Part VIII Vignette Experiments in Surveys 369
Allyson Holbrook and Paul J. Lavrakas
19 Are Factorial Survey Experiments Prone to Survey Mode Effects? 371
Katrin Auspurg, Thomas Hinz, and Sandra Walzenbach
19.1 Introduction 371
19.2 Idea and Scope of Factorial Survey Experiments 372
19.3 Mode Effects 373
19.4 Case Study 378
19.5 Conclusion 388
References 390
20 Validity Aspects of Vignette Experiments: Expected “What-If” Differences between Reports of Behavioral Intentions and Actual Behavior 393
Stefanie Eifler and Knut Petzold
20.1 Outline of the Problem 393
20.2 Research Findings from Our Experimental Work 399
20.3 Discussion 411
References 413
Part IX Introduction to Section on Analysis 417
Brady T. West and Courtney Kennedy
21 Identities and Intersectionality: A Case for Purposive Sampling in Survey-Experimental Research 419
Samara Klar and Thomas J. Leeper
21.1 Introduction 419
21.2 Common Techniques for Survey Experiments on Identity 420
21.3 How Limited are Representative Samples for Intersectionality Research? 426
21.4 Conclusions and Discussion 430
Author Biographies 431
References 431
22 Designing Probability Samples to Study Treatment Effect Heterogeneity 435
Elizabeth Tipton, David S. Yeager, Ronaldo Iachan, and Barbara Schneider
22.1 Introduction 435
22.2 Nesting a Randomized Treatment in a National Probability Sample: The NSLM 446
22.3 Discussion and Conclusions 451
Acknowledgments 453
References 453
23 Design-Based Analysis of Experiments Embedded in Probability Samples 457
Jan A. van den Brakel
23.1 Introduction 457
23.2 Design of Embedded Experiments 458
23.3 Design-Based Inference for Embedded Experiments with One Treatment Factor 460
23.4 Analysis of Experiments with Clusters of Sampling Units as Experimental Units 466
23.5 Factorial Designs 468
23.6 A Mixed-Mode Experiment in the Dutch Crime Victimization Survey 472
23.7 Discussion 477
Acknowledgments 478
References 478
24 Extending the Within-Persons Experimental Design: The Multitrait-Multierror (MTME) Approach 481
Alexandru Cernat and Daniel L. Oberski
24.1 Introduction 481
24.2 The Multitrait-Multierror (MTME) Framework 482
24.3 Designing the MTME Experiment 487
24.4 Statistical Estimation for the MTME Approach 489
24.5 Measurement Error in Attitudes toward Migrants in the UK 491
24.6 Results 494
24.7 Conclusions and Future Research Directions 497
Acknowledgments 498
References 498
Index 501
