A new and updated definitive resource for survey questionnaire testing and evaluation
Building on the success of the first Questionnaire Development, Evaluation, and Testing (QDET) conference in 2002, this book brings together leading papers from the Second International Conference on Questionnaire Design, Development, Evaluation, and Testing (QDET2) held in 2016. The volume assesses the current state of the art and science of QDET; examines the importance of methodological attention to the questionnaire in the present world of information collection; and ponders how the QDET field can anticipate new trends and directions as information needs and data collection methods continue to evolve.
Featuring contributions from international experts in survey methodology, Advances in Questionnaire Design, Development, Evaluation and Testing includes latest insights on question characteristics, usability testing, web probing, and other pretesting approaches, as well as:
- Recent developments in the design and evaluation of digital and self-administered surveys
- Strategies for comparing and combining questionnaire evaluation methods
- Approaches for cross-cultural and cross-national questionnaire development
- New data sources and methodological innovations during the last 15 years
- Case studies and practical applications
Advances in Questionnaire Design, Development, Evaluation and Testing serves as a forum to prepare researchers to meet the next generation of challenges, making it an excellent resource for researchers and practitioners in government, academia, and the private sector.
Table of Contents
List of Contributors xvii
Preface xxiii
Part I Assessing the Current Methodology for Questionnaire Design, Development, Testing, and Evaluation 1
1 Questionnaire Design, Development, Evaluation, and Testing: Where are We, and Where are We Headed? 3
Gordon B. Willis
1.1 Current State of the Art and Science of QDET 3
1.2 Relevance of QDET in the Evolving World of Surveys 11
1.3 Looking Ahead: Further Developments in QDET 16
1.4 Conclusion 19
References 20
2 Asking the Right Questions in the Right Way: Six Needed Changes in Questionnaire Evaluation and Testing Methods 25
Don A. Dillman
2.1 Personal Experiences with Cognitive Interviews and Focus Groups 25
2.2 My 2002 Experience at QDET 29
2.3 Six Changes in Survey Research that Require New Perspectives on Questionnaire Evaluation and Testing 33
2.4 Conclusion 42
References 43
3 A Framework for Making Decisions about Question Evaluation Methods 47
Roger Tourangeau, Aaron Maitland, Darby Steiger, and Ting Yan
3.1 Introduction 47
3.2 Expert Reviews 48
3.3 Laboratory Methods 51
3.4 Field Methods 55
3.5 Statistical Modeling for Data Quality 59
3.6 Comparing Different Methods 63
3.7 Recommendations 67
References 69
4 A Comparison of Five Question Evaluation Methods in Predicting the Validity of Respondent Answers to Factual Items 75
Aaron Maitland and Stanley Presser
4.1 Introduction 75
4.2 Methods 76
4.3 Results 79
4.4 Discussion 84
References 85
5 Combining Multiple Question Evaluation Methods: What Does tt Mean When the Data Appear to Conflict? 91
Jo d’Ardenne and Debbie Collins
5.1 Introduction 91
5.2 Questionnaire Development Stages 92
5.3 Selection of Case Studies 93
5.4 Case Study 1: Conflicting Findings Between Focus Groups and Cognitive Interviews 95
5.5 Case Study 2: Conflicting Findings Between Eye-Tracking, Respondent Debriefing Questions, and Interviewer Feedback 97
5.6 Case Study 3: Complementary Findings Between Cognitive Interviews and Interviewer Feedback 100
5.7 Case Study 4: Combining Qualitative and Quantitative Data to Assess Changes to a Travel Diary 104
5.8 Framework of QT Methods 110
5.9 Summary and Discussion 110
References 114
Part II Question Characteristics, Response Burden, and Data Quality 117
6 The Role of Question Characteristics in Designing and Evaluating Survey Questions 119
Jennifer Dykema, Nora Cate Schaeffer, Dana Garbarski, and Michael Hout
6.1 Introduction 119
6.2 Overview of Some of the Approaches Used to Conceptualize, Measure, and Code Question Characteristics 120
6.3 Taxonomy of Question Characteristics 127
6.4 Case Studies 132
6.5 Discussion 141
Acknowledgments 147
References 148
7 Exploring the Associations Between Question Characteristics, Respondent Characteristics, Interviewer Performance Measures, and Survey Data Quality 153
James M. Dahlhamer, Aaron Maitland, Heather Ridolfo, Antuane Allen, and Dynesha Brooks
7.1 Introduction 153
7.2 Methods 157
7.3 Results 174
7.4 Discussion 182
Disclaimer 191
References 191
8 Response Burden: What is it and What Predicts It? 193
Ting Yan, Scott Fricker, and Shirley Tsai
8.1 Introduction 193
8.2 Methods 197
8.3 Results 202
8.4 Conclusions and Discussion 206
Acknowledgments 210
References 210
9 The Salience of Survey Burden and Its Effect on Response Behavior to Skip Questions: Experimental Results from Telephone and Web Surveys 213
Frauke Kreuter, Stephanie Eckman, and Roger Tourangeau
9.1 Introduction 213
9.2 Study Designs and Methods 216
9.3 Manipulating the Interleafed Format 219
9.4 Discussion and Conclusion 224
Acknowledgments 226
References 227
10 A Comparison of Fully Labeled and Top-Labeled Grid Question Formats 229
Jolene D. Smyth and Kristen Olson
10.1 Introduction 229
10.2 Data and Methods 236
10.3 Findings 243
10.4 Discussion and Conclusions 253
Acknowledgments 254
References 255
11 The Effects of Task Difficulty and Conversational Cueing on Answer Formatting Problems in Surveys 259
Yfke Ongena and Sanne Unger
11.1 Introduction 259
11.2 Factors Contributing to Respondents’ Formatting Problems 262
11.3 Hypotheses 267
11.4 Method and Data 268
11.5 Results 275
11.6 Discussion and Conclusion 278
11.7 Further Expansion of the Current Study 281
11.8 Conclusions 282
References 283
Part III Improving Questionnaires on the Web and Mobile Devices 287
12 A Compendium of Web and Mobile Survey Pretesting Methods 289
Emily Geisen and Joe Murphy
12.1 Introduction 289
12.2 Review of Traditional Pretesting Methods 290
12.3 Emerging Pretesting Methods 294
References 308
13 Usability Testing Online Questionnaires: Experiences at the U.S. Census Bureau 315
Elizabeth Nichols, Erica Olmsted-Hawala, Temika Holland, and Amy Anderson Riemer
13.1 Introduction 315
13.2 History of Usability Testing Self-Administered Surveys at the US Census Bureau 316
13.3 Current Usability Practices at the Census Bureau 317
13.4 Participants: “Real Users, Not User Stories” 320
13.5 Building Usability Testing into the Development Life Cycle 323
13.6 Measuring Accuracy 327
13.7 Measuring Efficiency 331
13.8 Measuring Satisfaction 335
13.9 Retrospective Probing and Debriefing 337
13.10 Communicating Findings with the Development Team 339
13.11 Assessing Whether Usability Test Recommendations Worked 340
13.12 Conclusions 341
References 341
14 How Mobile Device Screen Size Affects Data Collected in Web Surveys 349
Daniele Toninelli and Melanie Revilla
14.1 Introduction 349
14.2 Literature Review 350
14.3 Our Contribution and Hypotheses 352
14.4 Data Collection and Method 355
14.5 Main Results 361
14.6 Discussion 368
Acknowledgments 369
References 370
15 Optimizing Grid Questions for Smartphones: A Comparison of Optimized and Non-Optimized Designs and Effects on Data Quality on Different Devices 375
Trine Dale and Heidi Walsoe
15.1 Introduction 375
15.2 The Need for Change in Questionnaire Design Practices 376
15.3 Contribution and Research Questions 378
15.4 Data Collection and Methodology 380
15.5 Main Results 386
15.6 Discussion 392
Acknowledgments 397
References 397
16 Learning from Mouse Movements: Improving Questionnaires and Respondents’ User Experience Through Passive Data Collection 403
Rachel Horwitz, Sarah Brockhaus, Felix Henninger, Pascal J. Kieslich, Malte Schierholz, Florian Keusch, and Frauke Kreuter
16.1 Introduction 403
16.2 Background 404
16.3 Data 409
16.4 Methodology 410
16.5 Results 415
16.6 Discussion 420
References 423
17 Using Targeted Embedded Probes to Quantify Cognitive Interviewing Findings 427
Paul Scanlon
17.1 Introduction 427
17.2 The NCHS Research and Development Survey 431
17.3 Findings 433
17.4 Discussion 445
References 448
18 The Practice of Cognitive Interviewing Through Web Probing 451
Stephanie Fowler and Gordon B. Willis
18.1 Introduction 451
18.2 Methodological Issues in the Use of Web Probing for Pretesting 452
18.3 Testing the Effect of Probe Placement 453
18.4 Analyses of Responses to Web Probes 455
18.5 Qualitative Analysis of Responses to Probes 459
18.6 Qualitative Coding of Responses 459
18.7 Current State of the Use of Web Probes 462
18.8 Limitations 465
18.9 Recommendations for the Application and Further Evaluation of Web Probes 466
18.10 Conclusion 468
Acknowledgments 468
References 468
Part IV Cross-Cultural and Cross-National Questionnaire Design and Evaluation 471
19 Optimizing Questionnaire Design in Cross-National and Cross-Cultural Surveys 473
Tom W. Smith
19.1 Introduction 473
19.2 The Total Survey Error Paradigm and Comparison Error 474
19.3 Cross-Cultural Survey Guidelines and Resources 477
19.4 Translation 478
19.5 Developing Comparative Scales 480
19.6 Focus Groups and Pretesting in Cross-National/Cultural Surveys 483
19.7 Tools for Developing and Managing Cross-National Surveys 484
19.8 Resources for Developing and Testing Cross-National Measures 485
19.9 Pre- and Post-Harmonization 486
19.10 Conclusion 488
References 488
20 A Model for Cross-National Questionnaire Design and Pretesting 493
Rory Fitzgerald and Diana Zavala-Rojas
20.1 Introduction 493
20.2 Background 493
20.3 The European Social Survey 495
20.4 ESS Questionnaire Design Approach 496
20.5 Critique of the Seven-Stage Approach 497
20.6 A Model for Cross-National Questionnaire Design and Pretesting 497
20.7 Evaluation of the Model for Cross-National Questionnaire Design and Pretesting Using the Logical Framework Matrix (LFM) 501
20.8 Conclusions 512
References 514
21 Cross-National Web Probing: An Overview of Its Methodology and Its Use in Cross-National Studies 521
Dorothée Behr, Katharina Meitinger, Michael Braun, and Lars Kaczmirek
21.1 Introduction 521
21.2 Cross-National Web Probing - Its Goal, Strengths, and Weaknesses 523
21.3 Access to Respondents Across Countries: The Example of Online Access Panels and Probability-Based Panels 526
21.4 Implementation of Standardized Probes 527
21.5 Translation and Coding Answers to Cross-Cultural Probes 532
21.6 Substantive Results 533
21.7 Cross-National Web Probing and Its Application Throughout the Survey Life Cycle 536
21.8 Conclusions and Outlook 538
Acknowledgments 539
References 539
22 Measuring Disability Equality in Europe: Design and Development of the European Health and Social Integration Survey Questionnaire 545
Amanda Wilmot
22.1 Introduction 545
22.2 Background 546
22.3 Questionnaire Design 548
22.4 Questionnaire Development and Testing 553
22.5 Survey Implementation 560
22.6 Lessons Learned 563
22.7 Final Reflections 566
Acknowledgments 567
References 567
Part V Extensions and Applications 571
23 Regression-Based Response Probing for Assessing the Validity of Survey Questions 573
Patrick Sturgis, Ian Brunton-Smith, and Jonathan Jackson
23.1 Introduction 573
23.2 Cognitive Methods for Assessing Question Validity 574
23.3 Regression-Based Response Probing 577
23.4 Example 1: Generalized Trust 579
23.5 Example 2: Fear of Crime 580
23.6 Data 581
23.7 Discussion 586
References 588
24 The Interplay Between Survey Research and Psychometrics, with a Focus on Validity Theory 593
Bruno D. Zumbo and José-Luis Padilla
24.1 Introduction 593
24.2 An Over-the-Shoulder Look Back at Validity Theory and Validation Practices with an Eye toward Describing Contemporary Validity Theories 595
24.3 An Approach to Validity that Bridges Psychometrics and Survey Design 602
24.4 Closing Remarks 606
References 608
25 Quality-Driven Approaches for Managing Complex Cognitive Testing Projects 613
Martha Stapleton, Darby Steiger, and Mary C. Davis
25.1 Introduction 613
25.2 Characteristics of the Four Cognitive Testing Projects 614
25.3 Identifying Detailed, Quality-Driven Management Approaches for Qualitative Research 615
25.4 Identifying Principles for Developing Quality-Driven Management Approaches 616
25.5 Applying the Concepts of Transparency and Consistency 617
25.6 The 13 Quality-Driven Management Approaches 618
25.7 Discussion and Conclusion 632
References 634
26 Using Iterative, Small-Scale Quantitative and Qualitative Studies: A Review of 15 Years of Research to Redesign a Major US Federal Government Survey 639
Joanne Pascale
26.1 Introduction 639
26.2 Measurement Issues in Health Insurance 641
26.3 Methods and Results 645
26.4 Discussion 660
26.5 Final Reflections 663
References 664
27 Contrasting Stylized Questions of Sleep with Diary Measures from the American Time Use Survey 671
Robin L. Kaplan, Brandon Kopp, and Polly Phipps
27.1 Introduction 671
27.2 The Sleep Gap 672
27.3 The Present Research 674
27.4 Study 1: Behavior Coding 675
27.5 Study 2: Cognitive Interviews 678
27.6 Study 3: Quantitative Study 682
27.7 Study 4: Validation Study 686
27.8 General Discussion 689
27.9 Implications and Future Directions 692
References 692
28 Questionnaire Design Issues in Mail Surveys of All Adults in a Household 697
Douglas Williams, J. Michael Brick, W. Sherman Edwards, and Pamela Giambo
28.1 Introduction 697
28.2 Background 698
28.3 The NCVS and Mail Survey Design Challenges 699
28.4 Field Test Methods and Design 704
28.5 Outcome Measures 706
28.6 Findings 708
28.7 Summary 716
28.8 Discussion 716
28.9 Conclusion 719
References 720
29 Planning Your Multimethod Questionnaire Testing Bento Box: Complementary Methods for a Well-Balanced Test 723
Jaki S. McCarthy
29.1 Introduction 723
29.2 A Questionnaire Testing Bento Box 725
29.3 Examples from the Census of Agriculture Questionnaire Testing Bento Box 733
29.4 Conclusion 743
References 744
30 Flexible Pretesting on a Tight Budget: Using Multiple Dependent Methods to Maximize Effort-Return Trade-Offs 749
Matt Jans, Jody L. Herman, Joseph Viana, David Grant, Royce Park, Bianca D.M. Wilson, Jane Tom, Nicole Lordi, and Sue Holtby
30.1 Introduction 749
30.2 Evolution of a Dependent Pretesting Approach for Gender Identity Measurement 752
30.3 Analyzing and Synthesizing Results 759
30.4 Discussion 764
Acknowledgments 766
References 766
Index 769
