Statistics for Data Science and Analytics is a comprehensive guide to statistical analysis using Python, presenting important topics useful for data science such as prediction, correlation, and data exploration. The authors provide an introduction to statistical science and big data, as well as an overview of Python data structures and operations.
A range of statistical techniques are presented with their implementation in Python, including hypothesis testing, probability, exploratory data analysis, categorical variables, surveys and sampling, A/B testing, and correlation. The text introduces binary classification, a foundational element of machine learning, validation of statistical models by applying them to holdout data, and probability and inference via the easy-to-understand method of resampling and the bootstrap instead of using a myriad of “kitchen sink” formulas. Regression is taught both as a tool for explanation and for prediction.
This book is informed by the authors’ experience designing and teaching both introductory statistics and machine learning at Statistics.com. Each chapter includes practical examples, explanations of the underlying concepts, and Python code snippets to help readers apply the techniques themselves.
Statistics for Data Science and Analytics includes information on sample topics such as: - Int, float, and string data types, numerical operations, manipulating strings, converting data types, and advanced data structures like lists, dictionaries, and sets- Experiment design via randomizing, blinding, and before-after pairing, as well as proportions and percents when handling binary data- Specialized Python packages like numpy, scipy, pandas, scikit-learn and statsmodels - the workhorses of data science - and how to get the most value from them- Statistical versus practical significance, random number generators, functions for code reuse, and binomial and normal probability distributions
Written by and for data science instructors, Statistics for Data Science and Analytics is an excellent learning resource for data science instructors prescribing a required intro stats course for their programs, as well as other students and professionals seeking to transition to the data science field.
Table of Contents
About the Authors xvii
Acknowledgments xix
About the Companion Website xxi
Introduction xxiii
1 Statistics and Data Science 1
1.1 Big Data: Predicting Pregnancy 2
1.2 Phantom Protection from Vitamin E 2
1.3 Statistician, Heal Thyself 3
1.4 Identifying Terrorists in Airports 4
1.5 Looking Ahead 5
1.6 Big Data and Statisticians 5
2 Designing and Carrying Out a Statistical Study 9
2.1 Statistical Science 9
2.2 Big Data 10
2.3 Data Science 10
2.4 Example: Hospital Errors 11
2.5 Experiment 12
2.6 Designing an Experiment 13
2.7 The Data 19
2.8 Variables and Their Flavors 21
2.9 Python: Data Structures and Operations 25
2.10 Are We Sure We Made a Difference? 34
2.11 Is Chance Responsible? The Foundation of Hypothesis Testing 34
2.12 Probability 36
2.13 Significance or Alpha Level 38
2.14 Other Kinds of Studies 40
2.15 When to Use Hypothesis Tests 42
2.16 Experiments Falling Short of the Gold Standard 42
2.17 Summary 43
2.18 Python: Iterations and Conditional Execution 44
2.19 Python: Numpy, scipy, and pandas - The Workhorses of Data Science 50
Exercises 56
3 Exploring and Displaying the Data 61
3.1 Exploratory Data Analysis 61
3.2 What to Measure - Central Location 62
3.3 What to Measure - Variability 65
3.4 What to Measure - Distance (Nearness) 69
3.5 Test Statistic 71
3.6 Examining and Displaying the Data 72
3.7 Python: Exploratory Data Analysis/Data Visualization 80
Exercises 88
4 Accounting for Chance - Statistical Inference 91
4.1 Avoid Being Fooled by Chance 91
4.2 The Null Hypothesis 92
4.3 Repeating the Experiment 93
4.4 Statistical Significance 99
4.5 Power 103
4.6 The Normal Distribution 103
4.7 Summary 105
4.8 Python: Random Numbers 105
Exercises 115
5 Probability 121
5.1 What Is Probability 121
5.2 Simple Probability 122
5.3 Probability Distributions 126
5.4 From Binomial to Normal Distribution 129
5.5 Appendix: Binomial Formula and Normal Approximation 133
5.6 Python: Probability 134
Exercises 141
6 Categorical Variables 143
6.1 Two-way Tables 143
6.2 Conditional Probability 144
6.3 Bayesian Estimates 147
6.4 Independence 150
6.5 Multiplication Rule 154
6.6 Simpson’s Paradox 156
6.7 Python: Counting and Contingency Tables 157
Exercises 163
7 Surveys and Sampling 167
7.1 Literary Digest - Sampling Trumps “All Data” 167
7.2 Simple Random Samples 170
7.3 Margin of Error: Sampling Distribution for a Proportion 172
7.4 Sampling Distribution for a Mean 174
7.5 The Bootstrap 176
7.6 Rationale for the Bootstrap 177
7.7 Standard Error 188
7.8 Other Sampling Methods 188
7.9 Absolute vs. Relative Sample Size 192
7.10 Python: Random Sampling Strategies 192
Exercises 202
8 More than Two Samples or Categories 207
8.1 Count Data - R × C Tables 207
8.2 The Role of Experiments (Many Are Costly) 208
8.3 Chi-Square Test 210
8.4 Single Sample - Goodness-of-Fit 215
8.5 Numeric Data: ANOVA 217
8.6 Components of Variance 222
8.7 Factorial Design 224
8.8 The Problem of Multiple Inference 226
8.9 Continuous Testing 228
8.10 Bandit Algorithms 229
8.11 Appendix: ANOVA, the Factor Diagram, and the F-Statistic 230
8.12 More than One Factor or Variable - From ANOVA to Statistical Models 237
8.13 Python: Contingency Tables and Chi-square Test 237
8.14 Python: ANOVA 241
Exercises 246
9 Correlation 249
9.1 Example: Delta Wire 249
9.2 Example: Cotton Dust and Lung Disease 251
9.3 The Vector Product Sum Test 252
9.4 Correlation Coefficient 256
9.5 Correlation is not Causation 260
9.6 Other Forms of Association 261
9.7 Python: Correlation 262
Exercises 269
10 Regression 271
10.1 Finding the Regression Line by Eye 272
10.2 Finding the Regression Line by Minimizing Residuals 274
10.3 Linear Relationships 276
10.4 Prediction vs. Explanation 280
10.5 Python: Linear Regression 284
Exercises 293
11 Multiple Linear Regression 295
11.1 Terminology 295
11.2 Example - Housing Prices 296
11.3 Interaction 301
11.4 Regression Assumptions 304
11.5 Assessing Explanatory Regression Models 306
11.6 Assessing Regression for Prediction 314
11.7 Python: Multiple Linear Regression 324
Exercises 332
12 Predicting Binary Outcomes 337
12.1 K-Nearest-Neighbors 337
12.2 Python: Classification 343
Exercises 346
Index 349
