An in-depth description of the state-of-the-art of 3D shape analysis techniques and their applications
This book discusses the different topics that come under the title of "3D shape analysis". It covers the theoretical foundations and the major solutions that have been presented in the literature. It also establishes links between solutions proposed by different communities that studied 3D shape, such as mathematics and statistics, medical imaging, computer vision, and computer graphics.
The first part of 3D Shape Analysis: Fundamentals, Theory, and Applications provides a review of the background concepts such as methods for the acquisition and representation of 3D geometries, and the fundamentals of geometry and topology. It specifically covers stereo matching, structured light, and intrinsic vs. extrinsic properties of shape. Parts 2 and 3 present a range of mathematical and algorithmic tools (which are used for e.g., global descriptors, keypoint detectors, local feature descriptors, and algorithms) that are commonly used for the detection, registration, recognition, classification, and retrieval of 3D objects. Both also place strong emphasis on recent techniques motivated by the spread of commodity devices for 3D acquisition. Part 4 demonstrates the use of these techniques in a selection of 3D shape analysis applications. It covers 3D face recognition, object recognition in 3D scenes, and 3D shape retrieval. It also discusses examples of semantic applications and cross domain 3D retrieval, i.e. how to retrieve 3D models using various types of modalities, e.g. sketches and/or images. The book concludes with a summary of the main ideas and discussions of the future trends.
3D Shape Analysis: Fundamentals, Theory, and Applications is an excellent reference for graduate students, researchers, and professionals in different fields of mathematics, computer science, and engineering. It is also ideal for courses in computer vision and computer graphics, as well as for those seeking 3D industrial/commercial solutions.
Table of Contents
Preface xv
Acknowledgments xvii
1 Introduction 1
1.1 Motivation 1
1.2 The 3D Shape Analysis Problem 2
1.3 About This Book 5
1.4 Notation 9
Part I Foundations 11
2 Basic Elements of 3D Geometry and Topology 13
2.1 Elements of Differential Geometry 13
2.1.1 Parametric Curves 13
2.1.2 Continuous Surfaces 15
2.1.2.1 Differential Properties of Surfaces 17
2.1.2.1.1 First Fundamental Form 17
2.1.2.1.2 Second Fundamental Form and Shape Operator 18
2.1.2.2 Curvatures 19
2.1.2.3 Laplace and Laplace-Beltrami Operators 21
2.1.3 Manifolds, Metrics, and Geodesics 22
2.1.4 Discrete Surfaces 24
2.1.4.1 Representations of Discrete Surfaces 24
2.1.4.2 Mesh Data Structures 28
2.1.4.3 Discretization of the Differential Properties of Surfaces 29
2.2 Shape, Shape Transformations, and Deformations 30
2.2.1 Shape-Preserving Transformations 31
2.2.1.1 Normalization for Translation 32
2.2.1.2 Normalization for Scale 32
2.2.1.3 Normalization for Rotation 32
2.2.1.3.1 Rotation Normalization Using Principal Component Analysis (PCA) 33
2.2.1.3.2 Rotation Normalization Using Planar Reflection Symmetry Analysis 34
2.2.2 Shape Deformations 35
2.2.3 Bending 35
2.2.4 Stretching 37
2.3 Summary and Further Reading 38
3 3D Acquisition and Preprocessing 41
3.1 Introduction 41
3.2 3D Acquisition 41
3.2.1 Contact 3D Acquisition 43
3.2.1.1 Coordinate Measuring Machine (CMM) 43
3.2.1.2 Arm-Based 3D Scanner 44
3.2.2 Noncontact 3D Acquisition 44
3.2.2.1 Time-of-Flight 44
3.2.2.1.1 Pulse-Based TOF 44
3.2.2.1.2 Phase Shift-Based TOF 45
3.2.2.2 Triangulation 45
3.2.2.3 Stereo 47
3.2.2.4 Structured Light 50
3.2.2.4.1 Temporal Coded Patterns 51
3.2.2.4.2 Spatial Coded Patterns 52
3.2.2.4.3 Direct Coded Patterns 55
3.2.2.5 Shape from X 55
3.3 Preprocessing 3D Models 56
3.3.1 Surface Smoothing and Fairing 57
3.3.1.1 Laplacian Smoothing 57
3.3.1.2 Taubin Smoothing 58
3.3.1.3 Curvature Flow Smoothing 58
3.3.2 Spherical Parameterization of 3D Surfaces 58
3.4 Summary and Further Reading 62
Part II 3D Shape Descriptors 65
4 Global Shape Descriptors 67
4.1 Introduction 67
4.2 Distribution-Based Descriptors 69
4.2.1 Point Sampling 69
4.2.2 Geometric Features 70
4.2.2.1 Geometric Attributes 70
4.2.2.2 Differential Attributes 71
4.2.3 Signature Construction and Comparison 72
4.3 View-Based 3D Shape Descriptors 73
4.3.1 The Light Field Descriptors (LFD) 74
4.3.2 Feature Extraction 75
4.3.3 Properties 76
4.4 Spherical Function-Based Descriptors 77
4.4.1 Spherical Shape Functions 78
4.4.2 Comparing Spherical Functions 80
4.4.2.1 Spherical Harmonic Descriptors 80
4.4.2.2 SphericalWavelet Transforms 81
4.4.2.2.1 Wavelet Coefficients as a Shape Descriptor 82
4.4.2.2.2 SphericalWavelet Energy Signatures 82
4.5 Deep Neural Network-Based 3D Descriptors 83
4.5.1 CNN-Based Image Descriptors 84
4.5.2 Multiview CNN for 3D Shapes 85
4.5.2.1 Network Architecture 86
4.5.2.2 View Aggregation using CNN 86
4.5.3 Volumetric CNN 87
4.6 Summary and Further Reading 89
5 Local Shape Descriptors 93
5.1 Introduction 93
5.2 Challenges and Criteria 94
5.2.1 Challenges 94
5.2.2 Criteria for 3D Keypoint Detection 95
5.2.3 Criteria for Local Feature Description 96
5.3 3D Keypoint Detection 96
5.3.1 Fixed-Scale Keypoint Detection 97
5.3.1.1 Curvature-Based Methods 97
5.3.1.1.1 Local Surface Patch (LSP) 98
5.3.1.2 Other Surface Variation-Based Methods 98
5.3.1.2.1 Matei’s Method 99
5.3.1.2.2 Intrinsic Shape Signatures (ISS) 99
5.3.1.2.3 Harris 3D 99
5.3.2 Adaptive-Scale Keypoint Detection 101
5.3.2.1 Extrinsic Scale-Space Based Methods 101
5.3.2.1.1 3D Shape Filtering 101
5.3.2.1.2 Multiscale Surface Variation 104
5.3.2.2 Intrinsic Scale-Space Based Methods 106
5.3.2.2.1 Scale-Space on 2D Parameterized Images 106
5.3.2.2.2 Scale-Space on 3D Shapes 109
5.3.2.2.3 Scale-Space on Transformed Domains 112
5.4 Local Feature Description 113
5.4.1 Signature-Based Methods 114
5.4.1.1 Splash 114
5.4.1.2 Point Signature 115
5.4.2 Histogram Based Methods 115
5.4.2.1 Histogram of Spatial Distributions 115
5.4.2.1.1 Spin Images 116
5.4.2.1.2 3D Shape Context 117
5.4.2.1.3 Intrinsic Shape Signature (ISS) 118
5.4.2.1.4 Rotational Projection Statistics (RoPS) 118
5.4.2.2 Histogram of Geometric Attributes 122
5.4.2.2.1 Point Feature Histograms (PFH) 122
5.4.2.2.2 Fast Point Feature Histograms (FPFH) 123
5.4.2.2.3 Signature of Histograms of Orientations (SHOT) 123
5.4.2.3 Histogram of Oriented Gradients 124
5.4.3 Covariance-Based Methods 124
5.5 Feature Aggregation Using Bag of Feature Techniques 126
5.5.1 Dictionary Construction 127
5.5.1.1 Feature Extraction 127
5.5.1.2 Codebook Construction 127
5.5.2 Coding and Pooling Schemes 128
5.5.2.1 Sparse Coding 128
5.5.2.2 Fisher Vectors 129
5.5.3 Vector of Locally Aggregated Descriptors (VLAD) 129
5.5.4 Vector of Locally Aggregated Tensors (VLAT) 130
5.6 Summary and Further Reading 131
5.6.1 Summary of 3D Keypoint Detection 131
5.6.2 Summary of Local Feature Description 132
5.6.3 Summary of Feature Aggregation 133
Part III 3D Correspondence and Registration 135
6 Rigid Registration 137
6.1 Introduction 137
6.2 Coarse Registration 138
6.2.1 Point Correspondence-Based Registration 138
6.2.1.1 The Typical Pipeline 139
6.2.1.2 Transformation Estimation from a Group of Correspondences 139
6.2.1.3 Transformation Estimation fromThree Correspondences 140
6.2.1.4 Transformation Estimation from Two Correspondences 141
6.2.1.5 Transformation Estimation from One Correspondence 142
6.2.2 Line-Based Registration 143
6.2.2.1 Line Matching Method 143
6.2.2.2 Line Clustering Method 144
6.2.2.2.1 Rotation Estimation 145
6.2.2.2.2 Translation Estimation 146
6.2.3 Surface-Based Registration 146
6.2.3.1 Principal Component Analysis (PCA) 146
6.2.3.2 RANSAC-Based DARCES 147
6.2.3.3 Four-Points Congruent Sets (4PCS) 149
6.2.3.3.1 Affine Invariants of 4-Points Set 149
6.2.3.3.2 Congruent 4-Points Extraction 151
6.2.3.3.3 The 4PCS Algorithm 151
6.3 Fine Registration 152
6.3.1 Iterative Closest Point (ICP) 153
6.3.1.1 Closest Point Search 153
6.3.1.2 Transformation Estimation 153
6.3.1.3 Summary of the ICP Method 154
6.3.2 ICP Variants 155
6.3.2.1 Point Selection 155
6.3.2.2 Point Matching 156
6.3.2.3 Point PairWeighting 156
6.3.2.4 Point Pair Rejection 156
6.3.2.5 Error Metrics 157
6.3.3 Coherent Point Drift 157
6.4 Summary and Further Reading 160
7 Nonrigid Registration 161
7.1 Introduction 161
7.2 Problem Formulation 162
7.3 Mathematical Tools 165
7.3.1 The Space of Diffeomorphisms 165
7.3.2 Parameterizing Spaces 166
7.4 Isometric Correspondence and Registration 168
7.4.1 Möbius Voting 168
7.4.2 Examples 170
7.5 Nonisometric (Elastic) Correspondence and Registration 171
7.5.1 Surface Deformation Models 171
7.5.1.1 Linear Deformation Model 171
7.5.1.2 Elastic Deformation Models 172
7.5.2 Square-Root Normal Fields (SRNF) Representation 173
7.5.3 Numerical Inversion of SRNF Maps 174
7.5.3.1 SRNF Inversion Algorithm 176
7.5.4 Correspondence 177
7.5.4.1 Optimization Over SO(3) 178
7.5.4.2 Optimization Over Γ 178
7.5.4.3 Differential of 𝜙 [184] 179
7.5.4.4 Initialization of the Gradient [184] 179
7.5.5 Elastic Registration and Geodesics 181
7.5.6 Coregistration 181
7.6 Summary and Further Reading 184
8 Semantic Correspondences 187
8.1 Introduction 187
8.2 Mathematical Formulation 188
8.3 Graph Representation 191
8.3.1 Characterizing the Local Geometry and the Spatial Relations 191
8.3.1.1 Unary Descriptors 192
8.3.1.2 Binary Descriptors 192
8.3.2 Cross Mesh Pairing of Patches 192
8.4 Energy Functions for Semantic Labeling 194
8.4.1 The Data Term 194
8.4.2 Smoothness Terms 194
8.4.2.1 Smoothness Constraints 194
8.4.2.2 Geometric Compatibility 195
8.4.2.3 Label Compatibility 196
8.4.3 The Intermesh Term 196
8.5 Semantic Labeling 196
8.5.1 Unsupervised Clustering 197
8.5.2 Learning the Labeling Likelihood 199
8.5.2.1 GentleBoost Classifier 199
8.5.2.2 Training GentleBoost Classifiers 200
8.5.3 Learning the Remaining Parameters 201
8.5.4 Optimization Using Graph Cuts 202
8.6 Examples 202
8.7 Summary and Further Reading 204
Part IV Applications 207
9 Examples of 3D Semantic Applications 209
9.1 Introduction 209
9.2 Semantics: Shape or Status 209
9.3 Semantics: Class or Identity 212
9.4 Semantics: Behavior 216
9.5 Semantics: Position 219
9.6 Summary and Further Reading 221
10 3D Face Recognition 223
10.1 Introduction 223
10.2 3D Face Recognition Tasks, Challenges and Datasets 224
10.2.1 3D Face Verification 224
10.2.2 3D Face Identification 225
10.2.3 3D Face Recognition Challenges 225
10.2.3.1 Intrinsic Transformations 225
10.2.3.2 Acquisition Conditions 226
10.2.3.3 Data Acquisition 226
10.2.3.4 Computation Time 227
10.2.4 3D Face Datasets 227
10.3 3D Face Recognition Methods 228
10.3.1 Holistic Approaches 232
10.3.1.1 Eigenfaces and Fisherfaces 232
10.3.1.1.1 Eigenfaces 232
10.3.1.1.2 Fisherfaces 233
10.3.1.2 Iterative Closest Point 234
10.3.1.3 Hausdorff Distance 234
10.3.1.4 Canonical Form 234
10.3.2 Local Feature-Based Matching 235
10.3.2.1 Keypoint-Based Methods 235
10.3.2.1.1 Landmark-Based Methods 235
10.3.2.1.2 SIFT-Like Keypoints 236
10.3.2.2 Curve-Based Features 237
10.3.2.3 Patch-Based Features 238
10.3.2.4 Other Features 239
10.4 Summary 239
11 Object Recognition in 3D Scenes 241
11.1 Introduction 241
11.2 Surface Registration-Based Object Recognition Methods 241
11.2.1 Feature Matching 242
11.2.2 Hypothesis Generation 242
11.2.2.1 Geometric Consistency-Based Hypothesis Generation 243
11.2.2.2 Pose Clustering-Based Hypothesis Generation 244
11.2.2.3 Constrained Interpretation Tree-Based Hypothesis Generation 244
11.2.2.4 RANdom SAmple Consensus-Based Hypothesis Generation 245
11.2.2.5 GameTheory-Based Hypothesis Generation 246
11.2.2.5.1 Preliminary on Game Theory 246
11.2.2.5.2 Matching Game for Transformation Hypothesis Generation 247
11.2.2.6 Generalized Hough Transform-Based Hypothesis Generation 248
11.2.3 Hypothesis Verification 249
11.2.3.1 Individual Verification 249
11.2.3.2 Global Verification 251
11.3 Machine Learning-Based Object Recognition Methods 255
11.3.1 Hough Forest-Based 3D Object Detection 255
11.3.1.1 3D Local Patch Extraction 255
11.3.1.2 3D Local Patch Representation 256
11.3.1.3 Hough Forest Training and Testing 256
11.3.1.3.1 Offline Training 256
11.3.1.3.2 Online detection 258
11.3.2 Deep Learning-Based 3D Object Recognition 260
11.3.2.1 Hand-crafted Feature-Based Methods 262
11.3.2.2 2D View-Based Methods 262
11.3.2.3 3D Voxel-Based Methods 263
11.3.2.4 3D Point Cloud-Based Methods 265
11.4 Summary and Further Reading 265
12 3D Shape Retrieval 267
12.1 Introduction 267
12.2 Benchmarks and Evaluation Criteria 270
12.2.1 3D Datasets and Benchmarks 270
12.2.2 Performance Evaluation Metrics 271
12.2.2.1 Precision 272
12.2.2.2 Recall 272
12.2.2.3 Precision-Recall Curves 273
12.2.2.4 F- and E-Measures 273
12.2.2.5 Area under Curve (AUC) or Average Precision (AP) 273
12.2.2.6 Mean Average Precision (mAP) 274
12.2.2.7 Cumulated Gain-Based Measure 274
12.2.2.8 Nearest Neighbor (NN), First-Tier (FT), and Second-Tier (ST) 275
12.3 Similarity Measures 275
12.3.1 Dissimilarity Measures 275
12.3.2 Hashing and Hamming Distance 277
12.3.3 Manifold Ranking 278
12.4 3D Shape Retrieval Algorithms 280
12.4.1 Using Handcrafted Features 280
12.4.2 Deep Learning-Based Methods 282
12.5 Summary and Further Reading 284
13 Cross-domain Retrieval 285
13.1 Introduction 285
13.2 Challenges and Datasets 287
13.2.1 Datasets 288
13.2.2 Training Data Synthesis 289
13.2.2.1 Photo Synthesis from 3D Models 289
13.2.2.2 2D Sketch Synthesis from 3D Models 290
13.3 Siamese Network for Cross-domain Retrieval 290
13.4 3D Shape-centric Deep CNN 292
13.4.1 Embedding Space Construction 293
13.4.1.1 Principal Component Analysis 295
13.4.1.2 Multi-dimensional Scaling 296
13.4.1.3 Kernel-Based Analysis 296
13.4.2 Learning Shapes from Synthesized Data 298
13.4.3 Image and Sketch Projection 298
13.5 Summary and Further Reading 300
14 Conclusions and Perspectives 301
References 303
Index 337
