Resource Management on Distributed Systems provides helpful guidance by describing algorithms and techniques for managing resources on parallel and distributed systems, including grids, clouds, and parallel processing-based platforms for big data analytics.
The book focuses on four general principles of resource management and their impact on system performance, energy usage, and cost, including end-of-chapter exercises. The text includes chapters on sensors, autoscaling on clouds, complex event processing for streaming data, and data filtering techniques for big data systems.
The book also covers results of applying the discussed techniques on simulated as well as real systems (including clouds and big data processing platforms), and techniques for handling errors associated with user predicted task execution times.
Written by a highly qualified academic with significant research experience in the field, Resource Management on Distributed Systems includes information on sample topics such as: - Attributes of parallel/distributed applications that have an intimate relationship with system behavior and performance, plus their related performance metrics.- Handling a lack of a prior knowledge of local operating systems on individual nodes in a large system.- Detection and management of complex events (that correspond to the occurrence of multiple raw events) on a platform for streaming analytics.- Techniques for reducing data latency for multiple operator-based queries in an environment processing large textual documents.
With comprehensive coverage of core topics in the field, Resource Management on Distributed Systems is a comprehensive guide to resource management in a single publication and is an essential read for professionals, researchers and students working with distributed systems.
Table of Contents
About the Author xv
Preface xvii
Acknowledgments xxiii
1 Introduction 1
1.1 Introduction to Distributed and Parallel Computing 1
1.2 Types of Computing Environments 2
1.3 Units of Computation 3
1.4 Principles Underlying Resource Management 5
1.5 Evolution of Distributed Systems 9
1.6 Summary 20
2 Characterization of Parallelism in Applications 25
2.1 Introduction 25
2.2 The Precedence Graph Model 26
2.3 Graph-Based Characteristics 27
2.4 Single-Point Characteristics 30
2.5 Performance Metrics 32
2.6 Impact of Parallelism Characteristics on Performance 36
2.7 Energy Performance Trade-Off 39
2.8 Summary 47
3 Resource Management Techniques for Distributed Computing Systems 51
3.1 Resource Allocation 52
3.2 Task/Process Scheduling 62
3.3 Grid Scheduling with Deadlines 67
3.4 Scheduling on Client-Server Systems 70
3.5 Summary 74
4 Resource Management on Systems Subjected to Uncertainties Associated with Workload and System Parameters 79
4.1 Introduction 79
4.2 Handling Errors Associated with User Estimates of Job Execution Times 80
4.3 Underestimation of Job Execution Times 82
4.4 Handling Uncertainties Associated with the Local Scheduling Policy 86
4.5 Any Schedulability Criterion 88
4.6 Matchmaking in the Dark: AS Criterion-Based Matchmaking 91
4.7 Soft Advance Reservation Requests 96
4.8 Summary 100
5 Resource Auto-Scaling 105
5.1 Introduction 105
5.2 Request Characteristics 107
5.3 Horizontal Auto-Scaling 108
5.4 Hybrid Auto-Scaling 117
5.5 Summary 120
6 Resource Management for Systems Running MapReduce Jobs 127
6.1 Introduction 127
6.2 MapReduce 128
6.3 Resource Management Techniques for MapReduce Job Requests to be Satisfied on a Best Effort Basis 130
6.4 Resource Management Techniques for MapReduce Job Requests with Service Level Agreements 132
6.5 The Constraint Programming-Based MapReduce Resource Management Technique 135
6.6 Errors Associated with User Estimates of Task Execution times 141
6.7 Summary 145
7 Energy Aware Resource Management 149
7.1 Introduction 149
7.2 DVFS-Based Resource Management Techniques 152
7.3 The EAMR-RM Algorithm 153
7.4 Configurable Resource Manager for Processing a Batch of MapReduce Jobs 158
7.5 Performance Analysis of CRM 161
7.6 Reducing the Number of Active Servers 164
7.7 Summary 166
8 Streaming Data and Complex Event Processing 169
8.1 Introduction 169
8.2 Management of Streaming Data 170
8.3 Dynamic Priority-Based Scheduling 171
8.4 Data-Driven Priority Scheduler (DDPS) 174
8.5 Multitennant Systems 179
8.6 Complex Event Processing 186
8.7 Remote Patient Monitoring System 191
8.8 Summary 196
9 Data Indexing and Filtering Techniques for Big Data Systems 201
9.1 Introduction 201
9.2 Harnessing Big Data 202
9.3 Data Indexing 203
9.4 Inverted Index 203
9.5 Graph-Based Indexing 205
9.6 Boolean AND Queries 207
9.7 Performance Analysis 210
9.8 Data Filtering 213
9.9 Parallel Processing Platforms 214
9.10 Motivations for Data Reduction 217
9.11 Data Filtering 219
9.12 Performance Analysis 224
9.13 Streaming Data 227
9.14 Handling User Preferences Comprising Keywords Connected by Boolean Operators 230
9.15 Summary 232
10 Sensor-Based Systems 237
10.1 Introduction 237
10.2 Middleware Services 239
10.3 Sensor-Based Bridge Management 242
10.4 Research Collaboration Platform for Management of Sensor-Based Smart Facilities 244
10.5 Resource Management on Wireless Sensor Networks 247
10.6 Scheduling on WSNs 250
10.7 Sensor Allocation 255
10.8 Summary 262
11 Summary 267
11.1 Chapter Entitled Introduction 267
11.2 Chapter Entitled Characterization of Parallelism in Applications 270
11.2.1 Graph-Based Characteristics 270
11.3 Chapter Entitled Resource Management Techniques for Distributed Computing Systems 271
11.4 Chapter Entitled Resource Management on Systems Subjected to Uncertainties Associated with Workload and System Parameters 272
11.5 Chapter Entitled Resource Auto-Scaling 274
11.6 Chapter Entitled Resource Management on Systems Running MapReduce Jobs 276
11.7 Chapter Entitled Energy-Aware Resource Management 278
11.8 Chapter Entitled Streaming Data and Complex Event Processing 279
11.9 Chapter Entitled Data Indexing and Filtering Techniques for Big Data Systems 281
11.10 Chapter Entitled Sensor-Based Systems 282
Index 285
