+353-1-416-8900REST OF WORLD
+44-20-3973-8888REST OF WORLD
1-917-300-0470EAST COAST U.S
1-800-526-8630U.S. (TOLL FREE)

Deep Reinforcement Learning and Its Industrial Use Cases. AI for Real-World Applications. Edition No. 1

  • Book

  • 416 Pages
  • October 2024
  • John Wiley and Sons Ltd
  • ID: 5996397
This book serves as a bridge connecting the theoretical foundations of DRL with practical, actionable insights for implementing these technologies in a variety of industrial contexts, making it a valuable resource for professionals and enthusiasts at the forefront of technological innovation.

Deep Reinforcement Learning (DRL) represents one of the most dynamic and impactful areas of research and development in the field of artificial intelligence. Bridging the gap between decision-making theory and powerful deep learning models, DRL has evolved from academic curiosity to a cornerstone technology driving innovation across numerous industries. Its core premise - enabling machines to learn optimal actions within complex environments through trial and error - has broad implications, from automating intricate decision processes to optimizing operations that were previously beyond the reach of traditional AI techniques.

“Deep Reinforcement Learning and Its Industrial Use Cases: AI for Real-World Applications” is an essential guide for anyone eager to understand the nexus between cutting-edge artificial intelligence techniques and practical industrial applications. This book not only demystifies the complex theory behind deep reinforcement learning (DRL) but also provides a clear roadmap for implementing these advanced algorithms in a variety of industries to solve real-world problems. Through a careful blend of theoretical foundations, practical insights, and diverse case studies, the book offers a comprehensive look into how DRL is revolutionizing fields such as finance, healthcare, manufacturing, and more, by optimizing decisions in dynamic and uncertain environments.

This book distills years of research and practical experience into accessible and actionable knowledge. Whether you’re an AI professional seeking to expand your toolkit, a business leader aiming to leverage AI for competitive advantage, or a student or academic researching the latest in AI applications, this book provides valuable insights and guidance. Beyond just exploring the successes of DRL, it critically examines challenges, pitfalls, and ethical considerations, preparing readers to not only implement DRL solutions but to do so responsibly and effectively.

Audience

The book will be read by researchers, postgraduate students, and industry engineers in machine learning and artificial intelligence, as well as those in business and industry seeking to understand how DRL can be applied to solve complex industry-specific challenges and improve operational efficiency.

Table of Contents

Preface xv

1 Deep Reinforcement Learning Applications in Real-World Scenarios: Challenges and Opportunities 1
Sunilkumar Ketineni and Sheela J.

1.1 Introduction 1

1.1.1 Problems with Real-World Implementation 2

1.2 Application to the Real World 3

1.2.1 Security and Robustness 3

1.2.2 Generalization 5

1.2.2.1 Overcoming Challenges in DRL 9

1.3 Possibilities for Making a Difference in the Real World 11

1.3.1 Transfer Learning and Domain Adaptation 11

1.4 Meta-Learning 12

1.5 Deep Reinforcement Learning (DRL) 13

1.5.1 Hybrid Approaches 14

1.6 Online vs. Offline Reinforcement Learning 15

1.7 Human-in-the-Loop Systems 15

1.8 Benchmarking and Standardization 16

1.9 Collaborative Multi-Agent Systems 18

1.10 Transfer Learning and Domain Adaptation 19

1.11 Hierarchical and Multimodal Learning 21

1.12 Imitation Learning and Human Feedback 22

1.13 Inverse Reinforcement Learning 23

1.14 Sim-to-Real Transfer 24

1.15 Conclusion 25

References 26

2 Deep Reinforcement Learning: A Key to Unlocking the Potential of Robotics and Autonomous Systems 29
Saksham and Chhavi Rana

2.1 Introduction 30

2.1.1 Significance of DRL Field 30

2.1.2 Transformative Advantages of DRL Field 32

2.2 Fields of Investigation 33

2.2.1 General Methods for Investigation 34

2.3 Background 36

2.3.1 Fundamentals of Deep Reinforcement Learning (DRL) 38

2.4 Deep Reinforcement Learning (DRL) in Robot Control 39

2.4.1 Navigation and Localization 40

2.4.2 Object Manipulation 42

2.5 Applications and Case Studies 43

2.6 Challenges and Future Directions 44

2.7 Evaluation and Metrics 46

2.8 Summary 47

References 48

3 Deep Reinforcement Learning Algorithms: A Comprehensive Overview 51
Shweta V. Bondre, Bhakti Thakre, Uma Yadav and Vipin D. Bondre

3.1 Introduction 52

3.1.1 How Reinforcement Learning Works? 53

3.2 Reinforcement Learning Algorithms 53

3.2.1 Value-Based Algorithms 53

3.2.1.1 Q-Learning 53

3.2.1.2 Deep Q-Networks (DQN) 57

3.2.1.3 Double DQN 58

3.2.1.4 Dueling DQN 58

3.3 Policy-Based 59

3.3.1 Policy Gradient Methods 59

3.3.2 REINFORCE (Monte Carlo Policy Gradient) 60

3.3.3 Actor-Critic Methods 61

3.3.4 Natural Policy Gradient Methods 62

3.4 Model-Based Reinforcement Learning 63

3.4.1 Probabilistic Ensembles with Trajectory Sampling (PETS) 63

3.4.2 Probabilistic Inference for Learning Control (PILCO) 64

3.4.3 Model Predictive Control (MPC) 65

3.4.4 Model-Agnostic Meta-Learning (MAML) 66

3.4.5 Soft Actor-Critic with Model Ensemble 67

3.4.6 Deep Deterministic Policy Gradients with Model (DDPG with Model) 68

3.5 Characteristics of Reinforcement Learning 69

3.6 DRL Algorithms and Their Advantages and Drawbacks 71

3.7 Conclusion 72

References 72

4 Deep Reinforcement Learning in Healthcare and Biomedical Applications 75
Balakrishnan D., Aarthy C., Nandhagopal Subramani, Venkatesan R. and Logesh T. R.

4.1 Introduction 76

4.2 Related Works 76

4.3 Deep Reinforcement Learning Framework 80

4.4 Deep Reinforcement Learning Applications in Healthcare and Biomedicine 81

4.5 Deep Reinforcement Learning Employs Efficient Algorithms 82

4.5.1 Deep Q-Networks 82

4.5.2 Policy Differentiation Techniques 82

4.5.3 Hindsight Experience Replay (HER) 82

4.5.4 Curiosity-Driven Exploration 82

4.5.5 Long Short-Term Memory Networks and Recurring Neural Network Designs 82

4.5.6 Multi-Agent DRL 83

4.6 Semi-Autonomous Control Based on Deep Reinforcement Learning for Robotic Surgery 83

4.6.1 Double Deep Q-Network (DDQN) 83

4.6.2 Materials and Methods 84

4.6.3 Results 86

4.6.4 Discussion 87

4.7 Conclusion 87

References 88

5 Application of Deep Reinforcement Learning in Adversarial Malware Detection 91
Manju and Chhavi Rana

5.1 Introduction 91

5.1.1 Background 95

5.1.2 Significance of Malware Detection 96

5.1.3 Challenges with Adversarial Attacks 96

5.2 Foundations of Deep Reinforcement Learning 97

5.2.1 Overview of Deep Reinforcement Learning 98

5.2.2 Core Concepts and Components 99

5.2.3 Relevance to Malware Detection 100

5.3 Malware Detection Landscape 101

5.3.1 Evolution of Malware Detection Techniques 102

5.3.2 Adversarial Attacks in Cybersecurity 103

5.3.3 Need for Advanced Detection Strategies 104

5.4 Deep Reinforcement Learning Techniques 104

5.4.1 Application of Deep Learning in Malware Detection 105

5.4.2 Reinforcement Learning Algorithms 106

5.5 Feature Selection Strategies 107

5.5.1 Importance of Feature Selection in Malware Detection 108

5.5.2 Techniques for Feature Selection 108

5.5.3 Optimization for Deep Reinforcement Learning Models 109

5.6 Datasets and Evaluation 110

5.7 Generating Adversarial Samples 111

Conclusion and Future Directions 112

Future Directions 112

References 112

6 Artificial Intelligence in Blockchain and Smart Contracts for Disruptive Innovation 115
Eashwar Sivakumar, Kiran Jot Singh and Paras Chawla

6.1 Introduction 115

6.1.1 Smart Contract 116

6.2 Literature Review 117

6.2.1 Blockchain and Smart Contracts in Digital Identity 117

6.2.2 Blockchain and Smart Contracts in Financial Security 118

6.2.3 Blockchain and Smart Contracts in Supply Chain Management 119

6.2.4 Blockchain and Smart Contracts in Insurance 120

6.2.5 Blockchain and Smart Contracts in Healthcare 121

6.2.6 Blockchain and Smart Contracts in Agriculture 121

6.2.7 Blockchain and Smart Contracts in Real Estate 122

6.2.8 Blockchain and Smart Contracts in Education and Research 123

6.2.9 Blockchain and Smart Contracts in Other Sectors 124

6.3 Critical Analysis of the Review 125

6.4 Blockchain and Artificial Intelligence 128

6.5 Discussion on the Reasoning for Implementation of Blockchain 129

6.6 Conclusion 130

References 130

7 Clinical Intelligence: Deep Reinforcement Learning for Healthcare and Biomedical Advancements 137
Keerthika K., Kannan M. and T. Saravanan

7.1 Introduction 138

7.2 Deep Reinforcement Learning Methods 138

7.2.1 Model-Free Methods 138

7.2.2 Policy Gradient Methods 139

7.2.3 Model-Based Methods 139

7.3 Applications of DRL in Healthcare 140

7.3.1 Tailored Treatment Recommendations 140

7.3.2 Optimization of Clinical Trials 141

7.3.3 Disease Diagnosis Support 142

7.3.4 Accelerated Drug Discovery and Design 142

7.3.5 Enhanced Robotic Surgery and Assistance 142

7.3.6 Health Management System 143

7.4 Challenges 143

7.5 Healthcare Data Types 144

7.5.1 Electronic Healthcare Records (EHRs) 144

7.5.2 Laboratory Data 145

7.5.3 Sensor Data 145

7.5.4 Biomedical Imaging Information 145

7.6 Guidelines for the Application of DRL 147

7.7 A Case Study: DRL in Healthcare and Biomedical Applications 147

7.7.1 Optimizing Radiation Therapy Dose Distribution in Cancer Treatment 147

7.7.2 Dose Strategy Model in Sepsis Patient Treatment 148

References 149

8 Cultivating Expertise in Deep and Reinforcement Learning Principles 151
Chilakalapudi Malathi and J. Sheela

8.1 Introduction 151

8.1.1 Reinforcement Learning’s Constituent Parts 152

8.1.2 Process of Markov Decisions (MDP) 152

8.1.3 Learning Reinforcement Methods 153

8.2 Intensive Learning Foundations 164

8.2.1 A Definition of Deep Learning 164

8.2.2 Deep Learning Elements 164

8.2.2.1 Different Kinds of Deep Learning Networks 165

8.3 Integrating Deep Learning and Reinforcement Learning 172

8.3.1 Deep Reinforcement Learning 172

8.3.2 Deep Reinforcement Learning Complexity Problems 174

Conclusion 175

References 175

9 Deep Reinforcement Learning in Healthcare and Biomedical Research 179
Shruti Agrawal and Pralay Mitra

9.1 Introduction 180

9.1.1 Reinforcement Learning 180

9.1.2 Deep Reinforcement Learning 181

9.2 Learning Methods in Bioinformatics with Applications in Healthcare and Biomedical Research 182

9.2.1 Protein Folding 182

9.2.2 Protein Docking 183

9.2.3 Protein-Ligand Binding 185

9.2.4 Binding Peptide Generation 187

9.2.5 Protein Design and Engineering 188

9.2.6 Drug Discovery and Development 190

9.3 Applications in Biological Data 192

9.3.1 Omics Data 192

9.3.2 Medical Imaging 192

9.3.3 Brain/Body-Machine Interfaces 193

9.4 Adaptive Treatment Approach in Healthcare 193

9.5 Diagnostic Tools in Healthcare and Biomedical Research 195

9.6 Scope of Deep Reinforcement Learning in Healthcare and Biomedical Applications 196

9.6.1 State and Action Space 196

9.6.2 Reward 197

9.6.3 Policy 198

9.6.4 Model Training 199

9.6.5 Exploration 199

9.6.6 Credit Assignment 200

9.7 Conclusions 200

References 201

10 Deep Reinforcement Learning in Robotics and Autonomous Systems 207
Uma Yadav, Shweta V. Bondre and Bhakti Thakre

10.1 Introduction 208

10.2 The Promise of Deep Reinforcement Learning (DRL) in Real-World Robotics 210

10.3 Preliminaries 211

10.4 Enhancing RL for Real-World Robotics 222

10.5 Reinforcement Learning for Various Robotic Applications 224

10.6 Problems Faced in RL for Robotics 231

10.7 RL in Robotics: Trends and Challenges 232

10.8 Conclusion 235

References 236

11 Diabetic Retinopathy Detection and Classification Using Deep Reinforcement Learning 239
H.R. Manjunatha and P. Sathish

11.1 Introduction 239

11.2 Literature Survey 243

11.3 Diabetic Retinopathy Detection and Classification 248

11.4 Result Analysis 256

11.5 Conclusion 260

References 260

12 Early Brain Stroke Detection Based on Optimized Cuckoo Search Using LSTM‐Gated Multi-Perceptron Neural Network 265
Anita Venaik, Asha A., Dhiyanesh B., Kiruthiga G., Shakkeera L. and Vinodkumar Jacob

12.1 Introduction 266

12.2 Literature Survey 268

12.2.1 Problem Statement 269

12.3 Proposed Methodology 270

12.3.1 Dataset Collection 270

12.3.2 Preprocessing 271

12.3.3 Genetic Feature Sequence Algorithm (GFSA) 275

12.3.4 Disease-Prone Factor (DPF) 281

12.3.5 Decision Tree-Optimized Cuckoo Search (DTOCS) 284

12.3.6 Long Short-Term Memory Gate Multilayer Perceptron Neural Network (LSTM-MLPNN) 289

12.4 Result and Discussion 293

12.4.1 Performance Matrix 293

12.5 Conclusion 296

References 297

13 Hybrid Approaches: Combining Deep Reinforcement Learning with Other Techniques 301
M. T. Vasumathi, Manju Sadasivan and Aurangjeb Khan

13.1 Introduction 302

13.1.1 Digital Twin - Introduction 302

13.1.2 Model of a Digital Twin 302

13.1.2.1 Steps Involved in Building a Digital Twin Prototype 303

13.1.3 Application Areas of Digital Twins 303

13.1.3.1 Digital Twin in Medical Field 304

13.1.3.2 Digital Twin in Smart City 304

13.1.3.3 Digital Twin in Sports 304

13.1.3.4 Digital Twin in Smart Manufacturing 305

13.2 Digital Twin Technologies 305

13.2.1 Data Acquisition and Sensors 306

13.2.2 Data Analytics and Machine Learning 306

13.2.3 Cloud Computing 307

13.2.4 Other Technologies 307

13.3 Integration of RL and Digital Twin 307

13.3.1 Motivation for Combining Digital Twin and RL 309

13.3.2 How RL Enhances Decision-Making Within Digital Twins 310

13.4 Challenges of Using RL in Digital Twins 311

13.5 Digital Twin Modeling with RL 312

13.6 Technology Underlying RL-Based Digital Twins 314

13.6.1 Integration of RL with Digital Twins in Four Stages 314

13.6.2 Tools and Libraries for Developing RL-Based Digital Twins 314

13.6.2.1 Simulation and Digital Twin Platforms 314

13.6.2.2 Reinforcement Learning Libraries 315

13.6.3 Integration with Existing Systems and IoT Devices for RL Deployment 315

13.6.3.1 Data Collection and Sensor Integration 315

13.6.3.2 Communication and Data Ingestion 316

13.6.3.3 Digital Twin Integration 316

13.6.3.4 RL Integration 316

13.6.3.5 Control and Actuation 316

13.6.3.6 Implementation of Feedback and Learning Process 316

13.6.3.7 Dashboard for Alert and Visualization 316

13.6.3.8 Ensuring the Security and Authentication 317

13.7 Industry-Specific Applications: A Case Study of DT in a Car Manufacturing Unit 317

13.7.1 IoT Components Required for Creating Digital Twin for the Manufacturing Unit 318

13.7.2 Architecture of the Proposed Digital Twin for Car Manufacturing Unit 318

13.7.3 Challenges and Opportunities in the Implementation of DTs for Car Manufacturing 320

13.8 Conclusion 321

References 322

14 Predictive Modeling of Rheumatoid Arthritis Symptoms: A High-Performance Approach Using HSFO-SVM and UNET-CNN 325
Anusuya V., Baseera A., Dhiyanesh B., Parveen Begam Abdul Kareem and Shanmugaraja P.

14.1 Introduction 326

14.1.1 Novelty of the Research 327

14.2 Related Work 328

14.2.1 Challenges and Problem Identification Factor 331

14.3 HSFO-SVM Based on LSTM-Gated Convolution Neural Network (lstmg-cnn) 332

14.3.1 C-Score and Cross-Fold Validation 332

14.3.2 Honey Scout Forager Optimization 335

14.3.3 Feature Selection Using SVM 336

14.3.4 UNET-CNN Classification 338

14.4 Result and Discussion 341

14.5 Conclusion 345

References 346

15 Using Reinforcement Learning in Unity Environments for Training AI Agent 349
Geetika Munjal and Monika Lamba

15.1 Introduction 349

15.2 Literature Review 351

15.3 Machine Learning 352

15.3.1 Categorization of Machine Learning 352

15.3.1.1 Supervised Learning 352

15.3.1.2 Unsupervised Learning 353

15.3.1.3 Reinforcement Learning 353

15.3.2 Classifying on the Basis of Envisioned Output 353

15.3.2.1 Classification 354

15.3.2.2 Regression 354

15.3.2.3 Clustering 354

15.3.3 Artificial Intelligence 354

15.4 Unity 354

15.4.1 Unity Hub 355

15.4.2 Unity Editor 355

15.4.3 Inspector 355

15.4.4 Game View 355

15.4.5 Scene View 355

15.4.6 Hierarchy 355

15.4.7 Project Window 356

15.5 Reinforcement Learning and Supervised Learning 356

15.5.1 Positive Reinforcement 357

15.5.2 Negative Reinforcement 357

15.5.3 Model-Free and Model-Based RL 357

15.6 Proposed Model 359

15.6.1 Setting Up a Virtual Environment 359

15.6.2 Setting Up of the Environment 360

15.6.2.1 Creating and Allocating Scripts for the Environment 361

15.6.2.2 Creating a Goal for the Agent 361

15.6.2.3 Reward-Driven Behavior 361

15.7 Markov Decision Process 362

15.8 Model-Based RL 362

15.9 Experimental Results 363

15.9.1 Machine Learning Models Used for the Environments 363

15.9.2 PushBlock 363

15.9.3 Hallway 365

15.9.4 Screenshots of the PushBlock Environment 368

15.9.5 Screenshots of the Hallway Environment 369

15.10 Conclusion 372

References 372

16 Emerging Technologies in Healthcare Systems 375
Ravi Kumar Sachdeva, Priyanka Bathla, Samriti Vij, Dishika, Madhur Jain, Lokesh Kumar, G. S. Pradeep Ghantasala and Rakesh Ahuja

16.1 Introduction 375

16.2 Personalized Medicine 376

16.3 AI and ML in Healthcare Sector 377

16.3.1 AI in Medical Diagnosis 378

16.3.2 Drug Discovery 378

16.3.3 Personalized Treatment Plans 379

16.3.4 Pattern Matching or Trend Detection 380

16.4 Immunotherapy 380

16.4.1 Monoclonal Antibodies 381

16.4.2 Checkpoint Inhibitors 381

16.4.3 CAR-T Cell Therapy 381

16.5 Regenerative Medicine 381

16.6 Digital Health (Use of Technology in Healthcare) 383

16.6.1 Wearable Devices 383

16.6.2 Telemedicine 384

16.6.3 Electronic Health Records 384

16.7 Health Inequity 385

16.7.1 Health Disparity 385

16.7.2 Health Equity 385

16.8 Future Directions in Healthcare Research 385

16.9 Challenges and Recommendations for Advanced Level of Modern Healthcare Technologies 386

16.9.1 Challenges 387

16.9.2 Recommendations 388

16.10 Healthcare Sector in Developing and Underdeveloped Countries 388

16.10.1 Healthcare Sector in Developing Countries 388

16.10.2 Healthcare Sector in Underdeveloped Countries 389

16.11 Comparison of Recent Progress and Future Mentoring in Healthcare Using Technology 389

16.12 Conclusion 391

References 392

Index 395

Authors

Shubham Mahajan Ajeekya D Y Patil University, India. Pethuru Raj Reliance Jio Platforms Ltd, Bangalore, India. Amit Kant Pandit Shri Mata Vaishno Devi University, India.