Linux Command Line and Shell Scripting Bible. Edition No. 4

Table of Contents

Acknowledgments xi

Part I: The Linux Command Line 1

Chapter 1: Starting with Linux Shells 3

Investigating Linux 3

Looking into the Linux kernel 4

System memory management 5

Software program management 6

Hardware management 8

Filesystem management 8

The GNU utilities 9

The core GNU utilities 10

The shell 10

The Linux desktop environment 11

The X Window software 12

The KDE Plasma desktop 12

The GNOME desktop 13

Other desktops 15

Examining Linux Distributions 17

Core Linux distributions 18

Specialized Linux distributions 19

Summary 20

Chapter 2: Getting to the Shell 21

Reaching the Command Line 21

Console terminals 22

Graphical terminals 22

Accessing CLI via a Linux Console Terminal 23

Accessing CLI via Graphical Terminal Emulation 26

Using the GNOME Terminal Emulator 27

Accessing GNOME Terminal 27

The menu bar 32

Using the Konsole Terminal Emulator 36

Accessing Konsole 36

The menu bar 38

Using the xterm Terminal Emulator 42

Accessing xterm 43

Command-line parameters 44

Summary 45

Chapter 3: Basic Bash Shell Commands 47

Starting the Shell 47

Using the Shell Prompt 48

Interacting with the Bash Manual 49

Navigating the Filesystem 53

Looking at the Linux filesystem 53

Traversing directories 57

Using absolute directory references 57

Using relative directory references 59

Listing Files and Directories 60

Displaying a basic listing 61

Displaying a long listing 63

Filtering listing output 64

Handling Files 66

Creating files 66

Copying files 66

Using command-line completion 69

Linking files 69

Renaming files 71

Deleting files 73

Managing Directories 74

Creating directories 74

Deleting directories 75

Viewing File Contents 77

Viewing the file type 77

Viewing the whole file 78

Using the cat command 78

Using the more command 78

Using the less command 80

Viewing parts of a file 80

Using the tail command 80

Using the head command 82

Summary 82

Chapter 4: More Bash Shell Commands 85

Monitoring Programs 85

Peeking at the processes 85

Unix-style parameters 86

BSD-style parameters 89

The GNU long parameters 91

Real-time process monitoring 93

Stopping processes 95

The kill command 95

The pkill command 96

Monitoring Disk Space 96

Mounting media 97

The mount command 97

The umount command 99

Using the df command 100

Using the du command 101

Working with Data Files 102

Sorting data 102

Searching for data 106

Compressing data 108

Archiving data 109

Summary 111

Chapter 5: Understanding the Shell 113

Investigating Shell Types 113

Exploring Parent and Child Shell Relationships 117

Looking at process lists 121

Creatively using subshells 123

Investigating background mode 123

Putting process lists into the background 125

Looking at co-processing 126

Understanding External and Built-In Commands 127

Looking at external commands 128

Looking at built-in commands 129

Using the history command 130

Using command aliases 134

Summary 135

Chapter 6: Using Linux Environment Variables 137

Exploring Environment Variables 137

Looking at global environment variables 138

Looking at local environment variables 140

Setting User-Defined Variables 141

Setting local user-defined variables 141

Setting global environment variables 142

Removing Environment Variables 144

Uncovering Default Shell Environment Variables 145

Setting the PATH Environment Variable 150

Locating System Environment Variables 152

Understanding the login shell process 152

Viewing the /etc/profi le file 152

Viewing the $HOME startup files 156

Understanding the interactive shell process 157

Understanding the noninteractive shell process 158

Making environment variables persistent 159

Learning about Variable Arrays 159

Summary 161

Chapter 7: Understanding Linux File Permissions 163

Exploring Linux Security 163

The /etc/passwd file 164

The /etc/shadow file 165

Adding a new user 166

Removing a user 169

Modifying a user 169

usermod 170

passwd and chpasswd 170

chsh, chfn, and chage 171

Using Linux Groups 173

The /etc/group file 173

Creating new groups 174

Modifying groups 175

Decoding File Permissions 176

Using file permission symbols 176

Default file permissions 178

Changing Security Settings 179

Changing permissions 180

Changing ownership 181

Sharing Files 182

Access Control Lists 184

Summary 186

Chapter 8: Managing Filesystems 189

Exploring Linux Filesystems 189

Exploring the Linux filesystem evolution 190

Looking at the ext filesystem 190

Looking at the ext2 filesystem 190

Digging into journaling filesystems 191

Looking at the ext3 filesystem 191

Looking at the ext4 filesystem 191

Looking at the JFS filesystem 192

Looking at ReiserFS 192

Looking at XFS 192

Understanding the volume-managing filesystems 192

Looking at the ZFS filesystem 193

Looking at the Btrfs filesystem 193

Looking at the Stratis filesystem 193

Working with Filesystems 194

Creating partitions 194

Looking at the fdisk utility 194

Working with gdisk 197

The GNU parted command 198

Creating a filesystem 199

Checking and repairing a filesystem 201

Managing Logical Volumes 202

Exploring LVM layout 203

Physical volume 203

Volume group 203

Logical volume 203

Understanding the LVM in Linux 203

Create the PVs 204

Create a VG 204

Create an LV 205

Using the Linux LVM 206

Format and mount an LV 206

Growing or shrinking your VGs and LVs 206

Summary 207

Chapter 9: Installing Software 209

Exploring Package Management 209

Inspecting the Debian-Based Systems 210

Managing packages with apt 211

Installing software packages with apt 213

Upgrading software with apt 215

Uninstalling software with apt 216

The apt repositories 218

The Red Hat-Based Systems 220

Listing installed packages 220

Installing software with dnf 221

Upgrading software with dnf 223

Uninstalling software with dnf 223

Dealing with broken dependencies 223

RPM repositories 224

Managing Software Using Containers 225

Using snap containers 225

Using flatpak containers 227

Installing from Source Code 229

Summary 232

Chapter 10: Working with Editors 235

Visiting the vim Editor 235

Checking your vim package 236

Exploring vim basics 237

Editing data 240

Copying and pasting 240

Searching and substituting 241

Navigating the nano Editor 242

Exploring the Emacs Editor 244

Checking your Emacs package 245

Using Emacs on the console 246

Exploring the basics of Emacs 247

Editing data 248

Copying and pasting 249

Searching and replacing 249

Using buffers in Emacs 250

Using windows in console mode Emacs 251

Using Emacs in a GUI 252

Exploring the KDE Family of Editors 253

Looking at the KWrite editor 253

Looking at the Kate editor 259

Exploring the GNOME Editor 263

Starting gedit 264

Understanding basic gedit features 265

Managing plugins 268

Summary 271

Part II: Shell Scripting Basics 273

Chapter 11: Basic Script Building 275

Using Multiple Commands 275

Creating a Script File 276

Displaying Messages 278

Using Variables 279

Environment variables 280

User variables 281

Command substitution 283

Redirecting Input and Output 284

Output redirection 285

Input redirection 285

Employing Pipes 287

Performing Math 289

The expr command 290

Using brackets 292

A floating-point solution 293

The basics of bc 293

Using bc in scripts 295

Exiting the Script 297

Checking the exit status 297

The exit command 298

Working through a Practical Example 300

Summary 301

Chapter 12: Using Structured Commands 303

Working with the if-then Statement 303

Exploring the if-then-else Statement 306

Nesting ifs 307

Trying the test Command 311

Using numeric comparisons 313

Using string comparisons 314

Looking at string equality 315

Looking at string order 316

Looking at string size 318

Using file comparisons 320

Checking directories 320

Checking whether an object exists 321

Checking for a file 322

Checking for read access 324

Checking for empty files 325

Checking whether you can write to a file 326

Checking whether you can run a file 327

Checking ownership 328

Checking default group membership 329

Checking file date 330

Considering Compound Testing 331

Working with Advanced if-then Features 332

Using single parentheses 332

Using double parentheses 333

Using double brackets 335

Considering the case Command 335

Working through a Practical Example 337

Summary 343

Chapter 13: More Structured Commands 345

Looking at the for Command 345

Reading values in a list 346

Reading complex values in a list 347

Reading a list from a variable 349

Reading values from a command 350

Changing the field separator 351

Reading a directory using wildcards 352

Trying the C-Style for Command 354

The C language for command 355

Using multiple variables 356

Exploring the while Command 357

Basic while format 357

Using multiple test commands 358

Using the until Command 359

Nesting Loops 361

Looping on File Data 364

Controlling the Loop 365

The break command 365

Breaking out of a single loop 365

Breaking out of an inner loop 366

Breaking out of an outer loop 367

The continue command 368

Processing the Output of a Loop 371

Working through a Few Practical Examples 372

Finding executable files 373

Creating multiple user accounts 374

Summary 375

Chapter 14: Handling User Input 377

Passing Parameters 377

Reading parameters 377

Reading the script name 380

Testing parameters 381

Using Special Parameter Variables 382

Counting parameters 382

Grabbing all the data 384

Being Shifty 386

Working with Options 388

Finding your options 388

Processing simple options 388

Separating options from parameters 389

Processing options with values 391

Using the getopt command 392

Looking at the command format 392

Using getopt in your scripts 393

Advancing to getopts 395

Standardizing Options 398

Getting User Input 399

Reading basics 399

Timing Out 401

Reading with no display 402

Reading from a file 403

Working through a Practical Example 404

Summary 408

Chapter 15: Presenting Data 411

Understanding Input and Output 411

Standard file descriptors 412

STDIN 412

STDOUT 413

STDERR 414

Redirecting errors 414

Redirecting errors only 414

Redirecting errors and data 415

Redirecting Output in Scripts 416

Temporary redirections 416

Permanent redirections 417

Redirecting Input in Scripts 418

Creating Your Own Redirection 419

Creating output file descriptors 419

Redirecting file descriptors 420

Creating input file descriptors 421

Creating a read/write file descriptor 422

Closing file descriptors 423

Listing Open File Descriptors 424

Suppressing Command Output 426

Using Temporary Files 427

Creating a local temporary file 427

Creating a temporary file in /tmp 428

Creating a temporary directory 429

Logging Messages 430

Working through a Practical Example 431

Summary 433

Chapter 16: Script Control 435

Handling Signals 435

Signaling the Bash shell 435

Generating signals 436

Interrupting a process 436

Pausing a process 437

Trapping signals 438

Trapping a script exit 440

Modifying or removing a trap 441

Running Scripts in Background Mode 443

Running in the background 443

Running multiple background jobs 445

Running Scripts without a Hang-up 447

Controlling the Job 448

Viewing jobs 448

Restarting stopped jobs 450

Being Nice 451

Using the nice command 452

Using the renice command 453

Running like Clockwork 454

Scheduling a job using the at command 454

Understanding the at command format 454

Retrieving job output 455

Listing pending jobs 457

Removing jobs 457

Scheduling regular scripts 458

Looking at the cron table 458

Building the cron table 459

Viewing cron directories 460

Looking at the anacron program 460

Starting scripts with a new shell 462

Working through a Practical Example 463

Summary 469

Part III: Advanced Shell Scripting 471

Chapter 17: Creating Functions 473

Exploring Basic Script Functions 473

Creating a function 474

Using functions 474

Returning a Value from a Function 477

The default exit status 477

Using the return command 478

Using function output 479

Using Variables in Functions 480

Passing parameters to a function 480

Handling variables in a function 482

Global variables 483

Local variables 484

Investigating Array Variables and Functions 485

Passing arrays to functions 485

Returning arrays from functions 487

Considering Function Recursion 488

Creating a Library 489

Using Functions on the Command Line 491

Creating functions on the command line 491

Defining functions in the bashrc file 492

Directly defining functions 492

Sourcing function files 493

Working Through a Practical Example 494

Downloading and installing 494

Building the library 495

The shtool library functions 496

Using the library 497

Summary 497

Chapter 18: Writing Scripts for Graphical Desktops 499

Creating Text Menus 499

Create the menu layout 499

Create the menu functions 501

Add the menu logic 502

Putting it all together 502

Using the select command 504

Doing Windows 505

The dialog package 506

The msgbox widget 507

The yesno widget 508

The inputbox widget 508

The textbox widget 510

The menu widget 511

The fselect widget 512

The dialog options 513

Using the dialog command in a script 515

Getting Graphic 516

The KDE environment 517

kdialog widgets 517

Using kdialog 519

The GNOME environment 521

zenity Widgets 521

Using zenity in scripts 523

Working Through a Practical Example 525

Summary 529

Chapter 19: Introducing sed and gawk 531

Manipulating Text 531

Getting to know the sed editor 532

Defining an editor command in the command line 533

Using multiple editor commands in the command line 534

Reading editor commands from a file 534

Getting to know the gawk program 535

Visiting the gawk command format 536

Reading the program script from the command line 536

Using data field variables 537

Using multiple commands in the program script 538

Reading the program from a file 539

Running scripts before processing data 539

Running scripts after processing data 540

Looking at the sed Editor Basic Commands 542

Introducing more substitution options 542

Substituting flags 542

Replacing characters 543

Using addresses 544

Addressing the numeric line 544

Using text pattern filters 545

Grouping commands 546

Deleting lines 547

Inserting and appending text 549

Changing lines 551

Transforming characters 552

Printing revisited 553

Printing lines 554

Printing line numbers 555

Listing lines 555

Using files with sed 556

Writing to a file 556

Reading data from a file 557

Working Through a Practical Example 559

Summary 564

Chapter 20: Regular Expressions 565

Exploring Regular Expressions 565

A definition 565

Types of regular expressions 567

Defining BRE Patterns 567

Plain text 567

Special characters 569

Anchor characters 570

Starting at the beginning 570

Looking for the ending 571

Combining anchors 572

The dot character 572

Character classes 573

Negating character classes 575

Using ranges 576

Special character classes 577

The asterisk 578

Trying Out Extended Regular Expressions 579

The question mark 579

The plus sign 580

Using braces 581

The pipe symbol 582

Grouping expressions 583

Working Through Some Practical Examples 584

Counting directory files 584

Validating a phone number 585

Parsing an email address 587

Summary 589

Chapter 21: Advanced sed 591

Looking at Multiline Commands 591

Navigating the next command 592

Using the single-line next command 592

Combining lines of text 593

Navigating the multiline delete command 595

Navigating the multiline print command 596

Holding Space 598

Negating a Command 599

Changing the Flow 602

Branching 603

Testing 605

Replacing via a Pattern 606

Using the ampersand 606

Replacing individual words 607

Placing sed Commands in Scripts 608

Using wrappers 608

Redirecting sed output 609

Creating sed Utilities 610

Spacing with double lines 610

Spacing files that may have blanks 611

Numbering lines in a file 612

Printing last lines 613

Deleting lines 614

Deleting consecutive blank lines 615

Deleting leading blank lines 616

Deleting trailing blank lines 616

Removing HTML tags 617

Working Through a Practical Example 619

Summary 624

Chapter 22: Advanced gawk k 627

Using Variables 627

Built-in variables 628

The field and record separator variables 628

Data variables 631

User-defined variables 634

Assigning variables in scripts 634

Assigning variables on the command line 635

Working with Arrays 636

Defining array variables 636

Iterating through array variables 637

Deleting array variables 638

Considering Patterns 638

Regular expressions 639

The matching operator 639

Mathematical expressions 640

Structured Commands 641

The if statement 641

The while statement 643

The do-while statement 644

The for statement 645

Printing with Formats 645

Using Built-in Functions 648

Mathematical functions 649

String functions 650

Time functions 652

Trying Out User-Defined Functions 653

Defining a function 653

Using your functions 654

Creating a function library 654

Working Through a Practical Example 655

Summary 657

Chapter 23: Working with Alternative Shells 659

Considering the Dash Shell 659

Looking at the Dash Shell Features 661

The Dash command-line parameters 661

The Dash environment variables 662

Default environment variables 662

Positional parameters 663

User-defined environment variables 663

The Dash built-in commands 664

Scripting in Dash 665

Creating Dash scripts 665

Things that won’t work 665

Using arithmetic 666

The test command 666

The function command 667

Exploring the zsh Shell 668

Viewing Parts of the zsh shell 668

Shell options 669

Built-in commands 670

Core built-in commands 670

Add-in modules 673

Viewing, adding, and removing modules 673

Scripting with zsh 674

Mathematical operations 674

Performing calculations 674

Mathematical functions 675

Structured commands 676

Functions 677

Working Through a Practical Example 677

Summary 678

Part IV: Creating and Managing Practical Scripts 679

Chapter 24: Writing Simple Script Utilities 681

Performing Backups 681

Backing up files daily 682

Obtaining the required functions 682

Creating a daily archive location 685

Creating a daily backup script 686

Running the daily archive script 688

Creating an hourly archive script 689

Running the hourly archive script 692

Managing Account Deletion 693

Obtaining the required functions 693

Getting the correct account name 693

Creating a function to get the correct account name 695

Verifying the entered account name 696

Determining whether the account exists 698

Removing any account processes 699

Finding account files 702

Removing the account 702

Creating the script 703

Running the script 708

Monitoring Your System 710

Obtaining the default shell audit functions 710

Obtaining the permission audit functions 714

Creating the script 716

Running the script 719

Summary 721

Chapter 25: Getting Organized 723

Understanding Version Control 723

Working directory 724

Staging area 725

Looking at the local repository 725

Exploring a remote repository 726

Branching 726

Cloning 726

Using Git for VCS 727

Setting Up Your Git Environment 727

Committing with Git 732

Summary 739

Appendix A: Quick Guide to Bash Commands 741

Appendix B: Quick Guide to sed and gawk 755

Index 767

Authors

Richard Blum Christine Bresnahan