Linux is everywhere, and it is one of the most important pieces of software humans have ever developed. It is an operating system that powers the majority of servers, embedded systems, and mobile devices.
As a web developer, it is an essential tool for my daily work and here I will share some notes and tips that I have gathered over the years.
Let's start with some basic terminology:
Kernel: The kernel acts as the core of the Linux operating system, managing the communication between hardware and software. It oversees hardware operations and enables applications to utilize these resources effectively.
Process: A process is an instance of a program that is being executed. A process can be a daemon or a service, but it can also be a user-initiated program.
Service: A service is a program that operates as a background process within the system.
Daemon: A daemon is a type of service that runs in the background and is not directly controlled by the user. Daemons are usually started at boot time and are controlled by the init process.
Shell: The shell is the command line interpreter that provides a user interface for the user to interact with the kernel. It is a program that takes commands from the keyboard and gives them to the operating system to perform.
The Filesystem: A filesystem is a method for storing and organizing files and directories on a storage device.
The X Window System: The X Window System serves as the foundational framework and protocol for creating graphical user interfaces on almost all Linux systems.
Distribution: A Linux distribution is a collection of software that is built on top of the Linux kernel and provides additional tools and features.
When you switch on the computer, the Basic Input/Output System (BIOS) begins initializing the hardware and conducts a test of the main memory. This routine is known as POST, which stands for Power On Self Test.
Once the POST is completed, system control passes from the BIOS to the Bootloader, which is a small program that is
responsible for loading the operating system(located under /boot).
It starts by loading the Kernel Image and the initial RAM disk into memory (which contains some critical files and device drivers needed to start the system) and then transfers the control to the kernel.
After the kernel has configured all the hardware and mounted the root filesystem, it executes /sbin/init. This process
then becomes the initial system process, subsequently initiating other processes to operationalize the system.
While most processes on the system ultimately derive from init, exceptions exist in the form of kernel processes.
These are initiated directly by the kernel and are tasked with handling the internal aspects of the operating system.
Systemd is a system and service manager for Linux operating systems. It is the first process that starts when the Linux system boots and is the last process that stops when the system shuts down.
To manage systemd services use systemctl:
# check status of a service
systemctl status cups
# start/stop/restart a service
systemctl start cups
systemctl stop cups
systemctl restart cups
# enable/disable a service on boot
systemctl enable cups
systemctl disable cups
Loading the graphical user interface is one of the final steps in the boot process. It allows users to interact with the system using graphical elements such as windows, icons, and menus.
A component known as the Display Manager is responsible for managing the displays available and initiates the X server (referred to as such because it delivers graphical services to applications, often referred to as X clients). Additionally, the Display Manager oversees graphical user logins and initiates the relevant desktop environment once a user has logged in.
A desktop environment comprises a session manager, which initiates and oversees the elements of the graphical session, and the window manager, which manages the positioning and movement of windows, including their title-bars and other controls.
The default display manager for GNOME is gdm.
/ The root directory: All other directories are subdirectories of root, which is the top of the directory tree structure./bin Essential user command binaries that need to be available for all users cat, ls, cp etc./boot Static files of the bootloader./dev Device files such as terminal devices, usb or any device attached to the system./etc Host-specific system configuration./home User home directories./lib Essential shared libraries and kernel modules./media Mount point for removable media./mnt Mount point for mounting a filesystem temporarily./opt Add-on application software packages./sbin Essential system binaries./srv Data for services provided by this system./tmp Temporary files./usr Secondary hierarchy for read-only user data; contains the majority of (multi-)user utilities and applications./var Variable data; files whose content is expected to continually change during normal operation of the system./root Home directory for the root user./proc Virtual filesystem providing process and kernel information as files.There are two types of package managers low and high level.
Low-level package managers handle the basic tasks of installing, removing, and managing individual packages. On the other hand, High-level package managers, like apt, simplify the process by automatically taking care of dependencies and providing easy access to a wide range of software through repositories, all within a user-friendly interface.
Low-level:
dpkg is a low-level package manager that is used to install, build, remove, and manage Debian packages.
# install a package
dpkg -i package.deb
# remove a package(keep configuration files)
dpkg -r package
# remove a package(remove configuration files)
dpkg -P package
# list installed packages
dpkg -l
# list files installed by a package
dpkg -L package
High-level:
apt is a high-level package manager that is used to install, remove, and update software packages.
# update the package index
apt update
# install a package
apt install package1
# upgrade packages
apt upgrade
# remove a package (keep configuration files)
apt remove package1
# remove a package (remove configuration files)
apt purge package1
# remove unused packages
apt autoremove
# list installed packages
apt list --installed
# list files installed by a package
apt list --installed | grep package1
# search for a package
apt search package1
# show package information
apt show package1
# concatenate files to standard output
cat file.txt
# see the contents of a file starting from the top
head file.txt
# see the contents of a file from the bottom
tail file.txt
# check a command manual
man cat
THey are called "virtual" because, although there can be multiple active terminals, only one terminal remains visible at a time.
One virtual terminal (usually VT 1 or VT 7) is reserved for the graphical environment, and text logins are enabled on the unused VTs.
To switch between VTs, press CTRL-ALT-function key for the VT.
# for Virtual Terminal 6 press:
CTRL + ALT + F6
The pipe symbol “man head | head” is used to pass the output of a program as an input for another from left to right.
man head | head
Shout down
# sends warning message and prevents users from logging in.
shotdown
shutdown -h 10:00 "Shutting down for scheduled maintenance."
# reboot
shotdown -r
$ which cat
/usr/bin/cat
$ whereis cat
cat: /usr/bin/cat /usr/share/man/man1/cat.1.gz
# find current path
$ pwd
# change directory
$ cd
# change to user home directory
$ cd ~
# go to parent directory
$ cd ..
df -h
Links are essentially shortcuts or references to a file or directory.
Hard links are simply additional names for an existing file.
$ ln /path/to/original /path/to/link
A symbolic link is like a shortcut to the original file or directory. If you delete the original file, the symbolic link will still exist but will point to a non-existing location.
$ ln -s /path/to/original /path/to/link
# changes your current directory to the root (/) directory.
$ cd /
# list the contents of the present working directory.
$ ls
# ls –a
$ list all files, including hidden files and directories.
# displays a tree view of the filesystem.
$ tree
# add a directory to the top of the directory stack, changing the current working directory to that directory.
$ pushd
# remove a directory from the top of the directory stack, changing the current working directory to that directory.
$ popd
# display the directory stack.
$ dirs
# modify timestamps of files or create new empty files.
$ touch file1.txt
# change the command line prompt to display the current working directory.
$ PS1="\w$ "
# change the command line prompt to display the current working directory and the time.
$ PS1="\w \t$ "
# change input stream
$ cat < file1.txt
# change output stream
$ cat file1.txt > file2.txt
# change output stream
$ cat file1.txt 1> file2.txt
# append to a file
$ cat file1.txt >> file2.txt
# redirect standard error to a file
$ command file1.txt 2> file2.txt
# redirect everything to a file
$ command file1.txt &> file2.txt
# executes a search utilizing an existing database of system files and directories.
$ locate file1.txt
# get a shorter list of results using grep to print only the lines that contain one or more specified strings
$ locate file1 | grep test
# search for files
$ find / -name file1.txt
# find files based on time
$ find / -ctime 3
# find files based on size
$ find / -size +100M
Grep is used as a primary text searching tool. It scans files for specified patterns and can be used with regular expressions, as well as simple strings:
# search for a pattern in a file and print all matching lines
grep [pattern] somefile.txt
# print all lines that do not match the pattern
grep -v [pattern] somefile.txt
# print the lines that contain the numbers 0 through 9
grep [0-9] somefile.txt
# print the context of lines
grep -C [pattern] somefile.txt
All running applications are processes.
A process is a running instance of one or more related tasks(threads). It is not the same thing as a program. A program or a command can start several simultaneous processes.
Some processes depend on each other and others are independent. A failed process may affect other processes running on the system.
In order to accomplish their task, they use system resources such as memory, CPU cycles and peripheral devices such as network cards displays etc. The operating system, particularly its kernel, plays a crucial role in distributing the right amount of these resources to each process while ensuring that the system as a whole is utilized efficiently.
Interactive Processes: These are user-initiated processes, started either via command line or graphical interface. Examples include 'bash' for command-line operations, 'firefox' as a web browser, 'top' for system monitoring, 'Slack' for communication etc.
Batch Processes: Automated and scheduled processes, running independently of user interaction and following a FIFO queue system. 'updatedb' and 'ldconfig' are common examples, often used for system maintenance tasks.
Daemons: Server processes that run in the background, often starting at system boot. They await system or user requests to provide services. Examples include 'httpd' for web services and 'sshd' for secure shell services.
Threads: Lightweight tasks running under a main process, sharing resources but individually managed. They enhance efficiency in complex applications like 'dconf-service' and 'gnome-terminal-server'.
Kernel Threads: Essential system tasks managed by the kernel, such as 'kthreadd' and 'migration'. These are not user-controlled and perform core system functions like resource management and process scheduling.
Running (R): The process is either actively running on a CPU or is ready to run.
Sleeping (S): The process is in a sleep state, waiting for an event or resource to become available.
Interruptible Sleep(D): The process is sleeping in a state that can be interrupted by signals.
Uninterruptible Sleep (D): This is similar to Interruptible Sleep, but in this state, the process cannot be interrupted by signals. This usually happens when the process is waiting for I/O operations to complete.
Stopped (T): The process has been stopped, either by a job control signal or because it is being traced.
Zombie (Z): The process has completed its execution but still exists in the process table because the parent process has not yet read its exit status.
Process ID (PID): Unique Process ID number.
Parent Process ID (PPID): Parent Process that initiated this process. Should the originating parent cease to function, the PPID will then point to an adoptive parent process; in contemporary kernel versions, this role is typically assumed by 'kthreadd', identified by a PPID of 2.
Thread ID (TID): For single-threaded processes, this identifier is identical to the PID. In contrast, within a multi-threaded process, although all threads share the same PID, each one possesses a distinct TID.
# find the process id
ps aux | grep example_process
# kill the process
kill -SIGKILL 1234
kill -9 1234
Each process is associated with user and group IDs, which determine the permissions and access levels of the process.
The nicen value of a process determines its priority level, with a lower nice value indicating a higher priority. Essential processes are usually given low niceness values, signifying high importance, whereas processes that can afford to wait are assigned higher values. A process with a higher niceness value, allows other processes to be executed first. In Linux, the niceness scale ranges from -20 for the highest priority to +19 for the lowest. This might seem counterintuitive, but this naming convention, where a 'nicer' process has a lower priority, is a legacy from the early days of UNIX.
# detail view of all processes
ps lf
# change the nice score of a process
renice +5 3037
Typically, jobs run in the foreground as a standard. However, appending an & symbol to a command, such as updatedb &, will execute the job in the background instead.
To suspend a job running in the foreground, you can press CTRL-Z, which suspends(SIGTSTP) and moves it to the background. To end the job entirely, use CTRL-C. If you have a suspended process, you can resume it in the background with the bg command, or bring a background process to the foreground using the fg command.
# start a process
gedit text.txt
# while on terminal hit CTRL+Z to suspend the process in the background
CTRL+Z
# with jobs -l you can see what processes have been launched from this terminal
jobs -l
[1] + 29554 suspended text.txt
# to run the process in the background use bg
bg
# to put the process in the foreground use fg
fg
# provides a detailed view of running processes
ps -l
# shows extensive details about all running processes
ps -aux
# displays real-time information on system processes and resource usage
top
# an interactive and advanced version of top
htop
# displays the processes running on the system in the form of a tree diagram
pstree
Cron is a time-based scheduling utility program.
# cron config file (cron table)
/etc/crontab
# anacron runs the jobs even if the system was down
# anacron config file
/etc/anacrontab
# make a script executalbe
chmod +x testat.sh
at now + 1 minute -f testat.sh
# see if the job is queued
atq
# make sure the job actually ran
cat /tmp/datestamp
# interactively
at now + 1 minute
# delays execution for a specific period
sleep 5 && gedit README.md
Everything is a file!
Linux filesystems are crucial for organizing and accessing data. Linux supports multiple filesystem types like ext4, XFS, Btrfs, FAT32, and NTFS, each with unique features.
The filesystems manage not only files but also metadata like permissions and support symbolic and hard links. Linux's filesystem hierarchy integrates all storage into a unified directory tree, starting from the root directory (/).
Additionally, Linux supports network-based filesystems, notably NFS (Network File System), which allows files to be shared across a network. NFS enables multiple clients to access data stored on a server as if it were locally mounted, enabling seamless file sharing across different systems and networks.
Mounting is essential in Linux, enabling access to filesystems at specific points in this directory tree.
Mounting refers to the process of making a filesystem accessible at a certain point in the directory structure. In short, you need to mount a filesystem to access its contents.
# mount the filesystem located on the partition /dev/sda5 to the directory /home
sudo mount /dev/sda5 /home
# unmount
sudo umount /home
If you want it to be automatically available every time the system starts up, you need to edit /etc/fstab accordingly.
Some filesystems, such as those mounted at /proc, are referred to as pseudo-filesystems because they don't have a physical representation on the disk.
The /proc filesystem is very important as it collects information on-demand without requiring disk storage.
You can find process statuses and other useful information.
# process 737 info
/proc/737/status
# cpu info
/proc/cpuinfo
# compare two files
diff file1.txt file2.txt
# a more readable format
diff -u file1.txt file2.txt
# side by side diff
diff -y file1.txt file2.txt
# recursive directory comparison
diff -r dir1 dir2
# add the differences into a patch file
diff -u file1.txt file2.txt > changes.patch
# apply the patches
patch original.txt changes.patch
Rsync stands for remote sync. It's a utility tool for efficient file transfer and synchronization.
# rsync a file from remote host to local with verbose and compress modes
rsync -vz user@remote_host:/path/to/remote/file /path/to/local/destination
# if specifies the input and of the output
# bs=64K sets the block size
# status=progress outputs the progress
dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync status=progress
VI is a text editor for Linux that can be utilized when GUI is not available. It is more advanced than the Nano editor but it is more complex, too.
The key point to grasp is that VI operates in three distinct modes:
# Edit some file
vi somefile.txt
# Recovery mode
vi -r somefile.txt
# In line mode
# write to a file
:q
# exit
:x or :wq
# quit
:q
# quit without saving
:q!
# Searching in VI
/pattern
# move to the next occurrence
n
# Move to previous occurrence
N
# Modifies the permissions for the user (owner) and others (o).
# The +x adds execute permissions.
chmod uo+x somefile.txt
# Modifies the permissions for the group (g).
# The -w removes write permissions for the group.
chmod g-w somefile.txt
# display the value of a specific variable
echo $SHELL
# export a variable
export VARIABLE_NAME="value"
# Add a variable permanently
1. edit ~/.bashrc and add the "export VARIABLE=value"
# reload the bash config
source ~/.bashrc or ~/.bashrc
The Path variable specifies directories where Linux shell searches for executable files to run commands without full paths.
You can add other things to Path variable:
export PATH=$HOME/bin:$PATH
# browse the list of executed commands
up/down keys
# execute previous command
!! (bang-bang)
# Search in command history
CTRL + R
# displays the contents of a file one page at a time
less somefile
cat somefile | less
# display the beginning of a text file
head –n 5 somefile
head -5 somefile
# display the end of a file
tail -15 somefile.log
tail -f somefile.log
ls -lR /tmp > /dev/null
ethtool ethtool is a Linux utility for network interface configuration, offering capabilities to view and change settings such as speed, duplex, and auto-negotiation.
netstat netstat is a tool for displaying network connections, routing tables, and interface statistics, useful for troubleshooting and monitoring network activity.
nmap Scans open ports on a network. Useful for security analysis.
tcpdump Dumps network traffic.
iptraf Monitors network traffic in text mode.
mtr Combines ipn and traceroute.
dig Tests DNS wokrings
# lock screen
super + l
# open terminal
ctrl + alt + t
# terminal
# clears the Screen
CTRL + L
# temporarily halt output
CTRL + S
# continue temporarily halted output
CTRL + Q
# exit current shell
CTRL + D
# put the current process into suspended background
CTR + Z
# kill the current process
CTRL + C
# go to the beginning of the line
CTRL + A
# delete the word before the cursor
CTRL + W
# delete from the beginning of line to cursor position
CTRL + U
# go to the end of the line
CTRL + E
References:
https://git-scm.com/docs/git-reflog https://stackoverflow.com/questions/1459150/how-to-undo-git-commit-amend-done-instead-of-git-commit https://gitready.com/intermediate/2009/02/09/reflog-your-safety-net.html