Operating systems

An operating system manages hardware resources and provides abstractions that programs can use safely: processes, virtual memory, files, sockets, timers, and device I/O. This chapter focuses on the concepts needed to diagnose applications, not kernel implementation catalogues.

For the guided experiments, use Module 2: How programs run.

Kernel and user space

Application code normally runs in user mode, where privileged hardware operations are restricted. The kernel runs with greater privilege and manages shared resources.

A program requests kernel work through a system call, such as opening a file, reading bytes, creating a process, allocating mapped memory, or sending network data. A library call may issue zero, one, or many system calls.

Transitions into the kernel have overhead, but minimizing their count is not automatically an optimization. Batch or avoid calls only after measurement identifies them as material.

Programs, processes, and threads

• A program is executable code and data stored somewhere.
• A process is a running instance with an address space, credentials, open resources, and one or more threads.
• A thread is an execution stream scheduled by the OS. Threads in one process share memory and most process resources.

A process commonly moves among runnable, running, sleeping or blocked, stopped, and terminated states. Exact names differ by OS.

The scheduler chooses which runnable thread uses a CPU. A context switch saves one execution context and restores another. It may also disturb caches and translation state, so its cost is workload- and machine-dependent.

More threads do not guarantee more throughput. They help when independent work exists and do not overwhelm CPU, memory, locks, I/O, or downstream capacity.

Process creation and termination

Processes have identifiers and parent relationships. Creation semantics differ across operating systems, but a new process receives its own execution context and resource handles according to the chosen API.

Processes terminate by returning an exit status, receiving a signal or equivalent event, or failing. Parent processes should collect child status where the OS requires it.

Normal service shutdown should:

1. stop accepting work;
2. finish or cancel in-flight work within a deadline;
3. close resources and settle transactions;
4. exit with a meaningful status.

A forced kill cannot be relied upon to run cleanup code.

Virtual memory

Each process sees a virtual address space. The OS and hardware map virtual pages to physical memory, files, shared mappings, or no current backing.

Common regions include:

• executable code and static data;
• thread stacks;
• dynamically allocated runtime objects, often called the heap;
• shared libraries;
• memory-mapped files and anonymous mappings.

The diagram is a model, not a guarantee of one exact layout. Runtimes manage frames and objects differently.

Pages and faults

Memory is managed in pages. A page fault occurs when an access needs kernel handling—for example, to establish a mapping, load file-backed data, or reject invalid access. A fault is not necessarily an application error.

Under memory pressure, an OS may reclaim cached pages or move eligible memory to secondary storage. Heavy faulting and swapping can make a process slow even when CPU use is low.

Useful measures:

• virtual size: address space mapped or reserved;
• resident set: pages currently resident in physical memory;
• runtime allocations: objects tracked by a language tool;
• peak use: highest observed value during an interval.

These measures answer different questions and should not be expected to match.

Concurrency and synchronization

A race condition occurs when correctness depends on timing or interleaving. A data race is unsynchronized conflicting memory access under the language's memory model.

Common primitives:

Primitive	Purpose
Mutex	one holder enters a critical section
Read-write lock	concurrent readers, exclusive writer
Semaphore	bound access to a counted resource
Condition variable or event	wait for a state change
Atomic operation	indivisible operation with defined memory-order behavior
Queue or channel	transfer ownership or messages between workers

Protect invariants, not individual lines. Keep critical sections small but complete.

The classic necessary conditions for deadlock are mutual exclusion, hold-and-wait, no forced preemption, and a cycle of waits. Preventing one condition—for example, acquiring locks in a consistent order—can prevent that class of deadlock.

Files and filesystems

A filesystem maps names to persistent objects and metadata. A process usually accesses an open object through a file descriptor or handle.

Separate these concepts:

• path: a name resolved through directories;
• open file: kernel state including access mode and current position;
• file data: bytes stored or cached;
• metadata: ownership, permissions, timestamps, and other attributes;
• directory entry: mapping from a name to a filesystem object.

Deleting a name does not necessarily invalidate an already open handle. Renaming within one filesystem is commonly atomic, making temporary-file replacement useful, but durability after a crash may require flushing file and directory state according to the filesystem contract.

Buffered writes can succeed before data reaches persistent media. “The write call returned” and “the data survives sudden power loss” are different guarantees.

I/O

I/O may be:

• blocking: the calling thread waits;
• non-blocking: the operation reports that it cannot proceed now;
• asynchronous: completion is reported later;
• buffered: data is copied through user or kernel buffers;
• memory-mapped: file data is accessed through virtual memory mappings.

The right model depends on concurrency, latency, throughput, cancellation, and library support. Asynchronous code does not make the device faster; it lets a thread do other work while operations wait.

Devices often use interrupts and DMA beneath the OS abstraction. Applications should reason first from observable system calls, waits, and throughput.

Inter-process communication

Processes can communicate through:

• pipes and named pipes;
• local or network sockets;
• shared memory plus synchronization;
• signals or event mechanisms;
• files and databases;
• OS-specific message queues.

Shared memory avoids serialization but creates synchronization and ownership problems. Message passing defines a clearer boundary but introduces encoding, buffering, and partial-failure concerns.

Permissions and isolation

The OS evaluates operations using process credentials, object permissions, and additional security policy. Apply least privilege:

• run services as a dedicated unprivileged identity;
• grant access only to required files, ports, and devices;
• separate writable data from immutable application files;
• do not treat a container boundary as equivalent to a virtual-machine boundary;
• keep secrets out of command arguments and world-readable files.

The kernel isolates processes, but privileged processes and kernel vulnerabilities can cross those boundaries. Isolation strength depends on configuration and threat model.

Virtual machines and containers

A virtual machine presents virtual hardware and runs a guest operating system. A container is a host process isolated with kernel mechanisms such as namespaces, resource controls, credentials, and filesystem views.

Containers share the host kernel. They improve packaging and isolation but do not remove the need for OS patching, least privilege, resource limits, or secure configuration.

Observation tools

Question	Linux examples	macOS examples
What processes run?	ps, top, /proc	ps, top
Which files or sockets are open?	lsof, /proc/PID/fd	lsof
Which system calls occur?	strace	platform tracing tools, often permission-restricted
Where is time spent?	time, profilers, perf where allowed	time, Instruments, sampling profilers
How much memory is resident?	ps, /proc/PID/status	ps, Activity Monitor tools

Tool output and field units vary by operating system. Record the command, OS, tool version, and workload with the observation.

Diagnostic workflow

When a program is slow or stuck:

1. reproduce the symptom and record the environment;
2. find the process and thread state;
3. compare elapsed time with CPU time;
4. inspect open files, sockets, and waits;
5. check memory pressure and storage behavior;
6. trace or profile only the implicated layer;
7. change one cause and repeat.

Avoid starting with scheduler tuning, kernel parameters, or exotic I/O APIs. Most application problems can be explained from process state, resource use, system calls, and application-level evidence.