Unit 1: Operating System Overview [4 Hours]

An Operating System (OS) is the foundational software layer that sits between computer hardware and the user. Without it, hardware is a collection of circuits with no way to run programs. This unit covers what an OS is, how it evolved, the different types available, its core functions, how programs communicate with it through system calls, how it organizes itself internally, and the role of the kernel and shell in making all of this work.

1.1 Introduction to Operating System

Definition

An Operating System (OS) is system software that manages computer hardware resources and provides services for application programs and users to interact with the machine.

The OS is the translator that stands between you and the hardware. When you click "Save" on a document, you never talk directly to the hard disk, the memory chip, or the processor. The OS intercepts that request, coordinates the hardware, and makes it happen. Without it, every program would have to contain its own code for managing memory, talking to disks, and handling keyboards, which would make software impossibly complex to build.

Detailed Explanation

A computer system can be divided into four layers:

• Hardware: the physical components, Central Processing Unit (CPU), memory, disks, input/output devices.
• Operating System: controls and coordinates use of hardware among various application programs and users.
• Application Programs: word processors, browsers, compilers, games — software that uses the OS to access hardware.
• Users: people or other machines interacting with the application layer.

The OS serves two primary roles simultaneously:

• Resource Manager: the OS allocates processor time, memory space, file storage, and input/output devices among competing processes efficiently and fairly.
• Extended Machine (Virtual Machine): the OS hides the complexity of hardware behind a clean, usable interface. From an application's perspective, it talks to the OS, not to raw circuits.

Goals of an Operating System

• Convenience: makes the computer easier to use by hiding low-level hardware details.
• Efficiency: ensures hardware resources are used optimally, reducing waste and idle time.
• Ability to evolve: allows new system functions to be developed and added without interfering with existing ones.

1.2 Evolution of Operating System

Definition

The evolution of operating systems describes how OS design has progressed through distinct generations, driven by changes in hardware technology, increasing system complexity, and the growing demands of users and applications.

Early computers had no OS at all. You fed instructions directly to the machine through physical switches or punched cards. Over decades, the OS evolved from a simple batch controller to a sophisticated multi-user, networked, real-time software platform. Each generation solved the problems the previous one created.

Detailed Explanation

Generation 1 (1945–1955): Vacuum Tubes, No OS

Computers used vacuum tubes and were operated directly by their designers. Programs were written in machine language or plugged in via physical wiring (plugboards). There was no operating system. A single user had full use of the machine. Jobs were run one at a time, and a significant amount of time was wasted setting up the next job manually.

Generation 2 (1955–1965): Transistors and Batch Systems

Transistors replaced vacuum tubes, making computers smaller and more reliable. Batch processing systems emerged to solve the idle-time problem. Jobs were grouped (batched) on punched cards, fed to the computer sequentially, and output collected as a batch. A simple resident monitor (the earliest form of an OS) read jobs from cards and transferred control to each program automatically. No user interaction was possible during execution.

Key limitation: while a job waited for Input/Output (I/O) operations, the CPU sat completely idle.

Generation 3 (1965–1980): Integrated Circuits and Multiprogramming

Integrated circuits (ICs) made hardware dramatically cheaper and faster. Two critical techniques emerged:

• Multiprogramming: multiple jobs are loaded into memory simultaneously. While one job waits for I/O, the CPU switches to another job. CPU utilization improves significantly.
• Time-sharing (Multitasking): each user gets a small slice of CPU time in rapid rotation. The system responds so quickly that each user feels they have a dedicated machine. UNIX, one of the most influential OS designs in history, was born in this era (1969–1970).

Generation 4 (1980–Present): Personal Computers and GUIs

Very Large Scale Integration (VLSI) circuits led to affordable personal computers (PCs). Operating systems moved from text-only command-line interfaces to Graphical User Interfaces (GUIs), pioneered by Xerox PARC and popularized by Apple Macintosh (1984) and later Microsoft Windows. Networks, distributed systems, and eventually mobile operating systems (Android, iOS) expanded the OS domain further.

1.3 Types of Operating System

Definition

Types of operating systems are the different categories into which OS designs are classified based on how they manage processes, handle user interaction, respond to time constraints, and support the number and nature of users they serve simultaneously.

Not every computer has the same job. A phone needs to respond to your touch instantly. A hospital's vital-signs monitor must respond within milliseconds or someone could die. A university server must handle hundreds of students at once. Different requirements have produced different OS types, each with design priorities that fit its environment.

Detailed Explanation

1. Batch Operating System

Jobs are collected, grouped into batches, and executed sequentially without user interaction during execution. The OS automatically moves from one job to the next. Used in payroll processing, bank statements, and other high-volume, repetitive tasks.

Advantages: efficient for large volumes of similar jobs; good CPU utilization for long-running tasks.

Disadvantages: no real-time feedback; if a job has an error, the user only discovers it after the full batch runs.

2. Time-Sharing (Multitasking) Operating System

Each user is given a small unit of CPU time called a time quantum or time slice. The CPU switches between users so rapidly that each user feels they have exclusive access. Example: early UNIX systems supporting multiple terminals.

Advantages: interactive response for multiple users simultaneously; efficient resource sharing.

Disadvantages: complex scheduling logic; security concerns when multiple users share the same machine.

3. Distributed Operating System

A Distributed Operating System (DOS) manages a group of independent computers networked together, making them appear as a single coherent system to the user. Tasks are split and processed across multiple machines. Examples: Google's infrastructure, cluster computing systems.

Advantages: resource sharing across machines; fault tolerance (if one node fails, others continue); scalability.

Disadvantages: complex to design; network failures can affect the entire system; security across nodes is harder.

4. Real-Time Operating System (RTOS)

A Real-Time Operating System (RTOS) guarantees that specific tasks are completed within strict, predefined time deadlines. Used in systems where delayed response can cause failure or harm.

• Hard RTOS: missing a deadline is considered a total system failure. Examples: aircraft flight control, anti-lock braking systems (ABS), cardiac pacemakers.
• Soft RTOS: missing an occasional deadline is acceptable but undesirable. Examples: video streaming, online gaming.

5. Multiprogramming Operating System

Multiple programs reside in memory simultaneously. When the running program waits for I/O, the OS switches the CPU to another program. The key goal is maximizing CPU utilization.

6. Multiprocessing Operating System

Uses two or more CPUs (processors) within a single computer system. Tasks are distributed across multiple processors for parallel execution, improving throughput and reliability. If one processor fails, others continue functioning.

7. Network Operating System (NOS)

A Network Operating System (NOS) runs on a server and provides services (file sharing, printer sharing, authentication) to client computers over a network. The OS on each computer remains independent; the NOS coordinates shared access. Examples: Novell NetWare, Windows Server.

1.4 Functions of Operating System

Definition

The functions of an operating system are the core responsibilities the OS carries out to manage hardware resources, coordinate running programs, protect data, and provide a usable environment for users and applications.

If the OS is the manager of a company, its functions are its job description. It decides who uses the CPU and when, allocates memory to each running program, controls access to files and devices, handles errors, and ensures the security of the whole system.

Detailed Explanation

1. Process Management

A process is a program in execution. The OS creates, schedules, suspends, resumes, and terminates processes. It handles:

• CPU Scheduling: deciding which process gets to run next and for how long.
• Process Synchronization: ensuring processes that share data don't corrupt each other's work.
• Deadlock Handling: detecting and resolving situations where processes are permanently blocked waiting for each other.

2. Memory Management

The OS tracks every byte of main memory (Random Access Memory, or RAM). It decides which process gets which memory locations, allocates memory when a process starts, and reclaims it when a process ends. Key tasks include protecting one process's memory from another and providing virtual memory when physical RAM is insufficient.

3. File System Management

The OS organizes data on storage devices into files and directories. It handles creating, reading, writing, and deleting files, enforcing access permissions, and managing disk space allocation. The file system abstraction means programs deal with logical file names rather than physical disk sectors.

4. Device Management (I/O Management)

The OS communicates with hardware devices (keyboard, mouse, printer, disk drives, network cards) through programs called device drivers. It provides a uniform interface so application programs don't need to contain device-specific code. The OS also manages device queues and handles interrupts generated by devices signaling they need attention.

5. Security and Protection

The OS protects system resources from unauthorized access. This includes user authentication (verifying identity), access control (determining what each user or process may do), and protection of one process's memory from interference by another. It also defends against malicious software.

6. User Interface (UI)

The OS provides a way for users to interact with the computer, either through a Command Line Interface (CLI) (the user types text commands) or a Graphical User Interface (GUI) (icons, windows, menus). The shell provides the CLI; the desktop environment provides the GUI.

7. Networking

Modern OS designs include network protocol support, enabling computers to communicate over local networks or the internet. The OS handles routing, socket management, and network device interfaces.

8. Error Detection and Handling

The OS continuously monitors the computer for errors in the CPU, memory, I/O devices, and software. When an error is detected, the OS takes appropriate action: notifying the user, terminating the offending process, or triggering a recovery mechanism.

1.5 System Call

Definition

A system call is a programmatic interface that allows a user-level application to request a service from the operating system's kernel, such as reading a file, creating a process, or allocating memory.

Every program runs in a restricted zone where it cannot directly touch hardware or critical system resources. When a program needs something beyond its own sandbox (like writing a file to disk), it makes a system call: a controlled, formal request handed up to the OS kernel, which performs the operation on the program's behalf and returns the result.

Detailed Explanation

Modern CPUs operate in (at least) two privilege modes:

• User mode: where normal application programs run. Direct hardware access is forbidden. Instructions that could damage the system are disabled.
• Kernel mode (Supervisor mode): where the OS kernel runs. All hardware instructions are permitted.

A system call is the boundary crossing from user mode to kernel mode. The sequence is:

Application calls system call function → CPU switches to kernel mode → Kernel validates the request → Kernel performs the operation → Result returned to application → CPU switches back to user mode

Categories of System Calls

1.6 Handling System Calls

Definition

Handling system calls refers to the mechanism by which the operating system receives, validates, executes, and returns results for requests made by user-space applications through the system call interface.

Making a system call is like pressing an emergency button. The system call handler is the trained crew that responds: it receives the signal, checks it is legitimate, performs exactly the requested action with full privileges, and then hands back the result before returning control to the program that pressed the button.

Detailed Explanation

Steps in Handling a System Call

1. User program invokes a system call: the program passes a system call number (identifying which service is requested) and any required parameters. Parameters are passed via registers, the stack, or a memory block whose address is given in a register.

2. Trap instruction is executed: a special CPU instruction called a trap (or software interrupt) is executed. This forces an immediate switch from user mode to kernel mode and jumps execution to a fixed address in the kernel called the trap handler.

3. System call dispatch: the kernel reads the system call number from the register and uses a system call table (a list mapping each number to the corresponding kernel routine) to dispatch to the correct handler function.

4. Kernel executes the service: the kernel handler validates parameters (checking for illegal values or memory addresses), verifies the process has permission to perform the operation, and then carries it out (reading a file, allocating memory, etc.).

5. Return to user mode: after execution, the return value (and any error code) is placed in a register. The kernel executes a return-from-trap instruction, restoring the saved state and switching the CPU back to user mode. Execution resumes in the user program at the instruction after the trap.

Role of the System Call Table

The system call table is an array in the kernel indexed by system call numbers. Each entry is a function pointer pointing to the kernel routine that handles that particular call. For example, on Linux, system call number 1 maps to sys_exit(), number 2 to sys_fork(), and so on.

Parameter Passing Methods

1.7 Operating System Structures

Definition

Operating System Structures describe the internal architectural designs used to organize the OS code, determining how the kernel is divided into components, how those components interact, and where OS services reside relative to the hardware and user programs.

Just as a building can be one open space or a carefully divided set of rooms, an OS can be a single block of code or a neatly layered and modular system. The structure chosen affects how easy the OS is to maintain, how fast it runs, and how secure it is.

Detailed Explanation

1. Monolithic Structure (Simple/Non-structured)

The entire OS runs as a single large program in kernel mode. All OS services (process management, memory management, file system, device drivers) exist in one address space and can call each other directly. Early UNIX and MS-DOS used this approach.

Advantages: fast (direct function calls with no communication overhead); simple to implement initially.

Disadvantages: difficult to maintain and debug; a bug anywhere can crash the entire system; no isolation between components.

2. Layered Structure

The OS is divided into a hierarchy of layers numbered from 0 (hardware) to N (user interface). Each layer uses services only from the layer directly below it and provides services only to the layer directly above it. Developed by Dijkstra's THE system (1968).

Advantages: easy to debug (one layer at a time); clean separation of concerns.

Disadvantages: performance overhead because a request must pass through multiple layers; strict layering is hard to enforce in practice.

3. Microkernel Structure

The kernel is reduced to the absolute minimum: only the most fundamental functions (Inter-Process Communication (IPC), basic scheduling, and minimal memory management) remain in kernel mode. Everything else (device drivers, file systems, networking) runs as separate processes in user mode called servers. Example: Mach microkernel, Minix.

Advantages: highly modular; a failing server process does not crash the kernel; easier to extend and port.

Disadvantages: performance overhead from frequent message-passing between user-mode servers and the kernel (context switches are expensive).

4. Modular Structure (Loadable Kernel Modules)

The kernel has a core set of components and can dynamically load additional modules at runtime. Modern Linux and Windows use this approach. Device drivers, file systems, and other extensions are loadable kernel modules (LKMs) that can be inserted and removed without rebooting.

Advantages: flexibility; system can be tailored without recompiling the kernel; cleaner than pure monolithic.

Disadvantages: less isolation than microkernel (modules run in kernel space); bugs in a module affect the whole kernel.

5. Hybrid Structure

Combines elements from multiple designs. Most modern operating systems use a hybrid approach: a large kernel (closer to monolithic) but with some microkernel principles and loadable modules. Examples: Windows NT (microkernel-influenced but not pure), macOS (Mach microkernel + BSD components + proprietary services).

1.8 Kernel and Its Types

Definition

The kernel is the core component of an operating system that resides permanently in memory and has complete control over all hardware. It is the bridge between hardware and software, directly managing the CPU, memory, and devices.

If the OS is a government, the kernel is the executive branch with full authority. Everything else (file managers, GUIs, device drivers in some designs) are agencies that operate under the kernel's authority and depend on it for access to the nation's (hardware's) resources.

Detailed Explanation

The kernel is always running while the computer is on. It loads into memory during startup and remains there. Its responsibilities include:

• CPU scheduling and process context switching
• Physical and virtual memory management
• Device driver management
• System call handling
• Interrupt handling

Types of Kernels

1. Monolithic Kernel

All OS services (process management, memory management, file systems, device drivers, system calls) are compiled into a single large executable that runs entirely in kernel mode. Communication between components is through direct function calls.

Examples: Traditional Linux kernel, early UNIX.

Advantage: high performance due to no IPC overhead. Disadvantage: large memory footprint; a bug in any component can bring down the entire kernel.

2. Microkernel

The kernel contains only the essentials: IPC, basic scheduling, minimal memory management. All other services (file system server, device driver server, networking server) run as separate processes in user space. They communicate with the kernel and each other via IPC (message passing).

Examples: Mach, Minix 3, QNX.

Advantage: stability (a crashing server does not crash the kernel), portability, security. Disadvantage: slower due to IPC overhead for every service request.

3. Hybrid Kernel

A microkernel design that allows certain non-essential services to run in kernel space for performance reasons. It is a pragmatic compromise: microkernel philosophy with monolithic performance optimizations.

Examples: Windows NT/10/11 kernel, macOS XNU (combining Mach microkernel and BSD components).

4. Exokernel

A research-oriented design where the kernel exposes hardware resources directly to applications with minimal abstraction. Applications manage their own resources. The goal is maximum performance and flexibility for specialized systems. The exokernel only enforces resource isolation and allocation.

Examples: MIT Exokernel (research project).

5. Nanokernel

An even more stripped-down kernel than the microkernel. It handles only interrupt handling and hardware abstraction at the most primitive level, delegating virtually everything else. Used in embedded systems where resource constraints are extreme. [verify: specific current production examples]

1.9 Shell

Definition

A shell is a user-space program that provides a text-based Command Line Interface (CLI) through which users can interact with the operating system by typing commands that the shell interprets and passes to the kernel for execution.

The shell is the outer layer of the OS that users actually touch. Just as a nut's shell protects and gives access to its core, the OS shell provides a controlled interface to the kernel's core. Type a command, the shell interprets it, assembles a system call or executes a program, and shows you the result.

Detailed Explanation

The shell sits between the user and the kernel. It reads text commands from the user, interprets them, and either executes a built-in command directly or launches an external program. The shell also supports:

• Scripting: sequences of commands written in a script file and executed automatically, enabling task automation.
• Piping (|): the output of one command is fed directly as the input to the next command.
• Redirection (>, <): sending output to a file or reading input from a file instead of the terminal.
• Variables and control flow: shells support variables, loops (for, while), and conditionals (if), making them full scripting environments.

Types of Shells

Shell vs. Kernel

The shell and the kernel are completely separate. The kernel cannot be replaced without reinstalling the OS. The shell is just a user-space program and can be changed without affecting the OS itself. A Linux system can run bash today and switch to zsh tomorrow with no impact on the kernel.

1.10 Open-Source Operating Systems

Definition

An open-source operating system is an OS whose source code is publicly available for anyone to inspect, modify, distribute, and use, typically under a license that preserves these freedoms.

In a closed-source OS like Windows, the source code is a trade secret. You can use the product but cannot see how it works. An open-source OS is a recipe shared with everyone: you can read it, improve it, adapt it for your needs, and share your version. Thousands of developers worldwide contribute to projects like Linux, leading to software that runs everything from personal laptops to supercomputers and smartphones.

Detailed Explanation

Key Open-Source Licenses

• GNU General Public License (GPL): requires that any modified version distributed publicly must also be open-source. Used by Linux kernel.
• MIT License: very permissive; allows use in proprietary software with minimal restrictions.
• BSD License: similar to MIT; also used for permissive distribution.

Major Open-Source Operating Systems

Advantages of Open-Source OS

• Transparency: anyone can inspect the code for security vulnerabilities or undesired behavior.
• Cost: generally free to use and distribute, reducing licensing costs.
• Community development: benefits from contributions of thousands of developers worldwide.
• Customizability: can be adapted for specific hardware or use cases without vendor permission.
• Longevity: not dependent on a single vendor's business decisions; if a company discontinues a product, the community can maintain it.

Disadvantages of Open-Source OS

• Support: community support can be inconsistent; enterprise support requires paid contracts.
• Compatibility: some proprietary hardware drivers and software are not available.
• Learning curve: advanced configuration typically requires command-line knowledge.

Exam Questions

1. What is an operating system? Explain its main functions.
2. Describe the evolution of operating systems through different generations.
3. What are the different types of operating systems? Explain with examples.
4. What is a system call? List the categories of system calls with examples.
5. Describe the steps involved in handling a system call.
6. Compare the monolithic structure with the microkernel structure of an operating system.
7. What is a kernel? Explain the different types of kernels.
8. What is a shell? How does it differ from the kernel? List types of shells.
9. What are open-source operating systems? List four examples with their uses.
10. Differentiate between multiprogramming and multiprocessing operating systems.
11. What is a real-time operating system? Distinguish between hard RTOS and soft RTOS.

Whole Chapter Summary

Operating System Introduction: The OS is the system software layer between hardware and users, acting as both a resource manager (allocating CPU, memory, and I/O) and an extended machine (hiding hardware complexity). Its goals are convenience, efficiency, and evolution.

Evolution: OS design progressed across four generations: no OS (vacuum tubes) → batch monitor (transistors, sequential job processing) → multiprogramming and time-sharing (ICs, UNIX) → GUI and personal computing (VLSI, modern OS). Each generation addressed idle-time and usability limitations of its predecessor.

Types of OS: Batch (sequential, no interaction), Time-sharing (CPU slices per user), Distributed (networked machines as one system), Real-Time (deadline guarantees: hard or soft), Multiprogramming (one CPU switching on I/O), Multiprocessing (multiple CPUs in parallel), Network OS (server-based resource sharing).

Functions: Process management (scheduling, synchronization, deadlock), Memory management (allocation, protection, virtual memory), File system (organized storage with permissions), Device management (drivers and I/O), Security, UI (CLI/GUI), Networking, and Error handling.

System Call: The formal interface for user programs to request kernel services. A trap instruction switches the CPU to kernel mode; the kernel dispatches via a system call table, executes the service, and returns the result. Six categories: process control, file management, device management, information maintenance, communication, and protection.

Handling System Calls: User invokes call → trap executes (switches to kernel mode) → system call table dispatches to handler → kernel validates and executes → return-from-trap restores user mode with result.

OS Structures: Monolithic (all in kernel, fast but fragile), Layered (hierarchy, debuggable but slow), Microkernel (minimal kernel + user-mode servers, stable but IPC-heavy), Modular (dynamic loadable modules, flexible), Hybrid (practical combination used by modern OS).

Kernel and Types: The kernel is the permanent, privileged OS core managing hardware. Types: Monolithic (all services in kernel space), Microkernel (minimal kernel + user-mode servers), Hybrid (best-of-both compromise), Exokernel (raw hardware to applications), Nanokernel (interrupt handling only, embedded).

Shell: A user-space CLI program (bash, zsh, PowerShell) that reads commands, interprets them, and communicates with the kernel. Supports scripting, piping, and redirection. Replaceable independently of the kernel.

Open-Source OS: OS with publicly available source code under licenses like GPL, MIT, or BSD. Linux (kernel) powers servers, supercomputers, and Android. Advantages: transparency, cost, customizability; disadvantages: variable support and occasional compatibility gaps.

Last-Minute Revision Sheet