Unix - System Calls and User–Kernel Mode Transition in UNIX

 

System calls are the fundamental interface through which user-level programs interact with the operating system kernel. In UNIX-based systems, applications do not directly access hardware resources such as CPU, memory, or devices. Instead, they request services from the kernel using system calls. This design ensures security, stability, and controlled access to system resources.

User Mode vs Kernel Mode

A UNIX system operates in two primary modes:

User mode is where regular applications execute. In this mode, programs have limited privileges and cannot directly access critical system resources or hardware. This restriction prevents accidental or malicious damage to the system.

Kernel mode is where the operating system core executes. In this mode, the kernel has full access to hardware and system memory. It manages processes, memory, file systems, and device communication.

The transition between these two modes is essential for maintaining system security and functionality.

What is a System Call

A system call is a controlled entry point into the kernel. When a program needs to perform an operation such as reading a file, creating a process, or sending data over a network, it invokes a system call. The kernel then executes the requested operation on behalf of the program and returns the result.

Examples of common system calls include:

  • open – to open a file

  • read – to read data from a file or device

  • write – to write data

  • fork – to create a new process

  • exec – to execute a new program

  • wait – to wait for process termination

These system calls form the backbone of UNIX system programming.

How System Calls Work

The process of executing a system call involves several steps:

  1. A user program calls a library function such as read or write. These functions are usually part of standard libraries like the C standard library.

  2. The library function prepares the necessary parameters and places a specific system call number in a register. This number uniquely identifies the requested service.

  3. A special instruction, often called a trap or software interrupt, is executed. This instruction causes the CPU to switch from user mode to kernel mode.

  4. The kernel identifies the system call using the system call number and executes the corresponding kernel function.

  5. Once the operation is complete, the kernel places the result in a register and switches the CPU back to user mode.

  6. Control is returned to the user program, which continues execution.

This mechanism ensures that all sensitive operations are handled securely by the kernel.

System Call Interface

The system call interface acts as a bridge between user programs and the kernel. It defines how system calls are invoked, including:

  • System call numbers

  • Argument passing mechanisms

  • Return values and error handling

Most UNIX systems use a standardized interface, making it easier for developers to write portable programs.

Performance Considerations

System calls are more expensive than regular function calls because they involve a mode switch and additional overhead. Frequent system calls can impact performance, so efficient programs minimize unnecessary system calls by buffering data or combining operations.

Security and Protection

System calls enforce strict access control. Since only the kernel can execute privileged instructions, user programs cannot directly manipulate hardware or sensitive data. The kernel validates all parameters passed through system calls to prevent misuse or security vulnerabilities.

Importance in UNIX

System calls are essential because they:

  • Provide a safe way for programs to interact with hardware

  • Enable multitasking and process management

  • Support file and device operations

  • Maintain system security and stability

Without system calls, user programs would not be able to function in a controlled and secure environment.