Unix - UNIX Memory-Mapped Files and the mmap() System Call

1. Introduction

Memory-mapped files are an important feature of UNIX and UNIX-like operating systems that allow a file or a specific region of a file to be mapped directly into the virtual address space of a process. This means that a program can access the contents of a file through memory addresses instead of using traditional file input/output operations such as read() and write() for every data access.

The mmap() system call is used to establish this mapping between a file and a process's virtual memory. Once the mapping is created, the program can access the mapped region as though it were an area of memory. The operating system manages the relationship between the memory region and the underlying file, including loading file contents into physical memory when necessary.

Memory mapping is particularly useful when applications need to access large files, work with structured data, share memory between processes, or perform frequent random access to file contents. It can simplify programming and may improve performance by reducing the need for explicit data-copying operations.

2. What Is a Memory-Mapped File?

A memory-mapped file is a file whose contents are associated with a region of a process's virtual memory. Instead of repeatedly requesting individual portions of the file through file-reading system calls, the application accesses the mapped contents using pointers.

For example, consider a program that needs to examine a large text file containing millions of records. With traditional file operations, the program might repeatedly call read() to obtain blocks of data. With memory mapping, it can map the file into its address space and access different portions using memory addresses.

The operating system does not necessarily load the entire file into physical RAM immediately. It typically brings the required portions into memory as the program accesses them, using mechanisms such as demand paging. This allows applications to work with files that are larger than the available physical memory, provided the application accesses valid mapped regions and sufficient system resources are available.

Memory-mapped files do not mean that a file permanently occupies a dedicated area of physical memory. Instead, the operating system establishes a virtual memory mapping and manages the movement of data between storage and physical memory.

3. Understanding the mmap() System Call

The mmap() system call creates a mapping in the virtual address space of the calling process. On systems that provide the POSIX interface, it is commonly declared in the sys/mman.h header file.

A typical function declaration is:

C

#include <sys/mman.h>

void *mmap(void *addr, size_t length,
           int prot, int flags,
           int fd, off_t offset);

The function accepts several arguments that determine how the mapping is created.

addr: This argument specifies a preferred starting address for the mapping. In most applications, NULL is passed so that the operating system can choose a suitable address.

length: This specifies the number of bytes to map. It determines the size of the requested virtual memory region.

prot: This defines the permitted memory protections. Common options include PROT_READ for reading, PROT_WRITE for writing, and PROT_EXEC for executing instructions from the mapped region. PROT_NONE prevents ordinary access.

flags: This determines the type of mapping. Common choices include MAP_SHARED and MAP_PRIVATE.

fd: This is the file descriptor of the file to be mapped. It is generally obtained by opening the file with open().

offset: This specifies the starting position in the file from which the mapping begins. It must satisfy the platform's mapping-alignment requirements, typically alignment to a page boundary.

If the operation succeeds, mmap() returns the starting address of the mapped region. If it fails, it returns MAP_FAILED. The program should check the return value before attempting to access the mapping.

4. Types of Memory Mapping

Memory mapping is commonly divided into two main types: shared mapping and private mapping.

A. Shared Mapping Using MAP_SHARED

A shared mapping allows changes made to the mapped region to be visible to other processes mapping the same underlying file region in a compatible way. Changes to a writable shared file mapping can also be propagated to the underlying file.

For example, two processes may map the same data file using MAP_SHARED. If one process modifies a mapped value, another process that maps the same region can observe the change, subject to synchronization and memory-consistency requirements.

Shared mappings are useful for applications that need to share file-backed data or coordinate access to common memory regions. However, they do not automatically make concurrent updates safe. Applications may need synchronization mechanisms such as mutexes, semaphores, or other suitable coordination methods.

B. Private Mapping Using MAP_PRIVATE

A private mapping provides copy-on-write behavior. Initially, the process can read the mapped file contents, but modifications to the mapping are private to the process and are not intended to update the underlying file or become visible as file changes to other processes.

For example, a program may use MAP_PRIVATE to examine a configuration file and temporarily modify values in memory without changing the original file.

The operating system can share physical pages until a process modifies them. When a write occurs, it can create a private copy of the affected page. This is known as copy-on-write.

Private mappings are useful when applications need to read file contents and perform temporary in-memory modifications without writing those modifications back to the file.

5. How Memory Mapping Works

The memory-mapping process involves several steps.

1. Open the file: The program opens the required file using open() and obtains a file descriptor with appropriate access permissions.

2. Create the mapping: The program calls mmap() with the desired file region, length, protection settings, and mapping type.

3. Receive a virtual address: If the mapping succeeds, the operating system returns the starting address of the mapped memory region.

4. Access the mapped data: The program reads or writes the region using ordinary memory operations, subject to the selected protections and mapping type.

5. Handle page faults: When the program accesses a page that is not currently resident in physical memory, the operating system may generate a page fault and load the required file data. A page fault is a normal part of demand paging, although invalid accesses can cause errors.

6. Synchronize changes if necessary: For a shared writable mapping, modified data may be written back according to the operating system's caching and write-back policies. The msync() system call can be used to request synchronization of mapped file changes with the underlying file.

7. Release the mapping: The program calls munmap() when the mapped region is no longer needed. It should also close the file descriptor when appropriate.

This mechanism lets programs interact with file contents through virtual memory while the operating system manages the underlying storage operations.

6. Example of Using mmap() in C

The following example demonstrates how to map a file for reading and display its contents. The file must exist, and the example assumes it is a non-empty regular file.

C

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>

int main(void)
{
    int fd = open("sample.txt", O_RDONLY);

    if (fd == -1) {
        perror("open");
        return 1;
    }

    struct stat st;

    if (fstat(fd, &st) == -1) {
        perror("fstat");
        close(fd);
        return 1;
    }

    if (st.st_size == 0) {
        printf("The file is empty.\n");
        close(fd);
        return 0;
    }

    size_t length = (size_t)st.st_size;

    char *data = mmap(NULL, length, PROT_READ,
                      MAP_PRIVATE, fd, 0);

    if (data == MAP_FAILED) {
        perror("mmap");
        close(fd);
        return 1;
    }

    if (fwrite(data, 1, length, stdout) != length) {
        perror("fwrite");
    }

    if (munmap(data, length) == -1) {
        perror("munmap");
    }

    close(fd);

    return 0;
}

Explanation of the Program

First, the program opens sample.txt in read-only mode. It then uses fstat() to obtain information about the file, including its size. Checking the size is important because mapping a zero-length file with a zero mapping length is not valid.

Next, the program calls mmap() to map the file into its virtual address space. The PROT_READ protection allows the program to read the mapped contents, while MAP_PRIVATE creates a private mapping.

The returned pointer, data, refers to the beginning of the mapped region. The program passes that pointer to fwrite() to display the file's contents. Using fwrite() with the explicit length allows the program to handle binary data and files that do not end with a null character.

After accessing the data, the program releases the mapped region using munmap() and closes the file descriptor using close().

The program demonstrates file access through a memory mapping without repeatedly calling read() to retrieve file blocks.

7. Advantages of Memory-Mapped Files

Memory-mapped files offer several advantages for application development and system performance.

Simplified data access: Applications can use pointers and ordinary memory operations to access file contents, which can make certain programs easier to design.

Efficient random access: Programs that frequently access different positions within a large file can use memory addresses to reach those regions directly, without explicitly seeking and reading each portion.

Demand paging: The operating system can load file pages as needed instead of requiring the application to read the entire file into a separate buffer.

Potential reduction in copying: Memory mapping can reduce some data-copying overhead associated with conventional file I/O, although actual performance depends on the workload and operating system.

Memory sharing: Multiple processes can map the same file region using MAP_SHARED, enabling them to access common file-backed data.

Convenient access to large files: A program can map a file region and access portions of it without allocating an equally large application-managed buffer.

8. Limitations and Precautions

Although memory mapping is useful, it has some limitations that programmers must understand.

Address-space limitations: The process must have enough available virtual address space for the requested mapping. Large mappings may fail, particularly in constrained environments.

File modification risks: If another process truncates a file while a process has it mapped, accessing mapped pages beyond the new end of the file can cause a SIGBUS signal on many UNIX-like systems. Applications should coordinate file-size changes carefully.

Memory protection: Attempting to write to a read-only mapping or accessing a region without the required permissions can cause a protection fault.

Synchronization requirements: Shared mappings do not automatically prevent race conditions. Concurrent writers need appropriate synchronization when consistency matters.

Persistence considerations: Changes to a shared mapping may not be written to storage immediately. msync() can request synchronization, but it is not by itself a universal guarantee of crash-safe or transactional updates. Applications requiring durable storage may need additional measures, such as appropriate file synchronization.

Platform differences: Although mmap() is widely supported by UNIX-like systems, available flags, implementation details, and some behaviors differ across platforms.

9. Applications of Memory-Mapped Files

Memory mapping is used in a variety of computing applications.

• Database systems: Database software may use memory mapping to access file-backed pages or structured data, depending on its storage architecture.

• Large-file processing: Applications that analyze logs, scientific datasets, or large binary files can map relevant file regions into memory.

• Shared data between processes: Multiple processes can use a shared mapping to access common data, with suitable synchronization.

• Executable loading: Operating systems commonly use memory-mapping mechanisms to load executable files and shared libraries into a process's address space.

• Search and indexing applications: Programs that frequently access portions of large indexes may benefit from mapped file regions.

• Binary data processing: Applications that examine structured binary formats can use mapped memory to access fields and records directly.

The performance benefits depend on the access pattern, storage device, memory pressure, file size, and operating-system implementation. Memory mapping is not automatically faster than traditional file I/O for every workload.

10. Difference Between Traditional File I/O and Memory-Mapped Files

Conclusion

UNIX memory-mapped files provide a way to access file contents through a process's virtual address space using the mmap() system call. By mapping a file or a portion of it into memory, applications can use ordinary memory operations to read data and, when permitted, modify it. The operating system manages page loading and the relationship between the mapped region and the underlying file.

The two principal mapping types, MAP_SHARED and MAP_PRIVATE, support different requirements for shared updates and private modifications. Although memory mapping can simplify programming and improve performance for certain workloads, developers must account for memory protection, synchronization, file-size changes, and data-persistence requirements.

A strong understanding of mmap(), virtual memory, demand paging, and mapping protections helps UNIX programmers build efficient applications for large-file processing, shared data access, and other memory-intensive tasks.