Unix - Core Dumps and Crash Analysis in UNIX

A core dump is a file generated by a UNIX operating system when a running program terminates abnormally, usually because of a serious error such as a segmentation fault, illegal instruction, or certain other fatal signals. The core file contains information about the state of the process at the moment it crashed. Developers and system administrators can examine this information to determine what caused the failure. Core dumps are therefore an important debugging and troubleshooting mechanism in UNIX systems.

1. What Is a Core Dump?

When a program crashes, the operating system can save a snapshot of the program's memory and execution state into a file called a core dump or core file.

For example, consider a program that attempts to access memory that it is not allowed to access:

int *ptr = NULL;
*ptr = 10;

The program may receive a SIGSEGV signal and terminate. If core dumping is enabled, UNIX can create a core file containing information about the failed process.

A core file can contain information such as:

  • Process memory

  • CPU register values

  • Stack contents

  • Program counter

  • Signal that caused the crash

  • Information about loaded libraries

  • Thread information

  • Process state at the time of failure

This information allows developers to investigate the problem after the program has stopped running.

2. Common Causes of Core Dumps

Core dumps are generally associated with abnormal program termination. Common causes include:

Segmentation Fault

A segmentation fault occurs when a program accesses memory incorrectly.

For example:

int *p = NULL;
*p = 100;

Here, the program attempts to write data through a null pointer. The operating system normally terminates the process.

Illegal Instructions

A program may attempt to execute an instruction that is invalid or unsupported by the processor. This can result in an illegal-instruction signal and potentially generate a core dump.

Bus Errors

A bus error can occur when a program performs an invalid memory access that cannot be handled correctly by the hardware or operating system.

Abort Signals

A program can explicitly terminate itself using the abort() function. This generates SIGABRT and may produce a core dump.

Stack Overflow

Excessive recursion or very large stack allocations can exhaust the available stack space and cause a program to terminate abnormally.

3. Core Dump Configuration

UNIX systems usually provide a mechanism to control whether processes are allowed to generate core files.

The ulimit command can be used in many UNIX shells to inspect the core-file size limit.

ulimit -c

A result of:

0

generally means that core dumping is disabled for the current shell environment.

A non-zero value indicates that a core dump can be generated up to the specified limit, depending on the operating system and configuration.

For example:

ulimit -c unlimited

allows core files to grow without a shell-imposed size limit.

The exact behavior and system-wide configuration depend on the UNIX variant being used.

4. Generating a Core Dump

Suppose a program called sample crashes because of a programming error.

After enabling core dumps, the operating system may generate a file with a name such as:

core

or:

core.12345

where the number can identify the process.

Modern Linux systems can also be configured to store core dumps in locations managed by systemd-coredump rather than placing a traditional core file directly in the program's working directory.

Therefore, the location and naming convention of core files vary between UNIX-like operating systems.

5. Using GDB to Analyze a Core Dump

One of the most commonly used debugging tools for UNIX and UNIX-like systems is GNU Debugger (GDB).

Suppose the executable is:

myprogram

and the core file is:

core

The following command can open the executable together with its core dump:

gdb ./myprogram core

GDB loads the program and information contained in the core file so that the developer can investigate the state of the process when it crashed.

A typical GDB session may show information similar to:

Program terminated with signal SIGSEGV, Segmentation fault.

This immediately identifies the signal responsible for the termination.

6. Examining the Backtrace

One of the most useful commands in GDB is:

backtrace

or its shorter form:

bt

A backtrace displays the sequence of function calls that led to the crash.

For example:

#0  process_data()
#1  calculate_value()
#2  main()

This indicates that main() called calculate_value(), which eventually called process_data(), where the failure occurred.

The backtrace is particularly useful for locating the area of the application that needs investigation.

7. Examining Variables

After identifying the function where the crash occurred, developers can inspect variables.

For example:

print variable

or:

p variable

If the program contains a pointer called ptr, the developer can use:

p ptr

to inspect its value.

This can help identify problems such as:

  • Null pointers

  • Unexpected values

  • Invalid addresses

  • Incorrect function arguments

  • Corrupted data

8. Examining the Current Stack Frame

GDB provides the frame command for examining a particular stack frame.

For example:

frame 0

selects the frame where the program was stopped.

The command:

info locals

can then display local variables available in that frame.

This is useful when trying to understand the state of the function at the exact moment of failure.

9. Importance of Debugging Symbols

Core dumps become significantly more useful when the program contains debugging symbols.

A program compiled with debugging information might be built using:

gcc -g program.c -o program

The -g option adds debugging information that allows tools such as GDB to associate machine instructions with source files, functions, and line numbers.

Without debugging symbols, GDB may only provide information such as memory addresses and function names, making the investigation more difficult.

For production applications, debugging symbols are often stored separately from the executable so that production binaries remain optimized while developers can still perform detailed crash analysis.

10. Core Dumps and Multithreaded Programs

Modern UNIX applications frequently use multiple threads. When a multithreaded application crashes, examining only the crashing thread may not be sufficient.

GDB can display information about all threads.

For example:

info threads

shows the threads known to the debugger.

Developers can then switch between threads and examine their individual stack traces.

A useful debugging approach is:

thread apply all bt

This requests a backtrace for all threads.

This can reveal situations where one thread caused the crash while other threads were waiting, processing data, or holding important locks.

11. Core Dumps and Shared Libraries

UNIX programs commonly depend on shared libraries. A crashing application may therefore involve code from libraries such as the C standard library or other external components.

During core analysis, GDB needs access to the appropriate executable and shared-library information to correctly interpret addresses and stack frames.

If the executable, libraries, or debugging symbols do not match the versions used when the crash occurred, the analysis may become inaccurate.

Therefore, developers should preserve:

  • The exact application binary

  • Matching debugging symbols

  • Relevant shared libraries

  • Build information

  • The core dump

for reliable post-crash investigation.

12. Security Considerations

Core dumps can contain sensitive information because they may capture large portions of a process's memory.

For example, a core file might contain:

  • Passwords temporarily stored in memory

  • Authentication tokens

  • API keys

  • Personal information

  • Database credentials

  • Application data

Consequently, unrestricted core dumping can create a security risk.

Production systems often restrict core dumps or configure controlled storage and access policies. Administrators should ensure that core files are protected with appropriate filesystem permissions and are removed when they are no longer required.

13. Core Dump Analysis Workflow

A typical crash-analysis process can follow these steps:

Step 1: Identify the Crash

Determine which application terminated unexpectedly and when the failure occurred.

Step 2: Locate the Core File

Find the core dump generated by the operating system or its crash-management service.

Step 3: Identify the Correct Binary

Use the exact executable version that generated the core dump.

Step 4: Load the Core Dump

Open the executable and core file with GDB:

gdb ./myprogram core

Step 5: Identify the Signal

Use:

info program

or examine the information displayed when GDB loads the core file.

Step 6: Generate a Backtrace

Run:

bt

to determine the function-call sequence.

Step 7: Inspect Variables

Use commands such as:

info locals
p variable

to investigate relevant data.

Step 8: Examine Threads

For multithreaded programs:

info threads

and:

thread apply all bt

can provide a broader picture.

Step 9: Identify the Root Cause

Use the collected information to determine whether the failure resulted from an invalid pointer, memory corruption, stack exhaustion, library problem, programming error, or another issue.

Step 10: Fix and Reproduce

After identifying the likely cause, developers modify the application, rebuild it, and attempt to reproduce the failure under controlled conditions.

14. Core Dump vs Log File

A core dump and a log file serve different purposes.

A log file generally contains messages intentionally written by an application or system service. Logs are useful for understanding what happened before and around an event.

A core dump, on the other hand, represents the state of a process when it terminated abnormally. It provides much deeper information about memory, threads, registers, and execution state.

For effective troubleshooting, developers often use both.

For example, logs may indicate:

Database connection failed

while a core dump may reveal that the application subsequently dereferenced an invalid pointer while handling the failure.

15. Advantages of Core Dump Analysis

Core dumps provide several important benefits.

Post-crash investigation: Developers can investigate a failure after the application has already terminated.

Detailed process information: The dump can contain memory, registers, stack information, and thread state.

Difficult-to-reproduce problems: Some crashes happen only in production or under unusual conditions. A core dump provides evidence from the actual failure.

Root-cause identification: Debuggers can help identify the function and source-code location associated with the crash.

Improved software reliability: Information obtained from crash analysis can be used to correct programming errors and prevent future failures.

16. Limitations of Core Dumps

Core dumps are not always sufficient by themselves.

They can be very large, particularly for applications with significant memory usage. They may also contain confidential information, creating security and storage concerns.

Optimized production programs can also make debugging more difficult because compiler optimizations may remove variables or rearrange instructions.

Another challenge occurs when the wrong executable or incompatible libraries are used during analysis. In such cases, the information presented by the debugger may be incomplete or misleading.

Conclusion

Core dumps and crash analysis are important UNIX debugging techniques for investigating abnormal application termination. A core dump preserves valuable information about a process at the moment it crashes, allowing developers to examine memory, registers, threads, stack frames, and program state after the failure.

Tools such as GDB make it possible to analyze this information through commands such as bt, info threads, info locals, and print. When combined with debugging symbols, application logs, and the correct executable and libraries, core-dump analysis can significantly reduce the time required to identify and fix difficult software failures.

For UNIX administrators and developers, understanding core dumps is especially valuable when diagnosing segmentation faults, memory-related failures, unexpected application termination, and production crashes.