Linux - Linux Core Dumps and Application Crash Analysis

A core dump is a file that contains information about the state of a running program at the moment it crashes unexpectedly. It can preserve details such as the process's memory, CPU register values, stack information, loaded libraries, and other diagnostic information. Developers and system administrators use core dumps to determine why an application terminated abnormally.

Core dumps are particularly useful for troubleshooting segmentation faults, memory-access violations, illegal instructions, application bugs, and unexpected process termination. Instead of trying to reproduce a problem repeatedly, an administrator can examine the saved state of the process after the crash.

1. What Is a Core Dump?

When a Linux application encounters a serious error, the operating system can terminate the process and optionally create a core dump.

For example, consider a program that attempts to access memory that it is not permitted to access:

Segmentation fault (core dumped)

The message indicates two things:

The process encountered a serious memory-related error.
Linux generated, or attempted to generate, a core dump.

A core dump is not simply a log file. It is a snapshot of the process at the time of failure. It can therefore provide information that ordinary application logs may not contain.

2. Why Core Dumps Are Important

Application logs usually contain information deliberately written by the application. A core dump can provide much deeper technical information about what the program was doing when it failed.

For example, suppose an application crashes while processing a customer request. The application log may show:

Processing customer request...
Request failed.

This does not necessarily reveal the underlying programming error.

A core dump examined with a debugger might reveal:

Program received signal SIGSEGV, Segmentation fault.

#0  process_request()
#1  handle_connection()
#2  main()

This provides a possible execution path leading to the crash.

Core dumps are therefore valuable for:

Finding the location of application crashes
Identifying segmentation faults
Investigating memory-access violations
Examining the call stack
Identifying problematic functions
Investigating crashes that cannot be reproduced easily
Troubleshooting production applications
Providing diagnostic information to software developers

3. Common Causes of Application Crashes

A Linux application can crash for many reasons. Some common causes include:

Segmentation Fault

A segmentation fault occurs when a program attempts to access memory incorrectly.

Common programming mistakes include:

Dereferencing a null pointer
Accessing memory that has already been released
Writing beyond an allocated memory region
Accessing invalid memory addresses

For example, a C program might contain:

int *ptr = NULL;
*ptr = 10;

The program attempts to write to an invalid memory location and may terminate with a segmentation fault.

Invalid Instructions

An application can also terminate if it attempts to execute an instruction that the processor cannot execute or that is invalid in the current context.

Abort

Applications sometimes deliberately terminate by calling functions such as abort() when they detect an unrecoverable internal condition.

Memory Corruption

Memory corruption can cause a program to behave unpredictably. The actual crash may happen much later than the original programming error, making a core dump particularly useful for investigation.

4. Linux Signals and Core Dumps

Core dumps are closely associated with Linux signals.

When certain serious signals terminate a process, Linux can generate a core dump if core dumping is enabled.

Examples include:

SIGSEGV
SIGABRT
SIGILL
SIGFPE
SIGBUS

SIGSEGV is commonly associated with segmentation faults.

SIGABRT is commonly generated when a program calls abort().

SIGILL indicates an illegal instruction.

SIGFPE is associated with arithmetic exceptions, although its exact causes can vary.

SIGBUS can occur because of certain invalid memory-access conditions.

Not every process termination produces a core dump. Core-dump generation depends on the signal, process configuration, system configuration, resource limits, and other factors.

5. Checking Whether Core Dumps Are Enabled

Linux uses resource limits to control whether a process can produce a core dump.

The ulimit command can be used to inspect the core-dump limit in a shell:

ulimit -c

Possible output might be:

A value of 0 generally means core dumping is disabled for processes launched from that shell.

Another possible output is:

unlimited

This means there is no shell-level size limit on the core dump.

You can temporarily enable core dumps in the current shell with:

ulimit -c unlimited

This setting affects processes launched from that shell and does not necessarily constitute a permanent system-wide configuration.

6. The Role of `core_pattern`

Linux determines how and where core dumps are handled through the kernel's core-pattern configuration.

You can inspect it with:

cat /proc/sys/kernel/core_pattern

The output can indicate a filename pattern or a program that receives the core-dump information.

For example, a system might have a configuration resembling:

core

In that case, a core file may be created using a name based on core.

On modern Linux systems using systemd, core dumps may instead be collected by systemd-coredump, meaning you may not find a traditional core file in the application's working directory.

7. `systemd-coredump`

Many modern Linux distributions use systemd-coredump to manage crash information.

Instead of simply leaving a large core file in the application's directory, the system can capture and manage the dump centrally.

The coredumpctl command can be used to inspect collected crash information.

For example:

coredumpctl list

This can display information about applications for which core dumps have been collected.

A typical entry may contain information such as:

TIME                         PID   UID   GID SIG COREFILE COMMAND
Mon 2026-09-07 10:30:15      2451  1000  1000 11  present  myapp

The exact output depends on the Linux distribution and system configuration.

8. Finding a Particular Crash

If a particular application has crashed, you can search the collected core dumps.

For example:

coredumpctl list myapp

This can help identify previous crashes involving myapp.

You can also obtain detailed information about a particular crash using its process ID:

coredumpctl info 2451

The output may contain:

Application name
Process ID
User ID
Signal responsible for termination
Timestamp
Executable path
Core-dump availability
Kernel information
Stack information
Related metadata

This information provides the first stage of crash investigation.

9. Examining a Core Dump with GDB

One of the most important tools for analyzing core dumps is GDB, the GNU Debugger.

If you have an executable called:

myapp

and a corresponding core file:

core

you can open both using:

gdb ./myapp core

GDB loads the executable and the saved process state.

Once inside GDB, the bt command can display the backtrace:

(gdb) bt

A backtrace shows the functions that were active when the application crashed.

For example:

#0  process_data()
#1  handle_request()
#2  server_loop()
#3  main()

This can help identify where the program was executing at the time of the crash.

10. Understanding a Backtrace

A backtrace represents the chain of function calls leading to the point where the program stopped.

Suppose an application follows this sequence:

main()
  |
  +-- server_loop()
        |
        +-- handle_request()
              |
              +-- process_data()

If process_data() crashes, the debugger may show:

#0 process_data()
#1 handle_request()
#2 server_loop()
#3 main()

Frame #0 is normally the current frame where the failure occurred.

The frames below it represent the functions that called it.

This makes the backtrace one of the most useful pieces of information in a core-dump investigation.

11. Debug Symbols

A core dump becomes significantly more useful when the application has been compiled with debugging symbols.

Without debug symbols, GDB might show something like:

#0  0x00007f... in ?? ()
#1  0x000055... in ?? ()

This provides limited information.

With appropriate debugging information, GDB may instead show:

#0 process_data() at parser.c:142
#1 handle_request() at server.c:87
#2 main() at server.c:35

Now the investigator can identify:

Function names
Source files
Source-code line numbers
Variables
Function arguments

This makes debugging considerably easier.

12. Useful GDB Commands

Several GDB commands are particularly useful when investigating a core dump.

To display the backtrace:

(gdb) bt

To display a more detailed backtrace:

(gdb) bt full

To select a particular stack frame:

(gdb) frame 0

To inspect local variables:

(gdb) info locals

To inspect function arguments:

(gdb) info args

To display source code around the current location:

(gdb) list

To inspect registers:

(gdb) info registers

To examine threads:

(gdb) info threads

These commands allow an administrator or developer to progressively investigate the state of the crashed application.

13. Analyzing Multithreaded Applications

Modern applications frequently use multiple threads. A crash in one thread does not necessarily mean that only that thread is relevant.

GDB can display the application's threads with:

(gdb) info threads

You may see something similar to:

Id   Target Id          Frame
1    Thread 0x...       main()
2    Thread 0x...       worker_thread()
3    Thread 0x...       network_thread()

You can switch to another thread with:

(gdb) thread 2

You can then examine its stack:

(gdb) bt

This is especially important when diagnosing race conditions, deadlocks, shared-memory problems, and multithreaded memory corruption.

14. Core Dumps and Shared Libraries

Applications often depend on shared libraries such as:

libc.so
libpthread
libssl

A crash may originate inside a shared library rather than directly inside the application's source code.

GDB can provide information about loaded shared libraries:

(gdb) info sharedlibrary

However, meaningful analysis may require the correct versions of the executable, libraries, and corresponding debug symbols.

A mismatch between the core dump and the binaries installed on the system can result in misleading or incomplete debugging information.

15. Using `coredumpctl` with GDB

On systems using systemd-coredump, you can often launch GDB against a collected crash using:

coredumpctl debug

For a particular process:

coredumpctl debug 2451

This can open the relevant executable and core information in GDB.

Once GDB starts, commands such as:

bt

and:

info threads

can be used to investigate the crash.

16. Practical Crash-Analysis Workflow

A systematic approach makes crash investigation easier.

First, identify the application that crashed.

coredumpctl list

Next, examine information about the crash:

coredumpctl info <PID>

Then open the crash in a debugger:

coredumpctl debug <PID>

Inside GDB, inspect the backtrace:

(gdb) bt

If the application is multithreaded, inspect all threads:

(gdb) info threads

Then investigate the relevant thread and frame:

(gdb) thread <thread-number>
(gdb) bt

If debugging symbols are available, inspect the source location and variables:

(gdb) list
(gdb) info locals
(gdb) info args

Finally, correlate the findings with application logs, recent deployments, configuration changes, library updates, and system events.

17. Core Dump Storage and Security

Core dumps can be very large because they may contain substantial portions of a process's memory.

They can also contain sensitive information, including:

Application data
User information
Authentication-related data held in memory
Database information
Encryption material
Internal configuration data

Therefore, core dumps should be treated as potentially sensitive files.

Administrators should consider:

Who can access core dumps
Where core dumps are stored
How long they are retained
Whether they should be transferred to another system
Whether storage capacity is sufficient
Whether sensitive applications should generate core dumps

18. Difference Between Logs and Core Dumps

Application logs and core dumps serve different purposes.

Application Logs	Core Dumps
Record events selected by the application	Capture process state at crash time
Usually text-based	Usually binary
Useful for understanding application activity	Useful for low-level crash analysis
Can explain what the application was doing	Can reveal execution state and memory information
Usually smaller	Can be very large
Generated intentionally by the application	Generated by the operating system/crash-handling system

In practice, the two should often be examined together.

19. Core Dumps Versus Crash Reports

A crash report may contain a summarized description of a failure, while a core dump can provide considerably more information for detailed debugging.

For example, a crash-reporting system might tell you:

Application crashed with SIGSEGV.

The core dump can potentially allow a developer to determine:

Which thread crashed
Which function was executing
Which source-code line was involved
What the call stack looked like
What certain variables contained
Which libraries were loaded

The amount of information available depends heavily on how the application and operating system were configured.

20. Important Considerations

A core dump does not automatically tell you the exact programming mistake. It provides evidence about the state of the process when it failed.

For example, a crash occurring inside a memory-management function might have been caused by memory corruption several seconds earlier. Therefore, the apparent crash location is not always the location where the original bug occurred.

For reliable analysis, developers may combine:

Core dumps
Debug symbols
GDB
Application logs
System logs
Source code
Recent software changes
Memory-analysis tools
Reproduction testing

Conclusion

Linux core dumps and application crash analysis provide a powerful method for investigating serious application failures. A core dump preserves important information about a process at the moment it crashes, while tools such as coredumpctl and GDB allow administrators and developers to examine that information.

The basic workflow is to identify the crash, inspect the available core-dump information, open the dump with a debugger, examine the backtrace and threads, inspect variables and registers when debugging symbols are available, and correlate the findings with application and system logs.

Understanding core dumps is particularly valuable for Linux administrators, developers, DevOps engineers, and system-support professionals because it transforms an unexplained application crash into a structured debugging investigation.

Linux - Linux Core Dumps and Application Crash Analysis

1. What Is a Core Dump?

2. Why Core Dumps Are Important

3. Common Causes of Application Crashes

Segmentation Fault

Invalid Instructions

Abort

Memory Corruption

4. Linux Signals and Core Dumps

5. Checking Whether Core Dumps Are Enabled

6. The Role of core_pattern

7. systemd-coredump

8. Finding a Particular Crash

9. Examining a Core Dump with GDB

10. Understanding a Backtrace

11. Debug Symbols

12. Useful GDB Commands

13. Analyzing Multithreaded Applications

14. Core Dumps and Shared Libraries

15. Using coredumpctl with GDB

16. Practical Crash-Analysis Workflow

17. Core Dump Storage and Security

18. Difference Between Logs and Core Dumps

19. Core Dumps Versus Crash Reports

20. Important Considerations

Conclusion

6. The Role of `core_pattern`

7. `systemd-coredump`

15. Using `coredumpctl` with GDB