Unix - UNIX Log Management and Log Rotation

UNIX systems generate a large amount of information about their activities. This information is stored in log files, which serve as records of system events, user actions, software operations, security incidents, and hardware performance. Log management is the process of collecting, storing, monitoring, analyzing, and maintaining these log files. Since log files can grow rapidly over time, log rotation is used to archive, compress, and remove old logs to prevent excessive disk space consumption.

Understanding Log Files in UNIX

A log file is a text file that contains records of events occurring within the operating system and its applications. Every significant action performed by the system is often recorded in a log file for future reference.

Log files help administrators:

  • Monitor system health

  • Detect errors and failures

  • Troubleshoot problems

  • Track user activities

  • Analyze system performance

  • Investigate security incidents

  • Maintain compliance with organizational policies

Most UNIX log files are stored in the /var/log directory.

Common log files include:

Log File Purpose
/var/log/messages General system messages
/var/log/syslog System-wide activity logs
/var/log/auth.log Authentication and login records
/var/log/secure Security-related events
/var/log/cron Scheduled task information
/var/log/maillog Mail server activities
/var/log/boot.log System startup information
/var/log/kern.log Kernel messages

The exact log file names may vary depending on the UNIX distribution.

Importance of Log Management

Proper log management is essential because log files provide valuable information about system behavior.

System Monitoring

Administrators can monitor system activities and identify unusual events before they become serious problems.

Troubleshooting

When an application crashes or a service stops functioning, logs provide detailed error messages that help identify the root cause.

Security Analysis

Logs record login attempts, authentication failures, privilege escalations, and suspicious activities. Security teams use this information to detect potential attacks.

Performance Evaluation

System logs reveal information about resource usage, hardware issues, and application performance trends.

Regulatory Compliance

Many organizations are required to maintain logs for auditing and compliance purposes.

Syslog Architecture

Traditional UNIX systems use a logging mechanism called Syslog.

The Syslog system consists of three main components:

Log Generator

Applications, services, and system processes generate log messages.

Examples include:

  • Kernel

  • Web servers

  • Mail servers

  • Database servers

  • User applications

Syslog Daemon

The Syslog daemon receives messages and determines where they should be stored.

Common daemons include:

  • syslogd

  • rsyslogd

  • syslog-ng

Log Files

Messages are written into appropriate log files according to predefined rules.

The daemon decides whether messages should be:

  • Stored locally

  • Sent to remote servers

  • Displayed on user terminals

  • Forwarded to monitoring systems

Log Levels

UNIX logging systems classify messages according to severity levels.

Emergency

System is unusable and requires immediate attention.

Alert

Immediate corrective action is needed.

Critical

Critical conditions affecting system operation.

Error

Errors that prevent proper functioning of applications or services.

Warning

Potential issues that should be monitored.

Notice

Normal but important events.

Informational

General informational messages.

Debug

Detailed diagnostic information used during troubleshooting.

These levels help administrators prioritize events and focus on critical problems.

Viewing Log Files

UNIX provides several commands for reading and analyzing log files.

cat

Displays the entire content of a log file.

cat /var/log/syslog

less

Allows viewing large files page by page.

less /var/log/syslog

tail

Displays the last few lines of a log file.

tail /var/log/syslog

Real-Time Monitoring

Administrators often monitor logs as new entries are added.

tail -f /var/log/syslog

This command continuously displays new log entries.

grep

Searches for specific keywords within log files.

grep "error" /var/log/syslog

This command helps identify error-related messages quickly.

Challenges in Log Management

As systems operate continuously, log files grow larger.

Common challenges include:

Disk Space Consumption

Large logs can consume significant storage space.

Performance Impact

Reading and processing massive log files can reduce system efficiency.

Data Retention

Organizations need to decide how long logs should be retained.

Security Risks

Sensitive information stored in logs may be exposed if access controls are weak.

Log Overload

Thousands of messages may be generated every minute, making analysis difficult.

Concept of Log Rotation

Log rotation is the process of managing growing log files by:

  • Renaming current logs

  • Creating new log files

  • Compressing old logs

  • Archiving historical data

  • Deleting outdated logs

Without log rotation, logs can eventually fill available disk space and affect system performance.

How Log Rotation Works

Suppose a system maintains a file called:

/var/log/syslog

When the file reaches a specified size or age:

  1. The current log file is renamed.

syslog.1

  1. A new empty log file is created.

syslog

  1. Older logs may be compressed.

syslog.1.gz

  1. Very old logs are removed automatically.

This process continues according to the defined retention policy.

Logrotate Utility

Most UNIX systems use the Logrotate utility to automate log rotation.

Logrotate can:

  • Rotate logs daily, weekly, or monthly

  • Compress archived logs

  • Delete old logs

  • Create new log files

  • Send rotated logs via email

  • Execute scripts after rotation

The main configuration file is often located at:

/etc/logrotate.conf

Additional configuration files may be stored in:

/etc/logrotate.d/

Common Logrotate Options

Rotate Count

Specifies how many old log files should be kept.

rotate 5

The system keeps five archived versions.

Compression

Compresses archived logs.

compress

This saves disk space.

Rotation Frequency

Rotate logs weekly.

weekly

Other options include:

daily
monthly
yearly

Missing Log Files

Ignore missing log files.

missingok

Create New Log

Create a new empty log after rotation.

create

Example Logrotate Configuration

/var/log/myapp.log {
    weekly
    rotate 4
    compress
    missingok
    create
}

This configuration:

  • Rotates logs every week

  • Keeps four archived versions

  • Compresses old logs

  • Ignores missing files

  • Creates a new log file after rotation

Centralized Log Management

In large organizations, logs from multiple servers are collected in a central location.

Benefits include:

  • Easier monitoring

  • Simplified troubleshooting

  • Improved security analysis

  • Long-term storage

  • Unified reporting

Centralized logging systems collect logs from various machines and store them in a dedicated server for analysis.

Security Considerations

Log files often contain sensitive information.

Administrators should:

  • Restrict log file permissions

  • Encrypt transmitted logs

  • Monitor unauthorized access

  • Protect archived logs

  • Regularly review security events

Failure to secure logs can expose passwords, usernames, IP addresses, and system details.

Best Practices for Log Management

  1. Enable logging for critical services.

  2. Review logs regularly.

  3. Implement automated log rotation.

  4. Compress archived logs.

  5. Store backups of important logs.

  6. Use centralized logging for large environments.

  7. Restrict access to log files.

  8. Monitor unusual events and failed login attempts.

  9. Define clear retention policies.

  10. Periodically verify log integrity.

Conclusion

UNIX Log Management is a critical administrative function that involves collecting, monitoring, analyzing, and maintaining system and application logs. Log files provide valuable insights into system performance, operational issues, and security events. As logs continuously grow, log rotation becomes necessary to control disk usage and maintain system efficiency. Tools such as Logrotate automate the rotation, compression, archiving, and deletion of logs, ensuring that systems remain organized, secure, and easy to manage. Effective log management enhances troubleshooting capabilities, strengthens security monitoring, and supports reliable system administration.