Unix - Troubleshooting Scenarios

(logrotate & journalctl)

Scenario 1: Disk Space Full Due to Logs

Problem

Server disk becomes 100% full, system slows down or crashes.

Possible Causes

  • Log rotation not configured

  • Logrotate not running

  • Huge log files in /var/log/

Troubleshooting Steps

du -sh /var/log/*

Check largest logs:

ls -lh /var/log/syslog

Solution

  • Configure logrotate properly

  • Force rotation:

logrotate -f /etc/logrotate.conf

  • Remove old logs if necessary

Scenario 2: Logrotate Not Rotating Logs

Problem

Log files are growing but not rotating.

Possible Causes

  • Syntax error in config

  • logrotate not scheduled

  • Wrong file path

Troubleshooting

logrotate -d /etc/logrotate.conf

(dry run – shows errors)

Check cron:

ls /etc/cron.daily/logrotate

Solution

  • Fix config errors

  • Ensure logrotate is executable

  • Restart cron

Scenario 3: Service Stops Logging After Rotation

Problem

After rotation, new logs are not written.

Possible Cause

  • Service still writing to old (renamed) log file

Troubleshooting

lsof | grep syslog

Solution

Add postrotate script:

postrotate
    systemctl reload rsyslog
endscript

Scenario 4: journalctl Logs Missing After Reboot

Problem

Old logs disappear after reboot.

Cause

  • Journal stored in volatile memory

Troubleshooting

ls /var/log/journal

Solution

Enable persistent logging:

mkdir -p /var/log/journal
systemctl restart systemd-journald

Scenario 5: journalctl Uses Too Much Disk Space

Problem

Journal logs consuming too much disk.

Troubleshooting

journalctl --disk-usage

Solution (Immediate Cleanup)

journalctl --vacuum-size=200M
journalctl --vacuum-time=7d

Permanent fix:

SystemMaxUse=500M

in /etc/systemd/journald.conf

Scenario 6: Unable to Read Logs (Permission Denied)

Problem

Normal user cannot read logs.

Cause

  • Restricted permissions (security feature)

Troubleshooting

ls -l /var/log/auth.log

Solution

  • Use sudo

  • Add user to log group:

usermod -aG adm username

Scenario 7: journalctl Shows Too Much Output

Problem

Hard to find specific errors.

Solution (Filtering)

By service:

journalctl -u ssh

By priority:

journalctl -p err

By time:

journalctl --since "1 hour ago"

Scenario 8: Logs Not Written to /var/log but Visible in journalctl

Problem

Logs visible in journalctl but not in syslog files.

Cause

  • journald not forwarding logs to rsyslog

Troubleshooting

grep ForwardToSyslog /etc/systemd/journald.conf

Solution

ForwardToSyslog=yes

Restart:

systemctl restart systemd-journald rsyslog

Scenario 9: Logrotate Deletes Important Logs Too Quickly

Problem

Logs needed for auditing are missing.

Cause

  • Low rotate value

  • Aggressive rotation

Solution

Increase retention:

rotate 30
monthly
compress