Log management is one of those essential sysadmin tasks that can make or break your server’s performance and disk space. Without proper log rotation, your system can quickly run out of storage, crash services, or become impossible to troubleshoot when you actually need those logs. Logrotate is Ubuntu’s built-in solution for automatically archiving, compressing, and cleaning up log files based on rules you define. This guide covers everything from basic setup to advanced configurations, common gotchas you’ll encounter, and how logrotate compares to other log management approaches.
How Logrotate Works Under the Hood
Logrotate operates through a combination of cron jobs and configuration files that define rotation policies. The main process runs daily via /etc/cron.daily/logrotate, checking all configured log files against their rotation criteria like size limits, age, or manual triggers.
The rotation process follows this sequence:
- Check if rotation criteria are met (size, time, or forced rotation)
- Execute pre-rotation scripts if defined
- Stop or signal the service writing to the log (optional)
- Move/rename the current log file
- Create a new empty log file with proper permissions
- Compress old log files if configured
- Remove old log files beyond the retention limit
- Execute post-rotation scripts
- Restart or signal the service to resume logging
Ubuntu 24 includes logrotate 3.21.0 by default, which introduced better handling of large files and improved error reporting compared to earlier versions.
Installation and Basic Setup
Logrotate comes pre-installed on Ubuntu 24, but let’s verify and check the setup:
# Check if logrotate is installed
dpkg -l | grep logrotate# Check version
logrotate --version# Install if missing (shouldn't be needed)
sudo apt update && sudo apt install logrotate
The default configuration structure includes:
/etc/logrotate.conf– Main configuration file/etc/logrotate.d/– Directory for service-specific configurations/var/lib/logrotate/status– Rotation status tracking/etc/cron.daily/logrotate– Daily cron job
Let’s examine the default global configuration:
# View main config
cat /etc/logrotate.conf# Check existing service configs
ls -la /etc/logrotate.d/# View rotation status
sudo cat /var/lib/logrotate/status
Creating Custom Log Rotation Configurations
The most common scenario is setting up rotation for application logs. Here’s a practical example for a web application:
# Create config for custom app logs
sudo nano /etc/logrotate.d/myapp# Basic configuration
/var/log/myapp/*.log {dailyrotate 30compressdelaycompressmissingoknotifemptycreate 0644 www-data www-datapostrotatesystemctl reload nginxendscript
}
For high-traffic applications that generate large logs quickly, you might need size-based rotation:
# Size-based rotation for busy applications
/var/log/myapp/access.log {size 100Mrotate 10compressdelaycompressmissingoknotifemptycreate 0644 www-data www-datacopytruncatepostrotate/usr/bin/systemctl reload myappendscript
}
Here’s a more complex example for database logs with custom compression and retention:
# Advanced MySQL log rotation
/var/log/mysql/*.log {dailyrotate 90compresscompresscmd /usr/bin/xzcompressext .xzcompressoptions -6delaycompressmissingoknotifemptycreate 0640 mysql admsharedscriptsprerotate/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logsendscriptpostrotate# Signal MySQL to reopen log filestest -x /usr/bin/mysqladmin && \/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf flush-logsendscript
}
Configuration Options and Directives
Understanding the available directives is crucial for effective log management. Here’s a comprehensive breakdown:
| Directive | Purpose | Example | Notes |
|---|---|---|---|
| daily/weekly/monthly | Rotation frequency | daily | Can also use specific times |
| size | Rotate when file reaches size | size 100M | Overrides time-based rotation |
| rotate | Number of old logs to keep | rotate 30 | 0 means delete immediately |
| compress | Compress rotated logs | compress | Uses gzip by default |
| copytruncate | Copy and truncate original | copytruncate | For apps that can’t reopen logs |
| create | Create new log file | create 0644 user group | Alternative to copytruncate |
| missingok | Don’t error if log missing | missingok | Useful for optional logs |
Real-World Use Cases and Examples
Let’s look at some practical scenarios you’ll encounter in production environments.
High-Volume Web Server Logs
# Nginx access logs for high-traffic site
/var/log/nginx/access.log {hourlyrotate 168 # Keep 1 week of hourly logssize 500Mcompressdelaycompressmissingoknotifemptycreate 0644 www-data admsharedscriptsprerotateif [ -d /etc/logrotate.d/httpd-prerotate ]; then \run-parts /etc/logrotate.d/httpd-prerotate; \fiendscriptpostrotateinvoke-rc.d nginx rotate >/dev/null 2>&1endscript
}
Application Logs with Centralized Logging
For applications that send logs to centralized systems like ELK stack, you might want faster local cleanup:
# Fast cleanup for centrally logged apps
/var/log/app/{error,access,debug}.log {dailyrotate 3 # Only keep 3 days locallysize 50Mcompressmissingoknotifemptycopytruncatepostrotate# Ensure log shipper picks up rotated filessystemctl restart filebeatendscript
}
System Service Logs
# Custom service with multiple log levels
/var/log/myservice/*.log {weeklyrotate 52compressdelaycompressmissingoknotifemptycreate 0640 myservice syslogpostrotate# Send SIGUSR1 to reopen logs/bin/kill -USR1 $(cat /run/myservice.pid 2>/dev/null) 2>/dev/null || trueendscript
}
Testing and Debugging Configurations
Before deploying rotation configs to production, always test them thoroughly:
# Test specific configuration without executing
sudo logrotate -d /etc/logrotate.d/myapp# Force rotation for testing (be careful in production)
sudo logrotate -f /etc/logrotate.d/myapp# Verbose output for troubleshooting
sudo logrotate -v /etc/logrotate.conf# Check what files would be affected
sudo logrotate -d /etc/logrotate.conf | grep "rotating pattern"
To test a configuration before the daily cron runs:
# Temporarily modify status file to force rotation
sudo cp /var/lib/logrotate/status /var/lib/logrotate/status.backup# Remove specific log entry to force rotation
sudo sed -i '/myapp/d' /var/lib/logrotate/status# Run logrotate manually
sudo logrotate -v /etc/logrotate.d/myapp# Restore status file if needed
sudo cp /var/lib/logrotate/status.backup /var/lib/logrotate/status
Common Issues and Troubleshooting
Here are the most frequent problems you’ll encounter and how to solve them:
Permission Issues
# Common error: permission denied
# Check log file ownership
ls -la /var/log/myapp/# Fix ownership if needed
sudo chown -R www-data:www-data /var/log/myapp/# Verify logrotate can read the directory
sudo -u www-data ls /var/log/myapp/
Service Not Reopening Log Files
This is probably the most common issue. Services need to be told to reopen their log files after rotation:
# Wrong approach - file handle remains open
/var/log/myapp/*.log {dailyrotate 7compress# Missing postrotate script
}# Correct approach - signal service to reopen
/var/log/myapp/*.log {dailyrotate 7compresspostrotatesystemctl reload myapp# Or send specific signal# kill -USR1 $(cat /run/myapp.pid)endscript
}
Logrotate Not Running
# Check if daily cron is working
sudo run-parts --test /etc/cron.daily# Check logrotate cron job
cat /etc/cron.daily/logrotate# Check system logs for errors
grep logrotate /var/log/syslog# Check logrotate status
sudo systemctl status cron
Configuration Syntax Errors
# Validate configuration syntax
sudo logrotate -d /etc/logrotate.conf# Common syntax issues:
# - Missing endscript after postrotate
# - Incorrect file permissions in create directive
# - Invalid size specifications (use M for MB, G for GB)
# - Duplicate directives in same config block
Performance Considerations and Optimization
Logrotate performance can impact system resources, especially with large files or many simultaneous rotations:
| Scenario | File Size | Rotation Time | CPU Usage | Optimization |
|---|---|---|---|---|
| Small logs (<10MB) | 1-10MB | <1 second | Low | Standard compression |
| Medium logs (10-100MB) | 10-100MB | 5-30 seconds | Medium | delaycompress |
| Large logs (100MB-1GB) | 100MB-1GB | 1-10 minutes | High | copytruncate + nice |
| Very large logs (>1GB) | >1GB | 10+ minutes | Very High | Size-based rotation |
For large files, consider using copytruncate to minimize service interruption:
# Optimized config for large files
/var/log/bigapp/massive.log {size 500M # Rotate before files get too largerotate 20compressdelaycompresscopytruncate # Avoids service restartmissingoknotifempty# Use nice to reduce CPU priorityprerotateecho "Starting rotation of large log file"endscriptpostrotate# Optional: signal external monitoringecho "Large log rotated" | wallendscript
}
Advanced Configuration Techniques
Custom Compression Settings
# Use different compression algorithms
/var/log/archive/*.log {dailyrotate 365compresscompresscmd /usr/bin/xzcompressext .xzcompressoptions -9 # Maximum compressionuncompresscmd /usr/bin/unxzdelaycompressmissingoknotifempty
}
Conditional Rotation Based on Time
# Rotate only during off-peak hours
/var/log/peak-app/*.log {dailyrotate 30compressmissingoknotifemptyprerotate# Only rotate between 2-4 AMhour=$(date +%H)if [ $hour -lt 2 ] || [ $hour -gt 4 ]; thenexit 1fiendscriptpostrotatesystemctl reload peak-appendscript
}
Log Archival to Remote Storage
# Archive old logs to S3 or remote storage
/var/log/important/*.log {dailyrotate 7 # Keep only 1 week locallycompressdelaycompressmissingoknotifemptycreate 0644 app applastaction# Archive logs older than 7 days to remote storagefind /var/log/important -name "*.log.*.gz" -mtime +7 -exec \aws s3 cp {} s3://my-log-bucket/$(hostname)/ \; -deleteendscript
}
Comparison with Alternative Log Management Solutions
| Solution | Best For | Pros | Cons | Resource Usage |
|---|---|---|---|---|
| logrotate | System logs, simple rotation | Built-in, reliable, lightweight | Limited features, cron-dependent | Very Low |
| rsyslog | Centralized logging, filtering | Real-time, powerful filtering | Complex configuration | Medium |
| journald | systemd services | Integrated with systemd, binary format | Not universal, requires systemd | Medium |
| Fluentd/Fluent Bit | Container environments, streaming | Real-time, many outputs | Resource intensive, complex | High |
Best Practices and Security Considerations
Follow these practices to maintain secure and efficient log rotation:
- Always use the
createdirective with appropriate permissions rather than relying on defaults - Set up log retention policies that comply with your organization’s requirements
- Use
delaycompressto allow log analysis tools to finish processing files - Monitor logrotate execution through system logs
- Test configurations in staging environments before production deployment
- Consider disk space carefully when setting rotation policies
- Use
sharedscriptswhen multiple log files share the same service - Implement proper error handling in pre/post rotation scripts
Security Configuration Example
# Secure log rotation with proper permissions
/var/log/secure/*.log {weeklyrotate 52compressdelaycompressmissingoknotifemptycreate 0600 root root # Restrict access to sensitive logssharedscriptsprerotate# Ensure only root can read rotated security logsumask 077endscriptpostrotate# Log rotation event for audit purposeslogger "Security logs rotated by logrotate"systemctl reload rsyslogendscript
}
Monitoring and Maintenance
Set up monitoring to ensure logrotate continues working properly:
# Create monitoring script
sudo nano /usr/local/bin/check-logrotate.sh#!/bin/bash
# Check if logrotate ran successfully today
last_run=$(stat -c %Y /var/lib/logrotate/status)
current_time=$(date +%s)
hours_since=$((($current_time - $last_run) / 3600))if [ $hours_since -gt 25 ]; thenecho "WARNING: logrotate hasn't run in $hours_since hours"exit 1
fi# Check for errors in syslog
if grep -q "logrotate.*error" /var/log/syslog; thenecho "ERROR: logrotate errors found in syslog"exit 1
fiecho "logrotate is working properly"
exit 0
For environments with VPS hosting or dedicated servers, consider setting up centralized log monitoring to track rotation status across multiple systems.
Regular maintenance tasks include:
- Reviewing disk usage trends to adjust retention policies
- Checking for failed rotations in system logs
- Updating configurations when adding new services
- Testing rotation configurations after system upgrades
- Monitoring compression ratios to optimize storage
With proper logrotate configuration, you can automate log management effectively while maintaining system performance and ensuring you have the log data you need for troubleshooting and compliance requirements.