Now that I’m managing Scrawlers’ server, I have the power to use a more appropriate Rails log rotation technique than I was allowed at Dreamhost. Specifically, I can use logrotate on my Ubuntu Slicehost slice.

In my setup, logrotate was already automatically running every night around 1:40AM. (Honestly, I’m not sure how, but I didn’t need to setup CRON.) My first attempt at configuring a new rotation was to handle all of my Rails apps with one configuration file. /etc/logrotate.d/rails_apps:

/var/www/apps/*/shared/log/*.log {
 daily
 missingok
 rotate 14
 compress
 delaycompress
 notifempty
 copytruncate
 postrotate
   /etc/init.d/mongrel_cluster restart
 endscript
}

The “daily” specification says rotate the logs daily. The “rotate 14” specification tells logrotate to keep 14 days of files. The “postrotate” specification is stopping and starting the mongrel cluster after each and every rotation.

You can execute a debug run of logrotate with:

logrotate -d /etc/logrotate.d/rails_apps

You can force an rotations with:

logrotate -f /etc/logrotate.d/rails_apps

Here is where the beautiful generality (see the ’*’ characters) hurts. The “postrotate” specification will execute for every log that is rotated in your Rails shared logs directory. In my case, this includes a production.log as well as a mongrel.log. And since I wildcarded what would be my base app directory, this would happen for every Rails app on the server. That means the entire Mongrel cluster is restarted multiple times. What you’ll see is while one restart is happening another kicks off and errors:

Restarting all mongrel_clusters...
Restarting all mongrel_clusters...
!!! PID file log/mongrel.8000.pid does not exist.  Not running?
mongrel::stop reported an error. Use mongrel_rails mongrel::stop -h to get help.

The fix is pretty simple. I needed to add a “sharedscripts” directive. This tells “postrotate” to only execute once for all logs referenced. So now we have:

/var/www/apps/*/shared/log/*.log {
 daily
 missingok
 rotate 14
 compress
 delaycompress
 notifempty
 copytruncate
 sharedscripts
 postrotate
   /etc/init.d/mongrel_cluster restart
 endscript
}

As with anything Linux, another option appears to be eliminating “postrotate” and “sharedscripts”, replacing them with the “lastaction” block directive, which also tells logrotate not to execute until all logs have been addressed. “lastaction” executes after “postrotate” and only if at least one log is rotated.

Of course, while I was at it, I also chose to start rotating my Apache and MySQL logs.

Apache:

/usr/local/apache2/logs/*_log {
 missingok
 rotate 10
 size=100k
 compress
 delaycompress
 notifempty
 copytruncate
}

In this case, I chose to rotate based on log size rather than daily. This should allow me to keep log data for longer in the Apache realm.

As for MySQL, apparently a logrotate definition was already setup on my server. I imagine this is from my deprec installation of Rails.

Aug 01, 2007 · Subscribe · Archive · Projects · Twitter · GitHub · Flickr