recycle-logs - (log) file recycler
recycle-logs [options...] [[--config] config-file1] [[--config] config-file2]...
A bain of Systems Administration work is managing log files, such as system
log messages (from syslogd), mail log messages (from
sendmail), printer spooling error logs, and so on.
Many operating systems provide some sort of script or scripts to assist
this task, such as logchecker, newsyslog on Solaris, or
skulker on AIX, however too often these are limited to one kind of log file, and
limited in the abilty to customize in the number of old files saved,
whether or not to compress the old file and so on.
recycle-logs attempts to overcome these limitations. File rotation and other
customization is based on control info specified in one or several
configuration files.
Print a usage message on standard error and exit with a return code of 100.
Print the version release on standard output and exit with a return code of 10.
Specify the program to use when the recycling method, lines-files, is ``lines.'' If you never recycle by lines, then this option is
meaningless.
Print debugging output. The higher the number, the more the output.
Don't actually do anything, but go through the motions as though we were
going to do something. Useful for checking what warnings would be produced,
and in conjunction with the --debug option.
Error output normally goes to standard error. Use this option if you want the output to by syslogged in addition.
Specify configuration file. This option can be specified many times to have multiple configuration files processed. If one is managing many computers which perform different roles, one may want to separate the common log-rotation parts into different files. For example, there might be a general part done on all computers, a part done only on databases, a part done only on Web servers, and so on.
The order that configuration files are processes is the order specified on the comand line; however files unprefaced with --config will appear after those prefaced with --config.
The format of a configuration file is a series file filenames enclosed in square bracket followed by a number of parameter names ``equal'' sign and value. That is:
# This is a comment line ; So is this. [filename1] parameter1 = value1 parameter2 = value2
[filename2] parameter1 = value3 parameter2 = value4
Comments start with # or ; and take effect to the end of the line.
This should be familiar to those who have worked with text-readible
Microsoft .INI files.
Note filenames, (filename1 and filename2 above) must be unique. However there are times when you may want to refer to the same file. One can use relative path specifiers, e.g. filename1 and ./filename1 which refer to the same file even though they appear to be different.
As quoted directly from the IniConf manual page:
Multiline or multivalued fields may also be defined ala UNIX ``here document'' syntax:
Parameter=<<EOT value/line 1 value/line 2 EOT
You may use any string you want in place of ``EOT''. Note that what follows the ``<<`` and what appears at the end of the text must match exactly, including any trailing whitespace.
All parameters below except count are optional. A description of what parameters and values are allowed follows.
This parameter can be used to further narrow whether rotation takes place.
The program is expected to be run from cron, but may happen that not every time it is run do you want to rotate some files.
One can specify time in seconds, minutes, hours, days, or weeks. The first letter in any case is used to specify each. One or more spaces however should appear after the digits and before the time specifier. Be careful `m', stands for minutes not months or milleniums, even though the program will happily accept these.
Examples:
age = 1 week age = 1 weeks age = 1 w age = 7 days age = 7 dia age = 07 d age = 168 hours age = 168 HOUR age = 168 h age = 10080 minutes age = 10080 minuto age = 10080 m age = 604800 seconds
All of the above mean the same thing -- one week.
This parameter, if given, specifies that the compression program named as the value is to berun after the file has been processed.
Example:
compress = /usr/local/gnu/bin/gzip -9
This parameter should be given if the compress parameter has been used for the file under consideration. If it is missing but the compression program is specifed, the program will try to infer what extension to use: ``.gz'' for gzip, ``.zip'' for ``zip'', and ``.Z'' for compress.
The extension parameter specifies a filename extension to be added after running the
compression program. It is meaningless if compression is not specified for this file. You should include the `.' as in .Z, or .gz or .zip.
Example:
extension = .gz
Normally, recycle-logs doesn't recycle an empty log file, If however you want to rotate the log even when the file is empty. The set this parameter to `true.'
Example:
ifempty = true
This parameter specifies the kind of rotation used. Currently the only two values allowd are `files' or `lines'. Value `files' means that the rotation is by a number of files. Value `lines' means the last n lines of the file are kept.
If recycling is done by files, the first file name recycle is the file name with a `.0' appended, the 2nd file has a `.1' appended and so on. Note that the file may have a further compression extension like `.gzip' too. See compress and extension below.
The number of lines or files to use is specified with the count parameter below.
Example:
lines-files = files
This specifies a value to be used in the lines-files parameter. If method is `lines' it is the number of lines to keep. If lines-files is `files', it is the number of files to keep.
Example:
count = 4
When rotating by files (lines-files equals ``files''), there are two ways that a file can be moved out of
production; the file can be moved or copied.
Copying a file generally is the safest approach, provided you have the additional disk space. As the name implies the existing log file is copied, and after this is done, the existing file is truncated. Many programs open a output file (standand output, standard error, or a log file) only once when the program is started and never reopen that file. For these programs, the copy method needs to be used or else after rotation the program will stop appending to the file.
A disadvantages of copying is that additional disk space needs to be used while the copying is going on. Also, it is possible to loose a couple of log records if these get written while the program is in copying the file.
The ``move'' method overcomes these two problem by moving the file instead
of copying it. However the program which recreates the log file may need to
be alerted to reopen its log file. Some programs like
syslogd will do this when you send a ``HUP'' signal to them. Other programs either
open the log file only before writing to it, closing the log file
afterwared. And some programs simple periodically reopen the log file.
If this parameter is not specified and recycling is by ``files'', the default value is ``move.''
Example:
method = copy
Normally if a log file specified in a configuration file is missing the program will complain. Set this parameter to false if you expect there to be situtations where the log file may not exist and you don't want to be bothered with a warning message.
Example:
missingok = true
This parameter, if given, specifies what program to run before recycling the file under consideration. It is similar to epilog described next (except it's done beforehand rather than afterwards).
This parameter, if given, specifies what program to run after recycling the file under consideration. It can be used for example to signal the program creating the file that a rotation has occured; perhaps the program will reopen its output files, or needs to restart itself or do some initial processing on the file.
Example:
epilog = /usr/bin/kill -HUP `cat /etc/syslog.pid`
The following is a sample config file:
# Save 4 files of /var/adm/messages which with file names # /var/adm/messages.0 to /var/adm/messages.3 # After moving the messages file, send # signal -HUP to syslogd. # Compress using /usr/ucb/compress and that adds the .Z extension # Replaces newsyslog script. [/var/adm/messages] count = 4 restart = /usr/bin/kill -HUP `cat /etc/syslog.pid` compress = /usr/ucb/compress -f extension = .Z
# Save the last 10000 lines of /var/adm/lpd-errs and # sends signal -HUP too. # [/var/adm/lpd-errs] count = 10000 # my syslog restart causes messages to stdout. We don't want these. # although we *do* want stderr messsages. epilog = /etc/init.d/syslogd.init restart >/dev/null method = lines
# Process accounting logs # Note how one can put insert multiple lines an action. [/var/adm/pacct] count = 7 epilog = <<EOT /usr/lib/acct/turnacct switch; /bin/rm -f /var/adm/pacct? EOT # the above is the same as # epilog = /usr/lib/acct/turnacct switch;/bin/rm -f /var/adm/pacct? compress = /usr/local/gnu/bin/gzip -9 extension = .gz
# Crontab activity log. Log file copied rather than moved, because # cron doesn't reopen its log file. # Replaces logchecker script. [/var/cron/log] count = 7 lines-files = files method = copy compress = /usr/local/gnu/bin/gzip -9 extension = .gz
# This shows how to rename using something other than xxx.0 xxx.1 # [/var/apache/logs/access] count = 1 lines-files = files method = copy epilog = <<EOF cd /var/log/apache; newfn=access-`date +%Y-%m-%d`.log && mv access.0 $newfn EOF
IniConf requires that each item named in a section (a log file here) be unique. If you want to do several things to a particular log file, you need to be clever how to specify the same file so it looks distinct, such as by using relative paths. (Symbolic links would work too.) The fact that muliple configuration files can be specified on the invocation complicates figuring out whether the log file names are unique. Alas, this program which relies on IniConf doesn't warn whether such as condition has occurred.
The Perl module IniConf(3) is used to parse configuration files. See also logrotate, skulker, newsyslog, logchecker, syslogd.
The current version is maintained (or not) by rocky@panix.com.
Copyright (C) 1997-2000 Rocky Bernstein, email: rocky@panix.com. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
$Id: recycle-logs.in,v 1.20 2000/02/11 09:37:28 rocky Exp $