[RC5-PROXYPER] Trimming log-files...

David Kelly dkelly at hiwaay.net
Fri Dec 5 23:13:39 EST 1997


Dirk Kuypers writes:

> > Now wait one second, just what log file are we talking about? Are you 
> > trying to rotate a file generated by redirecting the proxy's stdout to 
> > a file? Or are you attempting to rotate the proxy's logfile?
> 
> sorry, I should have clarified this: Of course I am talking about the 
> redirected STDOUT. The keylog-files are parsed under

Ah-ha! Now we're getting somewhere.

> http://www.comnets.rwth-aachen.de/~dk/rc5/rc5-stats.html
> 
> > The FreeBSD proxyper automatically rotates the log at 0000 GMT. A few 
> > minutes past I use cron to gzip that log and move to another directory.
> 
> So you mean the keylog-files where statistics are made about who finished 
> which block when under what OS and so on, don't you. I think I will have 
> to gzip them, too. But first I have to learn a bit more perl to read in a 
> gzipped file and split the data by emails;-)

I don't know enough perl to do that either, but basically you want to 
fork a system call to "zcat logfile.gz" and then simply read its stdout.

> > But what you seem to be describing is a case where you are attempting 
> > to rotate, pluck off the tail for saving, and delete the rest. If you 
> > attempt to do that to a file which a process has open, you will not 
> > release the disk space until the file is closed. Under FreeBSD, "fstat" 
> > will show you the processes that have a file open. SGI Irix uses 
> > "fuser", and "lsof" is a widely available utility to do the same.
> 
> Yes, I think normally I would have to send a HUP to the proxy before 
> doing so, but because I started it by using 'nohup rc5proxy' it is immune 
> now.

No. Its not because of the nohup, its because the proxy never closes/
open's its stdout. Let me suggest you need to write a logger to catch 
the proxy's stdout and invoke it something like this:

% rc5proxy |& logit &

Ok, I forgot exactly what the rc5 proxy name is this week. And my 
syntax is csh not sh or bash. Pipe stdout and stderr to logger.

Its probably easiest to write logger in /bin/sh. Or perl if that's your
thing. Logger simply copies its stdin to a file, meanwhile it has a
signal catcher to service HUP signals. Then send your HUP to logger and
let it close its current log file and open a new one.

#!/bin/sh
# beware! this code appears to work but hasn't been smoke tested.
# dmk 12/5/97

#echo $$ > ${HOME}/var/run/$0.PID
echo $$ > $0.PID

rotate() {
	# this may be too extreme for a timestamp
        DATE=`date +%y%m%d%H%M%S`
        LOGNAME=${DATE}.log
        echo "Log started at " $DATE >> ${LOGNAME}
}

# set up our trap for -HUP, signal 1
# execute our rotate proc on every HUP signal
trap rotate 1

# initialize our LOGNAME variable and file for the first time.
rotate

while read line
do
        echo ${line} >> $LOGNAME
done

Suggest you mkdir ~/var/run and have your logger write its PID in that 
directory to make it easier to kill -HUP. Then rather than have logger 
check the time, use cron to "kill -HUP `cat var/run/logit.PID`"

The above code didn't work worth a $#^@! when it was named "logger". 
Found out the hard way FreeBSD already had a "logger" and I was 
writting to my /var/log/messages via syslogd.

A simpler logit would be:

#!/bin/sh
while read line
do
        echo ${line} >> logfile
done

Notice it doesn't keep "logfile" open, but opens it to write each line 
then closes it while waiting for the next line. You could simply rotate 
this log by renaming it with cron. If cron renames it while its being 
written to, then it still works. The next time data is to be written a 
new file will be created.

> I tried it again now. Here is what I get before the trimming:
> 
> 13 [hera] dk> du -s .
> 62549   .
> 14 [hera] dk> df .
> Filesystem         1024-blocks  Used Available Capacity Mounted on
> dilbert:/u242H/home/dk
>                      1042600  605232   437368     58%   /home/dk
> 
> That's after the trimming:
> 
> 21 [hera] proxy> df .
> Filesystem         1024-blocks  Used Available Capacity Mounted on
> dilbert:/u242H/home/dk
>                      1042600  603560   439040     58%   /home/dk
> 24 [hera] dk> du -s
> 60899   .
> 
> So the file space seems to be freed, although ls -l gives
> 
> -rw-------   1 dk       AEG      15129397 Dec  5 09:58 nohup.out

When you trim the file the process writting it has its own pointer to 
where its going to write its next data. That's probably at 15129397 
right now. When you trimmed it, you freed space before that pointer and 
that shows up in the "df -k", while ls is showing the distance between 
the start of the file and the end of the file, which isn't the same 
thing as the size of the file. At least that's my guess.

--
David Kelly N4HHE, dkelly at nospam.hiwaay.net
=====================================================================
The human mind ordinarily operates at only ten percent of its
capacity -- the rest is overhead for the operating system.


--
To unsubcribe, send 'unsubscribe rc5-proxyper' to majordomo at llamas.net
rc5-proxyper-digest subscribers replace rc5-proxyper with rc5-proxyper-digest



More information about the proxyper mailing list