[stats-dev] log loader

Chris Hodson nerf at slacker.com
Fri Apr 22 15:16:51 EDT 2005


On 22-Apr-2005, Jim C. Nasby wrote:
> On Fri, Apr 22, 2005 at 03:03:04AM -0500, Jeff Lawson wrote:
> > 
> > > A few words about the pre-processor; If we use a C 
> > > pre-processor, it's obviously faster, but at a cost of 
> > > portability. 
> > 
> > The performance difference might not be that bad if you ensure that the
> > limiting rate is the I/O of the database insertions, and not your other
> > pre-processing activities.  For example, by using threading to continue to
> > parse log lines while you're waiting for the database to do the bulk insert
> > statement that is executing in another thread.  Of course threading in Perl
> > is a rather rarely used feature and some consider it to still be a little
> > experimental.
> 
> Nerf was actually suggesting that we just insert via perl, and not do a
> bulk copy. IMO we should use a bulk copy, as it will be much faster.
> 

I was wondering about how much of a speed difference it would be vs how much porability/readability we would be giving up.

Except for the initial load, I'm thinking this is either going to be run daily or hourly.  Either way it doesn't add up to a lot so extra processing time even if it's half as fast.

Just throwing out ideas at this point.

-Nerf


More information about the stats-dev mailing list