[stats-dev] log loader
nerf at slacker.com
Fri Apr 22 15:16:51 EDT 2005
On 22-Apr-2005, Jim C. Nasby wrote:
> On Fri, Apr 22, 2005 at 03:03:04AM -0500, Jeff Lawson wrote:
> > > A few words about the pre-processor; If we use a C
> > > pre-processor, it's obviously faster, but at a cost of
> > > portability.
> > The performance difference might not be that bad if you ensure that the
> > limiting rate is the I/O of the database insertions, and not your other
> > pre-processing activities. For example, by using threading to continue to
> > parse log lines while you're waiting for the database to do the bulk insert
> > statement that is executing in another thread. Of course threading in Perl
> > is a rather rarely used feature and some consider it to still be a little
> > experimental.
> Nerf was actually suggesting that we just insert via perl, and not do a
> bulk copy. IMO we should use a bulk copy, as it will be much faster.
I was wondering about how much of a speed difference it would be vs how much porability/readability we would be giving up.
Except for the initial load, I'm thinking this is either going to be run daily or hourly. Either way it doesn't add up to a lot so extra processing time even if it's half as fast.
Just throwing out ideas at this point.
More information about the stats-dev