[stats-dev] Incremental ogr-verify

[stats-dev at lists.distributed.net]

On 27-Jun-2003, stats-dev at lists.distributed.net wrote:
> On Fri, Jun 27, 2003 at 02:55:05PM -0400, stats-dev at lists.distributed.net wrote:
> > Here was my original idea.  I know there are some problems, which I will
> > address at the end.
> > 
> > Load the newly retired ids and new ids into ogr_idlookup and into a separate
> > table
> 
> See my original email.

I was agreeing on this point.

> 
> > Load the data from logdata into day_results
> 
> I'm pretty sure we don't want to do this. First, it means extra data to
> slog through, for no good reason. Second, I think if we pull today's
> data in before processing the new retires, there's a much better chance
> of screwing things up. There's no reason to touch today's data before
> processing retires; the existing code already handles that case
> perfectly fine.

I guess what I meant to say was "Load the data into day_results like
we're doing now"

> 
> > [snip]
>  
> IIRC, building ogr_complete is a relatively fast process. Because of
> this, we shouldn't screw with it for now. Even if we decided to do this
> incrementally, I think the updates would probably suck big time as
> triggers.

ogr_complete takes about a hour to create out of the ~4 hours that the
entire process takes.  I think that coming up with a solution that
reduces this would be a Good Thing.  As for deleting and inserting into
ogr_summary, we have to do two passes anyway - an update and an insert,
and an update is really a delete and an insert anyway.

I guess I'm not sure about how long the triggers 

> 
> I guess I managed not to update OGR_summary to account for old retires.
> Doh! There's no reason to delete though; it would be better to calculate
> the new information, then do an update on OGR_summary.
> [snip]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.distributed.net/pipermail/stats-dev/attachments/20030701/69b6ee12/attachment.bin