[RC5] [stats-dev] RC5-72 Sparkline stats

[Kevin McCoy]

Jim C. Nasby wrote:
> On Tue, Jun 20, 2006 at 05:00:56PM -0700, Kevin McCoy wrote:
>   
>> My stats server spiders a couple of the team pages.  It would be 
>> redundant to reload the phistory data over and over since you are only 
>> interested in new records.  I store the last 90 days of stats locally.
>>
>> My Team Warped stats server is at 
>> http://idk.serveftp.net/cowboy/participant.html
>>
>> This server displays stats down to the individual cow-level, as well as 
>> herd-level user stats, including team standings and overall rates for 
>> both contests.
>>
>> I have a lot of cows, so I wrote the Cowboy server to help me 
>> debug/manage them.  Cowboy monitors individual DNET cows all over the world.
>>     
>
> I assume it pulls from pproxy logs?
>   

No, I took a different approach, since a central pproxy can be 
problematic if it is located on the wrong side of a firewall.  
Additionally, Pproxy logs don't have sufficient detail to diagnose 
malfunctioning cows.  Instead, I have the cows email DNET logs either 
via standard email or to an integrated SMTP server I wrote.  The Cowboy 
server checks its email box about once an hour for log messages and 
accepts SMTP messages sent directly to it immediately.  It concatenates 
the emailed logs with any existing logs for a given cow, loads all the 
DNET events into an SQL database and analyzes the events.  It creates 
new graphs every two hours and regenerates the HTML tabular data. The 
end-user can view his/her DNET logs on line, by clicking on a web page 
link.  Cowboy flags any cows having network connectivity, random keys, 
improper shutdowns, expired beta clients, Etc.  It makes managing large 
herds, or geographically separated machines much easier. 

At some point I may redo the Cowboy server to dynamically generate the 
HTML directly from the SQL tables, but for now it has to be somewhat of 
a batch job.  It is pretty processor intense to generate 90-day running 
averages for lots of cows, so I am not sure it would be a good real-time 
web app.


Kevin