[RC5] d.net project: indexing the web

Robert A Fox bob at breezeholme.demon.co.uk
Fri Jul 9 22:07:33 EDT 1999

On Thu, 8 Jul 1999, Matt.Wilkie wrote:

> Harnessing individual computers to index [the web] as they browse could
> potentially greatly improve search engine's indices. I think d.net could
> help craft a solution to this problem because: 
> 1) (to the best of my knowledge) most well developed distributed 
> computing network in existence today

I'm not sure that the distributed.net client is suited to this task, well
developed though it is.  Looking for key words in text isn't difficult
work, so there's plenty of room to the distributed.net client and a
seperate indexer client.

To work with a web browser in this way, the client would have to act as a
proxy web server.  This would mean accepting incoming connections, and
could be a security risk.

> 2) d.net developers and participants are well aware of privacy 
> issues, and anything they build in this line will be privacy 'compliant'

What would we do with the information once we had it?  Use it to create a
new search engine, or give/sell it to existing one?  Search engines are
commercial organisations these days, and the last time it was suggested
that we worked on behalf of a profit-making company (distributed 
rendering), objections were raised by people ran clients on their
employer's computers.

> 3) a proprietary version of this is doomed to failure, nobody will
> trust it and it wouldn't get broadly enough used.

If one of the big portals like Yahoo! pushed the idea, I'm sure that it
would be a success.  It could be combined with existing "pay as you
surf" schemes, to pay a bounty on information about new sites.  A lot of
people would forget about the privacy aspect if they were making money.

I think it's a great idea, but I don't think that we're the best people to
do it.

Robert A Fox

To unsubscribe, send 'unsubscribe rc5' to majordomo at lists.distributed.net
rc5-digest subscribers replace rc5 with rc5-digest

More information about the rc5 mailing list