new thread
Erm, confession time.
I wanted to be able to search through the board using grep, so I got wget to download the board. However, I think it might have been a little over-zealous.
I don't think I've used up too much bandwidth or acctidently spammed anyone, but if I have I am truly very sorry.
I'll never wget the site again!
/me hangs head in shame.
For the convenience of users, the site has a lot of interconnectivity - different URLs will many times lead to the same results. As an example, you can get a form to write me by clicking on my name above, but you can get the same form by clicking on my name in other messages I have written or on my name in the users' directory. A second example is that the users' directory can be viewed in a variety of ways. A third is that the "other posts" list for a user is essentially the same no matter which of his posts you view it from.
Because of this, downloading every URL you can find is inefficient. It will give you the same information over and over. In any searches of the data you downloaded, this will probably be quite irritating.
Google in effect does what you did when they crawl the site, so the site has often seen it before. It has the bandwidth to handle it at the level it occurs, so worry not.
Google hits the site often so their copy is reasonably up to date, and using them is likely a better way to find material than crawling it occasionally yourself. Just enter site:the-light.com searchstring in their search box. "Searchstring" is what you'd normally enter, and the first part limits the search to material on this site.
Bill (site programmer)
