[Az-Geocaching] Suggestion for statistics
Gale Draper
listserv@azgeocaching.com
Fri, 5 Sep 2003 15:35:51 -0700 (PDT)
--0-679071366-1062801351=:80332
Content-Type: text/plain; charset=us-ascii
I'm glad things worked out okay. Will either of you be at the event? I'd like to make a donation to your site's expenses. I don't have a Paypal account and we've been wanting to donate for quite awhile.
Jason Poulter <polt@snaptek.com> wrote:
just got an email reply back from geocaching.com about the blocked
ip.... he has been busy and neglecting emails... but he did get the ip
UNBLOCKED so we can probably resume crawling normally....
he did metion that there are scripts inplace to detect ips /crawlers
that are beating the system to hard at anyone time.... so if i throttle
back the crawling we should be ok.... so i have a little fine tuning to
do until we get back to normal...
please be patient!!!!
thanks
Jason
Snaptek
AzGeocaching.com
Team Cache-Quest wrote:
>Most "crawlers" will obey a site's ROBOTS.TXT file.
>
>GEOCACHING.COM's ROBOTS.TXT disallows almost all crawling. Here is a
>copy...
>
>User-agent: *
># Disallow all unnecessary content from the search
>Disallow: /iis/*
>Disallow: /login/*
>Disallow: /admin/*
>Disallow: /map/*
>Disallow: /email/*
>Disallow: /my/*
>Disallow: /seek/nearest.asp*
>Disallow: /seek/nearest_cache.asp*
>Disallow: /seek/waypoint.asp*
>Disallow: /bait.asp
>
>Because of what Snaptek needs for azgeocaching.com, they have to ignore the
>robots.txt file. Unfortunately most web sites are smart enough now days to
>recognize unauthorized crawling and will automatically prevent it. Some
>consider it a form of hacking because of what it does to the server.
>
>You can't really blame geocaching.com. They probably have many folks trying
>to crawl the site and because of international interest there really isn't a
>good time to do it. What they really need is a decent interface to allow
>folks to get bulk information. The query generator was a step in the right
>direction, but it falls short. I'd like to simply get a list of all the
>caches I've found in XML format to do with what I want, but the Query
>Generator can't even do that.
>
>Jerry (Cache-Quest)
>
>----- Original Message -----
>From: "Regan L Smith"
>To:
>Sent: Friday, September 05, 2003 11:38 AM
>Subject: Re: [Az-Geocaching] Suggestion for statistics
>
>
>
>
>>There has to be other "crawlers" out there how are they dealing with it
>>
>>
>and
>
>
>>is there anything that we the beneficiaries of your fantastic work do???
>>----- Original Message -----
>>From: "Regan L Smith"
>>To:
>>Sent: Friday, September 05, 2003 10:42 AM
>>Subject: Re: [Az-Geocaching] Suggestion for statistics
>>
>>
>>
>>
>>>do they consider AZGeocaching a threat? I don't understand you guys make
>>>their stuff much easier to use, and understand.
>>>
>>>
>
>____________________________________________________________
>Az-Geocaching mailing list listserv@azgeocaching.com
>To edit your setting, subscribe or unsubscribe visit:
>http://listserv.azgeocaching.com/mailman/listinfo/az-geocaching
>
>Arizona's Geocaching Resource
>http://www.azgeocaching.com
>
>
____________________________________________________________
Az-Geocaching mailing list listserv@azgeocaching.com
To edit your setting, subscribe or unsubscribe visit:
http://listserv.azgeocaching.com/mailman/listinfo/az-geocaching
Arizona's Geocaching Resource
http://www.azgeocaching.com
---------------------------------
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
--0-679071366-1062801351=:80332
Content-Type: text/html; charset=us-ascii
<DIV>I'm glad things worked out okay. Will either of you be at the event? I'd like to make a donation to your site's expenses. I don't have a Paypal account and we've been wanting to donate for quite awhile.<BR><BR><B><I>Jason Poulter <polt@snaptek.com></I></B> wrote:</DIV>
<BLOCKQUOTE class=replbq style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">just got an email reply back from geocaching.com about the blocked <BR>ip.... he has been busy and neglecting emails... but he did get the ip <BR>UNBLOCKED so we can probably resume crawling normally....<BR><BR>he did metion that there are scripts inplace to detect ips /crawlers <BR>that are beating the system to hard at anyone time.... so if i throttle <BR>back the crawling we should be ok.... so i have a little fine tuning to <BR>do until we get back to normal...<BR><BR>please be patient!!!!<BR><BR>thanks<BR><BR>Jason<BR>Snaptek<BR>AzGeocaching.com<BR><BR><BR><BR>Team Cache-Quest wrote:<BR><BR>>Most "crawlers" will obey a site's ROBOTS.TXT file.<BR>><BR>>GEOCACHING.COM's ROBOTS.TXT disallows almost all crawling. Here is a<BR>>copy...<BR>><BR>>User-agent: *<BR>># Disallow all unnecessary content from the search<BR>>Disallow: /iis/*<BR>>Disallow:
/login/*<BR>>Disallow: /admin/*<BR>>Disallow: /map/*<BR>>Disallow: /email/*<BR>>Disallow: /my/*<BR>>Disallow: /seek/nearest.asp*<BR>>Disallow: /seek/nearest_cache.asp*<BR>>Disallow: /seek/waypoint.asp*<BR>>Disallow: /bait.asp<BR>><BR>>Because of what Snaptek needs for azgeocaching.com, they have to ignore the<BR>>robots.txt file. Unfortunately most web sites are smart enough now days to<BR>>recognize unauthorized crawling and will automatically prevent it. Some<BR>>consider it a form of hacking because of what it does to the server.<BR>><BR>>You can't really blame geocaching.com. They probably have many folks trying<BR>>to crawl the site and because of international interest there really isn't a<BR>>good time to do it. What they really need is a decent interface to allow<BR>>folks to get bulk information. The query generator was a step in the right<BR>>direction, but it falls short. I'd like to simply get a list of all
the<BR>>caches I've found in XML format to do with what I want, but the Query<BR>>Generator can't even do that.<BR>><BR>>Jerry (Cache-Quest)<BR>><BR>>----- Original Message ----- <BR>>From: "Regan L Smith" <BUGGERS@MINDSPRING.COM><BR>>To: <LISTSERV@AZGEOCACHING.COM><BR>>Sent: Friday, September 05, 2003 11:38 AM<BR>>Subject: Re: [Az-Geocaching] Suggestion for statistics<BR>><BR>><BR>> <BR>><BR>>>There has to be other "crawlers" out there how are they dealing with it<BR>>> <BR>>><BR>>and<BR>> <BR>><BR>>>is there anything that we the beneficiaries of your fantastic work do???<BR>>>----- Original Message ----- <BR>>>From: "Regan L Smith" <BUGGERS@MINDSPRING.COM><BR>>>To: <LISTSERV@AZGEOCACHING.COM><BR>>>Sent: Friday, September 05, 2003 10:42 AM<BR>>>Subject: Re: [Az-Geocaching] Suggestion for statistics<BR>>><BR>>><BR>>> <BR>>><BR>>>>do they consider
AZGeocaching a threat? I don't understand you guys make<BR>>>>their stuff much easier to use, and understand.<BR>>>> <BR>>>><BR>><BR>>____________________________________________________________<BR>>Az-Geocaching mailing list listserv@azgeocaching.com<BR>>To edit your setting, subscribe or unsubscribe visit:<BR>>http://listserv.azgeocaching.com/mailman/listinfo/az-geocaching<BR>><BR>>Arizona's Geocaching Resource<BR>>http://www.azgeocaching.com<BR>> <BR>><BR><BR><BR>____________________________________________________________<BR>Az-Geocaching mailing list listserv@azgeocaching.com<BR>To edit your setting, subscribe or unsubscribe visit:<BR>http://listserv.azgeocaching.com/mailman/listinfo/az-geocaching<BR><BR>Arizona's Geocaching Resource<BR>http://www.azgeocaching.com</BLOCKQUOTE><p><hr SIZE=1>
Do you Yahoo!?<br>
<a href="http://us.rd.yahoo.com/evt=10469/*http://sitebuilder.yahoo.com">Yahoo! SiteBuilder</a> - Free, easy-to-use web site design software
--0-679071366-1062801351=:80332--