Search found 2 matches

by wheel
Thu Jul 24, 2008 2:35 pm
Forum: Colocation & Dedicated Assistance
Topic: Bot/scrapers - stop them at the server level?
Replies: 2
Views: 34103

reasons to do this unfortunately. Not so sure about that anymore - that's why I'm investigating. I had a look at one site recently and a rough guess shows it was likely putting out 25-50 gigs in one month to scrapers. That's excluding the search engines - just places in korea, china, and the old ir...
by wheel
Thu Jul 24, 2008 12:51 pm
Forum: Colocation & Dedicated Assistance
Topic: Bot/scrapers - stop them at the server level?
Replies: 2
Views: 34103

Bot/scrapers - stop them at the server level?

Been doing some digging around on this lately....looking at my logs and feedback from other folks that monitor this type of thing, I'm starting to be suspicious that anywhere between 25 to 50% of web traffic on my server (and probably yours) is from bots or scrapers - not human visitors, and not eve...