Search Spider and Spam IP - Site slow, too much!


#1

I have a huge problem with my site.

I run a forum (Simple MAchine Forum) on a VPS.
It was perfect, no problem for 3 years.

Actually I experience a lot of problems.

Some times the site needs 40 seconds to load, sometimes just 1 second.

The problem is not the traffic.
The traffic is always the same.

Dreamhost support told me problems are:

  • slow access to mySql
  • a lot of active requests.

I copy what they wrote to me:

Looking at your Apache status: Current Time: Thursday, 23-Jan-2014 08:44:53 PST Restart Time: Thursday, 23-Jan-2014 07:27:54 PST Parent Server Generation: 0 Server uptime: 1 hour 16 minutes 59 seconds Total accesses: 29285 - Total Traffic: 17.7 MB CPU Usage: u2.5 s.25 cu0 cs0 - .0595% CPU load 6.34 requests/sec - 4024 B/second - 634 B/request 152 requests currently being processed, 2 idle workers

Then, the told me that I should have gone to the url


to block bad bots and offending IPs. This should help with the slowness
and increase stability.


Then i asked them to help me doing something, and the wrote me:

Since the last time the logs have been rotated, these are the number of
hits from the following IP’s:

#hits IP Address
2145 2.233.30.94
2296 79.25.175.48
2318 2.234.62.159
2463 79.1.202.190
2509 93.35.81.94
2614 79.31.60.174
2879 79.59.160.117
3029 2.230.118.234
3351 83.103.49.3
5111 217.22.208.50


I added them to my htaccess file.


Nothing changed.

2 questions:

  1. how can i see what IP are hitting my site, looking at the number of times they do it?
  2. what can i do to majke my site surfable??? actually it’s almost unusable… :frowning:

#2

I’ll defer to others here that can answer your questions better than I can. A suggestion however… post your .htaccess if you can and only clearly redact anything private…order can be important, .htaccess can be a pain and yours might not be doing what you think.


#3

This is my actual htaccess file

[code]#— DH-PHP handlers —
AddHandler fastcgi-script fcg fcgi fpl
AddHandler php-fastcgi .php
Action php-fastcgi /cgi-bin/dispatch.fcgi

USER IP BANNING

order allow,deny deny from 2.233.30.94 deny from 79.25.175.48 deny from 2.234.62.159 deny from 79.1.202.190 deny from 93.35.81.94 deny from 79.31.60.174 deny from 79.59.160.117 deny from 2.230.118.234 deny from 83.103.49.3 deny from 217.22.208.50 allow from all

PREVENT VIEWING OF .HTACCESS

<Files .htaccess>
order allow,deny
deny from all
[/code]

until some days ago it was so:

[code]#— DH-PHP handlers —
AddHandler fastcgi-script fcg fcgi fpl
AddHandler php-fastcgi .php
Action php-fastcgi /cgi-bin/dispatch.fcgi

<Files .htaccess>
order allow,deny
deny from all

RewriteEngine On
RewriteCond %{HTTP_REFERER} !^http://(.+.)?mydomain.com/ [NC]
RewriteCond %{HTTP_REFERER} !^$
RewriteRule .*.(jpe?g|gif|bmp|png)$ http://i.imgur.com/qX4w7.gif [L]

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule ^.* - [F,L]

USER IP BANNING

order allow,deny deny from 2.233.30.94 deny from 79.25.175.48 deny from 2.234.62.159 deny from 79.1.202.190 deny from 93.35.81.94 deny from 79.31.60.174 deny from 79.59.160.117 deny from 2.230.118.234 deny from 83.103.49.3 deny from 217.22.208.50 allow from all

INIZIO blocca IP di spammer e troll

#order allow,deny
#deny from all

FINE blocca IP di spammer e troll

ErrorDocument 403 "

Sito in costruzione[/code]