Page 1 of 1

Forum instability

Posted: Sun Nov 25, 2018 7:26 pm
by fluffy
Lately it seems that the forum is generating Internal Server Errors a lot, and I suspect this is simply Dreamhost having overallocated our server, as they tend to do. I'll see if there's anything we can do on our end but most likely we're going to need to get Dreamhost to migrate our site to a different server or something.

Re: Forum instability

Posted: Sun Nov 25, 2018 7:33 pm
by fluffy
Okay, it looks like we were being DoSed by some sort of botnet from China for some reason (like, we were getting thousands of hits a second from a single IP block all from spoofed mobile user agents with no referrers). I've blocked them from accessing the forum, hopefully that helps.

Re: Forum instability

Posted: Sun Nov 25, 2018 7:50 pm
by Lunkhead
fluffy wrote:Okay, it looks like we were being DoSed by some sort of botnet from China for some reason (like, we were getting thousands of hits a second from a single IP block all from spoofed mobile user agents with no referrers). I've blocked them from accessing the forum, hopefully that helps.
Ugh, that stinks. Thanks for handling that. Where would I go to see that kind of info? Is that in the boards admin/mods area somewhere or is that in the Dreamhost account admin?

Re: Forum instability

Posted: Sun Nov 25, 2018 7:56 pm
by fluffy
Neither, I went into the server logs:

Code: Select all

[scott]$ cd logs/songfight.net/https
[scott]$ cut -f1 -d\  access.log | sort | cut -f1-3 -d. |  uniq -c | sort -n | tail -n 20
    121 162.233.200
    127 109.70.100
    131 78.46.176
    156 40.77.167
    156 54.36.148
    160 76.121.51
    181 199.249.230
    223 176.31.101
    233 167.114.1
    298 144.76.38
    301 178.63.87
    356 144.76.14
    763 52.83.219
    831 157.55.39
    928 91.242.162
    993 188.138.188
   2275 207.46.13
   4241 46.229.168
 115257 220.243.136
 115335 220.243.135
And I blocked them by adding them to the songfight.net/.htaccess file.

Anyway the forum's already feeling WAY more responsive, go figure.

Re: Forum instability

Posted: Sun Nov 25, 2018 8:09 pm
by Lunkhead
Awesome, thanks again. I always wondered if there were access logs somewhere. I don't suppose we're running a log analyzer are we?

Re: Forum instability

Posted: Sun Nov 25, 2018 8:20 pm
by Lunkhead
Looks like there is an analyzer running, cool.

Re: Forum instability

Posted: Sun Nov 25, 2018 8:29 pm
by fluffy
Yeah, although it doesn't really do much useful (it's a crusty old instance of awstats that only does the same analysis that's trivial to do from the command line) and I don't know what the password to the dashboard is anyway.

Re: Forum instability

Posted: Mon Nov 26, 2018 7:17 am
by jb
Odd, I didn’t see any of the errors.

Re: Forum instability

Posted: Mon Nov 26, 2018 8:42 am
by fluffy
They mostly happened when searching or posting. Ken first reported it a couple days ago and it seemed transitory, then I started getting them a LOT. I could dig into the logs to see when this Chinese bot swarm started; the logs show there were hundreds of machines constantly requesting every forum URL at a rapid rate, purpose unknown.