Page 1 of 1
Forum instability
Posted: Sun Nov 25, 2018 7:26 pm
by fluffy
Lately it seems that the forum is generating Internal Server Errors a lot, and I suspect this is simply Dreamhost having overallocated our server, as they tend to do. I'll see if there's anything we can do on our end but most likely we're going to need to get Dreamhost to migrate our site to a different server or something.
Re: Forum instability
Posted: Sun Nov 25, 2018 7:33 pm
by fluffy
Okay, it looks like we were being DoSed by some sort of botnet from China for some reason (like, we were getting thousands of hits a second from a single IP block all from spoofed mobile user agents with no referrers). I've blocked them from accessing the forum, hopefully that helps.
Re: Forum instability
Posted: Sun Nov 25, 2018 7:50 pm
by Lunkhead
fluffy wrote:Okay, it looks like we were being DoSed by some sort of botnet from China for some reason (like, we were getting thousands of hits a second from a single IP block all from spoofed mobile user agents with no referrers). I've blocked them from accessing the forum, hopefully that helps.
Ugh, that stinks. Thanks for handling that. Where would I go to see that kind of info? Is that in the boards admin/mods area somewhere or is that in the Dreamhost account admin?
Re: Forum instability
Posted: Sun Nov 25, 2018 7:56 pm
by fluffy
Neither, I went into the server logs:
Code: Select all
[scott]$ cd logs/songfight.net/https
[scott]$ cut -f1 -d\ access.log | sort | cut -f1-3 -d. | uniq -c | sort -n | tail -n 20
121 162.233.200
127 109.70.100
131 78.46.176
156 40.77.167
156 54.36.148
160 76.121.51
181 199.249.230
223 176.31.101
233 167.114.1
298 144.76.38
301 178.63.87
356 144.76.14
763 52.83.219
831 157.55.39
928 91.242.162
993 188.138.188
2275 207.46.13
4241 46.229.168
115257 220.243.136
115335 220.243.135
And I blocked them by adding them to the songfight.net/.htaccess file.
Anyway the forum's already feeling WAY more responsive, go figure.
Re: Forum instability
Posted: Sun Nov 25, 2018 8:09 pm
by Lunkhead
Awesome, thanks again. I always wondered if there were access logs somewhere. I don't suppose we're running a log analyzer are we?
Re: Forum instability
Posted: Sun Nov 25, 2018 8:20 pm
by Lunkhead
Looks like there is an analyzer running, cool.
Re: Forum instability
Posted: Sun Nov 25, 2018 8:29 pm
by fluffy
Yeah, although it doesn't really do much useful (it's a crusty old instance of awstats that only does the same analysis that's trivial to do from the command line) and I don't know what the password to the dashboard is anyway.
Re: Forum instability
Posted: Mon Nov 26, 2018 7:17 am
by jb
Odd, I didn’t see any of the errors.
Re: Forum instability
Posted: Mon Nov 26, 2018 8:42 am
by fluffy
They mostly happened when searching or posting. Ken first reported it a couple days ago and it seemed transitory, then I started getting them a LOT. I could dig into the logs to see when this Chinese bot swarm started; the logs show there were hundreds of machines constantly requesting every forum URL at a rapid rate, purpose unknown.