Forum instability

Let us know when something isn't working correctly, or if you find a typo. Do not post complaints or suggestions here.
Post Reply
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Forum instability

Post by fluffy »

Lately it seems that the forum is generating Internal Server Errors a lot, and I suspect this is simply Dreamhost having overallocated our server, as they tend to do. I'll see if there's anything we can do on our end but most likely we're going to need to get Dreamhost to migrate our site to a different server or something.
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Forum instability

Post by fluffy »

Okay, it looks like we were being DoSed by some sort of botnet from China for some reason (like, we were getting thousands of hits a second from a single IP block all from spoofed mobile user agents with no referrers). I've blocked them from accessing the forum, hopefully that helps.
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Forum instability

Post by Lunkhead »

fluffy wrote:Okay, it looks like we were being DoSed by some sort of botnet from China for some reason (like, we were getting thousands of hits a second from a single IP block all from spoofed mobile user agents with no referrers). I've blocked them from accessing the forum, hopefully that helps.
Ugh, that stinks. Thanks for handling that. Where would I go to see that kind of info? Is that in the boards admin/mods area somewhere or is that in the Dreamhost account admin?
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Forum instability

Post by fluffy »

Neither, I went into the server logs:

Code: Select all

[scott]$ cd logs/songfight.net/https
[scott]$ cut -f1 -d\  access.log | sort | cut -f1-3 -d. |  uniq -c | sort -n | tail -n 20
    121 162.233.200
    127 109.70.100
    131 78.46.176
    156 40.77.167
    156 54.36.148
    160 76.121.51
    181 199.249.230
    223 176.31.101
    233 167.114.1
    298 144.76.38
    301 178.63.87
    356 144.76.14
    763 52.83.219
    831 157.55.39
    928 91.242.162
    993 188.138.188
   2275 207.46.13
   4241 46.229.168
 115257 220.243.136
 115335 220.243.135
And I blocked them by adding them to the songfight.net/.htaccess file.

Anyway the forum's already feeling WAY more responsive, go figure.
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Forum instability

Post by Lunkhead »

Awesome, thanks again. I always wondered if there were access logs somewhere. I don't suppose we're running a log analyzer are we?
User avatar
Lunkhead
You're No Good
Posts: 8104
Joined: Sat Sep 25, 2004 12:14 pm
Instruments: many
Recording Method: cubase/mac/tascam4x4
Submitting as: Berkeley Social Scene, Merisan, Tiny Robots
Pronouns: he/him
Location: Berkeley, CA
Contact:

Re: Forum instability

Post by Lunkhead »

Looks like there is an analyzer running, cool.
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Forum instability

Post by fluffy »

Yeah, although it doesn't really do much useful (it's a crusty old instance of awstats that only does the same analysis that's trivial to do from the command line) and I don't know what the password to the dashboard is anyway.
User avatar
jb
Hot for Teacher
Posts: 4158
Joined: Sat Sep 25, 2004 10:12 am
Instruments: Guitar, Cello, Keys, Uke, Vox, Perc
Recording Method: Logic X
Submitting as: The John Benjamin Band
Pronouns: he/him
Location: WASHINGTON, DC
Contact:

Re: Forum instability

Post by jb »

Odd, I didn’t see any of the errors.
blippity blop ya don’t stop heyyyyyyyyy
User avatar
fluffy
Eruption
Posts: 11028
Joined: Sat Sep 25, 2004 10:56 am
Instruments: sometimes
Recording Method: Logic Pro X
Submitting as: Sockpuppet
Pronouns: she/they
Location: Seattle-ish
Contact:

Re: Forum instability

Post by fluffy »

They mostly happened when searching or posting. Ken first reported it a couple days ago and it seemed transitory, then I started getting them a LOT. I could dig into the logs to see when this Chinese bot swarm started; the logs show there were hundreds of machines constantly requesting every forum URL at a rapid rate, purpose unknown.
Post Reply