The Song Fight Bulletin Board

I get a warning going to the front page in webroot somtimes. This morning it happened a couple of times. No idea what it is or why, I can't seem to find details. It's happened a couple of times in the past week or two but today it's every time.

Anyone else have this problem?

I haven't seen this happen, nor do I see what could be causing this. What's providing the warning, and what's the exact URL and browser you're using?

I have the front page bookmarked at: http://www.songfight.org/
I'm using Firefox 21.0 and I'm using Webroot as my security software atm, although that might change in two days when my subscription runs out... maybe.

And what's the entire warning you're getting?

I wonder if maybe Webroot has some outdated information (since we've had malware on the site in the distant past, but it's all been cleaned up), or is keying off of the IP address and complaining because something else on the same server has malware (which happens ALL THE TIME thanks to shared hosting).

Oops, although I DID just find a nasty backdoor lurking in the bushes. Removed. I wonder how long that's been sitting around...

Crap, according to the logs, it's been in use as of a few days ago by someone in Russia. Time to do a more thorough check of the site.

wow, a HUGE PORTION of the website is world-writeable. gee, I wonder how anyone would have managed to sneak anything into the hosting account.

facepalm

okay, I found and removed a whole bunch of insidiously-installed crap. Unfortunately the nature of shared hosting makes it pretty hard to figure out how it came about and I know from experience that Dreamhost support is really crappy about actually trying to do forensics and see about preventing these things from happening.

So far I've found the following:

A remote shell was installed in the legacy /zebra/ directory, and was set up to look like a 404 error if a particular cookie wasn't present

A few PHP scripts were installed to look like they were legit scripts (and interestingly enough they've stopped using blatant obfuscation tricks which makes them harder to find, although easier to dissect)

A .htaccess rule was set up to make it so that certain URLs would redirect to pharmacy spam sites (I only even found this because our Russian spammer friend was actually testing it and that was recent enough to show up in the access logs)

Aside from the accesses to the remote shell and the testing, I couldn't find any indication that the files were uploaded through web-based security holes. Many of the directories and files had group- and world-writeable permissions, so likely the vector was via someone else's compromised account being used as a shell. (Sadly this is a very common attack vector that's possible because of the nature of shared hosting; I've made several suggestions to Dreamhost about how they can mitigate this problem but so far they haven't implemented any of the solutions.) One of the things that facilitated this was that the account profile was set with a umask of 002 (which allows group-write permissions by default). I have, of course, changed this.

Also the site is full of old/abandoned PHP scripts and I'm not sure what should be here and what shouldn't, and we really should do a spring cleaning someday, Spud.

Now, the really funny thing is this stuff isn't stuff that should have been visible to Webroot just yet - it's very likely that it's only a coincidence, and it's just fortunate timing. Whatever damage the spammer was intending to do, he hadn't actually flipped the switch yet.

Would some non-shared hosting option help avoid these situations, like using a private virtual server?

It would help, yes, but AFAIK most VPS options would get very expensive very fast for a site with as much storage and traffic as songfight.org. I should just install my daily site integrity monitor script onto the fightmaster account and have it set to email me when new files go on the site or existing files change (it also specifically calls out things with bogus permissions).

How much traffic does the site generate in an average month? I know decent offers that include a few terabytes worth of unthrottled traffic per month.

Unfortunately, I don't have access to any bandwidth statistics. The HTTP access log doesn't record bandwidth stats for some reason (and only goes back one week anyway), and only JB has Dreamhost control panel access. Maybe JB would like to share the numbers, though...

Oh, the other big issue is storage. songfight.org has over 40GB of data and it's growing every day. LiNode provides a lot of bandwidth for cheap, but storage still costs a lot per month.

There are cheaper VPS options out there but I've been burned badly by their lack of reliability. Most cheap VPSes are fly-by-night operations that have no idea what they're doing, and the rest are fly-by-night operations that know EXACTLY what they're doing.

Heh, the Russian spammers were just trying to trigger the detonator. SORRY GUYS.

I think I just figured out what the endgame was, and how the malicious content warning happened... it looks like the spam stuff was installed in a way that it would NEVER purposefully affect the site as a whole, and the thing that served up spam content was just intended to be spidered separately. So Googlebot has been trying to index it today (and has probably been indexing it in the past as well), so it was just trying to boost other sites' pageranks without affecting the content of this site directly. That's pretty insidious, and even harder to notice than the last time something like this happened.

Fucking arms race, man.

There hasn't been a lot of conversation on this thread (I know I personally don't have much to add), but I wanted to take some time to thank you, fluffy, for helping to keep the site safe and clean. That's no small task, and I'm sure we all appreciate it.

awesome work, fluffy.
-bill