Main Menu

Server Outage

Started by Saga Musix, February 17, 2011, 18:48:59

Previous topic - Next topic

Saga Musix

Some of you might have noticed a server outage this noon at around 12:45 UTC. The server was shut down by the provider because it was consuming all resources. As far as I can see, there was no security breach; a problem with the mail server is the most plausible explanation at the moment. At this time of writing I cannot guaranetee that the server was not cracked (but I'd be surprised if it was, since all software is kept up-to-date, the last time I checked for system updates was actually yesterday), but if you want to feel really safe, you can of course change your passwords on the forum / issue tracker / etc. - I don't think it's necessary at the moment, as passwords on the forum and issue tracker are salted anyway, but of course it's a good idea in general to change passwords reguarily. Right now I'm looking into the mail server configuration to see if a bad configuration was indeed the reason for this server outage.
» No support, bug reports, feature requests via private messages - they will not be answered. Use the forums and the issue tracker so that everyone can benefit from your post.

Rakib

Do you use the mail server?
If not how about closing the mail server permanently?
^^

Saga Musix

The mail server is needed to send out confirmation mails and similar stuff; anyway, there was a rather silly misconfiguration which has been fixed now, so if the mail server was part of the problem or even caused it, this should not happen anymore. I still wonder a bit about a traffic peak (about 14MiB of in/out traffic combined in one hour, which doesn't sound much but actually is, in comparison to the usual site traffic) and excessive usage of CPU and/or RAM which occurred around the same time (probably caused by the mail server fuckup), but seeing that a similar situation was caused by a very stupid spider earlier tonight (a fix for this has also been applied by means of installing a proper robots.txt for the wiki), it doesn't seem too alarming anymore.
» No support, bug reports, feature requests via private messages - they will not be answered. Use the forums and the issue tracker so that everyone can benefit from your post.

LPChip

You're doing a great job Jojo!

Thanks for taking care of this.
"Heh, maybe I should've joined the compo only because it would've meant I wouldn't have had to worry about a damn EQ or compressor for a change. " - Atlantis
"yes.. I think in this case it was wishful thinking: MPT is makng my life hard so it must be wrong" - Rewbs

Saga Musix

Also looking at the forum stats right now, I'm rather confident that the outage was caused by some aggressive bot hammering ("Most Online Ever: 193 (Yesterday at 13:46:45)") - that is a whole lot more than usual, and that would also explain the traffic peak as well as the use of resources. I might have to tweak some Apache settings...
» No support, bug reports, feature requests via private messages - they will not be answered. Use the forums and the issue tracker so that everyone can benefit from your post.

Rakib

Cheap trick is to set a maximum clients to visit the site at once, but I don't know how popular the forum actually is.

http://httpd.apache.org/docs/trunk/mod/mpm_common.html#maxclients
^^

Saga Musix

MaxClients is actually set to a good value, and the website uses caching and similar techniques for reducing load actually, so I'm not sure if it would help much to lower the value even more. If a similar congestion happens again, further actions will be necessary, but at the moment I'll just watch the situation and see if anything changes.
» No support, bug reports, feature requests via private messages - they will not be answered. Use the forums and the issue tracker so that everyone can benefit from your post.