UMW Blogs, a.k.a. Old Faithful

Me and UMW Blogs are going on 5 years this Summer, she’s is the baddest of the bad and  meanest and leanest of the mean and lean. She’s a veritable titan of her kind, she’s an educational publishing platform of the very best kind, and she’s turning five. Five years ago from roughly May through August we brought together the early MistyLook themed WPMu and MediaWiki hybrid out into this wasteland of bad BlackBoard installs, and we shone a light.

A light of good publishing practices, a site for everyone regardless of his or her class status, and course spaces that actually looked good. We were already dreaming of fancy syndication, course aggregation, and a space attractive and user friendly enough that you would  actually want to have a stake in it.  It worked, five years later we have more than 6500 sites and 8500 users, and that number has steadily increased over these past five years. We run heavy traffic sites like UMW Bullet and EagleEye, or blogs for alumni 3 and 4 years out. We have aggregated blog posts from more than 40 UMW students who have written about their travels around the world, and some class sites that imagine a whole new use for online space discussion,  tags and the community in the virtual. There are student created research sites I’d put up against any university’s publicly open and shared work. Not to mention more than 35 original literary journals created by UMW students.

It’s been an amazing space to watch emerge and grow into a distributed and highly active, yet loosely bound, community of classes, students, faculty and staff. I continue to feel honored to be a part of it. Oddly enough, I also feel further and further away from it, so much happens there in a single day I can’t really keep track, we also have hundreds of plugins and themes, it’s a vertiable laboratory doubling as an enterprise system, it’s truly remarkable in that regard. We had almost 100% uptime this past academic school year, and save some recent brute force attacks these past two weeks, it has been a model for both reliability and possibility within a university platform.

Now I’m gonna talk numbers for a second because I continue to be blown away by them. This year alone we have more than 1.2 million unique visitors from all over the world and 4 million page views! This is mind blowing, umw.edu is a veritable brand with 100 years of history behind the idea and it gets 14 million a year. We’ve been doing open education out in the open for 5 years without packaging it—it’s an ongoing happening that others around the web can tap into, it is a model of online education that is not separated from a time, place, and most importantly the people that shape it. I love this system! It’s the proto-type for our upcoming Domain of One’s Own pilot, and it continues to evolve as an integral space wherein we all work.

 What’s more, as Notorious B.I.G. would have said if he was in my shoes, “Mo traffic, mo problems.” We’ve been getting brute force attacked this last week, and Zach Davis, per usual, comes up big and writes us a script to block too many attempts on the login page, which has been pulling us down. Such issues come with age, and it couldn’t have happened at a better time, but it also makes sense to share out Zach’s solution here. And to Zach’s credit, this is a someone trying to hack into WP, and this i where some expertise on our side for sys admining would give him Zach some freedom and force us to take a bit more responsibility. And while I want to in my heart of hearts, I’m not that good—and we can only run enterprise sites like UMW Blogs and the quickly emerging ds106.us successfully for so long until we start needing people who are good, who do it regularly, and know how to admin WordPress properly—I can hack my way through but it is ugly and I’ll be the first to admit  that my skills are not world class in this department :)

If you are having mini-DDOS attacks and your WordPress site is crashing as a result, then what follows may be very useful to you thanks to the great Zach Davis who got the original idea for this wp-login brute force login attack solution from this awesome blog post:

 I’ve done a number of things. I’ve tightened the firewall, tweaked the mod_security rules, reduced the number of apache processes available to the server, and installed an apache module called DOS Evasive. All of these will tighten things up, so keep an ear our for users having problems. We may need to loosen restrictions at some point.

_____________________

I don’t want to have to keep worrying about this server, which is crashing a few times per day, so I’ve gone ahead and implemented a solution to this brute force problem. I’ve added a mod_security rule to apache that looks like this:

# This has to be global, cannot exist within a directory or location clause . . .
SecDataDir /usr/local/apache/logs/data
SecAction phase:1,nolog,pass,initcol:ip=%{REMOTE_ADDR},initcol:user=%{REMOTE_ADDR}

<Location /wp-login.php>
# Setup brute force detection.

# React if block flag has been set.
SecRule user:bf_block “@gt 0″ “deny,status:401,log,msg:’ip address blocked for 5 minutes, more than 15 login $

# Setup Tracking. On a successful login, a 302 redirect is performed, a 200 indicates login failed.
SecRule RESPONSE_STATUS “^302″ “phase:5,t:none,nolog,pass,setvar:ip.bf_counter=0″
SecRule RESPONSE_STATUS “^200″ “phase:5,chain,t:none,nolog,pass,setvar:ip.bf_counter=+1,deprecatevar:ip.bf_co$
SecRule ip:bf_counter “@gt 15″ “t:none,setvar:user.bf_block=1,expirevar:user.bf_block=300,setvar:ip.bf_counte$
</Location>

(for future reference, the rule is stored in /etc/httpd/conf/modsec2.user.conf)

I got the idea here: http://www.frameloss.org/2011/07/29/stopping-brute-force-logins-against-wordpress/

Basically, what this does is assumes that after a successful login, WordPress will redirect the user (and send a 302 header). On an unsuccessful login, WordPress displays the login page with a regular 200 header. The rule tracks how many 302 headers a user gets while visiting the wp-login.php page and, if they exceed 15 10, it blocks the IP for 5 minutes. I tailed the mod security log and the request log at the same time and saw another attack come in (they are coming in constantly; within 5 minutes of blocking an IP, I see more requests, often as many as 3-4 per second from the same IP).

We need to watch this closely and make sure people can still login to WordPress. Normally I would want to test this before deploying it to production, but in this case I think it’s more important to get things locked down and then adjust as necessary. Please make sure that logins still work and let me know if you experience any problems.

How many of you get support like that from your host? It’s amazing, and this reminds me just how integral Zach has been to all of UMW’s successes with UMW Blogs, here’s to you hippie—it’s been an awesome 5 years and sooner or later we will let you out of our experimental prisoner. Nobody makes the bava like the Davis, NOBODY!

About Reverend

I am Jim Groom
This entry was posted in , dtlt, UMW Blogs, umwblogs, wmpued, Wordpress, wpmu, wpmu development. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *