Forum outage this morning

[Forest]

Hi folks,

Earlier this morning we had a brief outage on our server. There was a spike in incoming connections, which overwhelmed the web server, which crashed. We have programs that always monitor the web server, so they automatically restarted the server. Unfortunately, when this happened, some information from the forum wasn't saved correctly into the database, so when the server restarted, forum visitors received a cryptic error mesage.

It took a little while to diagnose and repair the problem, but everything is back up and running again. In this case, no data was lost, but we keep regular backups so that if any data is lost, we can get back up and running with a minimum of disruption.

If you ever notice any error messages in our software, please contact me immediately at ForestForTreesWeb@gmail.com so I can get right to work on it.

Thanks for your support of our nonprofit!

Best,
Forest

 

[Walt Oleksy (RIP 2021)]

Thanks for the update, Forest. You helm a very strong, steady-as-she-goes ship.

 

[Tennis Tom]

Don't worry about it Forest, it will give you TMS! I happened to be awake in my segmented "BBC" sleep pattern, part of my homeostatic routine perusing my favorite web sites, and saw the "error" message and Eric's happy face like he just stuck a 500 lb wild pig with a bow. I didn't find the message cryptic, just figured it was programmer jargon for: "We're sleeping, why aren't you?". Hopefully no pearls of TMS wisdom were lost during the server malfunction, parish the thought. Sorry I'll be missing you guys tonight, I have a crisis of my own to deal with (playing tennis with my coach). As a warning to attendees, if Herbie challenges you to an arm-wrestling match, just say "NO!".

Cheers,
tt/lsmft

 

[Forest]

TT, you absolutely crack me up. Yup, Eric is definitely someone who I would not want to mess with. His neck looks like a treetrunk. I get a bit nervous even talking about him like this. You're not listening, are you, Eric?

So what's the BBC sleeping pattern? I think I've run into something like that, myself.

Forest

 

[Forest]

I have to say, when I first discovered the outage, it gave me chills. While we did do regular backups, the backup was several weeks old, and it is just a fact of life online that if you don't have good backups, you can lose data that is important to you. Luckily, with the help of tech support from two companies we do business with, we were able to fix everything without any data being lost, but what if that hadn't been the case? It's vital to have a good backup process in place so that even in the worst case, only a small amount of data is lost.

To fix this, I've written a pair of scripts to automatically back up the forum. One runs at 2:05 AM every morning and backs up the forum and the other runs at 2:10 AM every Wednesday to back up the wiki pages (the wiki pages don't change as frequently as the forum, so don't need to be backed up as frequently). The two scripts keep the last ten and the last four backups around, respectively, in case data gets corrupted and we don't realize until later. Of course, the scripts store the backups on our server and what happens if the Rooskies bomb the data center where the server is housed? To avoid this type of situation, tech experts suggest having an off site backup. For this, as part of my "every Thursday routine," I'll copy the two backups onto my local hard disk, so the Rooskies will have to bomb both me and the data center. That seems unlikely.

What is all of this for? Well, if our computer ever does crash, I want us to be up and running again as soon as possible. The discussions that happen here are important, and none of us want them to simply disappear if something bad happens. This little nonprofit is my baby, and I want to make sure that nothing ever disrupts her services.