Recent Site Access Issue

[von Corax]

Over the past few days visitors to Brass Goggles might have been met with a 504 Gateway Timeout error message. When I contacted our hosting service, CanSpace Solutions, they were already aware of the problem and hard at work to mitigate it. The issue experienced was due to a concerted Denial of Service attack on the box where our site lives.

This morning I received the following missive from CanSpace's Tech Support desk:

CanSpace Solutions

Hello,

I'm writing to provide an update on the recent challenges we faced with your server, and to share the steps we've taken to address these issues.

Recently, your server was targeted by an unusually intense DDoS attack, which reached its peak yesterday, December 18th. Our data center is well-equipped to handle most DDoS attacks, but this particular instance was exceptional in both its scale and complexity.

This resulted in sporadic downtime as we were forced to shut down certain network services to keep other parts of the server online, as well as a few reboots as we added hardware to help the server cope with the extra traffic.

DDoS attacks are rare, but unfortunately they can happen to any provider at any time. We do our absolute best to mitigate these as fast as we possibly can. The peak of this attack actually began on the night of Sunday the 17th, and we had multiple agents (including myself) up all night well into Monday morning to try and get your websites and services back online as fast as we possibly could.

The good news is that this issue is now resolved, and should be permanently. We've managed to null-route all of the bad traffic, and we've also identified the sites that were the target of this attack and have moved them off the server altogether.

We understand the critical role your website and email services play in your business, and I just wanted to re-assure you that we were working as hard as we possibly could to get things back to normal as soon as possible.

As an added bonus, the extra hardware we added to your server to cope with the excess DDoS traffic remains in place, so your website should actually be loading even faster than before.

As always we greatly appreciate your business and I hope that the downtime was not too much of an inconvenience here. Merry Christmas and Happy Holidays!

-Bojan
CanSpace Solutions

----------------------------------------------
Ticket ID: #954146
Subject: 504 Errors
Status: Closed
Ticket URL: https://www.canspace.ca/clients/viewticket.php?tid=954146&c=Q7xMos0a
----------------------------------------------
visit our website | log in to your account | get support
Copyright © CanSpace Solutions, All rights reserved.

Please join me in thanking Bojan and his colleagues at CanSpace for their efforts on our behalf.

[Sorontar]

I must admit, that is the most I have ever heard from a hosting site in response to issues like that. Normally they tend to just say "Everything is under control" without indicating anything about how they have done that, or even what the problem was. Thanks for sharing it with us.
Sorontar

[SeVeNeVeS]

Are we being attacked again, can sometimes take 30 seconds or more of hanging before I can visit the forum and log in, then view a post. Slow.

[von Corax]

I'm not aware of anything. Is anyone else having trouble logging in?

[SeVeNeVeS]

Happened again yesterday, I couldn't access all day with time outs and firefox blocking the site because of risks from malicious activity and now it seems the last 3 days of posts and replies have mysteriously disappeared.

Strange indeed.

God knows this isn't exactly the most interactive forum with most people prefering just to lurk but not good that any replies that have been made are deleted. 

[Sorontar]

From our host:
"Ajax Issue
Friday, 12th April, 2024 09:02am

We have identified a hardware failure on ajax.canspace.ca and are working on replacing components at this moment. We apologize for the inconvenience and will have services restored to normal asap."

It would be nice to know what was affected.

[SeVeNeVeS]

It would be nice to know what was affected.

The morale of the mostly apathetic took a hit.

Surely as a web host they should have back-ups in place for such an eventuality?

Try and wangle some free hosting time, say 6 months as a gesture of good will on thier behalf. What are Admin paying for, if not reliabity of servive?

[von Corax]

I'm embarrassed to admit I was unaware of any problem until I received an email from CanSpace to the effect that service had been restored, although I did get an SSL warning yesterday afternoon after work.

According to their message, they suffered a catastrophic failure of the box on which BG resides; that box has now been replaced with a much beefier box with redundant RAID controllers which should prevent future data loss.

Can anyone give me any idea of how far back the data loss extends? The last manual backup I have is from Tuesday.

[SeVeNeVeS]

Around April 9th to the 11th appear to be missing..... can't be more specific than that I'm afraid.

Some of mine, yours, james, JW, LordWorthing from what I remember.

[von Corax]

I've just gotten my ansible playbook to back up the databases working again. Now I need to bullet-proof it a little and then I will set it up as a daily cron job.

[von Corax]

I've just gotten my ansible playbook to back up the databases working again. Now I need to bullet-proof it a little and then I will set it up as a daily cron job.

I've laid in a little armour plating, and now I'm testing it as a cron job; if this works I'll have it running by midnight.

[von Corax]

The invocations aren't working. I need to figure out how the daemon knows to ruAAAGHHHHkl;fdqno;fda

[von Corax]

The invocations aren't working. I need to figure out how the daemon knows to ruAAAGHHHHkl;fdqno;fda

fjkodanern';l;lmAGGGGHHHHHNNNNGGGGHah hah hah! (Pant pant.) I have done battle with the daemon CRON and have emerged victorious! The forum and wiki databases shall henceforth be backed up to my (offsite) local system daily at the midnight hour.

EDIT: I have posted the playbook to my GitHub repository at https://github.com/vonCorax/brassgoggles_ansible.git.

[von Corax]

...And just in bloody time, it would seem. Some time Thursday we suffered a database crash which took some time to recover from, using the backup that I first got to run automatically just the night before...

[Sorontar]

The power of antic.............
.
.
.
.
.
ipation.

Sorontar

[SeVeNeVeS]

Tekkie speak me no process..........layman's terms......we sorted it out after a major clusterf&ck and a little luck......that I uderstand.

Thanks to von Corax and Sorontar

[von Corax]

If anyone's interested, I've just pushed an update to my playbook that will email me on both completion and failure of the nightly backup.

Next step is to figure out how to leave the last few backups on the server, and automatically delete anything older than that.

[SeVeNeVeS]

If anyone's interested, I've just pushed an update to my playbook that will email me on both completion and failure of the nightly backup.

Next step is to figure out how to leave the last few backups on the server, and automatically delete anything older than that.

ME...

[von Corax]

Okay, I think I've figured out how to get Ansible to leave 2 weeks' worth of backups on the server. If it works, I'll know in two weeks. If it doesn't, I might know tomorrow.