Have you heard the saying, “You don’t need backup until you lose your data for the first time”?
I remember; it sounded just like a normal saying to me, but I grasped the real meaning when I first lost my data. I made a mistake – instead of formatting my USB flash drive, I accidentally formatted my 2 TB disk. When I realized it, it was too late.
Losing any type of data, especially when collected over a couple of years or even decades, is a traumatic experience. I would say it is even worse when it happens in corporate organizations. The data are relevant to the whole organization, and internal and external business operations.
That being said, it is crucial to develop a backup and recovery plan, perform regular backups, test them, and be ready, just in case you need it.
In this article, I would like to share with you five stories where big organizations suffered because of a lack of backup and recovery plan, human error, or because the backup just didn’t work as planned.
Grab your favorite cup of coffee, and join me on a journey.
GitLab Database Incident
In 2017, GitLab experienced a serious incident with one of its databases. According to their report, they lost six hours’ worth of database data, including issues, merge requests, users, comments, and snippets.
They identified three incidents.
In the first incident, attackers bombarded the database from 47,000 different IP addresses using the same account. They created snippets, causing it to become unstable and unresponsive; later, they locked the database for any write operations.
In the second incident, GitLab noticed that copying of data between databases fell too far behind and stopped working. This happened because there was a sudden increase in new writes that wasn’t handled quickly enough by the backup database.
In the third incident, an IT professional involved in troubleshooting mistakenly deleted a backup from the incorrect server. Upon realizing the error, he attempted to cancel the deletion process, but unfortunately, it was already too late.
Out of the original 300 gigabytes, only around 4.5 gigabytes of data remained.
This sounds like the personal experience I mentioned at the beginning of the article.
In their later reports, the team members pointed out that, based on what they observed, there were problems with the backup database.
The files it was creating were very small, just a few bytes, suggesting a possible issue with the backup procedure. Additionally, it was noted that backups to AWS S3 were not functioning correctly; the designated bucket was empty, indicating that the backups weren’t being stored as expected.
You can read the full report here GitLab.com database incident.
T-Mobile Sidekick Disaster
In 2009, T-Mobile experienced a significant data loss incident that affected approximately 800,000 smartphone users in the US. Users lost personal data, including emails, contacts, to-do lists, calendar entries, and photos from their mobile devices.
This event was considered one of the largest data loss disasters in cloud computing at that time.
The outage persisted for about two weeks. As cloud technology was still evolving during this period, the incident contributed to some people becoming skeptical about relying on the cloud for data storage.
Upon investigation, it was revealed that a Microsoft server failure had occurred, leading to the corruption of backup data.
Restoring the lost data proved to be a prolonged process, taking a few more weeks, and unfortunately, not all of the data could be fully recovered.
Here is what T-Mobile posted on the forums:
Dear valued T-Mobile Sidekick customers:
T-Mobile and the Sidekick data services provider, Danger, a subsidiary of Microsoft, are reaching out to express our apologies regarding the recent Sidekick data service disruption.
We appreciate your patience as Microsoft/Danger continues to work on maintaining platform stability, and restoring all services for our Sidekick customers.
Regrettably, based on Microsoft/Danger’s latest recovery assessment of their systems, we must now inform you that personal information stored on your device – such as contacts, calendar entries, to-do lists or photos – that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger. That said, our teams continue to work around-the-clock in hopes of discovering some way to recover this information. However, the likelihood of a successful outcome is extremely low. As such, we wanted to share this news with you and offer some tips and suggestions to help you rebuild your personal content. You can find these tips in our Sidekick Contacts FAQ. We encourage you to visit the Forums on a regular basis to access the latest updates as well as FAQs regarding this service disruption.
In addition, we plan to communicate with you on Monday (Oct. 12) the status of the remaining issues caused by the service disruption, including the data recovery efforts and the Download Catalog restoration which we are continuing to resolve. We also will communicate any additional tips or suggestions that may help in restoring your content.
We recognize the magnitude of this inconvenience. Our primary efforts have been focused on restoring our customers’ personal content. We also are considering additional measures for those of you who have lost your content to help reinforce how valuable you are as a T-Mobile customer.
We continue to advise customers to NOT reset their device by removing the battery or letting their battery drain completely, as any personal content that currently resides on your device will be lost.
Once again, T-Mobile and Microsoft/Danger regret any and all inconvenience this matter has caused.
Code Spaces Destroyed by Cyber Attack
In June 2014, Code Spaces fell victim to a cyber-attack. The hackers initiated a DDoS attack and later gained access to the AWS EC2 control panel, causing significant damage to their infrastructure and data.
The attackers took several destructive actions, including creating several unauthorized backup logins, deleting all Elastic Block Store (EBS) snapshots, wiping out an S3 bucket along with its data, removing all Amazon Machine Images (AMIs), and impacting multiple machines.
A big portion of Code Spaces’ data, backups, and configurations were either partially or completely erased.
An important observation here is that while Code Spaces had a backup system in place within AWS, they lacked an offsite backup and recovery plan.
If they had stored backups at an offsite location, even if the hackers deleted all data within AWS, copies of the backups could have been retrieved from an external source.
Unfortunately, this precautionary measure was not in place, emphasizing the significance of offsite backups in securing against such cyber threats.
| Date | Event |
|---|---|
| June 17, 2014 | Code Spaces is attacked by a DDoS (Denial of Service) attack. |
| June 18, 2014 | Code Spaces is shut down permanently. |
Unfortunately, this significant data loss led Code Spaces to permanently shut down their business.
You can read the report here Codespaces (2014) – Public Cloud Security Breaches and Code Spaces announcement to their customers.
Maersk and NotPetya Ransomware
Ransomware is everywhere, targeting organizations, their systems, and data with the aim of making them unusable. The attackers encrypt the data and demand money for the decryption key.
This is precisely what happened to Maersk, a worldwide shipping giant, in 2017. Their systems and data were encrypted by NotPetya, and the hacker group demanded 300 USD in bitcoin for the encryption key.
While they could repair machines and build new ones, they faced a major issue. During their investigation, they discovered they didn’t have a domain controller backup. To provide a complete picture, they had 150 domain controllers distributed worldwide.
Fortunately, they discovered that only one domain controller in Ghana remained unscathed. How did it escape the attack? Simple – it had been disconnected from the network before NotPetya got to it.
According to reports, DHCP and Active Directory were badly damaged, the Enterprise Service Bus and vCenter were also affected, 49 thousand laptops were destroyed, all print capabilities were disabled, and file sharing became unavailable. Additionally, 1200 applications became inaccessible.
They experienced a prolonged downtime.
How could they mitigate or avoid this issue? There are two important measures, including implementing strong security mechanisms and having a comprehensive backup and recovery plan that covers all critical points, including domain controllers.
You can read more in the presentation.
Toy Story and not-verified backup
Pixar was working on Toy Story 2 when a professional, intending to perform maintenance and clean up unnecessary files, accidentally deleted essential files from the main disk. That action resulted in the deletion of 90% of the film files.
Upon realizing the mistake, the professional attempted to restore the files from the backup. However, Pixar faced a setback as the backup copies turned out to be corrupted.
The root cause of this issue was the lack of verification for the backup copies.
It took an additional two months to rework the film files. This incident emphasizes that having a backup is not sufficient; it is crucial to verify the backup before attempting a restoration.
Elevate Your Data Security with BDRSuite
We’ve learned valuable lessons from five real-life stories.
Firstly, the absence of a backup and recovery plan, followed by inadequate backup configurations and a lack of verifications, resulted in millions of euros in losses for the companies.
What’s the solution?
Any company size requires a comprehensive backup solution with advanced features, compatible with both Windows and Linux, capable of backing up workloads to both onsite and offsite locations.
This solution should include features such as instant boot, verification, reporting, and notifications.
Look no further than BDRSuite for all your backup needs.
BDRSuite offers versatile backup solutions for your endpoint machines, servers, virtual machines, Exchange servers, databases, and Microsoft 365. Whether you choose to store your backups onsite or offsite is entirely up to you, but best practice dictates a combination of both optimal data security and accessibility. Have you heard about the example of Code Spaces!?
BDRSuite can back up up to 10 machines for free with the full feature set.
I have written an article that explains the 9 key features of why you should use BDRSuite; you can read it here 9 Key Features that Make BDRSuite the Ideal Backup Solution – Tech with Jasmin.
There are also more articles you can read here as well as BDRSuite Archives – Tech with Jasmin.
Conclusion
In this article, we emphasize the importance of having backups and highlight the costs organizations incur due to the lack of a proper backup and recovery plan.
We listed five true stories where companies lost their data, had a long downtime, and various degrees of suffering.
GitLab faced an outage and data loss due to an employee accidentally deleting the database. They didn’t have the most recent backup; some data was lost.
T-Mobile encountered a significant data loss incident affecting 800,000 smartphone users in the US. Their backup was corrupted, they couldn’t restore it.
Code Spaces suffered a complete business shutdown from a cyber-attack. All their backups were stored in one location without an offsite backup.
Maersk, a global shipping giant, had no data backup and lacked domain controller backups, leading them to rebuild everything from scratch.
Pixar, during the production of Toy Story 2, experienced a loss when a professional accidentally deleted film files. They didn’t have a backup.
Many problems share a common root: the lack of a robust backup system.
I encourage you to try BDRSuite. It is easy to use and provides all you need for your business workloads.
Download it from here and feel free to reach out with any questions.
Follow our Twitter and Facebook feeds for new releases, updates, insightful posts and more.
