Jump to content

Server Maintenance Tonight


Real Deal
 Share

Recommended Posts

  • Owner
Over the past couple of weeks we've had some performance issues with our servers mostly during the over-night period due to a couple of factors. I want to detail why these issues have happened, what we're doing to resolve them, as well as going over some scheduled maintenance for this evening. We always do our best to be as transparent and honest as possible and as such we're not going to keep any details from you.

 

As you may already know, we run CloudLinux on our servers to help keep things stable and to prevent the over-usage of some accounts affecting everybody else. The system is generally very solid and does a wonderful job of doing what it's supposed to do. Recently we, at the advice of CloudLinux, upgraded to a newer version of the system that was supposed to offer increased performance and reliability when in fact it turns out that we ran into a couple of serious issues with the new version of the software that were exasperated by our R1Soft backup system which can be very intensive all on it's own.

 

During normal use the new version of CloudLinux we were running on the servers performed well and did indeed give some performance gains especially when it came to hard drive access speeds on the server. The issues arose when our R1Soft backup system was run to back up your data and protect you from hardware failure and other data loss causes. This backup process tends to be very intensive as it needs to scan every bit on the disk to look for and record any changes to the data made to keep an up-to-date backup of your data. When this backup process ran, without any warning the servers locked up forcing us to not only reboot the systems but to also abort the backup process that had already been running for an hour or more. We have downgraded our CloudLinux installations across our servers back to a version we operated with for a very long time without issues and we don't anticipate having any major issues with the backup systems from this point forward.

 

We have always kept backups of our servers and we tend to use those backups fairly regularly to fix issues for customers such as when somebody accidentally deletes a file or drops the wrong database or database table. This process due to being very intensive does cause a few minutes of extremely slow performance every night when it runs however this usually clears up in 2 to 3 minutes. We realize that this can be annoying at best however it's the cost of keeping up to date off-server copies of all data and databases.

 

Due to all of the issues we've faced while performing backups over the last week we feel that we cannot rely upon the quality of the current backups stored of the systems and as such we need to perform fresh backups of all servers to ensure that if the need does arise to use the backups to restore data, that the data restored is accurate and problem free. We evaluated our options as far as when to run these fresh backups to cause the least impact in service performance for our customers and we determined that Sunday nights are the best times to do such intensive processes. The next decision was whether we should wait a week or two to perform this fresh backup or to go ahead and do the backup as soon as possible.

 

We have chosen to perform the seed backups tonight as we feel a week without reliable backups is far too long as hardware failure is not something that can be expected or planned for. We do run redundant disk arrays in our servers to help protect against drive failure (up to 2 drives can fail per server without data loss) however running redundant arrays is not a substitute for reliable daily backups of the data. This process we expect to take between 6 and 8 hours per server and once we've performed this fresh backup the backup system should not cause any major performance degradation in the future beyond the expected two to three minutes per night that it takes for the backup system to spin up per server and get started.

 

You may be frustrated with the issues we've faced on a couple of our servers over the last two weeks and we're right there with you on that frustration. We have posted a thread on the forum with the information contained from this email as well as some additional details about the backup process for this evening. If you you have any direct questions you would like to ask you are welcome to respond to this email or to visit the forums and publicly post your thoughts / questions / comments / suggestions as well.

Link to comment
Share on other sites

  • Owner

Maintenance could be tonight instead, not sure. I didn't see any downtime last night, so I'm assuming all of it will be checked tonight. Who knows at this point.

 

Hang in there, fellas. :)

Link to comment
Share on other sites

  • Owner

The maintenance IS tonight. They are performing backups, which is very important (we know this more than most, right?), and they want to be sure they have everything backed up ASAP.

 

We should be good tomorrow, if not late tonight (things are running a bit faster right now, anyway).

Link to comment
Share on other sites

 Share

×
×
  • Create New...