Announcement Maintenance/Downtime Notices And Announcements

[charlie]

We are back up. Will post more soon on what happened.

 

[charlie]

Regarding the downtime.

I woke up this morning and site was down. Tried to SSH into the server and got zero response. That's double ungood. So I immediately went to the hosting site and was going to restart the VPS and get things back online. Sign into VPS node and get a notice " Your service has been cancelled". Jump over to my email to see if I could see anything from hosting, find an email with "Service Suspension Notification", open it up to see we are shut down for a payment that is due on "06/03/2016". Immediately check date on computer to make sure I haven't slept an entire month.

4 hours after initial ticket submitted, welcome back RayPeatForum.

Needless to say, hosting done goofed, again.

Many stresses had this morning, so for now,

 

[charlie]

We are back online. Welcome back Ray Peat Forum. Full report tomorrow.

 

[charlie]

Quick note. We did lose a few hours of posts. Bright spot is, Saturday is our slowest day so we did not lose much. If you are able to go back and add what was lost, that would be sincerely appreciated. Apologies for any extra work this will cause you.

 

[tyw]

Hehehehehehe. DEVOPS!

Main difference of data center and cloud: in data center you are need 2 of everything, in cloud you are need 20 of everything -- DevOps Borat on Twitter

In devops are 10% get ***t done. Rest are get bull**** done -- DevOps Borat on Twitter

In devops optimist is person which is see glass as 99.999% full -- DevOps Borat on Twitter​

I hate devops ..... hope everything is fine now.

....

 

[charlie]

hope everything is fine now.

@tyw, thank you. :) Yes it looks like everything is sorted out now.

Around 8:00pm Saturday I just get home from a day of sun and fresh air, plan on checking the forum to see if all is well and then get some rest. A couple minutes into checking the forum, it goes dead. Tried to connect to server via SSH, nothing. Oh boy, did they shut us down for non payment again(we have never not paid)? Logged into VPS and try to restart, nothing. We are dead in the water and cannot reach server to even restart. Contacted hosting, hours later I get word that hosting lost contact with the node we were on. Our node was lost in space. Get another report that node was found and communication was established, everything back to normal. Great! Or so I thought.

I come and check out the forum, something is not right. Connect to server, run a check on database and it gives report that it's operational. However, when it tries to write, nothing happens.

So I get back with hosting, tell them database is messed up, can they please look into it. Get a response that they fixed it and all is well. Immediately test to see if things are working, nothing. It's does nothing, but sit there and look like its alive, but its not. Freaky. I am not even able to apply a backup because it simply will not write. Now its about 2am and I realize this isn't happening, need to get rest.

Wake up early, hosting still has no fix, even though they say its fixed. I jump on the bat phone and put out the call to the awesome server guy that has been helping me lately, Matt. Matt just got off holiday and is 2 hours away from home. In the meantime I push hosting to try and figure out a solution, nothing. We are completely dead in the water.

Matt gets home, jumps right in the saddle, off we ride to save the forum. He figures out that database and Innodb are broken, they are not communicating. When the node went offline, it started a chain reaction of corruptions. Thanks hosting.

Matt is all his awesomeness says, "I have an idea!" "Until we can get a new server going, with fresh install, I will host the forum for you on one of my dedicated servers." Yes! The answer is here! So Matt starts the transfer of all the information, it took around 3 hours I think. We have a huge amount of information here, many many gigabytes. To move it around, takes time. So he gives me the OK to switch the DNS over to his server, and here we go! Pull up the forum, it works for a moment, then breaks. We think we find a solution, apply the solution, it breaks worse. After several hours of Matt doing all he can to get our system back online, he calls it the day because he has a prior engagement.

There I am, broken forum, need a fresh install or a complete restore from backups.

Contact hosting, let them know them know we need a new VPS and then apply a full backup restore, or build from scratch and then I can apply the backups I have. Hosting says they will fire up new VPS, and pull their full backup and apply. Great! We are making progress now. New VPS comes up, backup applied, everything looks good, wooohoooo!

The hostings backup was from 4:30am that morning the forum went down. One of our backup solutions is hourly, so I pulled that info, tried to apply it, something is wrong. Some of the tables are missing. Investigate to see whats up, we had changed some things in the database and new tables were added. Little did I know, I needed to "opt-in" to have those tables recorded. I take full responsibility for that snafu even though I feel that's a design flaw of the backup software.

By now we are already around 24 hours of being down. I made the call to go ahead and go with the hostings backup, instead of pulling another source we had which would had only gained us a couple hours. With the speed that hosting was working at , it might have been another 24 hours to get it done.

Needless to say I learned a lot from this experience. This week I will be increasing our backup strategies, applying more and better solutions. If you would like to help with that funding please contact me. The information within the forum and the great work that is done here, I will do all I can to protect it. My sincere apologies for the few hours that were lost.

Matt the server guy did several hours of work, in all his kindness he only charged me 3 hours. I already had one hour paid, so I do owe him 2 hours. If you are able to help with that, please contact me. You will pay directly to him via paypal.

Big thanks for all your patience as we were working through this. Rest assured, if the forum is down that I am doing everything in my power to get it back online.

 

[Such_Saturation]

Is it time to pick a better offer

 

[tyw]

Go oldskool and rely on a simple ansible script and generic $20/month VPS (on something like Linode or DigitalOcean) with a simple cron script for hourly mysqldumps and backup syncs to another machine.

 

[charlie]

Go oldskool and rely on a simple ansible script and generic $20/month VPS (on something like Linode or DigitalOcean) with a simple cron script for hourly mysqldumps and backup syncs to another machine.

 

[charlie]

An updated version of the forum software became available this morning so I took the forum down for around 5 minutes to complete the update. I like to install these updates as soon as they are made available. Update went smooth and we are now on the most current version software version.

Tomorrow, Wednesday 5-25-16, the forum will go down for scheduled work on the server. It should only be around 1 hour of downtime at most. After this work is done I will go more into the details. Sincerely appreciate your patience while this work is being done to better the forum.

 

[charlie]

We are back up and with a new hosting service now.

 

[Such_Saturation]

 

[charlie]

My thoughts exactly.

Is it time to pick a better offer

You sir were correct. It was time to move on.

 

[Such_Saturation]

Cheers charlie and Matt

 

[charlie]

I am aware that the Wiki is down right now. The Litespeed server guys did some work on Litespeed last week and it looks like they locked access to the Wiki by mistake. Since its a holiday they are not responding today. The Wiki should be back up Tuesday at some point.

 

[charlie]

The Wiki is back up.

Ray Peat Forum Wiki

 

[charlie]

You might have noticed that your email notifications were not being mailed, I have been working on the email server the last couple of days making some much needed changes. Email notifications should be going out smoothly now. There might be a small hiccup or two tomorrow when I wrap up the rest of the work. But things should be pretty smooth from here on out.

 

[charlie]

The forum will be going down for a couple of hours Saturday morning around 9AM ET.

The new server node we are on has another client on it who has massive need of resources. Every morning at 8:10am that client has some kind of process that knocks our forum offline for a few minutes. The hosting has offered us a new node. So instead of dealing with this knock down every morning I decided to make the move. We have come this far, might as well keep going.

​

I sincerely apologize for this and hope this will be our last move. But I just cannot live with the fact that the forum will be knocked offline every day for a few minutes, so I must take action and make the move.

 

[heartnhands]

Hey I sincerely appreciate that.

Soon as we figure out how its going to be set up I will let everyone know. :)

 

[charlie]

We did not get the server built in time so the forum will not be going down today. Most likely will schedule it for next Saturday.