32 Hours of Posts Lost - July 2019

[iacas]

Roughly 32 hours of posts were lost when the RAID 10 array failed in spectacular fashion last night (just before the backup for the day's posts could be completed).

This means that any posts made from about 2:05am on the morning of July 23, 2019 until the site was restored at about 11:05am were lost. New member registrations, PMs, files uploaded, etc. are lost.

We successfully restored the site (all files, the databases, etc.) from the backup made early in the morning on July 23, and ask that you please re-create any posts that you felt were worthwhile that you lost in the 32 hours or so that we no longer have.

I'll be here in this topic to answer any other questions you might have.

Thank you.

[Jeremie Boop]

Ouch, gotta love the life of IT sometimes. We moved from nightly to a more granular hourly backup, but with our work losing 32 hours of data would result in probably a week or more of people redoing their work while trying to keep up with current work.

[David in FL]

Ugh.  So that’s what happened.

Sorry to hear that.  I can only imagine the scrambling you’ve been doing.

[iacas]

32 minutes ago, Jeremie Boop said:

Ouch, gotta love the life of IT sometimes. We moved from nightly to a more granular hourly backup, but with our work losing 32 hours of data would result in probably a week or more of people redoing their work while trying to keep up with current work.

Yeah, hourly would, IMO, be massive overkill for a site like TST, but I'm sure I'll review the policies and backup settings.

At the end of the day, I don't know that anyone said anything too important. And 32 hours is a pretty small amount of time, all things considered.

[Vinsk]

25 minutes ago, iacas said:

I don't know that anyone said anything too important.

I can certainly say, I did not.

[iacas]

I'm troubleshooting an error that occurs when you attempt to upload a file.

[Eric C]

I agree, Vinsk probably didn't.  OTOH, I'm sure I had said many brilliant things that I'll now never remember ..

Sorry for your troubles, Erik.  For a RAID 10 array to have taken a crap it must have been spectacular, and the stuff of nightmares for a guy like me.  As far as backups are concerned, most modern solutions backup at the block-level these days, and will also dedupe the data, which helps reduce both storage requirements and the time that making a backup takes.

[iacas]

15 minutes ago, Eric C said:

I agree, Vinsk probably didn't.  OTOH, I'm sure I had said many brilliant things that I'll now never remember ..

Sorry for your troubles, Erik.  For a RAID 10 array to have taken a crap it must have been spectacular, and the stuff of nightmares for a guy like me.  As far as backups are concerned, most modern solutions backup at the block-level these days, and will also dedupe the data, which helps reduce both storage requirements and the time that making a backup takes.

Yep to all of that. The backups are to a pretty recent standard and, yes, they're incremental.

[boogielicious]

35 minutes ago, Vinsk said:

I can certainly say, I did not.

I posted the cure for the shanks and it was brilliantly fool-proof and permanent. But I can't remember what I said now. Ah, old age. 

[Vinsk]

5 minutes ago, boogielicious said:

I posted the cure for the shanks and it was brilliantly fool-proof and permanent. But I can't remember what I said now. Ah, old age. 

Lol! Like when Lodka ( tv show Taxi) is being Alex at the shrink’s office and says he’s discovered the secret to Alex’s happiness and then suddenly reverts back to Lodka.....

[boogielicious]

1 minute ago, Vinsk said:

Lol! Like when Lodka ( tv show Taxi) is being Alex at the shrink’s office and says he’s discovered the secret to Alex’s happiness and then suddenly reverts back to Lodka.....

LOL! I am old enough to remember that!!

 

[Vinsk]

18 minutes ago, iacas said:

Yep to all of that.

Sighhhh....burned again.

35 minutes ago, Eric C said:

I agree, Vinsk probably didn't.

😐

[CarlSpackler]

Dang it! Now I need to repost my plan to practice hard and qualify for the senior tour in a few years. 😡

🤣🤣

[Wally Fairway]

1 hour ago, CarlSpackler said:

Dang it! Now I need to repost my plan to practice hard and qualify for the senior tour in a few years. 😡

🤣🤣

Are you planning on putting in 10,000 hours of practice?

[CarlSpackler]

54 minutes ago, Wally Fairway said:

Are you planning on putting in 10,000 hours of practice?

Nope. 1500 should do it. I'm a natural ball striker with a strong mental game. I have impeccable putting skills, but 1000 of those practice hours will be solidifying my 3-6' putts. You can't win on tour if you miss those. 😉

@iacas - I'm glad you got the site back up. It must have been a long night.

Edited July 24, 2019 by CarlSpackler
Spelling

[iacas]

10 minutes ago, CarlSpackler said:

@iacas - I'm glad you got the site back up. It must have been a long night.

Nothing happened until about 2:04am today. In fact, it may have been the backup process that runs at 2:05am that may have prompted the problems to finally "break" something, so really it's just been since I woke up at 7am that I've been doing this. And by 11:00 they'd rebuilt the RAID array and got it back online, and I've just been tacking down a few little things (like uploading through the site) since.

[Zippo]

I'm a retired Systems Administrator and I can guarantee you that RAID 20 is twice as good as RAID 10... 

[Pretzel]

1 hour ago, Zippo said:

I'm a retired Systems Administrator and I can guarantee you that RAID 20 is twice as good as RAID 10... 

The real pros all use RAID 9001 to go with their fancy ISO certifications.