Kevin Murray (claystorm) wrote,
Kevin Murray

WHY THE FUCK....(Server RAID Issues) Again.

Orginal Message Posted at 10:54am on June 9, 2005
Damn our server. Damn it to hell!

It is such a piece of shit, it's not even funny anymore.
So, RAID issues again. This time a different drive droped. Went to reboot and nothing. It sits at the RAID card screen and just takes for fucking ever. Finaly after letting it sit for like 10 or 20 min, it finaly goes past that screen. The problem is now it's having issues finding some Windows files it needs, thinking they are missing or messed. So this is going to be a really fun fun day.

So, if you hear of a computer being kicked out a window, and me breaking my foot cus this is a full tower computer, then you will know what happend.


PS, I put the updates and stats behind cut's to prevent this from being too long in peoples friends list. So, if you want to see the all the differnt progress status on things, click the "Updates" link below.

EDIT (06-09-2005 11:04am): And to make things worse, if this had gone down 3 days ago, I would have been able to just redirect them all over to the test server that was running 7.5. But no we upgraded that to 8.0. SHIT!

EDIT (06-09-2005 11:28am): We have a file, that is causing issues. LOL. So, the file that is fucked is c:\windows\system32\config\system. With the fact that it is taking close to 10 min to get thru the RAID shit, it's really slow going. Even just to reboot and hit the BIOS, 10 min. So, I have lots of time to sit and shit my pants. But, I'm hoping that I can repair this copy of windows and have everything just be dandy and fine, well that is what windows says I can do.
MUSIC: Third Eye Blind - Anything
MOOD: Moody, as is the server.

EDIT (06-09-2005 11:35am) : Changed BIOS boot order to 1) Floppy 2)CD 3)HD
MUSIC: Third Eye Blind - Wounded
MOOD: Hopefull?

EDIT (06-09-2005 11:43am) : Booted to Windows Server 2003 CD-ROM, but had issues finding disks.
MUSIC: Third Eye Blind - Deep Inside of You
MOOD: Pissed!

EDIT (06-09-2005 11:56am) : Booted to Windows Server 2003 CD-ROM again, but it can not seem to find Harddisks. Will try 3rd time, I will try something different this time, and if that does not, I will be entering the RAID Config.
MUSIC: Third Eye Blind - The Red Summer Sun
MOOD: Aggrvated

EDIT (06-09-2005 12:41pm) : Well, checked the RAID Config status, and well, it says that 3 out of 4 drives are rebuilding, which in my mind is not a good thing. But for right now, I think we are going to take the wait, and see method. Knowing that a rebuild can take anywhere from 6 to 10 hours, per drive, with a max of 2 drives running at once, we are looking at 8 to 16 hours min of rebuilding. So, it's going to end up being a long end of the week / weekend for Kevin, with little to no sleep. I guess in some ways I happy that its doing this now, and not a few days ago, cus this way, payroll for the week has been done, and next payroll will not happen until next Monday, in which, I should have the system back up (one way or another). Also, I'm glad this did not happen A) when I was in Mexico and B) right before I goto Mexico. That would just be bad, and really fuck me up. So, that is it for now, and there will prob not be another update for a while, since it is in the middle of a rebuild.
MUSIC: Natalie Imbruglia - One More Addiciton
MOOD: Disappointed

EDIT (06-09-2005 4:06pm) : Took a longer then normal lunch, just to get out of the office, and it was good. I mean, it's not like there is anything that I can do here, right now since it's rebuilding. Depending on what happends, I may rebuild the server on Friday if I feel like it has not made progress, or develope a way to use a boot disk, with the RAID driver and see if I can boot into the system that way. Next on my 2-do list is upgrade the old dynamics server (a dell poweredge 2300 server) to mirror the current server. That way, if this shit ever happens again, (_cus twice in 4 months is a little fucked up_) I can just redirect the clients to the backup server and have them up.
I need to price what it will cost to replace the current back-plane, RAID Card, and Harddisks to match the current Dynamics server. But, in other news, the test upgrade to 8.0 from 7.5 went fine, so that is good. Now all I have to do it get our current 7.5 server up and running. LOL. But, I think that since our current server has crashed twice in 2 months, I will have good reasons to spend the money to upgrade the old server.
So, that is it for now. It does look as if my plan to goto the Lake on Saturday is fucked beacuse of this. It will take some act of God to get that server up and running before Saturday. Hell, it's going to take some major work just to get it ready for Monday.
MUSIC: Everlast - Money (Dollar Bill)
MOOD: Drained, and it can only get worse.

EDIT (06-09-2005 5:31pm) : So, I'm getting ready to head home. I did restructure this post a little bit to keep this post smaller in friends list. I will prob start changing the time up in the date area, just to keep it on the top of my firends list. I should be back in 5 hours, to check on things, and we will see from there. If nothing has changed, then its back home for about 4 to 5 hours then back here again. So, that means 2 or 3 am. So, loads of fun to come. Wish me luck.
MUSIC: No Music, Just got done with Everlast (Whitey Ford Sings The Blues)
MOOD: Sleep, Who Needs That?.

EDIT (06-09-2005 10:02pm) : Well, back in the office. It's about 10pm and it still says it rebuilding. No fucking clue as to when it will be done which is hard on me, cus it means it could finish at 2am. Since it can not get to windows, I would have no fucking clue when it is done. I don't think I will hang around that long, as I can go back home and get some sleep, and try and get back in the office at 4am. That will make it 6 hours since I last checked on it, so that should be good. It also means, that if I hump my ass back home, I could get almost 5 hours of sleep! Then I could head back, and get another couple hours of sleep. But this could all change very quick if something starts to work, or just fucking fails. Right now I'm hoping for the working idea.
MUSIC: Moby - Lift Me Up
MOOD: It's no longer "Sleep, Who Needs That?" more like "Sleep, What is that?".

EDIT (06-10-2005 12:16am) : Well, you know back yesterday at 10PM, when I said I was not going to stick around that long, well I lied. But, now I think I am ready if I have to take a 2x4 to the server, and redo it all. I have all the historic databases so, makes that easy, plus, I'm hoping the backups ran right, as I did not have time to test them before the server fucked up. I have all the stuff needed to do a fresh dynamics install. I should have the codec for the DB somewhere around here (just found it), so I'm now all ready to go, with a fresh 7.5 install at noon if needed. I will bring JP backdown at noon, since he has gotten more sleep then me. I'm going to need someone to watch my back as I pull this little sun of a bitch back up. So, that should be load and loads of fun. Well, me going home now to sleep. I will prob just come back at normal time of 8 or 9am. Cus, to get back in by 6am is just too much.
MUSIC: Matchbox Twenty - Black and White People :: Matchbox Twenty - Crutch
MOOD: Super Nerd, He does not need sleep, he has Jolt Cola and Mt Dew. LOL.

EDIT (06-10-2005 9:42am) : So, I did get some sleep. I finaly got back home around 1am, slept till my alarm went off at 7:15, then just sat in bed for about 30 to 45 min cus I just was too tired to get up. Finaly got out of bed, took care of the dog, got dressed, feed the fish and got out of there. Went over to my apartment and picked up a CD I might need and also grabed my music CD colletion as I bored with the CD's I have in my car, and the MP3's I got here on my office computer. Went down to Costco to get some gas for my poor car, beacuse the gas gage was almost resting on empty, and I had almost 285 on my odomitor. 300 is about the max my car will go on a tank of gas, and that is out on the open road and not city driving. Then poped up the hill from Costco, and went to Starbucks to refill my need for caffine. At starbucks I also bought coldplay's new CD, so we will be poping that in very soon, and seeing what that is all about. I finaly got in to the office around 9:15 to 9:30.

So, I checked the server, and well, it still looks the same as lastnight. So, I thinking it's going to be a format and redo (as much as I don't want to do that). The good part is I have found 9/10 of the shit I need to pull it off, so that is good. If I do end up formating, I will replace the drive in Channel 0 as that is the drive that orignaly faild this time. I'm also tempted to just fucking replace all the drives, minus the one I had to buy the last time this happend. That way, if this happens again, I would know its the RAID card, or something else, but not the media, and could be to plop in new RAID card. So, I am starting to see a trip to CompUSA in my future. So, it's also offical now, that my plans for the weekend have been fucked. As I said before, I was scheduled to goto the lake with some friends on Saturday, and well, it really sounds like that is out the door. I need to call JP (Guy who did my job, who still helps sometimes), and get his thoughts on all of this, and see if he agrees with me about replaceing all the drives if I have to format. Well, that is my update for now, and really considering not much has changed, this has been a really long update.
MUSIC: Third Eye Blind - Losing a Whole Year :: Third Eye Blind - Jumper
MOOD: Starbucks, the stuff IT People are made of...

EDIT (06-10-2005 1:42pm) : Well, Well, Well, Around noon, I gave up trying to save the data on this disks. Its a gonner, so I took a trip up I-40 to CompUSA, and picked up 2 new Maxtor SATA ULTA 150, 250 GB drives, to match the other two new drive that got put in the server, so now ever drive in the server is new. So, if this happens again, we know that something else is up. Also, this gives me some drives to work with in a differnt computer. Then I went in, deleted the RAID config, redid it, and started the Windows Server 2003 install. So, that is running now. Next up is after the OS gets reinstalled, is to get the rest of the drives setup. Then add the support for the tape drive back in and bamm, we are going to kick it up a notch by reinstalling the backup software. LOL. Then I can start a fresh install of SQL Server 2000, and Dynamics 7.5. Once that is done, add Wennsoft into the mix, and we hope it will all run. Then its just a matter of restoring data back into the SQL server, and we are there (I make it sound so much faster and easer then its going to be). Right now, If I could just stop working for the pay period, I would have 40 hours in. So, my best guess is that this is going to be a 60 hour week (cus that just screams fun). But everything from here on is at time and half, so 18 dollars an hour. Well, now windows is back up and running, SO I GET TO SAY THIS:

IT'S ALIVE, ALIVE!!! (Ok, only like 10.125% way there, but still, it's progress.)

MUSIC: Goo Goo Dolls - Dizzy :: Goo Goo Dolls - Broadway
MOOD: No beer and sleep makes Kevin go something something.... Go Crazy? DON'T MIND IF I DO!!!!

EDIT (06-10-2005 3:11pm) : Windows installed and patch and running fine. Got the Anti-virus software installed. Got drives installed for everything. Basicly, I have a server again. I also got the backup software installed, so that is importing Wednesday nights backups right now, so, I can start the really fun process of getting all our shit back. I'm also going to import the P2003 company that we fixed on the test server, cus I'm not dealing with that shit again, and since that company has be reimported anyway. So, now that I know the backup is importing, I feel a little better. Still not out of the woods. I fell much better when I get that data back on the comptuer its self. So, that is the news for now, not that anyone is reading this but me. I'm also glad my boss is gone for the day, he left last night for Vegas, so it allows me to play my music way too loud. It's prob the only thing keeping my sorry ass going right now, as I have not had anything to eat or drink since about 10am this morning. I will be taking a trip to Costco, for soda, and RedBull. Cus, I will not be able to live on sleep alone.
MUSIC: Dave Matthews Band - Crash Into Me :: DMB - Too Much
MOOD: Just ready to pass out, but no!

EDIT (06-10-2005 7:32pm) : So, I went to Costco, my aprtment & the brugge's house. Did everything that I need to do in my personal life for right now. Right now, it's looking as if I still might get down to the lake this weekend, which would kick some major ass. I also forgot how fast the SQL server install can go when you have a very kick ass processer in it. I forgot that our server has a quad in it, which rocks my ass off right now. So things that on the test server took 40 min to one hour, are taking 20 min or less. Right now, SQL Server 2000, Service pack 3a is installing. Once that is done, dynamics will go on, and then it will be about time to meet my parents for dinner. When I get back, I will create all the historical companys (1997 to 2004), the CACI database, and the Dynamics database. Then it will be time to install Wennsoft and at which point, I can set it to upgrade each company, and leave for the night. Then all that is left is the reporting of all the company DB's and I'm done for the most part. There are a few cleanup items that I need to do, but really, it will be in working condition. So, I'm guessing a few hours on Saturday, like 1 to 3, then I can head to the lake. Also, if I really wanted too, I could put some of this off until Sunday, but I prob will not do that. So, that is the news for now. It's looking better and better, and for that I give thanks to the Computer Gods.
MUSIC: Dave Matthews Band - Rapunzel :: DMB - The Last Stop
MOOD: Kinda happy and excited!

EDIT (06-10-2005 9:27pm) : Back from dinner. It's creating the companys, at around 4 min a pop, so its going fast. So, the companys should be done creating around 10pm or there abouts. So, thats good. Once that is done, I have to finish up the GP install by doing all the Service packs, then install Wennsoft. Then it's just going about the waiting game, which I plan on sleep thru. So, I should be out of her around 11pm if all goes well. So, we shall find out.
MUSIC: Dave Matthews Band - Pig :: DMB - Spoon
MOOD: Tired, but its all good!

EDIT (06-10-2005 10:20pm) : It finished creating all the companys. Average time for each company was 4 min. Much faster then the 10 or so min on the test server. Again, its the quad processor we got running in that beast. It is really looking like I will get to spend the day at the lake, which is soo fucking cool. I will spend Sunday doing mop up work, like fixing user logins and bull shit like that. So, it will be done by Monday. Whats really funny, is that I have almost rebuilt our production server in one day. I did not know I could rebuild the server in one day.

My plan is, after a few days off and some time to reflect, I will do a full write up on this, and come up with solutions to keep this from happing again.
** My goal is to be able to lower downtime to 1 hour at the max.
** This means, having two production worthy servers, both able to run at a moments notice.
** This also means having the Maint Plan for SQL copy out the database a few times each day, not just once a day.
** I'm also going to change the backup process to include more static files, and/or start a once-a-month full server backup (meaning: ever little file on the server will get backup).

Also, now that Wennsoft is updating the Db's fine, I'm going to head home, see what the doggy has done, and head to bed.
MUSIC: Garbage - Dumb :: Garbage - Push It
MOOD: Tired, but its all good!

EDIT (06-11-2005 10:03pm) :


So, if you can't tell, the server is fixed in time for me to goto the lake. Wennsoft upgraded the tables fine, and then I came back, and imported the data, and that all went well. We lost about 1.5 days with of data, but that is not bad, and as I outlined above, I will fix that so, that it is much lower. There were things that I could have done if I had been thinking, but that is in the past, so now I have to move forward, and start a new. Right now the backup is running, and telling me it's about 30 gb, and this is everything, so thats cool. So, I have not restored the login's yet, cus I have stoped the SQL server to run the backups and all the fun stuff. Anyway, time to run away to the lake for some very much needed relaxing.
MUSIC: Mark Schultz - Do You Even Know Me Anymore :: Mark Shultz - Running Just to Catch Myself
MOOD: So happy, very very very very very very very happy.

Drive Droped: 06/09/2005 - 8:08 am
Noticed Error: 06/09/2005 - 10:15 am
Rebooted Server: 06/09/2005 - 10:30 am (downtime started)
Edit BIOS Settings: 06/09/2005 - 11:35am
Rebuild Started: 06/09/2005 - 10:45am - 11:45am (not sure, but somewhere in this time frame)
Server Rebuild Started: 06/10/2005 - 1:00pm
SQL Server Installed: 06/10/2005 - 7:15pm
Dynamics Instlled: 06/10/2005 - 7:45pm
Started Creating Companys: 06/10/2005 - 9:25pm
Finished Creating Companys: 06/10/2005 - 10:06pm
Started Installing Service Packs: 06/10/2005 - 10:10pm
Finished Installing Service Packs: 06/10/2005 - 10:15pm
Installed Wennsoft: 06/10/2005 - 10:16pm
Started Wennsoft table upgrades: 06/10/2005 - 10:17pm
Finsihed Wennsoft table upgrades: 06/11/2005 - 2:00am
Started import of data: 06/11/2005 - 9:36.50am
Finished import of data: 06/11/2005 - 9:54.00am
Server Rebuild Finished: 06/11/2005 - 10:00am (downtime ended)
Restored GP/Dynamcis Logins: 06/12/2005 - 4:00pm

TOTAL DOWNTIME: 47 hours, 30 min
  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.