This webpage is dedicated to one of the suckiest things I've ever had to do during my career as an IT Defender. I'm writing this to record it for posterity, and whining rights.

On Monday, November 14th, I came in to the office to find that the differential backups from Sunday night all failed. Some initial front-end troubleshooting revealed a hardware issue with the LTO, and so the fun began. But first, a word about my configuration.

We have a Windows 2003 Enterprise Server SP1 (on a Dell PowerEdge 2550), which is both the master and the media server running Veritas Netbackup 5.0MP5. The LTO is a Dell PowerVault 128T with one robot and two SCSI drives. Backups run over a private gigabit network, in our machine room.

The system error message that was coming back from the master/media server went as follows:

"App: E 'Mon Nov 14 08:41:29 2005': NetBackup AVR Daemon - " cannot get serial number for HPUltrium1-SCSI0 (device 0, __._Tape0) "
App: E 'Mon Nov 14 08:41:29 2005': NetBackup AVR Daemon - " DeviceIoControl() error on bus 0, target
0, lun 0: The device is not connected. "
App: E 'Mon Nov 14 08:41:26 2005': NetBackup TLD Daemon - " TLD(0) [988] timed out after waiting 305 seconds for ready, drive 1 "
App: E 'Mon Nov 14 08:41:26 2005': NetBackup TLD Daemon - " cannot get serial number for HPUltrium1-SCSI0 (device 0, __._Tape0) "
App: E 'Mon Nov 14 08:41:26 2005': NetBackup TLD Daemon - " DeviceIoControl() error on bus 0, target
0, lun 0: The device is not connected. "

I was not able to up the drive using the Netbackup Admin console (or the bp commands).

I went over to the machine room, and discovered that this was indeed a hardware problem. Inventorying the robot yielded a hardware error. I tried powercycling the LTO, which also did nothing.

I called Dell, and they had some basic troubleshooting ideas, including a very ghetto "stick a paperclip into it" fix. I tried all of their suggestions with no luck, but here they are anyway.

Suggestion #1: Eject the tape using the remote management card web interface.
Suggestion #2: Eject the tape through the front panel.
- From the Drive and Tape Operations menu, select Unload Tape from Drive.
- Use the [-] or [+] keys to select the drive you want to unload.
- Select Unload to move the tape from the drive module to the original slot where the tape was located. The tape is automatically rewound before it is unloaded.
- Press the eject button on the front of the tape drive which has the stuck tape.
Suggestion #3: Power off and power on the PowerVault 128T.
Suggestion #4: With a paper clip, press the reset button on the front of the tape drive.

(It should be noted that the paperclip reset is extremely difficult to do with the 128T architecture. The drive has to be plugged in, which means that it is all the way in the back of the LTO. You have to take out all the magazines, and blindly poke at the front panel hoping to get to the pinhole. Having a flashlight helps, but not that much).

Finally, I decided that the only thing left to do was to take the darn thing apart.

Here is the step by step, I followed to get the tape out.

1.

Important Note: The tape drives are hot pluggable. Ensure that a tape is not being loaded or removed.

Use the library front panel to take the drive off-line as follows:

  1. From the Main menu, select Service.
  2. Enter the service password then press the button below OK.
  3. Select the button below More.
  4. Select Online Drive Repair.
  5. Select the drive with the [-] or [+] keys, and press OK.
  6. Select Put Drive Off-line.

2.

Use a flat blade screwdriver or a Phillips-head screwdriver to loosen the two thumbscrews at the top and bottom of the tape drive (s). Finish loosening the thumbscrews with your hand.

3.

Using the tape drive handle, pull the tape drive(s) out from the back of the library.

4.

With the front of the tape drive facing you, remove the charcoal plastic cover from the right side of the tape drive sled to expose the interface board.

 

 

 

5.

Remove the screw that secures the interface board to the tape drive sled.

 

 

6.

Disconnect the three cables from the rear of the interface board.

 Note: The second time that I had to do this, I found that disconnecting the cables at the back of the board wasn't really necessary. If you are careful to not rip them out as you are spinning the wheel, it's probably better to just leave them as they are.

7.

Remove the interface board from the tape drive sled.

 Note: Again, as in step 7, this is not necessary if you are careful. I would suggest just working very carefully, and leaving the board in place. You only need to tilt it away to get at the thumb wheel.

8.

Turn the thumb wheel by hand towards the back of the drive as far as possible. There is no hard stop, but the wheel will begin to spring back. This positions the grabber into the parking position. The grabber may already be positioned in the parking position.

 

9.

Rotate the front reel motor using a T-10 Torx driver clockwise until you hear a click. The front reel motor will stop rotating clockwise when the tape is completely unthreaded.

 Note: I was not able to do this without removing the plate. I suppose if you had a different screwdriver, it might work. But my screwdriver got stuck before the Torx piece could reach its destination. Removing that plate voids the Dell warranty on the device. Our warranty already expired, so I just ripped it off.

10.

With the front of the tape drive facing you, turn the thumb wheel, on the right side of the tape drive, by hand towards the front of the drive until the cartridge is full ejected. There will be some resistance as the cartridge comes free of the magnet. 

11.

Remove the cartridge from the tape drive.

 

12.

Celebrate!

 

irina@columbia.edu