metareplace same drive

Hi,

Am running 2 x 5 fc drives in raid 5 config, one drive goes down
(maintenance mode) and needs to be replaced.

Am pretty sure that on an earlier version of Sol10, I could remove the
drive, insert another newfs'd and fsck'd drive in the same slot, reboot
-r to see the drive, metadb the new drive and do:

metareplace d0 c8t4d0s2 c8t4d0s2 for example

This worked and have it documented in a logbook, but with the latest
sol10 version, doing the same reports an error: old and new drives are
the same.

The only way I could get this working was to put the new drive in a
different slot and metareplace from there.

It's fairly fundamental that you may want to replace a drive to the same
slot, so what's the accepted method ?...

Regards,

Chris

On 2009-09-12, ChrisQ <> wrote:
> Hi,
>
> Am running 2 x 5 fc drives in raid 5 config, one drive goes down
> (maintenance mode) and needs to be replaced.
>
> Am pretty sure that on an earlier version of Sol10, I could remove the
> drive, insert another newfs'd and fsck'd drive in the same slot, reboot
> -r to see the drive, metadb the new drive and do:
>
> metareplace d0 c8t4d0s2 c8t4d0s2 for example
>
> This worked and have it documented in a logbook, but with the latest
> sol10 version, doing the same reports an error: old and new drives are
> the same.

O.K. First -- were you using FC drives previously?

I don't have experience with the meta* commands, but with zfs,
it would be:

zpool replace -f pool-name c1t10d0

(as long as it is in the same slot).

But -- with FC drives, you do need to reconfigure the drive
table, which would be done (without rebooting) with the command:

devfsadm -c disk -C

(with the "-c disk" meaning to limit the operations to disk devices
only, and the "-C" meaning to clean up old entries.

The FC disk entries include the drive's WWN in the tables.

================================================== ====================
> ls -laF /dev/rdsk/c1t10d0s0
lrwxrwxrwx 1 root root 74 Jun 15 21:57 /dev/rdsk/c1t10d0s0 -> \
.../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000020379007a4,0:a,raw
================================================== ====================

Note the long number after "/ssd@w" which (until the ',') is the WWN,
which can be found printed on each drive's label. This is unique to
each drive.

Note also that zfs reconfigures the drives to a new
partitioning, with slice 0 being the slice in use, and slice 8 (not
normally present) being used for keeping special information about the
drive and the array of which it is a part.

================================================== ====================
Part Tag Flag First Sector Size Last Sector
0 usr wm 34 34.18GB 71670951
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
8 reserved wm 71670952 8.00MB 71687335
================================================== ====================

> The only way I could get this working was to put the new drive in a
> different slot and metareplace from there.

Again -- I don't use the meta commands, but with zfs the
"zpool replace" does it without problems. And why are you still using
the meta commands given how well zfs works and how well integrated it is
in Solaris 10. (Granted, it may be that you are stuck with an existing
setup -- but if you get a spare tray for extra drives, you can migrate
over to zfs.) I don't know whether the meta commands allow this, but I
have migrated a zfs array from a set of 18 GB SCA to 36 GB FC one drive
at a time, and when the last drive was replaced, the size suddenly
jumped up to reflect the larger drives. I have subsequently moved the
same array from the 36 GB drives to 72 GB drives, again one at a time.
All without having to shut the system down. Some of these days, I need
to shut it down, so I can remove the HVD SCSI card from the system which
was used on the tray which handled all the SCA drives.

Obviously, you can't migrate from the meta array to the ZFS
array as easily, since there is no provision for mixing the two systems.

> It's fairly fundamental that you may want to replace a drive to the same
> slot, so what's the accepted method ?...

Yes -- I think that you should be able to do it with the
devfsadm command I described above -- and without having to reboot.

Good Luck,
DoN.

--
Email: <> | Voice (all times): (703) 938-4564
(too) near Washington D.C. | http://www.d-and-d.com/dnichols/DoN.html
--- Black Holes are where God is dividing by zero ---

You use "metareplace -e" when you replace a disk in the same slot, or
just re-enable a disk that for some reason has gone to maintenance
without really being broken.

See "man metareplace"

But with FC disks you probably need to use cfgadm to get the hardware
recognized correctly first.

DoN. Nichols wrote:
> On 2009-09-12, ChrisQ <> wrote:
>> Hi,
>>
>> Am running 2 x 5 fc drives in raid 5 config, one drive goes down
>> (maintenance mode) and needs to be replaced.
>>
>> Am pretty sure that on an earlier version of Sol10, I could remove the
>> drive, insert another newfs'd and fsck'd drive in the same slot, reboot
>> -r to see the drive, metadb the new drive and do:
>>
>> metareplace d0 c8t4d0s2 c8t4d0s2 for example
>>
>> This worked and have it documented in a logbook, but with the latest
>> sol10 version, doing the same reports an error: old and new drives are
>> the same.
>
> O.K. First -- were you using FC drives previously?
>

Don,

Thanks for the detailed reply. Have been using the same 10 disk fc box
for a couple of years now. It's an old emc box with all the drives 73 gb
hh and s/hand. I bought a whole bunch of drives and initially used a pc
running the seagate disk tools to read the logs. Most of the drives were
only 10's or a few hundred hours at most. This is the second drive that
failed, though it may just need reformatting.

The system is still on ufs as haven't had the time to get into zfs as
yet and want to completely understand the implications before updating.
Not to mention the time involved. So, its either the management console
or meta* commands. I'm pretty sure that I could do a metareplace into
the same slot last time, so stumped as to why it didn't work this time...

Regards,

Chris

On 2009-09-17, ChrisQ <> wrote:
> DoN. Nichols wrote:
>> On 2009-09-12, ChrisQ <> wrote:
>>> Hi,
>>>
>>> Am running 2 x 5 fc drives in raid 5 config, one drive goes down
>>> (maintenance mode) and needs to be replaced.
>>>
>>> Am pretty sure that on an earlier version of Sol10, I could remove the
>>> drive, insert another newfs'd and fsck'd drive in the same slot, reboot
>>> -r to see the drive, metadb the new drive and do:
>>>
>>> metareplace d0 c8t4d0s2 c8t4d0s2 for example
>>>
>>> This worked and have it documented in a logbook, but with the latest
>>> sol10 version, doing the same reports an error: old and new drives are
>>> the same.
>>
>> O.K. First -- were you using FC drives previously?
>>
>
> Don,
>
> Thanks for the detailed reply. Have been using the same 10 disk fc box
> for a couple of years now. It's an old emc box with all the drives 73 gb
> hh and s/hand.

O.K. So that did not change.

> I bought a whole bunch of drives and initially used a pc
> running the seagate disk tools to read the logs.

You can read the logs with smartctl, and using smartd, be given
e-mail notices any time the bad sector list grows. It does not come on
Solaris 10, but you can download the source and compile it.

================================================== ====================
== HOME ==
The home for smartmontools is located at:

http://smartmontools.sourceforge.net/
================================================== ====================

And the version which I am running is 5.38, which was the latest version
earlier this summer.

Here is an example of the outout of smarctl -a /dev/rdsk/c1t14d0s2

================================================== ====================
Katana:csu 21:37:26 # smartctl -a /dev/rdsk/c1t14d0s2
smartctl version 5.38 [sparc-sun-solaris2.10] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: SEAGATE SX3146807FC Version: D010
Serial number: 3HY0TDL4 7343QSVU
Device type: disk
Transport protocol: Fibre channel (FCP-2)
Local Time is: Thu Sep 17 21:38:20 2009 EDT
Device supports SMART and is Enabled
Temperature Warning Enabled
SMART Health Status: OK

Elements in grown defect list: 0
Vendor (Seagate) cache information
Blocks sent to initiator = 6914983988329
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 39773.07
number of minutes until next internal SMART test = 104

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 20328350 3 0 20328353 20328353 61762.251 0
write: 0 0 2 430457 430840 10032.625 0
verify: 247574 0 0 247574 247574 826.201 0

Non-medium error count: 61440

SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 37504 - [- - -]
# 2 Background short Completed - 37338 - [- - -]
# 3 Background short Completed - 37172 - [- - -]
# 4 Background short Completed - 37006 - [- - -]
# 5 Background short Completed - 36840 - [- - -]
# 6 Background short Completed - 36674 - [- - -]
# 7 Background short Completed - 36508 - [- - -]
# 8 Background short Completed - 36347 - [- - -]
# 9 Background short Completed - 36181 - [- - -]
#10 Background short Completed - 36015 - [- - -]
#11 Background short Completed - 35849 - [- - -]
#12 Background short Completed - 35683 - [- - -]
#13 Background short Completed - 35517 - [- - -]
#14 Background short Completed - 35351 - [- - -]
#15 Background short Completed - 35185 - [- - -]
#16 Background short Completed - 35020 - [- - -]
#17 Background short Completed - 34854 - [- - -]
#18 Background short Completed - 34688 - [- - -]
#19 Background short Completed - 34522 - [- - -]
#20 Background short Completed - 34356 - [- - -]
================================================== ====================

So -- 39,773+ total hours on that drive -- which happens to have
a ufs filesystem instead of being part of a ZFS array. I don't have
enough 146 GB drives to do things that way yet. :-)

IIRC, I used the Sun development set as part of the latest
Solaris 10 set of DVDs (Studio 12, IIRC).

To get the automatic notification, you will want to edit
/etc/smartd.conf to specify the disks which you want to monitor and the
level and frequency of testing.

> Most of the drives were
> only 10's or a few hundred hours at most. This is the second drive that
> failed, though it may just need reformatting.

Get smartmontools, compile them, and see what they tell you about
that drive (ideally in another slot, if you have a spare slot
somewhere).

> The system is still on ufs as haven't had the time to get into zfs as
> yet and want to completely understand the implications before updating.

Understood. I played with the latest version on both a Sun
Fire 280R, and a Sun Blade 1000 before I reinstalled on my Sun Blade
2000. (With a zfs raid array, export it first, and then import it to
the new OS.) The new OS is installed on a pair of 73 GB FC-AL internal
disks, while the old one was installed on a 36 GB FC-AL, with overflow
on a 146 GB FC-AL.

Some of these days I'll take down my main server and do the
same, but not yet. I really need a time when both my wife and I won't
need to access the world for quite a few hours. :-)

> Not to mention the time involved. So, its either the management console
> or meta* commands. I'm pretty sure that I could do a metareplace into
> the same slot last time, so stumped as to why it didn't work this time...

Do you remember whether you did a devfsadm before the
metareplace last time when it worked? That may be what is needed --
though if you did a reconfigure boot that should accomplish the same
thing.

And I think that I mentioned before -- zfs at least does not
care what slots the drives from its arrays are in -- it finds them and
uses them wherever that happens to be. (But be careful when you replace
to not accidentally expand the raid array to one of the hot spares
instead of just letting it use the hot spare on its own until you do the
"replace" with the new drive. I accidentally did that once, and there
was no way to back off using that extra drive other than backup of the
data to tape and rebuild the array. You can tell it to let go of a
drive used in a mirror, but not to back down from a 6-drive RAID array
to the desired 5-drive one. :-)

Good Luck,
DoN.

--
Email: <> | Voice (all times): (703) 938-4564
(too) near Washington D.C. | http://www.d-and-d.com/dnichols/DoN.html
--- Black Holes are where God is dividing by zero ---

DoN. Nichols wrote:

> You can read the logs with smartctl, and using smartd, be given
> e-mail notices any time the bad sector list grows. It does not come on
> Solaris 10, but you can download the source and compile it.
>


Don,

Thanks - downloaded and built smartmon and from a typical drive,
smartctl tells me:

darkstar: smartctl -a /dev/rdsk/c8t3d0s2
smartctl version 5.38 [sparc-sun-solaris2.10] Copyright (C) 2002-8 Bruce
Allen
Home page is http://smartmontools.sourceforge.net/

Device: SEAGATE ST373405FC Version: 0005
Serial number: 3EK0FT4Y00007203L6F7
Device type: disk
Transport protocol: Fibre channel (FCP-2)
Local Time is: Thu Sep 24 14:13:59 2009 WEST
Device supports SMART and is Enabled
Temperature Warning Enabled
SMART Health Status: OK

Current Drive Temperature: 28 C
Drive Trip Temperature: 65 C
Elements in grown defect list: 0
Vendor (Seagate) cache information
Blocks sent to initiator = 6929197
Blocks received from initiator = 10042042
Blocks read from cache and sent to initiator = 4277931
Number of read and write commands whose size <= segment size = 663979
Number of read and write commands whose size > segment size = 0
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 72.75
number of minutes until next internal SMART test = 100

Error counter log:
Errors Corrected by Total Correction
Gigabytes Total
ECC rereads/ errors algorithm
processed uncorrected
fast | delayed rewrites corrected invocations [10^9
bytes] errors
read: 836 0 0 836 836 91.529
0
write: 0 0 0 0 24 5.898
8

Non-medium error count: 1

[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']

SMART Self-test log
Num Test Status segment LifeTime
LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Completed - 9
- [- - -]
# 2 Background short Completed - 0
- [- - -]
# 3 Background short Completed - 0
- [- - -]

Long (extended) Self Test duration: 1840 seconds [30.7 minutes]

Whereas, from the failed one, I get:

darkstar: smartctl -a /dev/rdsk/c8t5d0s2
smartctl version 5.38 [sparc-sun-solaris2.10] Copyright (C) 2002-8 Bruce
Allen
Home page is http://smartmontools.sourceforge.net/

Device: SEAGATE ST373405FC Version: 0005
Serial number: 3EK0KJET00007212LFYK
Device type: disk
Local Time is: Thu Sep 24 14:29:12 2009 WEST
device is NOT READY (e.g. spun down, busy)

The drive is spun up and the green light on the emc box is lit, with no
yellow, so need to get the drive on a pc with the Seagate tools to see
if it's just a low level hardware format problem. The seagate pc tools
are quite good in that you can specify the sector size. ie: you could
most likely reformat the funny emc 538 byte sector drives to 512 byte
using this utility, thereby making them usable in the real world.

The only thing smartmon it doesn't say is the total drive hours since
dom, which the Seagate disk tools did, iirc, so it must be in the log
pages somewhere. Perhaps i'll have a go at modding the source at some
stage to read the relevant page.

Good stuff anyway and thanks again...

Regards,

Chris
devfsadm disk

On 2009-09-24, ChrisQ <> wrote:
> DoN. Nichols wrote:
>
>> You can read the logs with smartctl, and using smartd, be given
>> e-mail notices any time the bad sector list grows. It does not come on
>> Solaris 10, but you can download the source and compile it.
>>
>
>
> Don,
>
> Thanks - downloaded and built smartmon and from a typical drive,
> smartctl tells me:
>
> darkstar: smartctl -a /dev/rdsk/c8t3d0s2
> smartctl version 5.38 [sparc-sun-solaris2.10] Copyright (C) 2002-8 Bruce
> Allen
> Home page is http://smartmontools.sourceforge.net/
>
> Device: SEAGATE ST373405FC Version: 0005

[ ... ]

> Vendor (Seagate/Hitachi) factory information
> number of hours powered up = 72.75

Not long -- unless this drive resets the time every time the
power is removed.

[ ... ]

> Whereas, from the failed one, I get:
>
> darkstar: smartctl -a /dev/rdsk/c8t5d0s2
> smartctl version 5.38 [sparc-sun-solaris2.10] Copyright (C) 2002-8 Bruce
> Allen
> Home page is http://smartmontools.sourceforge.net/
>
> Device: SEAGATE ST373405FC Version: 0005
> Serial number: 3EK0KJET00007212LFYK
> Device type: disk
> Local Time is: Thu Sep 24 14:29:12 2009 WEST
> device is NOT READY (e.g. spun down, busy)

"Not ready" -- that is not a good sign.

> The drive is spun up and the green light on the emc box is lit, with no
> yellow, so need to get the drive on a pc with the Seagate tools to see
> if it's just a low level hardware format problem. The seagate pc tools
> are quite good in that you can specify the sector size. ie: you could
> most likely reformat the funny emc 538 byte sector drives to 512 byte
> using this utility, thereby making them usable in the real world.

You can format the EMC drives to normal Sun usage with the Sun
format program -- but you have to cross your eyes and hold your tongue
just right. :-)

In particular (going from memory from about a year and a half
ago):

format -e c1t??d0

when the format program agrees to tell you anything (it takes a few
minutes, IIRC), you need to select the auto-configure option for
defining the drive type -- *anything* else will set you back to where
you have to power the drive down and wait through format's start-up time
again.

Anyway -- *if* you use the auto-configure option, it will look
at the drive for a little while, then state that it needs to reformat
the drive to 512 bytes/sector. If you agree to this, it will do so, and
from then on you can happily use the drive. I did this with 19 out of
20 72 GB FC-AL drives from a hamfest. The 20th drive was beyond hope,
but all the rest were great.

Enjoy,
DoN.

--
Email: <> | Voice (all times): (703) 938-4564
(too) near Washington D.C. | http://www.d-and-d.com/dnichols/DoN.html
--- Black Holes are where God is dividing by zero ---