ops: offline dying sdg + resilver tuning; document ~18d structural ETA
This commit is contained in:
@@ -0,0 +1,5 @@
|
||||
ORIG zfs_resilver_min_time_ms=3000
|
||||
ORIG zfs_vdev_scrub_max_active=3
|
||||
--- applied ---
|
||||
NEW zfs_resilver_min_time_ms=8000
|
||||
NEW zfs_vdev_scrub_max_active=6
|
||||
@@ -0,0 +1,134 @@
|
||||
pool: tank
|
||||
state: DEGRADED
|
||||
status: One or more devices could not be used because the label is missing or
|
||||
invalid. Sufficient replicas exist for the pool to continue
|
||||
functioning in a degraded state.
|
||||
action: Replace the device using 'zpool replace'.
|
||||
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
|
||||
scan: resilvered 28K in 00:00:20 with 0 errors on Sun Mar 29 05:21:17 2026
|
||||
remove: Removal of vdev 38 copied 3.86T in 10h29m, completed on Mon Jun 15 08:56:22 2026
|
||||
35.5M memory used for removed device mappings
|
||||
config:
|
||||
|
||||
NAME STATE READ WRITE CKSUM
|
||||
tank DEGRADED 0 0 0
|
||||
mirror-0 ONLINE 0 0 0
|
||||
scsi-35000cca28ff26fc1 ONLINE 0 0 0
|
||||
scsi-35000cca2b0f08b3e ONLINE 0 0 0
|
||||
mirror-2 ONLINE 0 0 0
|
||||
scsi-35000c500e87c990f ONLINE 0 0 0
|
||||
scsi-35000c500e8b26db1 ONLINE 0 0 0
|
||||
mirror-3 ONLINE 0 0 0
|
||||
scsi-35000cca26fcf3f16 ONLINE 0 0 0
|
||||
scsi-35000cca27ac1a7a4 ONLINE 0 0 0
|
||||
mirror-4 ONLINE 0 0 0
|
||||
scsi-35000cca278cb9b64 ONLINE 0 0 0
|
||||
scsi-35000cca27ac19f3b ONLINE 0 0 0
|
||||
mirror-5 ONLINE 0 0 0
|
||||
scsi-35000c500e895992c ONLINE 0 0 0
|
||||
scsi-35000c500e8c07dfd ONLINE 0 0 0
|
||||
mirror-6 ONLINE 0 0 0
|
||||
scsi-35000c50074478f15 ONLINE 0 0 0
|
||||
scsi-35000c500e932bd6a ONLINE 0 0 0
|
||||
mirror-7 ONLINE 0 0 0
|
||||
wwn-0x5000cca253dd94fd ONLINE 0 0 0
|
||||
wwn-0x5000cca253ea02b8 ONLINE 0 0 0
|
||||
mirror-8 ONLINE 0 0 0
|
||||
wwn-0x5000cca26fc069f5 ONLINE 0 0 0
|
||||
wwn-0x5000cca26fc1e364 ONLINE 0 0 0
|
||||
mirror-9 ONLINE 0 0 0
|
||||
wwn-0x5000cca26fd1e331 ONLINE 0 0 0
|
||||
wwn-0x5000cca26fd26164 ONLINE 0 0 0
|
||||
mirror-10 ONLINE 0 0 0
|
||||
wwn-0x5000cca270dd33cd ONLINE 0 0 0
|
||||
wwn-0x5000cca270ec2421 ONLINE 0 0 1
|
||||
mirror-11 ONLINE 0 0 0
|
||||
wwn-0x5000cca270f30b9e ONLINE 0 0 0
|
||||
wwn-0x5000cca270f31483 ONLINE 0 0 0
|
||||
mirror-12 ONLINE 0 0 0
|
||||
wwn-0x5000cca270f4d497 ONLINE 0 0 0
|
||||
wwn-0x5000cca270f6d97b ONLINE 0 0 0
|
||||
mirror-13 ONLINE 0 0 0
|
||||
wwn-0x5000cca278cc837b ONLINE 0 0 0
|
||||
wwn-0x5000cca278cced39 ONLINE 0 0 0
|
||||
mirror-14 ONLINE 0 0 0
|
||||
wwn-0x5000cca278cdc415 ONLINE 0 0 0
|
||||
wwn-0x5000cca278cdc494 ONLINE 0 0 0
|
||||
mirror-15 ONLINE 0 0 0
|
||||
wwn-0x5000cca278cdd867 ONLINE 0 0 0
|
||||
wwn-0x5000cca278cf6e87 ONLINE 0 0 0
|
||||
mirror-16 ONLINE 0 0 0
|
||||
wwn-0x5000cca27ac2caab ONLINE 0 0 0
|
||||
wwn-0x5000cca27ac30531 ONLINE 0 0 0
|
||||
mirror-17 ONLINE 0 0 0
|
||||
wwn-0x5000cca27ac6e4f9 ONLINE 0 0 0
|
||||
wwn-0x5000cca27ad30bf4 ONLINE 0 0 0
|
||||
mirror-18 ONLINE 0 0 0
|
||||
scsi-35000c500c37f9e3b ONLINE 0 0 0
|
||||
scsi-35000c5007443fec9 ONLINE 0 0 1
|
||||
mirror-19 ONLINE 0 0 0
|
||||
scsi-35000c500e3f58d4f ONLINE 0 0 0
|
||||
scsi-35000c500c85de52c ONLINE 0 0 4
|
||||
mirror-20 ONLINE 0 0 0
|
||||
scsi-35000c500c6a58fa5 ONLINE 0 0 0
|
||||
scsi-35000c500b5ac5165 ONLINE 0 0 4
|
||||
mirror-21 ONLINE 0 0 0
|
||||
scsi-35000c500e4057bd5 ONLINE 0 0 0
|
||||
scsi-35000c50074440210 ONLINE 0 0 0
|
||||
mirror-22 ONLINE 0 0 0
|
||||
wwn-0x5000c500cb0c026b ONLINE 0 0 0
|
||||
wwn-0x5000c500cb156dd3 ONLINE 0 0 0
|
||||
mirror-23 ONLINE 0 0 0
|
||||
scsi-35000c500e405854b ONLINE 0 0 0
|
||||
scsi-35000c50074440264 ONLINE 0 0 0
|
||||
mirror-24 ONLINE 0 0 0
|
||||
wwn-0x5000c500cb0d7f23 ONLINE 0 0 0
|
||||
wwn-0x5000c500cb0d8e83 ONLINE 0 0 0
|
||||
mirror-25 ONLINE 0 0 0
|
||||
wwn-0x5000c500cb0d72db ONLINE 0 0 0
|
||||
wwn-0x5000c500c5909b99-part1 ONLINE 0 0 0
|
||||
mirror-26 ONLINE 0 0 0
|
||||
scsi-35000c500c59184b8 ONLINE 0 0 0
|
||||
scsi-35000c500c8f01421 ONLINE 0 0 0
|
||||
mirror-27 ONLINE 0 0 0
|
||||
scsi-35000c500dbdd0fa7 ONLINE 0 0 0
|
||||
scsi-35000c500dc367708 ONLINE 0 0 0
|
||||
mirror-28 ONLINE 0 0 0
|
||||
scsi-35000c5007444033c ONLINE 0 0 0
|
||||
scsi-35000c500e4265acf ONLINE 0 0 0
|
||||
mirror-29 ONLINE 0 0 0
|
||||
wwn-0x5000c500e892e467-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c500e9f1803c-part1 ONLINE 0 0 0
|
||||
mirror-30 ONLINE 0 0 0
|
||||
wwn-0x5000c500c974481f-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c500c4d54225-part1 ONLINE 0 0 1
|
||||
mirror-31 ONLINE 0 0 0
|
||||
wwn-0x5000c50074440021-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c500c8e46f13-part1 ONLINE 0 0 0
|
||||
mirror-32 ONLINE 0 0 0
|
||||
wwn-0x5000c5007443ff6e-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c500c2c92560-part1 ONLINE 0 0 0
|
||||
mirror-33 ONLINE 0 0 0
|
||||
wwn-0x5000c50074440141-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c500c7e1ac6e-part1 ONLINE 0 0 0
|
||||
mirror-34 ONLINE 0 0 0
|
||||
wwn-0x5000c500c660d0f1-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c500c367b982-part1 ONLINE 0 0 0
|
||||
mirror-35 DEGRADED 0 0 0
|
||||
4450364206979169854 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2PNZ99-part1
|
||||
wwn-0x5000c500c3663b1d-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c500c37f4eff ONLINE 18 0 0
|
||||
mirror-36 ONLINE 0 0 0
|
||||
wwn-0x5000c5007443e5a0-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c500db23c058-part1 ONLINE 0 0 0
|
||||
mirror-37 ONLINE 0 0 0
|
||||
wwn-0x5000c5007443fed3-part1 ONLINE 0 0 0
|
||||
wwn-0x5000c5007444028b-part1 ONLINE 0 0 0
|
||||
mirror-39 ONLINE 0 0 0
|
||||
wwn-0x5000c500c3e83a35-part1 ONLINE 0 0 0
|
||||
dm-uuid-mpath-35000c500d7461d87 ONLINE 0 0 0
|
||||
mirror-40 ONLINE 0 0 0
|
||||
wwn-0x5000cca278d02fdb ONLINE 0 0 0
|
||||
wwn-0x5000cca278dab334 ONLINE 0 0 0
|
||||
|
||||
errors: No known data errors
|
||||
@@ -95,16 +95,43 @@ mirror-35**, no new hardware. Started 2026-06-14 22:26.
|
||||
- **STEP 2 (when evacuation completes):**
|
||||
`zpool replace tank 4450364206979169854 wwn-0x5000c500c382f4ac-part1`
|
||||
— resilvers the freed disk into mirror-35's dead-leg slot.
|
||||
- **STEP 3:** per Seth, **KEEP the dying leg `wwn-0x5000c500c37f4eff` (sdg) as an
|
||||
extra copy — do NOT detach it.** Only the dead ZL2PNZ99 leg leaves (via the
|
||||
replace in step 2).
|
||||
- **Result:** mirror-35 = 3-way [sde healthy + c382f4ac healthy + sdg dying];
|
||||
ex-mirror-38 gone; pool ONLINE. Cost paid: ~14.6 TB usable + a 2nd indirect map.
|
||||
- Note: "4 legs" isn't possible — the dead ZL2PNZ99 is UNAVAIL (absent), not a
|
||||
copy; keeping it would only pin DEGRADED. Max meaningful legs = 3.
|
||||
- Verify after: `zpool status tank` ONLINE; `zpool list -v tank` mirror-35 3-way.
|
||||
- **STEP 1 DONE** (Jun 15 08:56): evacuation copied 3.86T in 10h29m.
|
||||
- **STEP 2 RUNNING** (since Jun 15 15:01): `zpool replace tank 4450364206979169854
|
||||
wwn-0x5000c500c382f4ac-part1` resilvering into mirror-35's dead-leg slot.
|
||||
- **STEP 2b — sdg OFFLINED, not detached** (Jun 16 ~17:35): the dying leg
|
||||
`wwn-0x5000c500c37f4eff` (sdg, ~29.6k pending sectors) was strangling the
|
||||
resilver (1.5 MB/s, ETA blew out to 18 days; ETA was *rising*). Seth chose
|
||||
`zpool offline` over `detach` to **keep the membership+DTL** so it can be
|
||||
`zpool online`-d later as a last-resort partial copy if sde ever fails mid-
|
||||
resilver. Offline gives the same speed win as detach but preserves that option.
|
||||
- **Consequence:** tank will report **DEGRADED indefinitely** (intentional offline
|
||||
member) — this is EXPECTED, not a fault. Don't treat DEGRADED as an alarm until
|
||||
sdg is dealt with.
|
||||
- **Result when resilver finishes:** real 2-way redundancy [sde + c382f4ac];
|
||||
sdg = OFFLINE bonus 3rd option (stale + bad sectors = partial rescue only).
|
||||
ex-mirror-38 gone. Cost: ~14.6 TB usable + a 2nd indirect map.
|
||||
- **sdg endgame (later):** when a real replacement disk exists, `zpool replace`
|
||||
sdg out and pull it for RMA. Until then leave it OFFLINE.
|
||||
|
||||
Check progress: `ssh pve173 'zpool status tank | grep -A3 remove:'`
|
||||
- **STEP 2 IS SLOW — ~18 day ETA, and this is STRUCTURAL, not fixable by knobs.**
|
||||
A ZFS healing resilver walks the entire 423T pool block-tree (work scales with
|
||||
pool size, not the 3.68T vdev) and politely yields to live media I/O
|
||||
(130–230 MB/s from 14 media CTs). 26h in: only ~518G of mirror-35's 3.68T
|
||||
resilvered. Pool is SAFE throughout (sde = complete healthy copy, 0 errors).
|
||||
- **Tuning applied (Jun 16)** to try to speed it — orig saved in
|
||||
`.backup/zfs-tunables-*.txt`. `zfs_resilver_min_time_ms` 3000→8000,
|
||||
`zfs_vdev_scrub_max_active` 3→6. Effect: scan phase ~2× faster, but completion
|
||||
ETA (issue-phase-gated) barely moved; media NOT squeezed. Left ON (harmless,
|
||||
may help issue phase after scan completes). **Revert if no benefit:**
|
||||
`ssh pve173 'echo 3000 >/sys/module/zfs/parameters/zfs_resilver_min_time_ms; echo 3 >/sys/module/zfs/parameters/zfs_vdev_scrub_max_active'`
|
||||
(NOTE: these are runtime-only — reset to default on reboot anyway.)
|
||||
- **Only real speed lever:** reduce media load (pause media CTs ~1 day) so the
|
||||
resilver stops yielding. Not done — pool is safe, not worth the outage.
|
||||
- **Re-check ~12h after Jun 16 tuning:** scan should be done (~78/423T at
|
||||
~5.5T/10min); the issue rate then gives a real completion ETA.
|
||||
|
||||
Check progress: `ssh pve173 'zpool status tank | grep -A3 "scan:"'`
|
||||
Pre-op snapshots: `.backup/zpool-pre-remove-*.txt`, `.backup/zpool-pre-replace-*.txt`, `.backup/zfs-tunables-*.txt`
|
||||
|
||||
## Immediate Next Steps (clar200 / F3 — separate thread, unblocked)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user