From fc744babad33a78339c10230757b7c3585452883 Mon Sep 17 00:00:00 2001 From: Seth Date: Tue, 16 Jun 2026 18:53:33 -0400 Subject: [PATCH] ops: offline dying sdg + resilver tuning; document ~18d structural ETA --- .backup/zfs-tunables-1781649690.txt | 5 + .backup/zpool-pre-replace-1781550115.txt | 134 ++++++++++++++++++ ...14-215802-f3-unlock-and-tank-redundancy.md | 45 ++++-- 3 files changed, 175 insertions(+), 9 deletions(-) create mode 100644 .backup/zfs-tunables-1781649690.txt create mode 100644 .backup/zpool-pre-replace-1781550115.txt diff --git a/.backup/zfs-tunables-1781649690.txt b/.backup/zfs-tunables-1781649690.txt new file mode 100644 index 0000000..6d07e92 --- /dev/null +++ b/.backup/zfs-tunables-1781649690.txt @@ -0,0 +1,5 @@ +ORIG zfs_resilver_min_time_ms=3000 +ORIG zfs_vdev_scrub_max_active=3 +--- applied --- +NEW zfs_resilver_min_time_ms=8000 +NEW zfs_vdev_scrub_max_active=6 diff --git a/.backup/zpool-pre-replace-1781550115.txt b/.backup/zpool-pre-replace-1781550115.txt new file mode 100644 index 0000000..7d537cf --- /dev/null +++ b/.backup/zpool-pre-replace-1781550115.txt @@ -0,0 +1,134 @@ + pool: tank + state: DEGRADED +status: One or more devices could not be used because the label is missing or + invalid. Sufficient replicas exist for the pool to continue + functioning in a degraded state. +action: Replace the device using 'zpool replace'. + see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J + scan: resilvered 28K in 00:00:20 with 0 errors on Sun Mar 29 05:21:17 2026 +remove: Removal of vdev 38 copied 3.86T in 10h29m, completed on Mon Jun 15 08:56:22 2026 + 35.5M memory used for removed device mappings +config: + + NAME STATE READ WRITE CKSUM + tank DEGRADED 0 0 0 + mirror-0 ONLINE 0 0 0 + scsi-35000cca28ff26fc1 ONLINE 0 0 0 + scsi-35000cca2b0f08b3e ONLINE 0 0 0 + mirror-2 ONLINE 0 0 0 + scsi-35000c500e87c990f ONLINE 0 0 0 + scsi-35000c500e8b26db1 ONLINE 0 0 0 + mirror-3 ONLINE 0 0 0 + scsi-35000cca26fcf3f16 ONLINE 0 0 0 + scsi-35000cca27ac1a7a4 ONLINE 0 0 0 + mirror-4 ONLINE 0 0 0 + scsi-35000cca278cb9b64 ONLINE 0 0 0 + scsi-35000cca27ac19f3b ONLINE 0 0 0 + mirror-5 ONLINE 0 0 0 + scsi-35000c500e895992c ONLINE 0 0 0 + scsi-35000c500e8c07dfd ONLINE 0 0 0 + mirror-6 ONLINE 0 0 0 + scsi-35000c50074478f15 ONLINE 0 0 0 + scsi-35000c500e932bd6a ONLINE 0 0 0 + mirror-7 ONLINE 0 0 0 + wwn-0x5000cca253dd94fd ONLINE 0 0 0 + wwn-0x5000cca253ea02b8 ONLINE 0 0 0 + mirror-8 ONLINE 0 0 0 + wwn-0x5000cca26fc069f5 ONLINE 0 0 0 + wwn-0x5000cca26fc1e364 ONLINE 0 0 0 + mirror-9 ONLINE 0 0 0 + wwn-0x5000cca26fd1e331 ONLINE 0 0 0 + wwn-0x5000cca26fd26164 ONLINE 0 0 0 + mirror-10 ONLINE 0 0 0 + wwn-0x5000cca270dd33cd ONLINE 0 0 0 + wwn-0x5000cca270ec2421 ONLINE 0 0 1 + mirror-11 ONLINE 0 0 0 + wwn-0x5000cca270f30b9e ONLINE 0 0 0 + wwn-0x5000cca270f31483 ONLINE 0 0 0 + mirror-12 ONLINE 0 0 0 + wwn-0x5000cca270f4d497 ONLINE 0 0 0 + wwn-0x5000cca270f6d97b ONLINE 0 0 0 + mirror-13 ONLINE 0 0 0 + wwn-0x5000cca278cc837b ONLINE 0 0 0 + wwn-0x5000cca278cced39 ONLINE 0 0 0 + mirror-14 ONLINE 0 0 0 + wwn-0x5000cca278cdc415 ONLINE 0 0 0 + wwn-0x5000cca278cdc494 ONLINE 0 0 0 + mirror-15 ONLINE 0 0 0 + wwn-0x5000cca278cdd867 ONLINE 0 0 0 + wwn-0x5000cca278cf6e87 ONLINE 0 0 0 + mirror-16 ONLINE 0 0 0 + wwn-0x5000cca27ac2caab ONLINE 0 0 0 + wwn-0x5000cca27ac30531 ONLINE 0 0 0 + mirror-17 ONLINE 0 0 0 + wwn-0x5000cca27ac6e4f9 ONLINE 0 0 0 + wwn-0x5000cca27ad30bf4 ONLINE 0 0 0 + mirror-18 ONLINE 0 0 0 + scsi-35000c500c37f9e3b ONLINE 0 0 0 + scsi-35000c5007443fec9 ONLINE 0 0 1 + mirror-19 ONLINE 0 0 0 + scsi-35000c500e3f58d4f ONLINE 0 0 0 + scsi-35000c500c85de52c ONLINE 0 0 4 + mirror-20 ONLINE 0 0 0 + scsi-35000c500c6a58fa5 ONLINE 0 0 0 + scsi-35000c500b5ac5165 ONLINE 0 0 4 + mirror-21 ONLINE 0 0 0 + scsi-35000c500e4057bd5 ONLINE 0 0 0 + scsi-35000c50074440210 ONLINE 0 0 0 + mirror-22 ONLINE 0 0 0 + wwn-0x5000c500cb0c026b ONLINE 0 0 0 + wwn-0x5000c500cb156dd3 ONLINE 0 0 0 + mirror-23 ONLINE 0 0 0 + scsi-35000c500e405854b ONLINE 0 0 0 + scsi-35000c50074440264 ONLINE 0 0 0 + mirror-24 ONLINE 0 0 0 + wwn-0x5000c500cb0d7f23 ONLINE 0 0 0 + wwn-0x5000c500cb0d8e83 ONLINE 0 0 0 + mirror-25 ONLINE 0 0 0 + wwn-0x5000c500cb0d72db ONLINE 0 0 0 + wwn-0x5000c500c5909b99-part1 ONLINE 0 0 0 + mirror-26 ONLINE 0 0 0 + scsi-35000c500c59184b8 ONLINE 0 0 0 + scsi-35000c500c8f01421 ONLINE 0 0 0 + mirror-27 ONLINE 0 0 0 + scsi-35000c500dbdd0fa7 ONLINE 0 0 0 + scsi-35000c500dc367708 ONLINE 0 0 0 + mirror-28 ONLINE 0 0 0 + scsi-35000c5007444033c ONLINE 0 0 0 + scsi-35000c500e4265acf ONLINE 0 0 0 + mirror-29 ONLINE 0 0 0 + wwn-0x5000c500e892e467-part1 ONLINE 0 0 0 + wwn-0x5000c500e9f1803c-part1 ONLINE 0 0 0 + mirror-30 ONLINE 0 0 0 + wwn-0x5000c500c974481f-part1 ONLINE 0 0 0 + wwn-0x5000c500c4d54225-part1 ONLINE 0 0 1 + mirror-31 ONLINE 0 0 0 + wwn-0x5000c50074440021-part1 ONLINE 0 0 0 + wwn-0x5000c500c8e46f13-part1 ONLINE 0 0 0 + mirror-32 ONLINE 0 0 0 + wwn-0x5000c5007443ff6e-part1 ONLINE 0 0 0 + wwn-0x5000c500c2c92560-part1 ONLINE 0 0 0 + mirror-33 ONLINE 0 0 0 + wwn-0x5000c50074440141-part1 ONLINE 0 0 0 + wwn-0x5000c500c7e1ac6e-part1 ONLINE 0 0 0 + mirror-34 ONLINE 0 0 0 + wwn-0x5000c500c660d0f1-part1 ONLINE 0 0 0 + wwn-0x5000c500c367b982-part1 ONLINE 0 0 0 + mirror-35 DEGRADED 0 0 0 + 4450364206979169854 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2PNZ99-part1 + wwn-0x5000c500c3663b1d-part1 ONLINE 0 0 0 + wwn-0x5000c500c37f4eff ONLINE 18 0 0 + mirror-36 ONLINE 0 0 0 + wwn-0x5000c5007443e5a0-part1 ONLINE 0 0 0 + wwn-0x5000c500db23c058-part1 ONLINE 0 0 0 + mirror-37 ONLINE 0 0 0 + wwn-0x5000c5007443fed3-part1 ONLINE 0 0 0 + wwn-0x5000c5007444028b-part1 ONLINE 0 0 0 + mirror-39 ONLINE 0 0 0 + wwn-0x5000c500c3e83a35-part1 ONLINE 0 0 0 + dm-uuid-mpath-35000c500d7461d87 ONLINE 0 0 0 + mirror-40 ONLINE 0 0 0 + wwn-0x5000cca278d02fdb ONLINE 0 0 0 + wwn-0x5000cca278dab334 ONLINE 0 0 0 + +errors: No known data errors diff --git a/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md b/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md index d44c1d6..83d22af 100644 --- a/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md +++ b/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md @@ -95,16 +95,43 @@ mirror-35**, no new hardware. Started 2026-06-14 22:26. - **STEP 2 (when evacuation completes):** `zpool replace tank 4450364206979169854 wwn-0x5000c500c382f4ac-part1` — resilvers the freed disk into mirror-35's dead-leg slot. -- **STEP 3:** per Seth, **KEEP the dying leg `wwn-0x5000c500c37f4eff` (sdg) as an - extra copy — do NOT detach it.** Only the dead ZL2PNZ99 leg leaves (via the - replace in step 2). -- **Result:** mirror-35 = 3-way [sde healthy + c382f4ac healthy + sdg dying]; - ex-mirror-38 gone; pool ONLINE. Cost paid: ~14.6 TB usable + a 2nd indirect map. -- Note: "4 legs" isn't possible — the dead ZL2PNZ99 is UNAVAIL (absent), not a - copy; keeping it would only pin DEGRADED. Max meaningful legs = 3. -- Verify after: `zpool status tank` ONLINE; `zpool list -v tank` mirror-35 3-way. +- **STEP 1 DONE** (Jun 15 08:56): evacuation copied 3.86T in 10h29m. +- **STEP 2 RUNNING** (since Jun 15 15:01): `zpool replace tank 4450364206979169854 + wwn-0x5000c500c382f4ac-part1` resilvering into mirror-35's dead-leg slot. +- **STEP 2b — sdg OFFLINED, not detached** (Jun 16 ~17:35): the dying leg + `wwn-0x5000c500c37f4eff` (sdg, ~29.6k pending sectors) was strangling the + resilver (1.5 MB/s, ETA blew out to 18 days; ETA was *rising*). Seth chose + `zpool offline` over `detach` to **keep the membership+DTL** so it can be + `zpool online`-d later as a last-resort partial copy if sde ever fails mid- + resilver. Offline gives the same speed win as detach but preserves that option. +- **Consequence:** tank will report **DEGRADED indefinitely** (intentional offline + member) — this is EXPECTED, not a fault. Don't treat DEGRADED as an alarm until + sdg is dealt with. +- **Result when resilver finishes:** real 2-way redundancy [sde + c382f4ac]; + sdg = OFFLINE bonus 3rd option (stale + bad sectors = partial rescue only). + ex-mirror-38 gone. Cost: ~14.6 TB usable + a 2nd indirect map. +- **sdg endgame (later):** when a real replacement disk exists, `zpool replace` + sdg out and pull it for RMA. Until then leave it OFFLINE. -Check progress: `ssh pve173 'zpool status tank | grep -A3 remove:'` +- **STEP 2 IS SLOW — ~18 day ETA, and this is STRUCTURAL, not fixable by knobs.** + A ZFS healing resilver walks the entire 423T pool block-tree (work scales with + pool size, not the 3.68T vdev) and politely yields to live media I/O + (130–230 MB/s from 14 media CTs). 26h in: only ~518G of mirror-35's 3.68T + resilvered. Pool is SAFE throughout (sde = complete healthy copy, 0 errors). +- **Tuning applied (Jun 16)** to try to speed it — orig saved in + `.backup/zfs-tunables-*.txt`. `zfs_resilver_min_time_ms` 3000→8000, + `zfs_vdev_scrub_max_active` 3→6. Effect: scan phase ~2× faster, but completion + ETA (issue-phase-gated) barely moved; media NOT squeezed. Left ON (harmless, + may help issue phase after scan completes). **Revert if no benefit:** + `ssh pve173 'echo 3000 >/sys/module/zfs/parameters/zfs_resilver_min_time_ms; echo 3 >/sys/module/zfs/parameters/zfs_vdev_scrub_max_active'` + (NOTE: these are runtime-only — reset to default on reboot anyway.) +- **Only real speed lever:** reduce media load (pause media CTs ~1 day) so the + resilver stops yielding. Not done — pool is safe, not worth the outage. +- **Re-check ~12h after Jun 16 tuning:** scan should be done (~78/423T at + ~5.5T/10min); the issue rate then gives a real completion ETA. + +Check progress: `ssh pve173 'zpool status tank | grep -A3 "scan:"'` +Pre-op snapshots: `.backup/zpool-pre-remove-*.txt`, `.backup/zpool-pre-replace-*.txt`, `.backup/zfs-tunables-*.txt` ## Immediate Next Steps (clar200 / F3 — separate thread, unblocked)