ops: offline dying sdg + resilver tuning; document ~18d structural ETA

2026-06-16 18:53:33 -04:00
parent 2b9c287d2c
commit fc744babad
3 changed files with 175 additions and 9 deletions
@@ -95,16 +95,43 @@ mirror-35**, no new hardware. Started 2026-06-14 22:26.
 - **STEP 2 (when evacuation completes):**
  `zpool replace tank 4450364206979169854 wwn-0x5000c500c382f4ac-part1`
  — resilvers the freed disk into mirror-35's dead-leg slot.
- **STEP 3:** per Seth, **KEEP the dying leg `wwn-0x5000c500c37f4eff` (sdg) as an
-  extra copy — do NOT detach it.** Only the dead ZL2PNZ99 leg leaves (via the
-  replace in step 2).
- **Result:** mirror-35 = 3-way [sde healthy + c382f4ac healthy + sdg dying];
-  ex-mirror-38 gone; pool ONLINE. Cost paid: ~14.6 TB usable + a 2nd indirect map.
- Note: "4 legs" isn't possible — the dead ZL2PNZ99 is UNAVAIL (absent), not a
-  copy; keeping it would only pin DEGRADED. Max meaningful legs = 3.
- Verify after: `zpool status tank` ONLINE; `zpool list -v tank` mirror-35 3-way.
+- **STEP 1 DONE** (Jun 15 08:56): evacuation copied 3.86T in 10h29m.
+- **STEP 2 RUNNING** (since Jun 15 15:01): `zpool replace tank 4450364206979169854
+  wwn-0x5000c500c382f4ac-part1` resilvering into mirror-35's dead-leg slot.
+- **STEP 2b — sdg OFFLINED, not detached** (Jun 16 ~17:35): the dying leg
+  `wwn-0x5000c500c37f4eff` (sdg, ~29.6k pending sectors) was strangling the
+  resilver (1.5 MB/s, ETA blew out to 18 days; ETA was *rising*). Seth chose
+  `zpool offline` over `detach` to **keep the membership+DTL** so it can be
+  `zpool online`-d later as a last-resort partial copy if sde ever fails mid-
+  resilver. Offline gives the same speed win as detach but preserves that option.
+- **Consequence:** tank will report **DEGRADED indefinitely** (intentional offline
+  member) — this is EXPECTED, not a fault. Don't treat DEGRADED as an alarm until
+  sdg is dealt with.
+- **Result when resilver finishes:** real 2-way redundancy [sde + c382f4ac];
+  sdg = OFFLINE bonus 3rd option (stale + bad sectors = partial rescue only).
+  ex-mirror-38 gone. Cost: ~14.6 TB usable + a 2nd indirect map.
+- **sdg endgame (later):** when a real replacement disk exists, `zpool replace`
+  sdg out and pull it for RMA. Until then leave it OFFLINE.

-Check progress: `ssh pve173 'zpool status tank | grep -A3 remove:'`
+- **STEP 2 IS SLOW — ~18 day ETA, and this is STRUCTURAL, not fixable by knobs.**
+  A ZFS healing resilver walks the entire 423T pool block-tree (work scales with
+  pool size, not the 3.68T vdev) and politely yields to live media I/O
+  (130–230 MB/s from 14 media CTs). 26h in: only ~518G of mirror-35's 3.68T
+  resilvered. Pool is SAFE throughout (sde = complete healthy copy, 0 errors).
+- **Tuning applied (Jun 16)** to try to speed it — orig saved in
+  `.backup/zfs-tunables-*.txt`. `zfs_resilver_min_time_ms` 3000→8000,
+  `zfs_vdev_scrub_max_active` 3→6. Effect: scan phase ~2× faster, but completion
+  ETA (issue-phase-gated) barely moved; media NOT squeezed. Left ON (harmless,
+  may help issue phase after scan completes). **Revert if no benefit:**
+  `ssh pve173 'echo 3000 >/sys/module/zfs/parameters/zfs_resilver_min_time_ms; echo 3 >/sys/module/zfs/parameters/zfs_vdev_scrub_max_active'`
+  (NOTE: these are runtime-only — reset to default on reboot anyway.)
+- **Only real speed lever:** reduce media load (pause media CTs ~1 day) so the
+  resilver stops yielding. Not done — pool is safe, not worth the outage.
+- **Re-check ~12h after Jun 16 tuning:** scan should be done (~78/423T at
+  ~5.5T/10min); the issue rate then gives a real completion ETA.
+
+Check progress: `ssh pve173 'zpool status tank | grep -A3 "scan:"'`
+Pre-op snapshots: `.backup/zpool-pre-remove-*.txt`, `.backup/zpool-pre-replace-*.txt`, `.backup/zfs-tunables-*.txt`

 ## Immediate Next Steps (clar200 / F3 — separate thread, unblocked)