From 2b9c287d2c81231ebb72ebebf14be2e849ff4a9f Mon Sep 17 00:00:00 2001 From: Seth Date: Sun, 14 Jun 2026 22:27:34 -0400 Subject: [PATCH] ops: start ex-mirror-38 evacuation; record tank redundancy-restore plan --- .backup/zpool-pre-remove-1781490349.txt | 258 ++++++++++++++++++ ...14-215802-f3-unlock-and-tank-redundancy.md | 24 +- 2 files changed, 281 insertions(+), 1 deletion(-) create mode 100644 .backup/zpool-pre-remove-1781490349.txt diff --git a/.backup/zpool-pre-remove-1781490349.txt b/.backup/zpool-pre-remove-1781490349.txt new file mode 100644 index 0000000..16d0786 --- /dev/null +++ b/.backup/zpool-pre-remove-1781490349.txt @@ -0,0 +1,258 @@ + pool: tank + state: DEGRADED +status: One or more devices could not be used because the label is missing or + invalid. Sufficient replicas exist for the pool to continue + functioning in a degraded state. +action: Replace the device using 'zpool replace'. + see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J + scan: resilvered 28K in 00:00:20 with 0 errors on Sun Mar 29 05:21:17 2026 +remove: Removal of vdev 1 copied 10.9T in 22h22m, completed on Thu Feb 12 19:49:25 2026 + 16.3M memory used for removed device mappings +config: + + NAME STATE READ WRITE CKSUM + tank DEGRADED 0 0 0 + mirror-0 ONLINE 0 0 0 + scsi-35000cca28ff26fc1 ONLINE 0 0 0 + scsi-35000cca2b0f08b3e ONLINE 0 0 0 + mirror-2 ONLINE 0 0 0 + scsi-35000c500e87c990f ONLINE 0 0 0 + scsi-35000c500e8b26db1 ONLINE 0 0 0 + mirror-3 ONLINE 0 0 0 + scsi-35000cca26fcf3f16 ONLINE 0 0 0 + scsi-35000cca27ac1a7a4 ONLINE 0 0 0 + mirror-4 ONLINE 0 0 0 + scsi-35000cca278cb9b64 ONLINE 0 0 0 + scsi-35000cca27ac19f3b ONLINE 0 0 0 + mirror-5 ONLINE 0 0 0 + scsi-35000c500e895992c ONLINE 0 0 0 + scsi-35000c500e8c07dfd ONLINE 0 0 0 + mirror-6 ONLINE 0 0 0 + scsi-35000c50074478f15 ONLINE 0 0 0 + scsi-35000c500e932bd6a ONLINE 0 0 0 + mirror-7 ONLINE 0 0 0 + wwn-0x5000cca253dd94fd ONLINE 0 0 0 + wwn-0x5000cca253ea02b8 ONLINE 0 0 0 + mirror-8 ONLINE 0 0 0 + wwn-0x5000cca26fc069f5 ONLINE 0 0 0 + wwn-0x5000cca26fc1e364 ONLINE 0 0 0 + mirror-9 ONLINE 0 0 0 + wwn-0x5000cca26fd1e331 ONLINE 0 0 0 + wwn-0x5000cca26fd26164 ONLINE 0 0 0 + mirror-10 ONLINE 0 0 0 + wwn-0x5000cca270dd33cd ONLINE 0 0 0 + wwn-0x5000cca270ec2421 ONLINE 0 0 1 + mirror-11 ONLINE 0 0 0 + wwn-0x5000cca270f30b9e ONLINE 0 0 0 + wwn-0x5000cca270f31483 ONLINE 0 0 0 + mirror-12 ONLINE 0 0 0 + wwn-0x5000cca270f4d497 ONLINE 0 0 0 + wwn-0x5000cca270f6d97b ONLINE 0 0 0 + mirror-13 ONLINE 0 0 0 + wwn-0x5000cca278cc837b ONLINE 0 0 0 + wwn-0x5000cca278cced39 ONLINE 0 0 0 + mirror-14 ONLINE 0 0 0 + wwn-0x5000cca278cdc415 ONLINE 0 0 0 + wwn-0x5000cca278cdc494 ONLINE 0 0 0 + mirror-15 ONLINE 0 0 0 + wwn-0x5000cca278cdd867 ONLINE 0 0 0 + wwn-0x5000cca278cf6e87 ONLINE 0 0 0 + mirror-16 ONLINE 0 0 0 + wwn-0x5000cca27ac2caab ONLINE 0 0 0 + wwn-0x5000cca27ac30531 ONLINE 0 0 0 + mirror-17 ONLINE 0 0 0 + wwn-0x5000cca27ac6e4f9 ONLINE 0 0 0 + wwn-0x5000cca27ad30bf4 ONLINE 0 0 0 + mirror-18 ONLINE 0 0 0 + scsi-35000c500c37f9e3b ONLINE 0 0 0 + scsi-35000c5007443fec9 ONLINE 0 0 1 + mirror-19 ONLINE 0 0 0 + scsi-35000c500e3f58d4f ONLINE 0 0 0 + scsi-35000c500c85de52c ONLINE 0 0 4 + mirror-20 ONLINE 0 0 0 + scsi-35000c500c6a58fa5 ONLINE 0 0 0 + scsi-35000c500b5ac5165 ONLINE 0 0 4 + mirror-21 ONLINE 0 0 0 + scsi-35000c500e4057bd5 ONLINE 0 0 0 + scsi-35000c50074440210 ONLINE 0 0 0 + mirror-22 ONLINE 0 0 0 + wwn-0x5000c500cb0c026b ONLINE 0 0 0 + wwn-0x5000c500cb156dd3 ONLINE 0 0 0 + mirror-23 ONLINE 0 0 0 + scsi-35000c500e405854b ONLINE 0 0 0 + scsi-35000c50074440264 ONLINE 0 0 0 + mirror-24 ONLINE 0 0 0 + wwn-0x5000c500cb0d7f23 ONLINE 0 0 0 + wwn-0x5000c500cb0d8e83 ONLINE 0 0 0 + mirror-25 ONLINE 0 0 0 + wwn-0x5000c500cb0d72db ONLINE 0 0 0 + wwn-0x5000c500c5909b99-part1 ONLINE 0 0 0 + mirror-26 ONLINE 0 0 0 + scsi-35000c500c59184b8 ONLINE 0 0 0 + scsi-35000c500c8f01421 ONLINE 0 0 0 + mirror-27 ONLINE 0 0 0 + scsi-35000c500dbdd0fa7 ONLINE 0 0 0 + scsi-35000c500dc367708 ONLINE 0 0 0 + mirror-28 ONLINE 0 0 0 + scsi-35000c5007444033c ONLINE 0 0 0 + scsi-35000c500e4265acf ONLINE 0 0 0 + mirror-29 ONLINE 0 0 0 + wwn-0x5000c500e892e467-part1 ONLINE 0 0 0 + wwn-0x5000c500e9f1803c-part1 ONLINE 0 0 0 + mirror-30 ONLINE 0 0 0 + wwn-0x5000c500c974481f-part1 ONLINE 0 0 0 + wwn-0x5000c500c4d54225-part1 ONLINE 0 0 1 + mirror-31 ONLINE 0 0 0 + wwn-0x5000c50074440021-part1 ONLINE 0 0 0 + wwn-0x5000c500c8e46f13-part1 ONLINE 0 0 0 + mirror-32 ONLINE 0 0 0 + wwn-0x5000c5007443ff6e-part1 ONLINE 0 0 0 + wwn-0x5000c500c2c92560-part1 ONLINE 0 0 0 + mirror-33 ONLINE 0 0 0 + wwn-0x5000c50074440141-part1 ONLINE 0 0 0 + wwn-0x5000c500c7e1ac6e-part1 ONLINE 0 0 0 + mirror-34 ONLINE 0 0 0 + wwn-0x5000c500c660d0f1-part1 ONLINE 0 0 0 + wwn-0x5000c500c367b982-part1 ONLINE 0 0 0 + mirror-35 DEGRADED 0 0 0 + 4450364206979169854 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2PNZ99-part1 + wwn-0x5000c500c3663b1d-part1 ONLINE 0 0 0 + wwn-0x5000c500c37f4eff ONLINE 18 0 0 + mirror-36 ONLINE 0 0 0 + wwn-0x5000c5007443e5a0-part1 ONLINE 0 0 0 + wwn-0x5000c500db23c058-part1 ONLINE 0 0 0 + mirror-37 ONLINE 0 0 0 + wwn-0x5000c5007443fed3-part1 ONLINE 0 0 0 + wwn-0x5000c5007444028b-part1 ONLINE 0 0 0 + wwn-0x5000c500c382f4ac-part1 ONLINE 0 0 0 + mirror-39 ONLINE 0 0 0 + wwn-0x5000c500c3e83a35-part1 ONLINE 0 0 0 + dm-uuid-mpath-35000c500d7461d87 ONLINE 0 0 0 + mirror-40 ONLINE 0 0 0 + wwn-0x5000cca278d02fdb ONLINE 0 0 0 + wwn-0x5000cca278dab334 ONLINE 0 0 0 + +errors: No known data errors +##### LIST ##### +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +tank 549T 423T 126T - - - 77% 1.00x DEGRADED - + mirror-0 10.9T 10.9T 19.2G - - 13% 99.8% - ONLINE + scsi-35000cca28ff26fc1 12.7T - - - - - - - ONLINE + scsi-35000cca2b0f08b3e 10.9T - - - - - - - ONLINE + indirect-1 - - - - - - - - ONLINE + mirror-2 20T 20.0T 16G - - - 99.9% - ONLINE + scsi-35000c500e87c990f 20.0T - - - - - - - ONLINE + scsi-35000c500e8b26db1 20.0T - - - - - - - ONLINE + mirror-3 10.9T 10.9T 16G - - - 99.8% - ONLINE + scsi-35000cca26fcf3f16 10.9T - - - - - - - ONLINE + scsi-35000cca27ac1a7a4 10.9T - - - - - - - ONLINE + mirror-4 10.9T 10.9T 16G - - - 99.8% - ONLINE + scsi-35000cca278cb9b64 10.9T - - - - - - - ONLINE + scsi-35000cca27ac19f3b 10.9T - - - - - - - ONLINE + mirror-5 20T 20.0T 16G - - - 99.9% - ONLINE + scsi-35000c500e895992c 20.0T - - - - - - - ONLINE + scsi-35000c500e8c07dfd 20.0T - - - - - - - ONLINE + mirror-6 20T 20.0T 16G - - - 99.9% - ONLINE + scsi-35000c50074478f15 20.0T - - - - - - - ONLINE + scsi-35000c500e932bd6a 20.0T - - - - - - - ONLINE + mirror-7 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca253dd94fd 10.9T - - - - - - - ONLINE + wwn-0x5000cca253ea02b8 10.9T - - - - - - - ONLINE + mirror-8 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca26fc069f5 10.9T - - - - - - - ONLINE + wwn-0x5000cca26fc1e364 10.9T - - - - - - - ONLINE + mirror-9 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca26fd1e331 10.9T - - - - - - - ONLINE + wwn-0x5000cca26fd26164 10.9T - - - - - - - ONLINE + mirror-10 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca270dd33cd 10.9T - - - - - - - ONLINE + wwn-0x5000cca270ec2421 10.9T - - - - - - - ONLINE + mirror-11 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca270f30b9e 10.9T - - - - - - - ONLINE + wwn-0x5000cca270f31483 10.9T - - - - - - - ONLINE + mirror-12 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca270f4d497 10.9T - - - - - - - ONLINE + wwn-0x5000cca270f6d97b 10.9T - - - - - - - ONLINE + mirror-13 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca278cc837b 10.9T - - - - - - - ONLINE + wwn-0x5000cca278cced39 10.9T - - - - - - - ONLINE + mirror-14 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca278cdc415 10.9T - - - - - - - ONLINE + wwn-0x5000cca278cdc494 10.9T - - - - - - - ONLINE + mirror-15 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca278cdd867 10.9T - - - - - - - ONLINE + wwn-0x5000cca278cf6e87 10.9T - - - - - - - ONLINE + mirror-16 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca27ac2caab 10.9T - - - - - - - ONLINE + wwn-0x5000cca27ac30531 10.9T - - - - - - - ONLINE + mirror-17 10.9T 10.9T 16G - - - 99.8% - ONLINE + wwn-0x5000cca27ac6e4f9 10.9T - - - - - - - ONLINE + wwn-0x5000cca27ad30bf4 10.9T - - - - - - - ONLINE + mirror-18 14.5T 14.5T 16.0G - - - 99.9% - ONLINE + scsi-35000c500c37f9e3b 14.6T - - - - - - - ONLINE + scsi-35000c5007443fec9 14.6T - - - - - - - ONLINE + mirror-19 14.5T 14.5T 16G - - - 99.9% - ONLINE + scsi-35000c500e3f58d4f 14.6T - - - - - - - ONLINE + scsi-35000c500c85de52c 14.6T - - - - - - - ONLINE + mirror-20 14.5T 14.5T 16G - - - 99.9% - ONLINE + scsi-35000c500c6a58fa5 14.6T - - - - - - - ONLINE + scsi-35000c500b5ac5165 14.6T - - - - - - - ONLINE + mirror-21 14.5T 14.5T 16G - - - 99.9% - ONLINE + scsi-35000c500e4057bd5 14.6T - - - - - - - ONLINE + scsi-35000c50074440210 14.6T - - - - - - - ONLINE + mirror-22 14.5T 14.5T 16G - - - 99.9% - ONLINE + wwn-0x5000c500cb0c026b 14.6T - - - - - - - ONLINE + wwn-0x5000c500cb156dd3 14.6T - - - - - - - ONLINE + mirror-23 14.5T 14.5T 16G - - - 99.9% - ONLINE + scsi-35000c500e405854b 14.6T - - - - - - - ONLINE + scsi-35000c50074440264 14.6T - - - - - - - ONLINE + mirror-24 14.5T 14.5T 16G - - - 99.9% - ONLINE + wwn-0x5000c500cb0d7f23 14.6T - - - - - - - ONLINE + wwn-0x5000c500cb0d8e83 14.6T - - - - - - - ONLINE + mirror-25 14.5T 14.5T 16G - - - 99.9% - ONLINE + wwn-0x5000c500cb0d72db 14.6T - - - - - - - ONLINE + wwn-0x5000c500c5909b99-part1 14.6T - - - - - - - ONLINE + mirror-26 14.5T 14.5T 16G - - - 99.9% - ONLINE + scsi-35000c500c59184b8 14.6T - - - - - - - ONLINE + scsi-35000c500c8f01421 14.6T - - - - - - - ONLINE + mirror-27 14.5T 14.5T 16G - - - 99.9% - ONLINE + scsi-35000c500dbdd0fa7 14.6T - - - - - - - ONLINE + scsi-35000c500dc367708 14.6T - - - - - - - ONLINE + mirror-28 14.5T 14.5T 74.3G - - 44% 99.5% - ONLINE + scsi-35000c5007444033c 14.6T - - - - - - - ONLINE + scsi-35000c500e4265acf 14.6T - - - - - - - ONLINE + mirror-29 20T 11.6T 8.36T - - 1% 58.2% - ONLINE + wwn-0x5000c500e892e467-part1 20.0T - - - - - - - ONLINE + wwn-0x5000c500e9f1803c-part1 20.0T - - - - - - - ONLINE + mirror-30 14.5T 3.44T 11.1T - - 1% 23.6% - ONLINE + wwn-0x5000c500c974481f-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c500c4d54225-part1 14.6T - - - - - - - ONLINE + mirror-31 14.5T 3.86T 10.7T - - 0% 26.5% - ONLINE + wwn-0x5000c50074440021-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c500c8e46f13-part1 14.6T - - - - - - - ONLINE + mirror-32 14.5T 3.17T 11.4T - - 1% 21.8% - ONLINE + wwn-0x5000c5007443ff6e-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c500c2c92560-part1 14.6T - - - - - - - ONLINE + mirror-33 14.5T 3.22T 11.3T - - 0% 22.1% - ONLINE + wwn-0x5000c50074440141-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c500c7e1ac6e-part1 14.6T - - - - - - - ONLINE + mirror-34 14.5T 3.56T 11.0T - - 0% 24.5% - ONLINE + wwn-0x5000c500c660d0f1-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c500c367b982-part1 14.6T - - - - - - - ONLINE + mirror-35 14.5T 3.68T 10.9T - - 0% 25.3% - DEGRADED + 4450364206979169854 - - - - - - - - UNAVAIL + wwn-0x5000c500c3663b1d-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c500c37f4eff 14.6T - - - - - - - ONLINE + mirror-36 14.5T 3.74T 10.8T - - 0% 25.7% - ONLINE + wwn-0x5000c5007443e5a0-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c500db23c058-part1 14.6T - - - - - - - ONLINE + mirror-37 14.5T 3.79T 10.8T - - 1% 26.1% - ONLINE + wwn-0x5000c5007443fed3-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c5007444028b-part1 14.6T - - - - - - - ONLINE + wwn-0x5000c500c382f4ac-part1 14.6T 3.86T 10.7T - - 0% 26.5% - ONLINE + mirror-39 14.5T 3.99T 10.6T - - 0% 27.4% - ONLINE + wwn-0x5000c500c3e83a35-part1 14.6T - - - - - - - ONLINE + dm-uuid-mpath-35000c500d7461d87 14.6T - - - - - - - ONLINE + mirror-40 10.9T 2.69T 8.22T - - 0% 24.6% - ONLINE + wwn-0x5000cca278d02fdb 10.9T - - - - - - - ONLINE + wwn-0x5000cca278dab334 10.9T - - - - - - - ONLINE diff --git a/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md b/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md index 8668e1f..d44c1d6 100644 --- a/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md +++ b/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md @@ -84,7 +84,29 @@ yet** — stopped at "ready, awaiting bench hardware + go-ahead." ## Pending Work -## Immediate Next Steps +## ⏳ IN PROGRESS — tank redundancy restore (no new disk path) + +Seth chose to fix tank redundancy by **moving the bare ex-mirror-38 disk into +mirror-35**, no new hardware. Started 2026-06-14 22:26. + +- **STEP 1 RUNNING:** `zpool remove tank wwn-0x5000c500c382f4ac-part1` — + evacuating 3.86 TB, ~11h ETA (~103 MB/s). Pre-state snapshot: + `.backup/zpool-pre-remove-1781490349.txt`. **Irreversible.** +- **STEP 2 (when evacuation completes):** + `zpool replace tank 4450364206979169854 wwn-0x5000c500c382f4ac-part1` + — resilvers the freed disk into mirror-35's dead-leg slot. +- **STEP 3:** per Seth, **KEEP the dying leg `wwn-0x5000c500c37f4eff` (sdg) as an + extra copy — do NOT detach it.** Only the dead ZL2PNZ99 leg leaves (via the + replace in step 2). +- **Result:** mirror-35 = 3-way [sde healthy + c382f4ac healthy + sdg dying]; + ex-mirror-38 gone; pool ONLINE. Cost paid: ~14.6 TB usable + a 2nd indirect map. +- Note: "4 legs" isn't possible — the dead ZL2PNZ99 is UNAVAIL (absent), not a + copy; keeping it would only pin DEGRADED. Max meaningful legs = 3. +- Verify after: `zpool status tank` ONLINE; `zpool list -v tank` mirror-35 3-way. + +Check progress: `ssh pve173 'zpool status tank | grep -A3 remove:'` + +## Immediate Next Steps (clar200 / F3 — separate thread, unblocked) 1. **Flash the bridge:** `cd ~/bin/clar200-unlock/firmware && ./flash.sh /dev/ttyACM0` (installs arduino-cli + esp32 core first run). Then loopback-test (jumper