diff --git a/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md b/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md new file mode 100644 index 0000000..8668e1f --- /dev/null +++ b/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md @@ -0,0 +1,168 @@ +# Handoff: F3 unlock of CLAR200 drives + tank redundancy restore + +## Session Metadata +- Created: 2026-06-14 21:58:02 +- Project: /home/claude/bin/clar200-unlock +- Branch: main +- Session duration: ~1 session (tank diagnosis → project bootstrap) + +### Recent Commits (for context) + - 947d5cf init: F3 unlock project — ESP32-C3 bridge + procedure for CLAR200 Exos X18 + +## Handoff Chain + +- **Continues from**: [2026-06-14-bootstrap.md](./2026-06-14-bootstrap.md) + - Previous title: Handoff — 2026-06-14 — clar200-unlock bootstrap +- **Supersedes**: None (bootstrap is the inline build-time handoff; this is the + session-close handoff. Both current; read bootstrap for the blow-by-blow.) + +> Review the previous handoff for full context before filling this one. + +## Current State Summary + +Two threads, one root cause. **tank** (pve173, 549 TB ZFS) is DEGRADED: +mirror-35 is a 3-way mirror effectively running on ONE healthy disk, and +ex-mirror-38 is a bare single-disk vdev — both need **14.6 TB+** replacements, +of which **none are free** on pve173. Seth re-added an EMC-locked 16 TB drive +(SN ZRS015V7) to fill the gap, but it's the same CLAR200-locked Exos X18 a prior +session couldn't unlock. I confirmed the lock is firmware-level (no host fix), +then bootstrapped the **clar200-unlock** project: an ESP32-C3 USB-serial bridge ++ full F3-terminal procedure to attempt an out-of-band reformat. Firmware and +docs are written, committed, and pushed. **Nothing flashed, no drive benched +yet** — stopped at "ready, awaiting bench hardware + go-ahead." + +## Codebase Understanding + +### Architecture Overview + +- Recovery rides the drive's **F3 service UART**, bypassing the SCSI layer (every + SCSI cmd incl. READ CAPACITY returns Hardware Error on these EMC drives). +- Bridge = **ESP32-C3** (native USB-CDC host side) ↔ **UART1 GPIO4(TX)/GPIO5(RX)** + to the drive. Live `~N` baud menu avoids reflashing during baud sweep. +- Build flag `CDCOnBoot=cdc` is mandatory (makes `Serial` == USB-CDC). + +### Critical Files + +| File | Purpose | Relevance | +|------|---------|-----------| +| `PROCEDURE.md` | bench→wire→flash→F3→format→verify runbook | **primary** — work from here | +| `firmware/clar200_bridge.ino` | ESP32-C3 bridge sketch | flash to the C3 | +| `firmware/flash.sh` | build+flash (installs arduino-cli+esp32 core) | run first | +| `docs/f3-command-reference.md` | F3 dialect, confirmed vs 7200.11-only | consult before any format cmd | +| `~/bin/EMC_CLAR200_DRIVE_FIX_NOTES.md` | exhaustive prior SCSI-side failure log | why host tools can't work | + +### Key Patterns Discovered + +- These C3 boards: onboard LED GPIO8 (active-low); safe free UART pins GPIO4/5 + (avoid strapping 2/8/9, console 20/21, USB 18/19). +- `esptool` already on steel141; `python` is **python3** only (no `python` alias). + +## Work Completed + +### Tasks Finished + +- [x] Diagnosed tank mirror-35 (3-way: 1 dead + 1 dying + 1 healthy; 3.68 TB live data) +- [x] Confirmed CLAR200 drive lock is firmware-level (new HBA + bus reset both fail) +- [x] Identified the attached cable as ESP32-C3 (`/dev/ttyACM0`, MAC 58:8c:81:ac:f0:4c) +- [x] Built clar200-unlock project: firmware, flash script, procedure, F3 ref, docs +- [x] Created + pushed Gitea repo; wrote bootstrap + this handoff + +### Files Modified + +| File | Changes | Rationale | +|------|---------|-----------| +| (new project `~/bin/clar200-unlock/*`) | created all | bootstrap; see commit 947d5cf | + +### Decisions Made + +| Decision | Options Considered | Rationale | +|----------|-------------------|-----------| +| F3 UART path | SCSI tools, firmware DL, F3, RMA | only F3 bypasses the locked SCSI layer | +| ESP32-C3 as bridge | buy USB-TTL vs use attached C3 | C3 already here, 3.3 V native-safe | +| Bench, not in-JBOD | in-chassis vs bench | never risk live tank during F3 work | +| Discover F3 cmds via `?` | copy 7200.11 recipes vs map live | Exos dialect differs; blind `m0,` risky | + +## Pending Work + +## Immediate Next Steps + +1. **Flash the bridge:** `cd ~/bin/clar200-unlock/firmware && ./flash.sh /dev/ttyACM0` + (installs arduino-cli + esp32 core first run). Then loopback-test (jumper + GPIO4↔GPIO5, confirm echo in `screen /dev/ttyACM0 115200`). +2. **Bench drive ZRS015V7:** pull from pve173 JBOD, give standalone SAS power. +3. **Find F3 pinout** (multimeter), wire TX/RX/GND, get boot banner + `F3 T>`. +4. **Map format cmd via `?`**, format to 512, verify `sg_readcap`/`smartctl`. +5. **If healthy → tank:** `zpool replace tank 4450364206979169854 ` + (replace BEFORE detaching the dying leg), then re-mirror ex-mirror-38. +6. Repeat whole procedure for **ZR511KWK**. + +### Blockers/Open Questions + +- [ ] Does Seth have **standalone SAS power** (SFF-8482→SATA-power / SAS dock), + jumper wires, and a multimeter on the bench? (gates Step 2–3) +- [ ] RMA to ServerPartDeals first vs. attempt F3 first? (leaning: F3 first) +- [ ] Real Exos X18 F3 format token is unknown until the live `?` session + +### Deferred Items + +- BSY isolation/tape trick — drive isn't BSY-hung; contingency only. +- Further SCSI-level attempts — exhausted, rejected. +- `zpool scrub tank` — do AFTER mirror-35 is on healthy disks (scrub hammers + the dying sdg); minor CKSUM counts on mirror-10/18/19/20/30 noted. + +## Context for Resuming Agent + +## Important Context + +The drive ZRS015V7 is on **pve173** at `/dev/sg121` + `/dev/sg122` (dual SAS +paths, 7:0:43:0 and 8:0:11:0), model "SS160520 CLAR200", fake serial +"Maxtor 5A320J0…RAMB1TV0". The ESP32-C3 is on **steel141** `/dev/ttyACM0` — a +DIFFERENT machine — which is *fine* because F3 work is bench-based; the USB side +plugs into whatever host runs `screen`, and the GPIO side wires to the benched +drive. **tank mirror-35's dying leg `wwn-0x5000c500c37f4eff` (/dev/sdg, 30k +pending sectors) is the ONLY redundancy besides the one healthy leg — do NOT +detach it until a real replacement disk has resilvered in.** The dead leg to +replace is GUID `4450364206979169854`. + +### Assumptions Made + +- The C3 board has onboard LED on GPIO8 (harmless if not). +- 38400 8N1 is the F3 baud first guess (sweep if garbled). +- Drive boots clean (answers INQUIRY) so `F3 T>` reachable without isolation trick. + +### Potential Gotchas + +- `python` → use `python3` on steel141. +- USB-CDC baud is virtual; the **drive** baud is set with the firmware's `~N` + menu, not the host terminal baud. +- Never wire ESP32 VCC/5V to the drive — TX/RX/GND only; drive self-powered. +- `flash.sh` first run downloads ~1 GB esp32 core. + +## Environment State + +### Tools/Services Used + +- pve173 (tank host): `zpool`, `smartctl`, `sg_*`, `sg_reset` via `ssh pve173`. +- steel141: `esptool` (installed), ESP32-C3 on `/dev/ttyACM0`. arduino-cli NOT + yet installed (flash.sh installs it). +- Gitea: repo https://git.sethpc.xyz/Seth/clar200-unlock (pushed). + +### Active Processes + +- None started/left running. No flashing done. No zpool mutations performed + (diagnosis was read-only except a harmless SCSI bus reset on the dead drive). + +### Environment Variables + +- `HOMELAB_PASSWORD` (used indirectly via SSH aliases) — name only. + +## Related Resources + +- Repo: https://git.sethpc.xyz/Seth/clar200-unlock +- `~/bin/EMC_CLAR200_DRIVE_FIX_NOTES.md` — prior exhaustive attempt +- `~/bin/CLAUDE.md` — storage section (tank, drive gotchas, ZL2PNZ99/ZL21R9ZF) +- Bootstrap handoff: `./2026-06-14-bootstrap.md` + +--- + +**Security Reminder**: Before finalizing, run `validate_handoff.py` to check for accidental secret exposure.