clar200-unlock/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md

# Handoff: F3 unlock of CLAR200 drives + tank redundancy restore

## Session Metadata
- Created: 2026-06-14 21:58:02
- Project: /home/claude/bin/clar200-unlock
- Branch: main
- Session duration: ~1 session (tank diagnosis → project bootstrap)

### Recent Commits (for context)
  - 947d5cf init: F3 unlock project — ESP32-C3 bridge + procedure for CLAR200 Exos X18

## Handoff Chain

- **Continues from**: [2026-06-14-bootstrap.md](./2026-06-14-bootstrap.md)
  - Previous title: Handoff — 2026-06-14 — clar200-unlock bootstrap
- **Supersedes**: None (bootstrap is the inline build-time handoff; this is the
  session-close handoff. Both current; read bootstrap for the blow-by-blow.)

> Review the previous handoff for full context before filling this one.

## Current State Summary

Two threads, one root cause. **tank** (pve173, 549 TB ZFS) is DEGRADED:
mirror-35 is a 3-way mirror effectively running on ONE healthy disk, and
ex-mirror-38 is a bare single-disk vdev — both need **14.6 TB+** replacements,
of which **none are free** on pve173. Seth re-added an EMC-locked 16 TB drive
(SN ZRS015V7) to fill the gap, but it's the same CLAR200-locked Exos X18 a prior
session couldn't unlock. I confirmed the lock is firmware-level (no host fix),
then bootstrapped the **clar200-unlock** project: an ESP32-C3 USB-serial bridge
+ full F3-terminal procedure to attempt an out-of-band reformat. Firmware and
docs are written, committed, and pushed. **Nothing flashed, no drive benched
yet** — stopped at "ready, awaiting bench hardware + go-ahead."

## Codebase Understanding

### Architecture Overview

- Recovery rides the drive's **F3 service UART**, bypassing the SCSI layer (every
  SCSI cmd incl. READ CAPACITY returns Hardware Error on these EMC drives).
- Bridge = **ESP32-C3** (native USB-CDC host side) ↔ **UART1 GPIO4(TX)/GPIO5(RX)**
  to the drive. Live `~N` baud menu avoids reflashing during baud sweep.
- Build flag `CDCOnBoot=cdc` is mandatory (makes `Serial` == USB-CDC).

### Critical Files

| File | Purpose | Relevance |
|------|---------|-----------|
| `PROCEDURE.md` | bench→wire→flash→F3→format→verify runbook | **primary** — work from here |
| `firmware/clar200_bridge.ino` | ESP32-C3 bridge sketch | flash to the C3 |
| `firmware/flash.sh` | build+flash (installs arduino-cli+esp32 core) | run first |
| `docs/f3-command-reference.md` | F3 dialect, confirmed vs 7200.11-only | consult before any format cmd |
| `~/bin/EMC_CLAR200_DRIVE_FIX_NOTES.md` | exhaustive prior SCSI-side failure log | why host tools can't work |

### Key Patterns Discovered

- These C3 boards: onboard LED GPIO8 (active-low); safe free UART pins GPIO4/5
  (avoid strapping 2/8/9, console 20/21, USB 18/19).
- `esptool` already on steel141; `python` is **python3** only (no `python` alias).

## Work Completed

### Tasks Finished

- [x] Diagnosed tank mirror-35 (3-way: 1 dead + 1 dying + 1 healthy; 3.68 TB live data)
- [x] Confirmed CLAR200 drive lock is firmware-level (new HBA + bus reset both fail)
- [x] Identified the attached cable as ESP32-C3 (`/dev/ttyACM0`, MAC 58:8c:81:ac:f0:4c)
- [x] Built clar200-unlock project: firmware, flash script, procedure, F3 ref, docs
- [x] Created + pushed Gitea repo; wrote bootstrap + this handoff

### Files Modified

| File | Changes | Rationale |
|------|---------|-----------|
| (new project `~/bin/clar200-unlock/*`) | created all | bootstrap; see commit 947d5cf |

### Decisions Made

| Decision | Options Considered | Rationale |
|----------|-------------------|-----------|
| F3 UART path | SCSI tools, firmware DL, F3, RMA | only F3 bypasses the locked SCSI layer |
| ESP32-C3 as bridge | buy USB-TTL vs use attached C3 | C3 already here, 3.3 V native-safe |
| Bench, not in-JBOD | in-chassis vs bench | never risk live tank during F3 work |
| Discover F3 cmds via `?` | copy 7200.11 recipes vs map live | Exos dialect differs; blind `m0,` risky |

## Pending Work

## ✅ COMPLETE — tank redundancy restore (Jun 21) — no new disk used

**Done Jun 21 05:53**: resilver of mirror-35 finished, 4.12T in 5d14h52m, 0 errors.
- mirror-35 = healthy 2-way [`c382f4ac` ONLINE + `sde` (`c3663b1d`) ONLINE] + `sdg`
  (`c37f4eff`) OFFLINE. Dead ZL2PNZ99 leg gone. Was effectively 1 healthy disk; now 2.
- ex-mirror-38 evacuated & gone. **Both redundancy holes closed, zero new disks.**
- Pool reports DEGRADED — ONLY because `sdg` is intentionally OFFLINE. Not a fault.
- Capacity cost: pool 549T→535T (~14T), 103T free @ 80%. Tunables reverted to stock.

### Remaining decisions
1. **`sdg` — RESOLVED Jun 22:** detached. Pool now **ONLINE** (clean 2-way
   mirror-35). Drive flagged for RMA & physical pull:
   - **Serial `ZL21HMR2`** (Seagate Exos X16 ST16000NM001G-2KK103, 16 TB, fw SN04,
     WWN 5000c500c37f4eff) — the failing one (30k pending sectors). Check Seagate
     warranty by this serial.
   - Physical location: **enclosure `6:0:15:0`, slot 8** (SAS expander
     0x50060480012df53f, phy15). Locate LED set ON Jun 22; clears on pull/reset.
   - Post-pull: nothing needed in ZFS (already detached). tank stays ONLINE.
2. CLAR200 F3 unlock — still pending, unblocked, needs bench (see below).
   Reminder: an unlocked CLAR200 16 TB would recover the ~14 TB capacity spent
   here by re-adding it as a new vdev.

---
## (historical) IN PROGRESS — tank redundancy restore (no new disk path)

Seth chose to fix tank redundancy by **moving the bare ex-mirror-38 disk into
mirror-35**, no new hardware. Started 2026-06-14 22:26.

- **STEP 1 RUNNING:** `zpool remove tank wwn-0x5000c500c382f4ac-part1` —
  evacuating 3.86 TB, ~11h ETA (~103 MB/s). Pre-state snapshot:
  `.backup/zpool-pre-remove-1781490349.txt`. **Irreversible.**
- **STEP 2 (when evacuation completes):**
  `zpool replace tank 4450364206979169854 wwn-0x5000c500c382f4ac-part1`
  — resilvers the freed disk into mirror-35's dead-leg slot.
- **STEP 1 DONE** (Jun 15 08:56): evacuation copied 3.86T in 10h29m.
- **STEP 2 RUNNING** (since Jun 15 15:01): `zpool replace tank 4450364206979169854
  wwn-0x5000c500c382f4ac-part1` resilvering into mirror-35's dead-leg slot.
- **STEP 2b — sdg OFFLINED, not detached** (Jun 16 ~17:35): the dying leg
  `wwn-0x5000c500c37f4eff` (sdg, ~29.6k pending sectors) was strangling the
  resilver (1.5 MB/s, ETA blew out to 18 days; ETA was *rising*). Seth chose
  `zpool offline` over `detach` to **keep the membership+DTL** so it can be
  `zpool online`-d later as a last-resort partial copy if sde ever fails mid-
  resilver. Offline gives the same speed win as detach but preserves that option.
- **Consequence:** tank will report **DEGRADED indefinitely** (intentional offline
  member) — this is EXPECTED, not a fault. Don't treat DEGRADED as an alarm until
  sdg is dealt with.
- **Result when resilver finishes:** real 2-way redundancy [sde + c382f4ac];
  sdg = OFFLINE bonus 3rd option (stale + bad sectors = partial rescue only).
  ex-mirror-38 gone. Cost: ~14.6 TB usable + a 2nd indirect map.
- **sdg endgame (later):** when a real replacement disk exists, `zpool replace`
  sdg out and pull it for RMA. Until then leave it OFFLINE.

- **STEP 2 IS SLOW — ~18 day ETA, and this is STRUCTURAL, not fixable by knobs.**
  A ZFS healing resilver walks the entire 423T pool block-tree (work scales with
  pool size, not the 3.68T vdev) and politely yields to live media I/O
  (130–230 MB/s from 14 media CTs). 26h in: only ~518G of mirror-35's 3.68T
  resilvered. Pool is SAFE throughout (sde = complete healthy copy, 0 errors).
- **Tuning applied (Jun 16)** to try to speed it — orig saved in
  `.backup/zfs-tunables-*.txt`. `zfs_resilver_min_time_ms` 3000→8000,
  `zfs_vdev_scrub_max_active` 3→6. Effect: scan phase ~2× faster, but completion
  ETA (issue-phase-gated) barely moved; media NOT squeezed. Left ON (harmless,
  may help issue phase after scan completes). **Revert if no benefit:**
  `ssh pve173 'echo 3000 >/sys/module/zfs/parameters/zfs_resilver_min_time_ms; echo 3 >/sys/module/zfs/parameters/zfs_vdev_scrub_max_active'`
  (NOTE: these are runtime-only — reset to default on reboot anyway.)
- **Only real speed lever:** reduce media load (pause media CTs ~1 day) so the
  resilver stops yielding. Not done — pool is safe, not worth the outage.
- **Re-check ~12h after Jun 16 tuning:** scan should be done (~78/423T at
  ~5.5T/10min); the issue rate then gives a real completion ETA.

Check progress: `ssh pve173 'zpool status tank | grep -A3 "scan:"'`
Pre-op snapshots: `.backup/zpool-pre-remove-*.txt`, `.backup/zpool-pre-replace-*.txt`, `.backup/zfs-tunables-*.txt`

## Immediate Next Steps (clar200 / F3 — separate thread, unblocked)

1. **Flash the bridge:** `cd ~/bin/clar200-unlock/firmware && ./flash.sh /dev/ttyACM0`
   (installs arduino-cli + esp32 core first run). Then loopback-test (jumper
   GPIO4↔GPIO5, confirm echo in `screen /dev/ttyACM0 115200`).
2. **Bench drive ZRS015V7:** pull from pve173 JBOD, give standalone SAS power.
3. **Find F3 pinout** (multimeter), wire TX/RX/GND, get boot banner + `F3 T>`.
4. **Map format cmd via `?`**, format to 512, verify `sg_readcap`/`smartctl`.
5. **If healthy → tank:** `zpool replace tank 4450364206979169854 <newdisk>`
   (replace BEFORE detaching the dying leg), then re-mirror ex-mirror-38.
6. Repeat whole procedure for **ZR511KWK**.

### Blockers/Open Questions

- [ ] Does Seth have **standalone SAS power** (SFF-8482→SATA-power / SAS dock),
  jumper wires, and a multimeter on the bench? (gates Step 2–3)
- [ ] RMA to ServerPartDeals first vs. attempt F3 first? (leaning: F3 first)
- [ ] Real Exos X18 F3 format token is unknown until the live `?` session

### Deferred Items

- BSY isolation/tape trick — drive isn't BSY-hung; contingency only.
- Further SCSI-level attempts — exhausted, rejected.
- `zpool scrub tank` — do AFTER mirror-35 is on healthy disks (scrub hammers
  the dying sdg); minor CKSUM counts on mirror-10/18/19/20/30 noted.

## Context for Resuming Agent

## Important Context

The drive ZRS015V7 is on **pve173** at `/dev/sg121` + `/dev/sg122` (dual SAS
paths, 7:0:43:0 and 8:0:11:0), model "SS160520 CLAR200", fake serial
"Maxtor 5A320J0…RAMB1TV0". The ESP32-C3 is on **steel141** `/dev/ttyACM0` — a
DIFFERENT machine — which is *fine* because F3 work is bench-based; the USB side
plugs into whatever host runs `screen`, and the GPIO side wires to the benched
drive. **tank mirror-35's dying leg `wwn-0x5000c500c37f4eff` (/dev/sdg, 30k
pending sectors) is the ONLY redundancy besides the one healthy leg — do NOT
detach it until a real replacement disk has resilvered in.** The dead leg to
replace is GUID `4450364206979169854`.

### Assumptions Made

- The C3 board has onboard LED on GPIO8 (harmless if not).
- 38400 8N1 is the F3 baud first guess (sweep if garbled).
- Drive boots clean (answers INQUIRY) so `F3 T>` reachable without isolation trick.

### Potential Gotchas

- `python` → use `python3` on steel141.
- USB-CDC baud is virtual; the **drive** baud is set with the firmware's `~N`
  menu, not the host terminal baud.
- Never wire ESP32 VCC/5V to the drive — TX/RX/GND only; drive self-powered.
- `flash.sh` first run downloads ~1 GB esp32 core.

## Environment State

### Tools/Services Used

- pve173 (tank host): `zpool`, `smartctl`, `sg_*`, `sg_reset` via `ssh pve173`.
- steel141: `esptool` (installed), ESP32-C3 on `/dev/ttyACM0`. arduino-cli NOT
  yet installed (flash.sh installs it).
- Gitea: repo https://git.sethpc.xyz/Seth/clar200-unlock (pushed).

### Active Processes

- None started/left running. No flashing done. No zpool mutations performed
  (diagnosis was read-only except a harmless SCSI bus reset on the dead drive).

### Environment Variables

- `HOMELAB_PASSWORD` (used indirectly via SSH aliases) — name only.

## Related Resources

- Repo: https://git.sethpc.xyz/Seth/clar200-unlock
- `~/bin/EMC_CLAR200_DRIVE_FIX_NOTES.md` — prior exhaustive attempt
- `~/bin/CLAUDE.md` — storage section (tank, drive gotchas, ZL2PNZ99/ZL21R9ZF)
- Bootstrap handoff: `./2026-06-14-bootstrap.md`

---

**Security Reminder**: Before finalizing, run `validate_handoff.py` to check for accidental secret exposure.