169 lines
7.3 KiB
Markdown
169 lines
7.3 KiB
Markdown
# Handoff: F3 unlock of CLAR200 drives + tank redundancy restore
|
||
|
||
## Session Metadata
|
||
- Created: 2026-06-14 21:58:02
|
||
- Project: /home/claude/bin/clar200-unlock
|
||
- Branch: main
|
||
- Session duration: ~1 session (tank diagnosis → project bootstrap)
|
||
|
||
### Recent Commits (for context)
|
||
- 947d5cf init: F3 unlock project — ESP32-C3 bridge + procedure for CLAR200 Exos X18
|
||
|
||
## Handoff Chain
|
||
|
||
- **Continues from**: [2026-06-14-bootstrap.md](./2026-06-14-bootstrap.md)
|
||
- Previous title: Handoff — 2026-06-14 — clar200-unlock bootstrap
|
||
- **Supersedes**: None (bootstrap is the inline build-time handoff; this is the
|
||
session-close handoff. Both current; read bootstrap for the blow-by-blow.)
|
||
|
||
> Review the previous handoff for full context before filling this one.
|
||
|
||
## Current State Summary
|
||
|
||
Two threads, one root cause. **tank** (pve173, 549 TB ZFS) is DEGRADED:
|
||
mirror-35 is a 3-way mirror effectively running on ONE healthy disk, and
|
||
ex-mirror-38 is a bare single-disk vdev — both need **14.6 TB+** replacements,
|
||
of which **none are free** on pve173. Seth re-added an EMC-locked 16 TB drive
|
||
(SN ZRS015V7) to fill the gap, but it's the same CLAR200-locked Exos X18 a prior
|
||
session couldn't unlock. I confirmed the lock is firmware-level (no host fix),
|
||
then bootstrapped the **clar200-unlock** project: an ESP32-C3 USB-serial bridge
|
||
+ full F3-terminal procedure to attempt an out-of-band reformat. Firmware and
|
||
docs are written, committed, and pushed. **Nothing flashed, no drive benched
|
||
yet** — stopped at "ready, awaiting bench hardware + go-ahead."
|
||
|
||
## Codebase Understanding
|
||
|
||
### Architecture Overview
|
||
|
||
- Recovery rides the drive's **F3 service UART**, bypassing the SCSI layer (every
|
||
SCSI cmd incl. READ CAPACITY returns Hardware Error on these EMC drives).
|
||
- Bridge = **ESP32-C3** (native USB-CDC host side) ↔ **UART1 GPIO4(TX)/GPIO5(RX)**
|
||
to the drive. Live `~N` baud menu avoids reflashing during baud sweep.
|
||
- Build flag `CDCOnBoot=cdc` is mandatory (makes `Serial` == USB-CDC).
|
||
|
||
### Critical Files
|
||
|
||
| File | Purpose | Relevance |
|
||
|------|---------|-----------|
|
||
| `PROCEDURE.md` | bench→wire→flash→F3→format→verify runbook | **primary** — work from here |
|
||
| `firmware/clar200_bridge.ino` | ESP32-C3 bridge sketch | flash to the C3 |
|
||
| `firmware/flash.sh` | build+flash (installs arduino-cli+esp32 core) | run first |
|
||
| `docs/f3-command-reference.md` | F3 dialect, confirmed vs 7200.11-only | consult before any format cmd |
|
||
| `~/bin/EMC_CLAR200_DRIVE_FIX_NOTES.md` | exhaustive prior SCSI-side failure log | why host tools can't work |
|
||
|
||
### Key Patterns Discovered
|
||
|
||
- These C3 boards: onboard LED GPIO8 (active-low); safe free UART pins GPIO4/5
|
||
(avoid strapping 2/8/9, console 20/21, USB 18/19).
|
||
- `esptool` already on steel141; `python` is **python3** only (no `python` alias).
|
||
|
||
## Work Completed
|
||
|
||
### Tasks Finished
|
||
|
||
- [x] Diagnosed tank mirror-35 (3-way: 1 dead + 1 dying + 1 healthy; 3.68 TB live data)
|
||
- [x] Confirmed CLAR200 drive lock is firmware-level (new HBA + bus reset both fail)
|
||
- [x] Identified the attached cable as ESP32-C3 (`/dev/ttyACM0`, MAC 58:8c:81:ac:f0:4c)
|
||
- [x] Built clar200-unlock project: firmware, flash script, procedure, F3 ref, docs
|
||
- [x] Created + pushed Gitea repo; wrote bootstrap + this handoff
|
||
|
||
### Files Modified
|
||
|
||
| File | Changes | Rationale |
|
||
|------|---------|-----------|
|
||
| (new project `~/bin/clar200-unlock/*`) | created all | bootstrap; see commit 947d5cf |
|
||
|
||
### Decisions Made
|
||
|
||
| Decision | Options Considered | Rationale |
|
||
|----------|-------------------|-----------|
|
||
| F3 UART path | SCSI tools, firmware DL, F3, RMA | only F3 bypasses the locked SCSI layer |
|
||
| ESP32-C3 as bridge | buy USB-TTL vs use attached C3 | C3 already here, 3.3 V native-safe |
|
||
| Bench, not in-JBOD | in-chassis vs bench | never risk live tank during F3 work |
|
||
| Discover F3 cmds via `?` | copy 7200.11 recipes vs map live | Exos dialect differs; blind `m0,` risky |
|
||
|
||
## Pending Work
|
||
|
||
## Immediate Next Steps
|
||
|
||
1. **Flash the bridge:** `cd ~/bin/clar200-unlock/firmware && ./flash.sh /dev/ttyACM0`
|
||
(installs arduino-cli + esp32 core first run). Then loopback-test (jumper
|
||
GPIO4↔GPIO5, confirm echo in `screen /dev/ttyACM0 115200`).
|
||
2. **Bench drive ZRS015V7:** pull from pve173 JBOD, give standalone SAS power.
|
||
3. **Find F3 pinout** (multimeter), wire TX/RX/GND, get boot banner + `F3 T>`.
|
||
4. **Map format cmd via `?`**, format to 512, verify `sg_readcap`/`smartctl`.
|
||
5. **If healthy → tank:** `zpool replace tank 4450364206979169854 <newdisk>`
|
||
(replace BEFORE detaching the dying leg), then re-mirror ex-mirror-38.
|
||
6. Repeat whole procedure for **ZR511KWK**.
|
||
|
||
### Blockers/Open Questions
|
||
|
||
- [ ] Does Seth have **standalone SAS power** (SFF-8482→SATA-power / SAS dock),
|
||
jumper wires, and a multimeter on the bench? (gates Step 2–3)
|
||
- [ ] RMA to ServerPartDeals first vs. attempt F3 first? (leaning: F3 first)
|
||
- [ ] Real Exos X18 F3 format token is unknown until the live `?` session
|
||
|
||
### Deferred Items
|
||
|
||
- BSY isolation/tape trick — drive isn't BSY-hung; contingency only.
|
||
- Further SCSI-level attempts — exhausted, rejected.
|
||
- `zpool scrub tank` — do AFTER mirror-35 is on healthy disks (scrub hammers
|
||
the dying sdg); minor CKSUM counts on mirror-10/18/19/20/30 noted.
|
||
|
||
## Context for Resuming Agent
|
||
|
||
## Important Context
|
||
|
||
The drive ZRS015V7 is on **pve173** at `/dev/sg121` + `/dev/sg122` (dual SAS
|
||
paths, 7:0:43:0 and 8:0:11:0), model "SS160520 CLAR200", fake serial
|
||
"Maxtor 5A320J0…RAMB1TV0". The ESP32-C3 is on **steel141** `/dev/ttyACM0` — a
|
||
DIFFERENT machine — which is *fine* because F3 work is bench-based; the USB side
|
||
plugs into whatever host runs `screen`, and the GPIO side wires to the benched
|
||
drive. **tank mirror-35's dying leg `wwn-0x5000c500c37f4eff` (/dev/sdg, 30k
|
||
pending sectors) is the ONLY redundancy besides the one healthy leg — do NOT
|
||
detach it until a real replacement disk has resilvered in.** The dead leg to
|
||
replace is GUID `4450364206979169854`.
|
||
|
||
### Assumptions Made
|
||
|
||
- The C3 board has onboard LED on GPIO8 (harmless if not).
|
||
- 38400 8N1 is the F3 baud first guess (sweep if garbled).
|
||
- Drive boots clean (answers INQUIRY) so `F3 T>` reachable without isolation trick.
|
||
|
||
### Potential Gotchas
|
||
|
||
- `python` → use `python3` on steel141.
|
||
- USB-CDC baud is virtual; the **drive** baud is set with the firmware's `~N`
|
||
menu, not the host terminal baud.
|
||
- Never wire ESP32 VCC/5V to the drive — TX/RX/GND only; drive self-powered.
|
||
- `flash.sh` first run downloads ~1 GB esp32 core.
|
||
|
||
## Environment State
|
||
|
||
### Tools/Services Used
|
||
|
||
- pve173 (tank host): `zpool`, `smartctl`, `sg_*`, `sg_reset` via `ssh pve173`.
|
||
- steel141: `esptool` (installed), ESP32-C3 on `/dev/ttyACM0`. arduino-cli NOT
|
||
yet installed (flash.sh installs it).
|
||
- Gitea: repo https://git.sethpc.xyz/Seth/clar200-unlock (pushed).
|
||
|
||
### Active Processes
|
||
|
||
- None started/left running. No flashing done. No zpool mutations performed
|
||
(diagnosis was read-only except a harmless SCSI bus reset on the dead drive).
|
||
|
||
### Environment Variables
|
||
|
||
- `HOMELAB_PASSWORD` (used indirectly via SSH aliases) — name only.
|
||
|
||
## Related Resources
|
||
|
||
- Repo: https://git.sethpc.xyz/Seth/clar200-unlock
|
||
- `~/bin/EMC_CLAR200_DRIVE_FIX_NOTES.md` — prior exhaustive attempt
|
||
- `~/bin/CLAUDE.md` — storage section (tank, drive gotchas, ZL2PNZ99/ZL21R9ZF)
|
||
- Bootstrap handoff: `./2026-06-14-bootstrap.md`
|
||
|
||
---
|
||
|
||
**Security Reminder**: Before finalizing, run `validate_handoff.py` to check for accidental secret exposure.
|