Seth/clar200-unlock
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fc744babad33a78339c10230757b7c3585452883
clar200-unlock/.claude/handoffs/2026-06-14-215802-f3-unlock-and-tank-redundancy.md
T

11 KiB
Raw Blame History

Handoff: F3 unlock of CLAR200 drives + tank redundancy restore

Session Metadata

  • Created: 2026-06-14 21:58:02
  • Project: /home/claude/bin/clar200-unlock
  • Branch: main
  • Session duration: ~1 session (tank diagnosis → project bootstrap)

Recent Commits (for context)

  • 947d5cf init: F3 unlock project — ESP32-C3 bridge + procedure for CLAR200 Exos X18

Handoff Chain

  • Continues from: 2026-06-14-bootstrap.md
    • Previous title: Handoff — 2026-06-14 — clar200-unlock bootstrap
  • Supersedes: None (bootstrap is the inline build-time handoff; this is the session-close handoff. Both current; read bootstrap for the blow-by-blow.)

Review the previous handoff for full context before filling this one.

Current State Summary

Two threads, one root cause. tank (pve173, 549 TB ZFS) is DEGRADED: mirror-35 is a 3-way mirror effectively running on ONE healthy disk, and ex-mirror-38 is a bare single-disk vdev — both need 14.6 TB+ replacements, of which none are free on pve173. Seth re-added an EMC-locked 16 TB drive (SN ZRS015V7) to fill the gap, but it's the same CLAR200-locked Exos X18 a prior session couldn't unlock. I confirmed the lock is firmware-level (no host fix), then bootstrapped the clar200-unlock project: an ESP32-C3 USB-serial bridge

  • full F3-terminal procedure to attempt an out-of-band reformat. Firmware and docs are written, committed, and pushed. Nothing flashed, no drive benched yet — stopped at "ready, awaiting bench hardware + go-ahead."

Codebase Understanding

Architecture Overview

  • Recovery rides the drive's F3 service UART, bypassing the SCSI layer (every SCSI cmd incl. READ CAPACITY returns Hardware Error on these EMC drives).
  • Bridge = ESP32-C3 (native USB-CDC host side) ↔ UART1 GPIO4(TX)/GPIO5(RX) to the drive. Live ~N baud menu avoids reflashing during baud sweep.
  • Build flag CDCOnBoot=cdc is mandatory (makes Serial == USB-CDC).

Critical Files

File Purpose Relevance
PROCEDURE.md bench→wire→flash→F3→format→verify runbook primary — work from here
firmware/clar200_bridge.ino ESP32-C3 bridge sketch flash to the C3
firmware/flash.sh build+flash (installs arduino-cli+esp32 core) run first
docs/f3-command-reference.md F3 dialect, confirmed vs 7200.11-only consult before any format cmd
~/bin/EMC_CLAR200_DRIVE_FIX_NOTES.md exhaustive prior SCSI-side failure log why host tools can't work

Key Patterns Discovered

  • These C3 boards: onboard LED GPIO8 (active-low); safe free UART pins GPIO4/5 (avoid strapping 2/8/9, console 20/21, USB 18/19).
  • esptool already on steel141; python is python3 only (no python alias).

Work Completed

Tasks Finished

  • Diagnosed tank mirror-35 (3-way: 1 dead + 1 dying + 1 healthy; 3.68 TB live data)
  • Confirmed CLAR200 drive lock is firmware-level (new HBA + bus reset both fail)
  • Identified the attached cable as ESP32-C3 (/dev/ttyACM0, MAC 58:8c:81:ac:f0:4c)
  • Built clar200-unlock project: firmware, flash script, procedure, F3 ref, docs
  • Created + pushed Gitea repo; wrote bootstrap + this handoff

Files Modified

File Changes Rationale
(new project ~/bin/clar200-unlock/*) created all bootstrap; see commit 947d5cf

Decisions Made

Decision Options Considered Rationale
F3 UART path SCSI tools, firmware DL, F3, RMA only F3 bypasses the locked SCSI layer
ESP32-C3 as bridge buy USB-TTL vs use attached C3 C3 already here, 3.3 V native-safe
Bench, not in-JBOD in-chassis vs bench never risk live tank during F3 work
Discover F3 cmds via ? copy 7200.11 recipes vs map live Exos dialect differs; blind m0, risky

Pending Work

⏳ IN PROGRESS — tank redundancy restore (no new disk path)

Seth chose to fix tank redundancy by moving the bare ex-mirror-38 disk into mirror-35, no new hardware. Started 2026-06-14 22:26.

  • STEP 1 RUNNING: zpool remove tank wwn-0x5000c500c382f4ac-part1 — evacuating 3.86 TB, ~11h ETA (~103 MB/s). Pre-state snapshot: .backup/zpool-pre-remove-1781490349.txt. Irreversible.

  • STEP 2 (when evacuation completes): zpool replace tank 4450364206979169854 wwn-0x5000c500c382f4ac-part1 — resilvers the freed disk into mirror-35's dead-leg slot.

  • STEP 1 DONE (Jun 15 08:56): evacuation copied 3.86T in 10h29m.

  • STEP 2 RUNNING (since Jun 15 15:01): zpool replace tank 4450364206979169854 wwn-0x5000c500c382f4ac-part1 resilvering into mirror-35's dead-leg slot.

  • STEP 2b — sdg OFFLINED, not detached (Jun 16 ~17:35): the dying leg wwn-0x5000c500c37f4eff (sdg, ~29.6k pending sectors) was strangling the resilver (1.5 MB/s, ETA blew out to 18 days; ETA was rising). Seth chose zpool offline over detach to keep the membership+DTL so it can be zpool online-d later as a last-resort partial copy if sde ever fails mid- resilver. Offline gives the same speed win as detach but preserves that option.

  • Consequence: tank will report DEGRADED indefinitely (intentional offline member) — this is EXPECTED, not a fault. Don't treat DEGRADED as an alarm until sdg is dealt with.

  • Result when resilver finishes: real 2-way redundancy [sde + c382f4ac]; sdg = OFFLINE bonus 3rd option (stale + bad sectors = partial rescue only). ex-mirror-38 gone. Cost: ~14.6 TB usable + a 2nd indirect map.

  • sdg endgame (later): when a real replacement disk exists, zpool replace sdg out and pull it for RMA. Until then leave it OFFLINE.

  • STEP 2 IS SLOW — ~18 day ETA, and this is STRUCTURAL, not fixable by knobs. A ZFS healing resilver walks the entire 423T pool block-tree (work scales with pool size, not the 3.68T vdev) and politely yields to live media I/O (130–230 MB/s from 14 media CTs). 26h in: only ~518G of mirror-35's 3.68T resilvered. Pool is SAFE throughout (sde = complete healthy copy, 0 errors).

  • Tuning applied (Jun 16) to try to speed it — orig saved in .backup/zfs-tunables-*.txt. zfs_resilver_min_time_ms 3000→8000, zfs_vdev_scrub_max_active 3→6. Effect: scan phase ~2× faster, but completion ETA (issue-phase-gated) barely moved; media NOT squeezed. Left ON (harmless, may help issue phase after scan completes). Revert if no benefit: ssh pve173 'echo 3000 >/sys/module/zfs/parameters/zfs_resilver_min_time_ms; echo 3 >/sys/module/zfs/parameters/zfs_vdev_scrub_max_active' (NOTE: these are runtime-only — reset to default on reboot anyway.)

  • Only real speed lever: reduce media load (pause media CTs ~1 day) so the resilver stops yielding. Not done — pool is safe, not worth the outage.

  • Re-check ~12h after Jun 16 tuning: scan should be done (~78/423T at ~5.5T/10min); the issue rate then gives a real completion ETA.

Check progress: ssh pve173 'zpool status tank | grep -A3 "scan:"' Pre-op snapshots: .backup/zpool-pre-remove-*.txt, .backup/zpool-pre-replace-*.txt, .backup/zfs-tunables-*.txt

Immediate Next Steps (clar200 / F3 — separate thread, unblocked)

  1. Flash the bridge: cd ~/bin/clar200-unlock/firmware && ./flash.sh /dev/ttyACM0 (installs arduino-cli + esp32 core first run). Then loopback-test (jumper GPIO4↔GPIO5, confirm echo in screen /dev/ttyACM0 115200).
  2. Bench drive ZRS015V7: pull from pve173 JBOD, give standalone SAS power.
  3. Find F3 pinout (multimeter), wire TX/RX/GND, get boot banner + F3 T>.
  4. Map format cmd via ?, format to 512, verify sg_readcap/smartctl.
  5. If healthy → tank: zpool replace tank 4450364206979169854 <newdisk> (replace BEFORE detaching the dying leg), then re-mirror ex-mirror-38.
  6. Repeat whole procedure for ZR511KWK.

Blockers/Open Questions

  • Does Seth have standalone SAS power (SFF-8482→SATA-power / SAS dock), jumper wires, and a multimeter on the bench? (gates Step 2–3)
  • RMA to ServerPartDeals first vs. attempt F3 first? (leaning: F3 first)
  • Real Exos X18 F3 format token is unknown until the live ? session

Deferred Items

  • BSY isolation/tape trick — drive isn't BSY-hung; contingency only.
  • Further SCSI-level attempts — exhausted, rejected.
  • zpool scrub tank — do AFTER mirror-35 is on healthy disks (scrub hammers the dying sdg); minor CKSUM counts on mirror-10/18/19/20/30 noted.

Context for Resuming Agent

Important Context

The drive ZRS015V7 is on pve173 at /dev/sg121 + /dev/sg122 (dual SAS paths, 7:0:43:0 and 8:0:11:0), model "SS160520 CLAR200", fake serial "Maxtor 5A320J0…RAMB1TV0". The ESP32-C3 is on steel141 /dev/ttyACM0 — a DIFFERENT machine — which is fine because F3 work is bench-based; the USB side plugs into whatever host runs screen, and the GPIO side wires to the benched drive. tank mirror-35's dying leg wwn-0x5000c500c37f4eff (/dev/sdg, 30k pending sectors) is the ONLY redundancy besides the one healthy leg — do NOT detach it until a real replacement disk has resilvered in. The dead leg to replace is GUID 4450364206979169854.

Assumptions Made

  • The C3 board has onboard LED on GPIO8 (harmless if not).
  • 38400 8N1 is the F3 baud first guess (sweep if garbled).
  • Drive boots clean (answers INQUIRY) so F3 T> reachable without isolation trick.

Potential Gotchas

  • python → use python3 on steel141.
  • USB-CDC baud is virtual; the drive baud is set with the firmware's ~N menu, not the host terminal baud.
  • Never wire ESP32 VCC/5V to the drive — TX/RX/GND only; drive self-powered.
  • flash.sh first run downloads ~1 GB esp32 core.

Environment State

Tools/Services Used

  • pve173 (tank host): zpool, smartctl, sg_*, sg_reset via ssh pve173.
  • steel141: esptool (installed), ESP32-C3 on /dev/ttyACM0. arduino-cli NOT yet installed (flash.sh installs it).
  • Gitea: repo https://git.sethpc.xyz/Seth/clar200-unlock (pushed).

Active Processes

  • None started/left running. No flashing done. No zpool mutations performed (diagnosis was read-only except a harmless SCSI bus reset on the dead drive).

Environment Variables

  • HOMELAB_PASSWORD (used indirectly via SSH aliases) — name only.

Related Resources

  • Repo: https://git.sethpc.xyz/Seth/clar200-unlock
  • ~/bin/EMC_CLAR200_DRIVE_FIX_NOTES.md — prior exhaustive attempt
  • ~/bin/CLAUDE.md — storage section (tank, drive gotchas, ZL2PNZ99/ZL21R9ZF)
  • Bootstrap handoff: ./2026-06-14-bootstrap.md

Security Reminder: Before finalizing, run validate_handoff.py to check for accidental secret exposure.

Reference in New Issue View Git Blame Copy Permalink