From 9982cf5666ffc7d51e931abd7c35ab297ac58293 Mon Sep 17 00:00:00 2001 From: YeonGyu-Kim Date: Sat, 30 May 2026 19:12:12 +0900 Subject: [PATCH] docs(omo-codex): batch 71 (6 files) --- .../references/runtimes/bundled-js-binary.md | 415 +++++++++++++++ .../debugging/references/runtimes/go.md | 252 +++++++++ .../references/runtimes/native-binary.md | 484 ++++++++++++++++++ .../debugging/references/runtimes/node.md | 260 ++++++++++ .../debugging/references/runtimes/python.md | 248 +++++++++ .../debugging/references/runtimes/rust.md | 234 +++++++++ 6 files changed, 1893 insertions(+) create mode 100644 packages/omo-codex/plugin/skills/debugging/references/runtimes/bundled-js-binary.md create mode 100644 packages/omo-codex/plugin/skills/debugging/references/runtimes/go.md create mode 100644 packages/omo-codex/plugin/skills/debugging/references/runtimes/native-binary.md create mode 100644 packages/omo-codex/plugin/skills/debugging/references/runtimes/node.md create mode 100644 packages/omo-codex/plugin/skills/debugging/references/runtimes/python.md create mode 100644 packages/omo-codex/plugin/skills/debugging/references/runtimes/rust.md diff --git a/packages/omo-codex/plugin/skills/debugging/references/runtimes/bundled-js-binary.md b/packages/omo-codex/plugin/skills/debugging/references/runtimes/bundled-js-binary.md new file mode 100644 index 000000000..a64c80736 --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/runtimes/bundled-js-binary.md @@ -0,0 +1,415 @@ +# Bundled-JS / Embedded-Source Binaries (Bun SEA, Node SEA, Deno compile, pkg, Electron, PyInstaller) + +A growing class of "binaries" are not stripped C/C++ at all — they are a runtime VM glued onto a high-level-language bundle. The bundle is **plaintext or trivially-decodable** inside the binary. + +If you reach for `native-binary.md` workflow on these (Ghidra → pwndbg → hex), you will waste hours decompiling a runtime you don't care about while the actual logic sits exposed three megabytes away. + +**This reference exists because the workflow is fundamentally different from stripped C.** + +--- + +## When to use this reference instead of `native-binary.md` + +Open this if `file ./target` shows a generic Mach-O / ELF / PE BUT any of: + +- Size is suspiciously large (50 MB+ for a "simple CLI") +- `strings -n 8 ./target | rg -i "node_modules|webpack|esbuild|bun|pkg/lib|electron|pyinstaller"` returns hits +- The binary's CLI flags include things like `--inspect`, `--unhandled-rejections`, npm-style help text +- Vendor docs say it's built with Bun / pkg / nexe / Deno compile / PyInstaller / Electron / Tauri (UI shell) +- `head -c 4 ./target | xxd` shows a known runtime magic for an embedded archive section + +If yes → **stop following `native-binary.md` and follow this**. Triage and dynamic tracing are the same. Static analysis is completely different. + +--- + +## The workflow + +``` + [1] Triage → identify the bundler (Bun? pkg? Deno? Electron? PyInstaller?) + [2] Locate the bundle → find where the embedded source archive starts + [3] Extract → dump source to disk so you can grep / read it + [4] Source-level static analysis (rg + Read, NOT Ghidra) + [5] Runtime verification → debug logs, --inspect, partial-evidence patterns + [6] Fix / report +``` + +Step 3 is the unlock — once you have plaintext source on disk, the rest is normal codebase exploration. + +--- + +## [1] Identify the bundler — 30-second fingerprint + +```bash +# Look for runtime-specific markers in plaintext strings +strings -n 12 ./target 2>/dev/null | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|NODE_SEA_BLOB|tauri|ESZIP_V2|denort' | head -20 +``` + +| Marker pattern | Bundler | Source format | +|---|---|---| +| `@oven/bun-darwin`, `bun-lockfile-format-v`, `// @bun` | **Bun SEA** (compiled via `bun build --compile`) | Plaintext JS, single big bundle | +| `NODE_SEA_BLOB` + `NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1` | **Node SEA** (`node --build-sea` or `--experimental-sea-config`) | Plaintext JS, or V8 code cache (when `useCodeCache: true`), or startup snapshot (when `useSnapshot: true`) — the latter two are NOT plaintext | +| `pkg/lib/bootstrap.js`, `pkg/prelude`, `PAYLOAD_POSITION` | **pkg** (vercel/pkg) | Plaintext or v8 cached data | +| `ESZIP_V2`, `denort`, `deno_runtime` | **Deno compile** (`deno compile`) | TS/JS in eszip archive — readable but needs `eszip` crate to walk; not pure plaintext | +| `Electron`, `app.asar`, `chrome.dll`, `Squirrel.Mac` | **Electron** | `app.asar` archive (TAR-like with JSON header). Source is plaintext JS once extracted | +| `PyInstaller`, `pyz`, `_MEIPASS`, `pyi-os-utils` | **PyInstaller** | Compressed `.pyc` bytecode — needs `pyinstxtractor` + `decompyle3` to recover Python source | +| `nexe-`, `nexe_compile`, `:::nexe::` | **nexe** | Plaintext JS appended to node binary | +| `Tauri`, `tao`, `wry`, `tauri::generate_context` | **Tauri** (Rust shell + JS UI) | **Two worlds**: JS frontend in resource section is extractable here; Rust commands / core logic are native and require [native-binary.md](native-binary.md) | + +If multiple match (e.g. Tauri + Bun): the outer shell is the first one (Tauri/Electron). The inner JS is the second one's format. **For Tauri specifically, expect to use both this reference (for the UI bundle) and `native-binary.md` (for the Rust binary side).** + +> **Source-format reality check**: only Bun SEA, pkg (when not using `--public-packages`), nexe, and Electron `.asar` are reliably plaintext. Node SEA with code-cache or snapshot, PyInstaller `.pyc`, and Deno eszip require additional tooling. Don't assume `strings` will find readable code — verify the bundler first. + +--- + +## [2] Locate the bundle + +### Bun SEA — JS is just embedded plaintext + +The JS source is concatenated into the binary as a giant template literal / string. No decoding needed. + +```bash +# Verify by searching for typical JS bundle markers +strings -n 8 ./target | rg "function|var |let |const |async function" | head -5 + +# Find where the bundle starts (look for "use strict" or banner comment) +LC_ALL=C grep -aob '"use strict"' ./target | head -5 +LC_ALL=C grep -aob '#!/usr/bin/env bun' ./target | head -5 +``` + +### Node SEA — `NODE_SEA_BLOB` resource/segment + activated fuse + +Per the [Node.js SEA docs](https://nodejs.org/api/single-executable-applications.html), a Node-built SEA contains: +- A resource (PE), section in `NODE_SEA` segment (Mach-O), or note (ELF) named `NODE_SEA_BLOB` +- The fuse string `NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1` (with trailing `:1` indicating injected; `:0` means a copy of the node binary that has not yet had a blob injected) + +```bash +# Confirm it is a SEA at all +LC_ALL=C grep -aob 'NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1' ./target | head -1 + +# Find the blob resource/section +LC_ALL=C grep -aob 'NODE_SEA_BLOB' ./target | head + +# On Mach-O, inspect the segment directly +otool -l ./target | grep -A4 'NODE_SEA' +``` + +The blob format is documented but non-trivial to walk by hand. For extraction, **use postject in reverse** (carve the section bytes) or read the blob via `node:sea` API from inside a debug build of the same binary. Plain `strings` will get you the embedded JS only when the SEA was built without `useCodeCache` and without `useSnapshot` — both of those replace plaintext with V8 cache data or startup snapshot bytes. + +`node --build-sea sea-config.json` and `node --experimental-sea-config sea-config.json` *generate* SEA blobs; neither inspects an existing executable. + +### Deno compile — eszip archive section + +```bash +# Deno-compile binaries embed an eszip v2 archive +LC_ALL=C grep -aob 'ESZIP_V2' ./target | head -3 +# Also confirm the runtime +LC_ALL=C grep -aob 'denort' ./target | head -1 +``` + +To extract, use the `eszip` Rust crate (or the `@deno/eszip` JS port) to parse the archive after carving it out at the offset above. There is no stable Deno CLI flag that inspects compiled-executable eszip contents as of 2026-04 — `deno info` only works on source files. + +### pkg — `PAYLOAD_POSITION` marker + +```bash +LC_ALL=C grep -aob 'PAYLOAD_POSITION' ./target | head +LC_ALL=C grep -aob 'pkg/prelude' ./target | head +``` + +For source extraction, use the `pkg-extract` tooling community projects or carve based on the offset reported by the `PAYLOAD_POSITION:` value. + +### Electron — `app.asar` is usually a separate file + +Most Electron apps ship `app.asar` next to the binary, not embedded inside. Extract it with the official tool: + +```bash +# macOS layout +ls -la /Applications/MyApp.app/Contents/Resources/app.asar +npx @electron/asar extract app.asar ./extracted/ +# or older: +npx asar extract app.asar ./extracted/ +``` + +For single-file builds where the asar is embedded inside the executable, **do not pattern-match arbitrary 4-byte sequences** (the asar format starts with a Pickle-encoded uint32 header size + JSON metadata, and the same bytes appear elsewhere in any binary). Instead, use a Pickle-aware extractor that validates the JSON header before claiming a match — the `asar` npm package's programmatic `extractAll()` API does this. Carve the asar bytes by scanning for a candidate Pickle header (4-byte size + 4-byte payload size + `{"files":` prefix), validate the JSON parses, then feed the carved buffer to `extractAll()`. + +### PyInstaller — use `pyinstxtractor`, NOT runtime self-extraction + +```bash +# Recover the embedded archive without running the binary +python3 pyinstxtractor.py ./target +# Output: ./target_extracted/ with .pyc files + +# Decompile the .pyc files back to Python source +decompyle3 ./target_extracted/main.pyc # Python 3.7+ +uncompyle6 ./target_extracted/main.pyc # older Python +``` + +If `pyinstxtractor` cannot read the archive (e.g. non-standard PyInstaller version), use the official `pyi-archive_viewer` tool that ships with PyInstaller. Avoid the "run-the-binary-and-snoop-`/tmp/_MEI*`" approach: it only catches what runs in the time window between `_MEIPASS` extraction and cleanup, and it executes potentially untrusted code. + +--- + +## [3] Extract source to disk — DO NOT skip this + +**The single biggest mistake** with bundled-JS reverse engineering is trying to read the source out of `strings` output or `xxd` dumps. You will lose data. See "Gotchas" below. + +### For Bun SEA / nexe / single-string-blob bundlers + +Read the binary as bytes, find the JS section, save to a `.js` file: + +```python +# extract_bundled_js.py +import sys + +if len(sys.argv) < 2: + raise SystemExit("usage: extract_bundled_js.py ") + +with open(sys.argv[1], 'rb') as f: + data = f.read() + +markers = [b'// @bun', b'"use strict"', b"'use strict'", b'#!/usr/bin/env'] +start = -1 +for m in markers: + p = data.find(m) + if p != -1 and (start == -1 or p < start): + start = p + +if start == -1: + raise SystemExit( + "no bundle marker found — binary may not be Bun/nexe, " + "or markers were stripped. Try strings(1) for hints." + ) + +# Heuristic end: look for a long null run AFTER start. +# This is a heuristic, NOT a guarantee. Verify the tail of the output +# looks like JS (closing braces, EOF) before trusting it. +end = data.find(b'\x00' * 1024, start) +if end == -1: + end = len(data) + +bundle = data[start:end] +print(f'Extracted {len(bundle)} bytes from offset {start} to {end}', file=sys.stderr) +sys.stdout.buffer.write(bundle) +``` + +```bash +python3 extract_bundled_js.py ./target > extracted-bundle.js +wc -c extracted-bundle.js +# Sanity check the tail is JS, not random binary +tail -c 200 extracted-bundle.js +``` + +### For PyInstaller + +Use `pyinstxtractor` then `uncompyle6` / `decompyle3` on the `.pyc` files. + +### For Electron .asar + +```bash +npx asar extract app.asar ./extracted/ +# Now ./extracted/ has a normal node_modules + your source layout +``` + +### For Deno compile + +Use the `eszip` Rust crate or the `@deno/eszip` JS port to walk the archive after carving the eszip section out at the offset reported by the `ESZIP_V2` magic search. There is no stable Deno CLI as of 2026-04 that inspects compiled-binary eszip contents directly. + +--- + +## [4] Source-level static analysis — `rg` + `Read`, not Ghidra + +Once you have the source on disk, treat it as a normal codebase: + +```bash +# Find function definitions +rg -n "^function |^const \w+ = (function|\(.*\) =>)" extracted-bundle.js | head + +# Find specific behavior +rg -n "claude-opus-4-7|reasoning_effort|api_key" extracted-bundle.js + +# Resolve minified identifiers — they show up as `var XYZ="value"` +rg -aoP 'var \w+="[^"]+"' extracted-bundle.js | head -50 +``` + +For minified bundles, use a template-literal-aware parser to extract specific functions or template strings. Example skeleton: + +```python +def find_template_end(data, start): + """Walk a JS template literal preserving ${...} interpolation depth. + Returns position of closing backtick.""" + i = start + while i < len(data): + c = data[i:i+1] + if c == b'\\': + i += 2; continue + if c == b'$' and data[i+1:i+2] == b'{': + depth = 1; i += 2 + while i < len(data) and depth > 0: + cc = data[i:i+1] + if cc == b'\\': i += 2; continue + if cc == b'`': + j = find_template_end(data, i+1) + i = j + 1; continue + if cc == b'{': depth += 1 + elif cc == b'}': depth -= 1 + elif cc in (b'"', b"'"): + q = cc; i += 1 + while i < len(data) and data[i:i+1] != q: + if data[i:i+1] == b'\\': i += 2 + else: i += 1 + i += 1; continue + i += 1 + continue + if c == b'`': return i + i += 1 + return -1 +``` + +For function-body extraction, **track the parameter list separately** before tracking body braces. The naive approach mis-counts destructuring `function f({a, b, ...c})` as the body `{` and exits early. + +--- + +## [5] Runtime verification + +You usually cannot single-step JS inside a Bun-compiled binary the way you would with `node --inspect`. Workarounds: + +### Bun-compiled + +Bun's inspector takes `--inspect[=:[/]]` on the command line. For env-var control of compiled binaries, the form is the same minus the leading `--`: + +```bash +# Default port (6499) auto-prefix +./target --inspect +# → ws://localhost:6499/ (paste into https://debug.bun.sh) + +# Explicit host:port[/prefix] +./target --inspect=localhost:9229/dbg +# Or via env var (if --inspect cannot be passed) +BUN_INSPECT=localhost:9229/dbg ./target +``` + +**For HTTP request tracing without an interactive debugger** (highest-value Bun-specific runtime evidence): + +```bash +# Print every fetch() / node:http request as a curl command + full headers/body +BUN_CONFIG_VERBOSE_FETCH=curl ./target ... + +# Or just print the request/response without curl-format +BUN_CONFIG_VERBOSE_FETCH=true ./target ... +``` + +Plus generic env-var-based debug logging if the app supports it: + +```bash +APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target +``` + +### Node SEA / pkg / nexe +```bash +# These usually accept --inspect since they are real Node +./target --inspect +# Then chrome://inspect or node --inspect-brk +``` + +### Electron +```bash +./target.app/Contents/MacOS/target --inspect=9229 --remote-debugging-port=9223 +# Renderer process is at chrome://inspect, main process via the inspector port +``` + +### When you cannot make a real call +The target's API may require credentials, network access, or paid quota you don't have. **You are not stuck** — see [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for the fallback patterns. + +--- + +## ⚠️ Gotchas — read these before extracting + +### G1. `strings -n N` silently drops short identifier interpolations + +`strings` outputs runs of printable characters of length **≥ N**. Default is 4 on most systems; many references (including older versions of `native-binary.md`) recommend `-n 8` for less noise. + +**With `-n 8`, short template-literal interpolations like `${x}`, `${i}`, `${R}` are silently dropped** because they are 4 chars surrounded by non-printable bytes (newlines or section padding). The result looks like: + +```text +expected: \n${x}\n +strings: \n ← ${x} is gone, no warning +``` + +A consumer reading the strings output would conclude the template is empty. + +**Mitigation**: +1. Use `strings` only for **fingerprinting** (Phase 1 triage), never as the source of extracted text. +2. For actual extraction, **read the binary as bytes** with `python3 -c "open('./target','rb').read()"` and grep / parse from there. +3. If you must use `strings`, try `strings -n 1 -t x ./target` and post-filter — but byte-level reads are still more reliable. + +### G2. Stale cached binary ≠ latest features + +Bundled-app installers often check a remote version and skip download if a cached binary exists. If you reverse-engineered an old version and the user reports behavior you don't see in the source, **re-run the installer** (or fetch the version manifest manually) before assuming the source is current. + +```bash +# Example pattern - varies by tool +curl -fsSL https://example.com/install.sh | head -50 # find version-fetch URL +curl -fsSL https://static.example.com/cli/cli-version.txt +./your-tool --version +# Compare. If different, re-install. +``` + +### G3. APFS / NTFS case-insensitivity silently overwrites files + +When extracting many minified function bodies (`cVR`, `CVR`, `dpr`, `DPR`, …) and saving each to its own file, **macOS APFS and Windows NTFS treat `cVR.txt` and `CVR.txt` as the same file**. The second write silently overwrites the first. + +**Mitigation**: prefix filenames with something case-distinguishing, e.g. `mode-cVR.txt`, `mode-CVR.txt`, or use a hash suffix. + +### G4. Bun's runtime adds 30-50 MB of unrelated symbols + +A 70 MB Bun-compiled binary is **mostly Bun runtime** (~50 MB) plus your app (~20 MB). When fingerprinting, you will see thousands of strings like `tree-sitter-typescript`, `react-native-stylex` etc. that the user's actual app doesn't use — these are package names baked into Bun's package-resolution data. + +**Mitigation**: when grepping for "what does this app do?", filter out runtime noise: +```bash +strings -n 8 ./target | rg -v 'node_modules|@oven/bun|package-lock|tree-sitter|ffmpeg-installer' | head +``` + +### G5. Source maps usually NOT shipped + +Bundled apps strip source maps for production. Variable names are minified to `T`, `R`, `a`, `r`, etc. Treat the bundle like an obfuscated codebase: identify constants by tracing assignments (`var T="actual-name"`) and resolve interpolations manually. + +### G6. The "extract" file is not legally redistributable + +If reverse-engineering proprietary software, the extracted source is the vendor's IP. Use it for understanding behavior, **never commit it to git**, never post snippets in public issues. Cleanup your `extracted-bundle.js` files in Phase 9. + +--- + +## Silent-failure patterns specific to bundled JS + +| Pattern | Why it's silent | +|---|---| +| Bundle includes unreachable dead code from tree-shaking failures | You read code that never runs — verify with runtime trace | +| `process.env.X` resolved at BUILD time, not RUNTIME | Setting the env var at runtime has no effect; the value is baked in | +| `import.meta.url` in compiled binary returns `bun://...` not a real path | File-relative resolution silently breaks | +| Worker threads spawn from embedded code, look for sub-bundle inside main bundle | Workers may have their own copy of dependencies | +| Minified identifiers with case variants used in same module | Easy to confuse `cVR` with `CVR` when reading fast | + +--- + +## Phase 9 cleanup specifics for bundled-JS work + +```bash +# Remove extracted bundles — they may contain proprietary source +rm -f /tmp/extracted-bundle.js /tmp/extracted-*.js +rm -rf /tmp/asar-extracted/ +rm -rf /tmp/_MEI* + +# Remove strings dumps +rm -f /tmp/*-strings.txt /tmp/*-strings-v*.txt + +# Remove Python helper scripts created for parsing +rm -f /tmp/extract_bundled_js.py /tmp/parse_template.py + +# Verify the extraction directory is gone (if you used a workspace dir) +ls /Users/$USER/local-workspaces/*-extracted/ 2>/dev/null +# rm -rf only after journal review confirms nothing important is there +``` + +--- + +## When to escalate back to `native-binary.md` + +If extraction reveals the "bundle" is actually compiled to v8 cached data (pkg with `--public-packages` or PyInstaller with bytecode-only mode), and decompilation is non-trivial, **switch back to `native-binary.md` workflow** (Ghidra against the runtime + careful tracing). Bundled-JS workflow only helps when the high-level source is recoverable as readable text. diff --git a/packages/omo-codex/plugin/skills/debugging/references/runtimes/go.md b/packages/omo-codex/plugin/skills/debugging/references/runtimes/go.md new file mode 100644 index 000000000..afed82eda --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/runtimes/go.md @@ -0,0 +1,252 @@ +# Go Debugging + +Covers goroutines, `dlv` (Delve), `pprof`, the race detector, and the fact that Go's concurrency model means most bugs are about goroutines doing something quiet and wrong. + +--- + +## Environment detection (Phase 0) + +```bash +go version +cat go.mod | head -5 + +# Delve installed? +which dlv +dlv version + +# Build constraints +grep -r '// +build\|//go:build' cmd/ internal/ pkg/ 2>/dev/null | head + +# pprof wired up? +grep -r 'net/http/pprof\|runtime/pprof' --include='*.go' | head -3 +``` + +--- + +## Delve (`dlv`) — the Go debugger + +Go's gc compiler emits DWARF, but plain gdb barely understands goroutines. **Use dlv, not gdb.** Plain gdb on a Go binary will miss goroutine state and print garbage for interface values. + +### The five `dlv` launch modes + +```bash +# Build and launch under debugger (equivalent to `go run` + debug) +dlv debug ./cmd/server -- --port=8080 + +# Debug a test binary +dlv test ./internal/handler/ # enters the test package under debug + +# Debug an existing binary (must be built with -gcflags="all=-N -l" for best results) +dlv exec ./bin/myserver + +# Attach to a running process +dlv attach $(pgrep myserver) + +# Headless mode (IDE / remote attach) — default port 2345 +dlv debug --headless --listen=:2345 --api-version=2 ./cmd/server +``` + +### Building a debuggable binary + +The compiler inlines and optimizes aggressively in normal builds, which makes stepping confusing. For serious debugging: + +```bash +go build -gcflags="all=-N -l" -o ./bin/server ./cmd/server +# -N disables optimization +# -l disables inlining +``` + +Then `dlv exec ./bin/server`. + +### Essential dlv commands + +``` +(dlv) b main.main # breakpoint at function +(dlv) b handler.go:42 # breakpoint at file:line +(dlv) b pkg/foo.Bar # breakpoint at type method (Go path syntax) +(dlv) c / continue # continue until next break +(dlv) n / next # step over +(dlv) s / step # step into +(dlv) so / stepout # step out +(dlv) bt / stack # stack trace of current goroutine +(dlv) goroutines # list all goroutines +(dlv) goroutine # switch to goroutine N +(dlv) goroutine bt # stack of a specific goroutine +(dlv) locals # all locals in frame +(dlv) args # function args +(dlv) p # print value (understands interfaces, maps, slices) +(dlv) vars # package vars matching regex +(dlv) regs # registers (rare in Go debugging) +(dlv) on print # auto-print on breakpoint hit (powerful!) +(dlv) trace # like breakpoint but just logs, doesn't stop +``` + +The `trace` command is underused — it's like a logpoint, no stepping required. + +--- + +## Goroutine-centric debugging + +Goroutine leaks and deadlocks are the most common Go bugs. `dlv`'s `goroutines` command is the starting point. + +``` +(dlv) goroutines -t # with truncated stack +(dlv) goroutines -s # sorted by stack +(dlv) goroutines -with user # filter user-spawned goroutines +``` + +Common patterns: + +| You see in `goroutines` | Usually means | +|---|---| +| 100s of goroutines stuck at `chan receive` | Producer died; consumers leak | +| 100s stuck at `semacquire` | Lock contention; a holder probably deadlocked | +| One stuck at `select` with no default | Missing case or closed channel scenario | +| Stuck at `netpoll` | External I/O not responding — not a Go bug, check downstream | +| Growing count over time | Goroutine leak — need to find who's spawning without cleanup | + +### Panic signals in Go + +```go +// Without recovery, panics crash the program with a stack trace of ALL goroutines +// With recovery, they're silent unless explicitly logged: +defer func() { + if r := recover(); r != nil { + log.Printf("recovered panic: %v\n%s", r, debug.Stack()) // GOOD + // log.Printf("recovered") // BAD — silent + } +}() +``` + +**Always check for silent recovers** in Phase 8. Grep: +```bash +rg 'recover\(\)' --type go +``` + +And inspect each site for whether the panic is actually surfaced. + +--- + +## Race detector — ALWAYS run when the bug is intermittent + +```bash +go test -race ./... +go run -race ./cmd/server +go build -race ./cmd/server +``` + +The race detector wraps memory accesses and catches concurrent read/write without synchronization. **Run this before attaching dlv** if intermittency is involved — it often finds the bug directly. + +Output shape: +``` +WARNING: DATA RACE +Read at 0x00c0001a0080 by goroutine 7: + main.(*Counter).Value() + /path/to/counter.go:14 +0x3c +Previous write at 0x00c0001a0080 by goroutine 6: + main.(*Counter).Inc() + /path/to/counter.go:10 +0x5f +``` + +Both stacks. Both goroutines. The race is obvious from the line pair. + +--- + +## pprof — for perf, memory, goroutine leaks + +### Wire it up (idempotent; usually already present) + +```go +import _ "net/http/pprof" + +func main() { + go func() { + log.Println(http.ListenAndServe("localhost:6060", nil)) + }() + // ... rest of your server +} +``` + +### Queries + +```bash +# CPU profile (30s) +go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30 + +# Heap snapshot +go tool pprof http://localhost:6060/debug/pprof/heap + +# Goroutine snapshot — find leaks +go tool pprof http://localhost:6060/debug/pprof/goroutine + +# Block profile — find blocking ops (needs runtime.SetBlockProfileRate) +go tool pprof http://localhost:6060/debug/pprof/block + +# Mutex profile — find lock contention (needs runtime.SetMutexProfileFraction) +go tool pprof http://localhost:6060/debug/pprof/mutex +``` + +Inside pprof: +``` +(pprof) top # top functions by self time +(pprof) list main.handler # annotated source of a function +(pprof) web # SVG callgraph in browser (requires graphviz) +(pprof) traces # sample traces +``` + +For goroutine leaks, **take two snapshots 30s apart** and diff: +```bash +go tool pprof -base prof1.pb.gz prof2.pb.gz +``` + +Goroutines that appear in prof2 but not prof1 are new; if they stick around, they're leaking. + +--- + +## `GODEBUG` — runtime-level observability + +```bash +GODEBUG=gctrace=1 ./myserver # print GC stats +GODEBUG=schedtrace=1000 ./myserver # scheduler trace every 1000ms +GODEBUG=scheddetail=1,schedtrace=1000 # detailed scheduler state +GODEBUG=allocfreetrace=1 ./myserver # every alloc/free (noisy!) +GODEBUG=memprofilerate=1 ./myserver # profile every allocation +``` + +Useful for diagnosing GC pressure, goroutine starvation, or memory pattern issues. + +--- + +## Silent-failure patterns in Go + +| Pattern | Why it's silent | +|---|---| +| `if err != nil { return err }` that returns to a caller that ignores | Error bubbles up, then gets discarded at the top | +| `defer func() { recover() }()` — bare recover, no log | Panic swallowed, program continues with state corruption | +| `_, _ = conn.Write(data)` | Intentionally discarded error | +| Buffered channel send that blocks forever | Sender hangs; hard to see if no deadlock detection | +| `time.Sleep` in a test | "Works on my machine"; test passes locally, fails in CI | +| `go func() { ... }()` with no error path | Goroutine dies silently on panic unless recover+log | +| Context canceled but operation continues | Ignored `ctx.Err()` check | +| `json.Unmarshal` of zero-value struct field | Input missing the key; silently zero | +| Closed channel read returning zero value | Consumer doesn't check `ok`; reads forever | + +--- + +## Phase 9 cleanup specifics + +```bash +# Kill dlv sessions +pkill -f 'dlv' || true +lsof -iTCP:2345 -sTCP:LISTEN -nP 2>/dev/null # dlv default + +# Kill pprof HTTP endpoint if you started it just for this session +lsof -iTCP:6060 -sTCP:LISTEN -nP 2>/dev/null + +# Revert any `fmt.Println("DEBUG: ...")` or `log.Printf("DEBUG: ...")` additions +git diff | grep -E '(fmt\.Println\("DEBUG|log\.Printf\("DEBUG|println!)' +git checkout + +# Unset env vars +unset GODEBUG +``` diff --git a/packages/omo-codex/plugin/skills/debugging/references/runtimes/native-binary.md b/packages/omo-codex/plugin/skills/debugging/references/runtimes/native-binary.md new file mode 100644 index 000000000..b7c303a91 --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/runtimes/native-binary.md @@ -0,0 +1,484 @@ +# Native Binary Debugging (No Source / Reverse Engineering) + +For binaries where you don't have trustworthy source: stripped production builds, third-party closed libs, malware, CTF challenges, firmware, vendored libs whose docs lie. The workflow is specific; doing it out of order wastes days. + +This reference **coordinates** the triage and dynamic work. The heavy tools each have their own reference: +- **Static decompilation** → [tools/ghidra.md](../tools/ghidra.md) +- **Interactive debugging** → [tools/pwndbg.md](../tools/pwndbg.md) +- **Scripted interaction / exploitation** → [tools/pwntools.md](../tools/pwntools.md) + +Read those before using them — especially Ghidra, which has a surprising amount of workflow that's not obvious. + +--- + +## ⚠️ STOP — is this actually a stripped C/C++ binary? + +A growing share of "binaries" are actually **bundled high-level apps** — Bun SEA, Node SEA, Deno compile, pkg, nexe, Electron, Tauri, PyInstaller. Their workflow is completely different: the high-level source is recoverable with the right per-bundler tool (often plaintext, sometimes V8 cache / `.pyc` / eszip needing extra tooling), and Ghidra against the runtime VM wastes hours. + +Quick check: + +```bash +file ./target # Mach-O / ELF / PE - inconclusive +du -h ./target # 50 MB+ for a "simple CLI" → suspect bundled +strings -n 12 ./target | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|tauri' | head -5 +``` + +**If any hits** → close this file, open [bundled-js-binary.md](bundled-js-binary.md) instead. Following the Ghidra/pwndbg path on a bundled-app binary wastes hours decompiling the runtime VM while the app-level bundle is recoverable with the right per-bundler tool (plaintext for Bun/pkg/nexe/Electron-asar; eszip / V8-cache / `.pyc` for Deno / Node SEA / PyInstaller). + +If `file` says "Mach-O" or "ELF", `du` is < 20 MB, and the strings check is empty → continue here. + +--- + +## The workflow (do these in order) + +Every step's output is input to the next. Skipping steps means guessing later. + +``` + [1] Triage → what kind of binary is this? + [2] Dynamic tracing → what syscalls / libcalls does it make? + [3] Static analysis → what does it DO, in readable form? (Ghidra) + [4] Dynamic debug → confirm hypotheses at runtime (pwndbg) + [5] Scripted repro → lock the bug with a pwntools script + [6] TDD + fix / report +``` + +Steps 1 and 2 are fast (minutes). Step 3 is slow (tens of minutes to hours depending on size). Don't skip 1-2 and go straight to Ghidra — the triage output tells you what to focus on inside Ghidra. + +--- + +## [1] Triage — 5-minute fingerprint + +```bash +# Basic identity +file ./target +# elf, mach-o, pe? 32/64-bit? dynamically linked? stripped? + +# Architecture details +readelf -h ./target # ELF header: entry point, arch, type +lipo -info ./target 2>/dev/null # macOS: universal binary? + +# Interesting strings (often leaks function names, error messages, URLs, API keys) +strings -n 8 ./target | head -100 +strings -n 8 ./target | grep -iE '(http|/api/|error|debug|version)' + +# Imported symbols (what does it link against?) +nm -D ./target 2>/dev/null # dynamic symbols +objdump -T ./target 2>/dev/null # same, alternate tool +readelf -d ./target # dynamic section (NEEDED libs) +ldd ./target 2>/dev/null # resolved library paths + +# Security posture (affects what exploits / bugs are possible) +checksec --file=./target # requires pwntools or installing checksec +# NX, PIE, RELRO, stack canary, FORTIFY + +# Is it stripped? +nm ./target 2>/dev/null | head # empty? stripped. full? not stripped. +file ./target # will say "stripped" or "not stripped" +``` + +### ⚠️ `strings -n N` silently drops short content + +`strings` prints runs of printable characters of length **≥ N**. With `-n 8`, **anything shorter than 8 chars sandwiched between non-printable bytes is dropped silently**. This includes: + +- Short identifier interpolations in templates (`${x}`, `${i}`, `${R}`) +- Short embedded constants (`v3`, `null`, integer immediates as bytes) +- Short error codes between binary padding + +Real example: a JavaScript template literal `\n${x}\n` came out of `strings -n 8` as `\n` — the `${x}` (4 chars) was dropped. A consumer reading the dump would conclude the template was empty. It is not. + +**Use `strings` only for fingerprinting (Phase 1).** For any extraction whose correctness matters, **read bytes directly**: + +```bash +# Count occurrences of a needle +LC_ALL=C grep -aoc 'NEEDLE' ./target + +# Find offsets +LC_ALL=C grep -aob 'NEEDLE' ./target | head + +# Or via Python for byte-precise context +python3 -c " +import sys +data = open('./target','rb').read() +needle = b'NEEDLE' +pos = data.find(needle) +print(repr(data[max(0,pos-100):pos+200])) +" +``` + +If you must keep using `strings`, lower the threshold: `strings -n 1 -t x ./target | rg ...`. The signal-to-noise drops sharply but short content is preserved. + +Write the triage summary to the journal: + +```markdown +## Binary triage +- Type: +- Arch: +- Libs: +- Security: +- Interesting strings: +- First hypothesis surface: +``` + +--- + +## [2] Dynamic tracing — what does it actually call? + +These are cheap — run them before Ghidra to orient yourself. + +### Linux: strace + ltrace + +```bash +# System calls +strace -f -o trace.out ./target arg1 arg2 +strace -f -e trace=network ./target # filter to network syscalls +strace -f -e trace=file ./target # filter to file ops + +# Library calls (less useful when stripped but still informative) +ltrace -f -o ltrace.out ./target +ltrace -f -e 'str*+mem*' ./target # filter to string/mem functions +``` + +### macOS: Mach-O specifics + +**SIP block reality check.** With System Integrity Protection enabled (default on every modern macOS), `dtruss` / `dtrace` will **silently fail** to attach to: +- Anything in `/usr`, `/bin`, `/sbin`, `/System` +- Apple-signed binaries (Xcode CLT, Homebrew formulae from Apple-distributed taps) +- Notarized vendor binaries (Bun, Deno, Docker Desktop, etc.) + +`dtruss ./target` will appear to run but produce zero events. This is not a bug; it is the SIP design. Disabling SIP requires a Recovery Mode reboot — usually not worth it. Use the alternatives below. + +```bash +# dtruss — works only when SIP allows it (your own unsigned binaries) +sudo dtruss -f ./target 2>&1 | head -20 # equivalent to strace +# If output is suspiciously empty → SIP blocked it. Switch to lldb or app-level logging. +``` + +**Mach-O metadata inspection (no SIP issues, no debugger needed):** + +```bash +# Architecture and slices +file ./target # arm64 / x86_64 / universal +lipo -info ./target # which architectures included +lipo -thin arm64 ./target -output ./target-arm64 # extract one slice for analysis + +# Headers & load commands (segments, dylibs, code-signature pointer) +otool -h ./target # Mach header (cputype, ncmds, flags) +otool -l ./target | head -100 # load commands; entitlements live in code-signature blob, see codesign below + +# Dynamic library dependencies (macOS equivalent of ldd) +otool -L ./target # linked dylibs with versions +dyld_info ./target # macOS 13+, more detailed than otool -L + +# Disassembly +otool -tv ./target | head -200 # quick disassembly without Ghidra +otool -tV ./target # with symbol-resolved branches + +# Imported / exported symbols (Apple `nm`, NOT GNU) +nm -u ./target # undefined references = imports +nm -gU ./target # external defined = exports +# Note: GNU `-D`/dynamic flags are not honored on Apple `nm`; use the above forms. +symbols -fullSourcePath -onlyWithDebugInfo ./target # if any debug info survives + +# Code signature & entitlements (entitlements come from codesign, NOT otool) +codesign -dv --entitlements :- ./target 2>&1 # signature info + entitlements XML on stdout +spctl --assess --type execute -vv ./target # Gatekeeper assessment + +# Cert chain — extract to a temp dir to avoid creating files named -0/-1 in cwd +tmp=$(mktemp -d) +codesign -dvv --extract-certificates="$tmp/cert" ./target 2>&1 +ls -la "$tmp" +# rm -rf "$tmp" # journal first, clean up later + +# Strings inside specific segments only (less noise than full-binary strings) +otool -s __TEXT __cstring ./target # C string section +otool -s __TEXT __const ./target # constants section +``` + +**Interactive debugging on macOS — use `lldb`, not `gdb`.** + +GDB on macOS requires a self-signed code-signing certificate (`codesign --entitlements gdb.entitlements --sign gdb-cert /opt/homebrew/bin/gdb`) and even then is unreliable on arm64. **Use `lldb` directly** — it ships with Xcode CLT and works without configuration. + +```bash +# Start lldb +lldb ./target + +# Set arguments +(lldb) settings set target.run-args arg1 arg2 + +# Run with breakpoints +(lldb) breakpoint set --name function_name # symbol-based +(lldb) breakpoint set --address 0x1000034c0 # address-based +(lldb) breakpoint set --regex '.*decode.*' # regex over symbols + +# Run / step / inspect +(lldb) run +(lldb) bt # backtrace +(lldb) frame variable # locals +(lldb) register read # all registers +(lldb) memory read --size 8 --format x --count 16 $sp # 16 qwords from stack +(lldb) disassemble --frame # current function +(lldb) image list # loaded modules +(lldb) image lookup -a 0x1000034c0 # which module + symbol owns this address + +# Process attach to running process +(lldb) process attach --pid 12345 +(lldb) process attach --name target # attach by name + +# Print Mach-O specific +(lldb) image dump sections ./target +(lldb) image dump symtab ./target +``` + +**Function interception via `DYLD_INSERT_LIBRARIES`** (macOS equivalent of `LD_PRELOAD`): + +```bash +# Build a shim dylib that overrides specific functions +# Then run target with it preloaded +DYLD_INSERT_LIBRARIES=./shim.dylib DYLD_FORCE_FLAT_NAMESPACE=1 ./target +``` + +DYLD_INSERT works in the unrestricted case but is blocked in three distinct scenarios — distinguish them when diagnosing why your shim didn't load: + +1. **SIP / restricted process** (target has the `__RESTRICT,__restrict` section, is setuid/setgid, or is a platform/Apple-signed binary): dyld unconditionally strips all `DYLD_*` env vars before the process starts. Nothing you set will reach the target. +2. **Hardened runtime + library validation** (`CS_RUNTIME` flag set, `com.apple.security.cs.disable-library-validation` entitlement absent): the process accepts `DYLD_INSERT_LIBRARIES` but **rejects** loading any dylib that isn't signed by the same Team ID or by Apple. Symptom: shim is found but not loaded; check `log show --predicate 'eventMessage CONTAINS "library validation failed"'`. +3. **Notarization / Gatekeeper translocation**: the binary may be running from a translocated path; relative paths in `DYLD_INSERT_LIBRARIES` won't resolve. Use absolute paths. + +Check each: + +```bash +# Restrict segment present? (case 1) +otool -l ./target | grep -A2 __RESTRICT +# Hardened runtime flag? (case 2) +codesign -d --verbose=4 ./target 2>&1 | grep -iE 'flags=|CodeDirectory' +# Look for "0x10000(runtime)" or similar in the flags line. +# Disable-library-validation entitlement? +codesign -d --entitlements :- ./target 2>&1 | grep disable-library-validation +``` + +**App-level debug logging (always works, ignores SIP):** + +When debugger attach is blocked, fall back to maximizing the app's own logging: + +```bash +# Try common patterns +APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target +NSDebugEnabled=YES ./target # Cocoa apps +OS_ACTIVITY_MODE=debug ./target # os_log subsystem + +# Then read os_log unified logging stream live +log stream --predicate 'process == "target"' --level debug + +# Or extract historical logs +log show --predicate 'process == "target"' --last 1h --info --debug +``` + +This is the **partial-runtime-evidence path** for macOS. See [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for how to combine app-level logs with static analysis when wire-level capture is blocked. + +**Network capture on macOS (TLS-decrypted):** + +```bash +# 1. Find the active network service (don't assume "Wi-Fi"): +# Map the default-route interface to the matching networksetup service name. +networksetup -listallnetworkservices # show options +DEFAULT_IF=$(route -n get default 2>/dev/null | awk '/interface:/ {print $2}') +echo "Default-route interface: $DEFAULT_IF" +# Match the interface (en0, en1, ...) back to a service name: +SERVICE=$(networksetup -listallhardwareports | awk -v iface="$DEFAULT_IF" ' + /^Hardware Port:/ { hp = substr($0, index($0,$3)) } + /^Device:/ { if ($2 == iface) print hp } +') +if [ -z "$SERVICE" ]; then + echo "Could not auto-detect active service. Pick one from -listallnetworkservices manually." >&2 + echo "Aborting proxy setup." >&2 + false # signal failure but stay safe at top level +else + echo "Using service: $SERVICE" +fi + +# 2. JOURNAL the original proxy state before changing it (REQUIRED for safe rollback): +networksetup -getwebproxy "$SERVICE" # save this output to journal +networksetup -getsecurewebproxy "$SERVICE" # save this too + +# 3. Start mitmproxy with persistent CA at ~/.mitmproxy/ +mitmproxy --listen-host 127.0.0.1 --listen-port 8888 & + +# 4. Trust the mitmproxy CA system-wide if the target uses URLSession or any framework +# that ignores HTTPS_PROXY/SSL_CERT_FILE (most macOS-native apps do): +sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/.mitmproxy/mitmproxy-ca-cert.pem + +# 5. Two routing options. Try env-var first; fall back to system proxy: +# 5a. Apps that honor env vars (most CLIs): +HTTPS_PROXY=http://127.0.0.1:8888 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem ./target ... + +# 5b. Apps that use URLSession / system network config (most GUI apps, Bun, some CLIs): +networksetup -setwebproxy "$SERVICE" 127.0.0.1 8888 +networksetup -setsecurewebproxy "$SERVICE" 127.0.0.1 8888 + +# 6. Cleanup — RESTORE original state from journal, untrust CA: +networksetup -setwebproxystate "$SERVICE" off +networksetup -setsecurewebproxystate "$SERVICE" off +sudo security delete-certificate -c "mitmproxy" /Library/Keychains/System.keychain +``` + +**Critical**: forgetting step 6 leaves all your subsequent traffic mis-routed and silently MITM-able. Journal every step. + +### What to look for + +| Observation | Hypothesis | +|---|---| +| `open("/etc/secret-config", ...)` | Reads unexpected config; look at what it does with contents | +| `connect(... 1.2.3.4:443)` | Phones home or depends on an external service | +| `getenv("FOO")` returning NULL | Env var expected but not set | +| Repeated `poll`/`epoll_wait` with no progress | Stuck on I/O; check downstream | +| `SIGSEGV` caught by signal handler | Custom crash recovery — often hides the real bug | +| `dlopen("libfoo.so.42")` | Dynamic plugin loading; check plugin path | + +--- + +## [3] Static analysis with Ghidra + +When triage + tracing have narrowed you to "something in function X" or "the crypto routine is weird", open Ghidra. + +**Open [tools/ghidra.md](../tools/ghidra.md) before launching Ghidra** — the import / analyze / decompile workflow is not obvious and first-time users waste an hour figuring it out. + +Ghidra's decompiler turns machine code into readable-ish C. That's usually what you want. Stay in the Decompiler view; drop to Listing (disassembly) only when the decompiler punts. + +--- + +## [4] Dynamic debugging with pwndbg + +Once static analysis gives you a hypothesis ("this branch at 0x401234 is where the validation fails"), confirm it at runtime with pwndbg. + +**Open [tools/pwndbg.md](../tools/pwndbg.md) before launching gdb.** Pwndbg gives you the context view (registers / stack / disasm / code all visible at once) which is essential for binary debugging. + +Typical pwndbg flow: + +``` +$ gdb ./target # pwndbg loads automatically if installed +pwndbg> break *0x401234 # break at the address static analysis flagged +pwndbg> run arg1 arg2 +# At the breakpoint: +pwndbg> context # registers + stack + disasm +pwndbg> telescope $rdi # walk pointers at $rdi +pwndbg> x/20xw $rsp # raw dump of stack +pwndbg> ni / si # step next / step instruction +``` + +--- + +## [5] Scripted reproduction with pwntools + +Once you have a hypothesis with a concrete repro input, lock it down with pwntools. This is the "failing test" equivalent for binaries. + +**Open [tools/pwntools.md](../tools/pwntools.md)** — the Process/Remote/ELF/context APIs are the foundation. + +```python +from pwn import * + +context.binary = elf = ELF('./target') + +p = process('./target') +p.sendlineafter(b'> ', b'') +result = p.recvall(timeout=3) +assert b'expected-output-when-fixed' in result, f'bug repro: {result}' +``` + +This script is now your "red test". When the fix is applied, the script should pass (or the assertion should be inverted for negative tests — e.g. "the crash string should NOT appear"). + +--- + +## [6] Fixing a binary bug you can't recompile + +Three options, in preference order: + +### Option A: Patch at the source (if you have it) + +If the bug is in your own code and source is available, fix it there and rebuild. Standard TDD path. + +### Option B: Binary patch + +For tiny fixes (one byte, one branch inversion): + +```bash +# Identify the exact byte offset +# e.g. Ghidra says the bug is at 0x401234 = file offset 0x1234 +printf '\x90\x90' | dd of=./target bs=1 seek=$((0x1234)) conv=notrunc +``` + +Journal the exact `dd` command and the original bytes so you can revert. + +### Option C: Wrap / shim + +If you can't patch the binary, write a shim library (LD_PRELOAD on Linux, DYLD_INSERT_LIBRARIES on macOS) that overrides the buggy function. pwntools has examples. + +### Option D: Report upstream + +If it's a third-party binary and none of the above are feasible, the "fix" is a high-quality bug report with: +- Full triage summary +- Reproducible pwntools script +- Ghidra decompilation of the buggy function +- Hypothesis about the root cause +- Recommended patch sketch (in C or pseudocode) + +--- + +## Silent-failure patterns in native binaries + +| Pattern | Why it's silent | +|---|---| +| Ignored libc return codes (`read`, `write`, `malloc`) | Bug continues with garbage data; no check | +| Signal handler swallows SIGSEGV | Crash converted to "something didn't work"; no log | +| `setjmp`/`longjmp` unwinding over cleanup | Resources leak silently | +| Thread-local error state never read (`errno`, `GetLastError`) | Error happened, nobody asked | +| Recovered assertion failure in release build | `assert` compiled out; precondition violations silently corrupt | +| Dangling pointer reads after free | Often looks like valid data until it doesn't | + +--- + +## Phase 9 cleanup specifics + +```bash +# Kill debugger sessions +pkill -f 'gdb' || true +pkill -f 'lldb' || true + +# Ghidra scratch projects (if made just for this session) +# Named something like ~/ghidra-projects/debug-: +ls -la ~/ghidra-projects/ 2>/dev/null +# rm -rf ~/ghidra-projects/debug-scratch # only if the journal says to + +# Core dumps left from crashes +rm -f ./core ./core.* ~/core.* + +# strace/ltrace output files +rm -f trace.out ltrace.out + +# If you made a binary patch (Option B above), confirm revert +# The journal should have the original bytes — restore them: +# printf '' | dd of=./target bs=1 seek= conv=notrunc + +# Trace-output files +rm -f /tmp/debug-*.bin /tmp/debug-*.strace /tmp/debug-*.ltrace + +# macOS-specific: +# Restore proxy settings if you set them (CRITICAL — leaves system traffic mis-routed otherwise) +# Use the SAME $SERVICE you used when enabling the proxy (read it from the journal). +# Do NOT hardcode "Wi-Fi" — many machines route traffic over Ethernet, USB tether, or a VPN service. +[ -n "$SERVICE" ] && { + networksetup -setwebproxystate "$SERVICE" off 2>/dev/null + networksetup -setsecurewebproxystate "$SERVICE" off 2>/dev/null +} +# Or restore explicitly from the journaled original state — see the proxy section above. + +# Stop mitmproxy +pkill -f 'mitmproxy' 2>/dev/null + +# Remove DYLD shim libraries you built +rm -f /tmp/*-shim.dylib + +# Clear extracted strings dumps (these can be huge and may contain secrets) +rm -f /tmp/*-strings*.txt + +# Verify hostname resolution returns to normal (mitmproxy can leave entries) +scutil --dns | head -20 +``` diff --git a/packages/omo-codex/plugin/skills/debugging/references/runtimes/node.md b/packages/omo-codex/plugin/skills/debugging/references/runtimes/node.md new file mode 100644 index 000000000..aac7fb179 --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/runtimes/node.md @@ -0,0 +1,260 @@ +# Node.js / tsx / ts-node / Bun / Deno Debugging + +Covers Node 18+, tsx, ts-node, Bun, Deno. Launch recipes, inspector protocol usage, the `node inspect` CLI, and the **tsx source-map silent-failure** that costs people days. + +--- + +## Environment detection (Phase 0) + +```bash +node --version +cat package.json | head -40 + +# Which JS runtime launches the app? (order them; the first match wins) +ls node_modules/.bin/tsx 2>/dev/null && echo 'has tsx' +ls node_modules/.bin/ts-node 2>/dev/null && echo 'has ts-node' +ls node_modules/.bin/vitest 2>/dev/null && echo 'has vitest' +which bun 2>/dev/null && bun --version +which deno 2>/dev/null && deno --version + +# Source-map situation +grep -E '"sourceMap"|"inlineSources"' tsconfig.json 2>/dev/null +grep -l '//# sourceMappingURL' dist/*.js 2>/dev/null | head -3 + +# Debug-relevant ports +lsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null +lsof -iTCP:9230 -sTCP:LISTEN -nP 2>/dev/null +``` + +--- + +## 🚨 The tsx + `node inspect` CLI silent-failure (READ THIS) + +`tsx` transpiles each `.ts` file on the fly and emits an inline source map. V8 Inspector registers the module with its `.ts` path (so it shows up in the debugger's `scripts` list), **but the `node inspect` CLI REPL does not resolve source-map line numbers reliably**. Setting `sb('session.ts', 285)` will show a "pending" breakpoint that **never fires even after the module loads**. + +The breakpoint list will happily display it, so you think it's set. It isn't. + +### Three reliable workarounds + +| Workaround | When to use | Downside | +|---|---|---| +| **`debugger;` statement in source** | You can edit the source, CLI required | Requires source edit + revert | +| **Chrome DevTools GUI** (`chrome://inspect`) | CLI not required, faster iteration | Not usable if user specifically asked for CLI | +| **Debug the built `dist/` JS** | Source maps are working end-to-end | Requires `npm run build` on every source change | + +The `debugger;` statement is the most reliable. Journal the edit — revert at Phase 9. + +--- + +## Launch recipes by runtime + +### Node (plain JS / compiled TS) + +```bash +# Break on first line, wait for debugger to attach +node --inspect-brk=9229 dist/index.js + +# Attach immediately, don't block startup — pair with debugger; statements +node --inspect=9229 dist/index.js + +# Wait for debugger to attach, THEN run (new in Node 20.15+) +node --inspect-wait=9229 dist/index.js + +# Source maps in stack traces (always a good idea in debug builds) +node --enable-source-maps --inspect dist/index.js +``` + +### tsx + +```bash +# The tsx runner is --import-compatible, so these work: +node --inspect-brk=9229 --import tsx index.ts +node --inspect=9229 --import tsx index.ts + +# If user prefers invoking tsx directly, this also works but is less explicit: +NODE_OPTIONS='--inspect-brk=9229' npx tsx index.ts + +# ⚠️ tsx watch + inspector = inspector reloads per file change +# Debug without watch: +node --inspect=9229 --import tsx index.ts # (no `watch`) +``` + +### ts-node (legacy but still encountered) + +```bash +node --inspect-brk -r ts-node/register src/index.ts +# ESM (ts-node's ESM loader is fragile — if possible, migrate to tsx): +node --inspect --loader ts-node/esm src/index.ts +``` + +### Bun (WebKit Inspector Protocol, NOT V8) + +```bash +bun --inspect src/index.ts # opens debug.bun.sh URL +bun --inspect-brk src/index.ts # break on start +bun --inspect-wait src/index.ts # wait for attach +bun test --inspect-brk # debug test runner +``` + +**Critical**: Bun uses WebKit Inspector Protocol, not V8. `chrome://inspect` cannot connect directly. Use `debug.bun.sh` or the (currently buggy, per Bun docs) VS Code extension. + +### Deno (native V8, Chrome DevTools / VS Code compatible) + +```bash +deno run --inspect-brk --allow-all src/main.ts +deno test --inspect-brk --filter "auth" +``` + +Deno is the smoothest TS debugging experience — native V8 inspector, no source-map workarounds. + +### Vitest + +```bash +# Single worker required — inspector can't attach to multiple workers +vitest --inspect-brk --no-file-parallelism +vitest --inspect-brk --browser --no-file-parallelism # browser mode +``` + +Without `--no-file-parallelism`, breakpoints won't fire because the process Vitest spawns workers in isn't the one listening on the inspector port. + +--- + +## Attaching with `node inspect` CLI + +```bash +node inspect 127.0.0.1:9229 # attach to an existing --inspect process +``` + +Core commands at the `debug>` prompt: + +``` +cont, c resume until next break / debugger; +next, n step over +step, s step into +out, o step out +pause pause a running process +bt backtrace +scripts list all modules V8 has loaded (incl. tsx-transpiled .ts) +sb(N) set breakpoint at line N of current file +sb('file', N) set breakpoint at line N of matching file (⚠️ unreliable with tsx) +sb(func) set breakpoint at function reference +cb(N), cb('file', N) clear breakpoint +breakpoints list breakpoints (shows pending ones, doesn't tell you they'll never fire) +watch('expr') persistent watch expression +watchers show watchers +exec('expr') evaluate expression in paused frame's scope +repl drop into full REPL with frame's scope +restart restart the debuggee +kill kill the debuggee +``` + +**`exec('expr')` is the most powerful tool in this CLI** — it evaluates any JS in the paused frame and returns the value. Use it heavily. + +--- + +## `exec()` patterns that resolve hypotheses fast + +At a breakpoint, these queries resolve most LLM / agent / async bugs in one line each: + +```js +// Agent / LLM state +exec('this.agent.state.messages.length') +exec('this.agent.state.messages.map(m => m.role)') +exec('JSON.stringify(this.agent.state.messages.at(-1)).substring(0, 500)') +exec('this.agent.state.messages.at(-1).errorMessage') // silent-error sentinel +exec('this.agent.state.messages.at(-1).stopReason') +exec('JSON.stringify(this.agent.state.usage)') // undefined / all-zero = failed call +exec('this.agent.state.model.baseUrl') // catch hardcoded vs env-var + +// Env / config at runtime +exec('process.env.RELEVANT_VAR') +exec('Object.keys(process.env).filter(k => k.startsWith("ANTHROPIC"))') +exec('this.config') + +// Async / timing +exec('Date.now() - this._turnStartedAt') +exec('this._activePromises?.size') + +// HTTP request/response in-flight +exec('JSON.stringify(req.body).length') +exec('res.statusCode') +exec('res.headersSent') + +// What's actually running +exec('process.version') +exec('process.cwd()') +exec('process.argv') +``` + +--- + +## Silent-failure patterns in Node + +These are the patterns that most commonly look like success but aren't. Always check when a response is "too fast" or "too empty": + +| Signal | What it means | +|---|---| +| HTTP 200 + `content: ""` | Silent error swallowed | +| HTTP 200 + response in <1s for an LLM call | Too fast for a real Claude/GPT call; something short-circuited | +| `usage: { totalTokens: 0 }` | LLM SDK returned a stub without making the call | +| `stopReason: "error" + content: []` | SDK packaged an error into a "success" message | +| Unhandled promise rejection with no log | Caller forgot to `await`, or `.catch(() => {})` | +| `try { await x(); } catch {}` | Error eaten, no log | +| `void somePromise()` | Explicit opt-out of error propagation; often a bug | +| Callback-style API where callback never fires | Error happened before callback scheduled | +| Handler returns `res.json(...)` twice | Second call is silent on some Express versions | + +When you find one, add a temporary `console.error('[DEBUG]', ...)` to make it loud — journal it, revert at Phase 9. + +--- + +## tmux session layout (two sessions, one purpose each) + +```bash +# Long-running inspected process +tmux new-session -d -s debug-server -c "$PWD" +tmux send-keys -t debug-server 'node --inspect=9229 --import tsx index.ts' Enter + +# Interactive debugger client (separate pane for readability) +tmux new-session -d -s debug-client -c "$PWD" +tmux send-keys -t debug-client 'node inspect 127.0.0.1:9229' Enter + +# Non-blocking pane inspection from the outside +tmux capture-pane -p -t debug-server -S -50 +``` + +Journal both session names. Kill both at Phase 9: + +```bash +tmux kill-session -t debug-server +tmux kill-session -t debug-client +``` + +--- + +## When to abandon the CLI and switch to Chrome DevTools + +The user's preference for CLI is valid and should be respected. But you may recommend a switch in one short sentence if ANY of these hold: + +- You hit source-map resolution failures (`sb('file', line)` not firing) AND the fix is time-sensitive +- You need to watch many values simultaneously (GUI watch panel is faster to scan) +- You're stepping through async-heavy code where CLI step semantics get murky across microtask boundaries + +Phrase as a note, not a request: "I can push through with `debugger;` statements in CLI. If we hit three or more of these in a row, switching to `chrome://inspect` GUI would cut cycle time in half — your call." + +--- + +## Phase 9 cleanup specifics + +```bash +# Revert source-level debug statements +git diff | grep -E '(debugger;|console\.log\(.*DEBUG|\[ARBITER-DEBUG|\[DEBUG)' +# Revert any matching files: +git checkout + +# Kill inspector-attached processes +pkill -f 'node --inspect' || true +pkill -f 'bun --inspect' || true +pkill -f 'deno.*--inspect' || true +lsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null +``` diff --git a/packages/omo-codex/plugin/skills/debugging/references/runtimes/python.md b/packages/omo-codex/plugin/skills/debugging/references/runtimes/python.md new file mode 100644 index 000000000..e16a9ad08 --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/runtimes/python.md @@ -0,0 +1,248 @@ +# Python Debugging + +Covers CPython 3.9+, pytest, asyncio, Django, FastAPI. Setup commands, attach mechanisms, state-query patterns, gotchas, silent-failure signatures. + +--- + +## Environment detection (Phase 0) + +```bash +# Which Python will actually run the code? +which python; which python3 +python --version + +# Is there a project env manager in play? +ls poetry.lock uv.lock Pipfile.lock requirements*.txt .python-version 2>/dev/null + +# Installed debuggers / profilers in this env? +python -c 'import pdb, sys; print("pdb", "built-in"); print("python", sys.executable)' +pip list 2>/dev/null | grep -iE '^(ipdb|pudb|debugpy|py-spy|memray|rich)\s' + +# asyncio debug mode available? +python -c 'import asyncio; print(asyncio.__version__)' +``` + +**Wrapper gotchas** (these change how flags propagate): + +- `poetry run python ...` — args after `python` are fine; args before `poetry run` go to poetry, not python +- `uv run python ...` — similar; prefer `uv run -- python -X dev` if flags collide +- `pipenv run` — same story +- `./manage.py ` (Django) — shebang resolution; make sure it points to the right venv +- `pytest` — loads `conftest.py` at collection; breakpoints inside collection need `pytest --pdb-trace` not `--pdb` + +--- + +## The four ways to attach + +| Method | When to use | Command | +|---|---|---| +| **`breakpoint()` inline** (Python 3.7+) | You can edit the source and restart. Most reliable. | Add `breakpoint()` to source. Run normally. It invokes `pdb` by default. | +| **`python -m pdb