docs(omo-codex): batch 71 (6 files)
This commit is contained in:
YeonGyu-Kim
@@ -0,0 +1,415 @@
|
||||
# Bundled-JS / Embedded-Source Binaries (Bun SEA, Node SEA, Deno compile, pkg, Electron, PyInstaller)
|
||||
|
||||
A growing class of "binaries" are not stripped C/C++ at all — they are a runtime VM glued onto a high-level-language bundle. The bundle is **plaintext or trivially-decodable** inside the binary.
|
||||
|
||||
If you reach for `native-binary.md` workflow on these (Ghidra → pwndbg → hex), you will waste hours decompiling a runtime you don't care about while the actual logic sits exposed three megabytes away.
|
||||
|
||||
**This reference exists because the workflow is fundamentally different from stripped C.**
|
||||
|
||||
---
|
||||
|
||||
## When to use this reference instead of `native-binary.md`
|
||||
|
||||
Open this if `file ./target` shows a generic Mach-O / ELF / PE BUT any of:
|
||||
|
||||
- Size is suspiciously large (50 MB+ for a "simple CLI")
|
||||
- `strings -n 8 ./target | rg -i "node_modules|webpack|esbuild|bun|pkg/lib|electron|pyinstaller"` returns hits
|
||||
- The binary's CLI flags include things like `--inspect`, `--unhandled-rejections`, npm-style help text
|
||||
- Vendor docs say it's built with Bun / pkg / nexe / Deno compile / PyInstaller / Electron / Tauri (UI shell)
|
||||
- `head -c 4 ./target | xxd` shows a known runtime magic for an embedded archive section
|
||||
|
||||
If yes → **stop following `native-binary.md` and follow this**. Triage and dynamic tracing are the same. Static analysis is completely different.
|
||||
|
||||
---
|
||||
|
||||
## The workflow
|
||||
|
||||
```
|
||||
[1] Triage → identify the bundler (Bun? pkg? Deno? Electron? PyInstaller?)
|
||||
[2] Locate the bundle → find where the embedded source archive starts
|
||||
[3] Extract → dump source to disk so you can grep / read it
|
||||
[4] Source-level static analysis (rg + Read, NOT Ghidra)
|
||||
[5] Runtime verification → debug logs, --inspect, partial-evidence patterns
|
||||
[6] Fix / report
|
||||
```
|
||||
|
||||
Step 3 is the unlock — once you have plaintext source on disk, the rest is normal codebase exploration.
|
||||
|
||||
---
|
||||
|
||||
## [1] Identify the bundler — 30-second fingerprint
|
||||
|
||||
```bash
|
||||
# Look for runtime-specific markers in plaintext strings
|
||||
strings -n 12 ./target 2>/dev/null | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|NODE_SEA_BLOB|tauri|ESZIP_V2|denort' | head -20
|
||||
```
|
||||
|
||||
| Marker pattern | Bundler | Source format |
|
||||
|---|---|---|
|
||||
| `@oven/bun-darwin`, `bun-lockfile-format-v`, `// @bun` | **Bun SEA** (compiled via `bun build --compile`) | Plaintext JS, single big bundle |
|
||||
| `NODE_SEA_BLOB` + `NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1` | **Node SEA** (`node --build-sea` or `--experimental-sea-config`) | Plaintext JS, or V8 code cache (when `useCodeCache: true`), or startup snapshot (when `useSnapshot: true`) — the latter two are NOT plaintext |
|
||||
| `pkg/lib/bootstrap.js`, `pkg/prelude`, `PAYLOAD_POSITION` | **pkg** (vercel/pkg) | Plaintext or v8 cached data |
|
||||
| `ESZIP_V2`, `denort`, `deno_runtime` | **Deno compile** (`deno compile`) | TS/JS in eszip archive — readable but needs `eszip` crate to walk; not pure plaintext |
|
||||
| `Electron`, `app.asar`, `chrome.dll`, `Squirrel.Mac` | **Electron** | `app.asar` archive (TAR-like with JSON header). Source is plaintext JS once extracted |
|
||||
| `PyInstaller`, `pyz`, `_MEIPASS`, `pyi-os-utils` | **PyInstaller** | Compressed `.pyc` bytecode — needs `pyinstxtractor` + `decompyle3` to recover Python source |
|
||||
| `nexe-`, `nexe_compile`, `:::nexe::` | **nexe** | Plaintext JS appended to node binary |
|
||||
| `Tauri`, `tao`, `wry`, `tauri::generate_context` | **Tauri** (Rust shell + JS UI) | **Two worlds**: JS frontend in resource section is extractable here; Rust commands / core logic are native and require [native-binary.md](native-binary.md) |
|
||||
|
||||
If multiple match (e.g. Tauri + Bun): the outer shell is the first one (Tauri/Electron). The inner JS is the second one's format. **For Tauri specifically, expect to use both this reference (for the UI bundle) and `native-binary.md` (for the Rust binary side).**
|
||||
|
||||
> **Source-format reality check**: only Bun SEA, pkg (when not using `--public-packages`), nexe, and Electron `.asar` are reliably plaintext. Node SEA with code-cache or snapshot, PyInstaller `.pyc`, and Deno eszip require additional tooling. Don't assume `strings` will find readable code — verify the bundler first.
|
||||
|
||||
---
|
||||
|
||||
## [2] Locate the bundle
|
||||
|
||||
### Bun SEA — JS is just embedded plaintext
|
||||
|
||||
The JS source is concatenated into the binary as a giant template literal / string. No decoding needed.
|
||||
|
||||
```bash
|
||||
# Verify by searching for typical JS bundle markers
|
||||
strings -n 8 ./target | rg "function|var |let |const |async function" | head -5
|
||||
|
||||
# Find where the bundle starts (look for "use strict" or banner comment)
|
||||
LC_ALL=C grep -aob '"use strict"' ./target | head -5
|
||||
LC_ALL=C grep -aob '#!/usr/bin/env bun' ./target | head -5
|
||||
```
|
||||
|
||||
### Node SEA — `NODE_SEA_BLOB` resource/segment + activated fuse
|
||||
|
||||
Per the [Node.js SEA docs](https://nodejs.org/api/single-executable-applications.html), a Node-built SEA contains:
|
||||
- A resource (PE), section in `NODE_SEA` segment (Mach-O), or note (ELF) named `NODE_SEA_BLOB`
|
||||
- The fuse string `NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1` (with trailing `:1` indicating injected; `:0` means a copy of the node binary that has not yet had a blob injected)
|
||||
|
||||
```bash
|
||||
# Confirm it is a SEA at all
|
||||
LC_ALL=C grep -aob 'NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1' ./target | head -1
|
||||
|
||||
# Find the blob resource/section
|
||||
LC_ALL=C grep -aob 'NODE_SEA_BLOB' ./target | head
|
||||
|
||||
# On Mach-O, inspect the segment directly
|
||||
otool -l ./target | grep -A4 'NODE_SEA'
|
||||
```
|
||||
|
||||
The blob format is documented but non-trivial to walk by hand. For extraction, **use postject in reverse** (carve the section bytes) or read the blob via `node:sea` API from inside a debug build of the same binary. Plain `strings` will get you the embedded JS only when the SEA was built without `useCodeCache` and without `useSnapshot` — both of those replace plaintext with V8 cache data or startup snapshot bytes.
|
||||
|
||||
`node --build-sea sea-config.json` and `node --experimental-sea-config sea-config.json` *generate* SEA blobs; neither inspects an existing executable.
|
||||
|
||||
### Deno compile — eszip archive section
|
||||
|
||||
```bash
|
||||
# Deno-compile binaries embed an eszip v2 archive
|
||||
LC_ALL=C grep -aob 'ESZIP_V2' ./target | head -3
|
||||
# Also confirm the runtime
|
||||
LC_ALL=C grep -aob 'denort' ./target | head -1
|
||||
```
|
||||
|
||||
To extract, use the `eszip` Rust crate (or the `@deno/eszip` JS port) to parse the archive after carving it out at the offset above. There is no stable Deno CLI flag that inspects compiled-executable eszip contents as of 2026-04 — `deno info` only works on source files.
|
||||
|
||||
### pkg — `PAYLOAD_POSITION` marker
|
||||
|
||||
```bash
|
||||
LC_ALL=C grep -aob 'PAYLOAD_POSITION' ./target | head
|
||||
LC_ALL=C grep -aob 'pkg/prelude' ./target | head
|
||||
```
|
||||
|
||||
For source extraction, use the `pkg-extract` tooling community projects or carve based on the offset reported by the `PAYLOAD_POSITION:<n>` value.
|
||||
|
||||
### Electron — `app.asar` is usually a separate file
|
||||
|
||||
Most Electron apps ship `app.asar` next to the binary, not embedded inside. Extract it with the official tool:
|
||||
|
||||
```bash
|
||||
# macOS layout
|
||||
ls -la /Applications/MyApp.app/Contents/Resources/app.asar
|
||||
npx @electron/asar extract app.asar ./extracted/
|
||||
# or older:
|
||||
npx asar extract app.asar ./extracted/
|
||||
```
|
||||
|
||||
For single-file builds where the asar is embedded inside the executable, **do not pattern-match arbitrary 4-byte sequences** (the asar format starts with a Pickle-encoded uint32 header size + JSON metadata, and the same bytes appear elsewhere in any binary). Instead, use a Pickle-aware extractor that validates the JSON header before claiming a match — the `asar` npm package's programmatic `extractAll()` API does this. Carve the asar bytes by scanning for a candidate Pickle header (4-byte size + 4-byte payload size + `{"files":` prefix), validate the JSON parses, then feed the carved buffer to `extractAll()`.
|
||||
|
||||
### PyInstaller — use `pyinstxtractor`, NOT runtime self-extraction
|
||||
|
||||
```bash
|
||||
# Recover the embedded archive without running the binary
|
||||
python3 pyinstxtractor.py ./target
|
||||
# Output: ./target_extracted/ with .pyc files
|
||||
|
||||
# Decompile the .pyc files back to Python source
|
||||
decompyle3 ./target_extracted/main.pyc # Python 3.7+
|
||||
uncompyle6 ./target_extracted/main.pyc # older Python
|
||||
```
|
||||
|
||||
If `pyinstxtractor` cannot read the archive (e.g. non-standard PyInstaller version), use the official `pyi-archive_viewer` tool that ships with PyInstaller. Avoid the "run-the-binary-and-snoop-`/tmp/_MEI*`" approach: it only catches what runs in the time window between `_MEIPASS` extraction and cleanup, and it executes potentially untrusted code.
|
||||
|
||||
---
|
||||
|
||||
## [3] Extract source to disk — DO NOT skip this
|
||||
|
||||
**The single biggest mistake** with bundled-JS reverse engineering is trying to read the source out of `strings` output or `xxd` dumps. You will lose data. See "Gotchas" below.
|
||||
|
||||
### For Bun SEA / nexe / single-string-blob bundlers
|
||||
|
||||
Read the binary as bytes, find the JS section, save to a `.js` file:
|
||||
|
||||
```python
|
||||
# extract_bundled_js.py
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
raise SystemExit("usage: extract_bundled_js.py <target>")
|
||||
|
||||
with open(sys.argv[1], 'rb') as f:
|
||||
data = f.read()
|
||||
|
||||
markers = [b'// @bun', b'"use strict"', b"'use strict'", b'#!/usr/bin/env']
|
||||
start = -1
|
||||
for m in markers:
|
||||
p = data.find(m)
|
||||
if p != -1 and (start == -1 or p < start):
|
||||
start = p
|
||||
|
||||
if start == -1:
|
||||
raise SystemExit(
|
||||
"no bundle marker found — binary may not be Bun/nexe, "
|
||||
"or markers were stripped. Try strings(1) for hints."
|
||||
)
|
||||
|
||||
# Heuristic end: look for a long null run AFTER start.
|
||||
# This is a heuristic, NOT a guarantee. Verify the tail of the output
|
||||
# looks like JS (closing braces, EOF) before trusting it.
|
||||
end = data.find(b'\x00' * 1024, start)
|
||||
if end == -1:
|
||||
end = len(data)
|
||||
|
||||
bundle = data[start:end]
|
||||
print(f'Extracted {len(bundle)} bytes from offset {start} to {end}', file=sys.stderr)
|
||||
sys.stdout.buffer.write(bundle)
|
||||
```
|
||||
|
||||
```bash
|
||||
python3 extract_bundled_js.py ./target > extracted-bundle.js
|
||||
wc -c extracted-bundle.js
|
||||
# Sanity check the tail is JS, not random binary
|
||||
tail -c 200 extracted-bundle.js
|
||||
```
|
||||
|
||||
### For PyInstaller
|
||||
|
||||
Use `pyinstxtractor` then `uncompyle6` / `decompyle3` on the `.pyc` files.
|
||||
|
||||
### For Electron .asar
|
||||
|
||||
```bash
|
||||
npx asar extract app.asar ./extracted/
|
||||
# Now ./extracted/ has a normal node_modules + your source layout
|
||||
```
|
||||
|
||||
### For Deno compile
|
||||
|
||||
Use the `eszip` Rust crate or the `@deno/eszip` JS port to walk the archive after carving the eszip section out at the offset reported by the `ESZIP_V2` magic search. There is no stable Deno CLI as of 2026-04 that inspects compiled-binary eszip contents directly.
|
||||
|
||||
---
|
||||
|
||||
## [4] Source-level static analysis — `rg` + `Read`, not Ghidra
|
||||
|
||||
Once you have the source on disk, treat it as a normal codebase:
|
||||
|
||||
```bash
|
||||
# Find function definitions
|
||||
rg -n "^function |^const \w+ = (function|\(.*\) =>)" extracted-bundle.js | head
|
||||
|
||||
# Find specific behavior
|
||||
rg -n "claude-opus-4-7|reasoning_effort|api_key" extracted-bundle.js
|
||||
|
||||
# Resolve minified identifiers — they show up as `var XYZ="value"`
|
||||
rg -aoP 'var \w+="[^"]+"' extracted-bundle.js | head -50
|
||||
```
|
||||
|
||||
For minified bundles, use a template-literal-aware parser to extract specific functions or template strings. Example skeleton:
|
||||
|
||||
```python
|
||||
def find_template_end(data, start):
|
||||
"""Walk a JS template literal preserving ${...} interpolation depth.
|
||||
Returns position of closing backtick."""
|
||||
i = start
|
||||
while i < len(data):
|
||||
c = data[i:i+1]
|
||||
if c == b'\\':
|
||||
i += 2; continue
|
||||
if c == b'$' and data[i+1:i+2] == b'{':
|
||||
depth = 1; i += 2
|
||||
while i < len(data) and depth > 0:
|
||||
cc = data[i:i+1]
|
||||
if cc == b'\\': i += 2; continue
|
||||
if cc == b'`':
|
||||
j = find_template_end(data, i+1)
|
||||
i = j + 1; continue
|
||||
if cc == b'{': depth += 1
|
||||
elif cc == b'}': depth -= 1
|
||||
elif cc in (b'"', b"'"):
|
||||
q = cc; i += 1
|
||||
while i < len(data) and data[i:i+1] != q:
|
||||
if data[i:i+1] == b'\\': i += 2
|
||||
else: i += 1
|
||||
i += 1; continue
|
||||
i += 1
|
||||
continue
|
||||
if c == b'`': return i
|
||||
i += 1
|
||||
return -1
|
||||
```
|
||||
|
||||
For function-body extraction, **track the parameter list separately** before tracking body braces. The naive approach mis-counts destructuring `function f({a, b, ...c})` as the body `{` and exits early.
|
||||
|
||||
---
|
||||
|
||||
## [5] Runtime verification
|
||||
|
||||
You usually cannot single-step JS inside a Bun-compiled binary the way you would with `node --inspect`. Workarounds:
|
||||
|
||||
### Bun-compiled
|
||||
|
||||
Bun's inspector takes `--inspect[=<host>:<port>[/<prefix>]]` on the command line. For env-var control of compiled binaries, the form is the same minus the leading `--`:
|
||||
|
||||
```bash
|
||||
# Default port (6499) auto-prefix
|
||||
./target --inspect
|
||||
# → ws://localhost:6499/<auto-prefix> (paste into https://debug.bun.sh)
|
||||
|
||||
# Explicit host:port[/prefix]
|
||||
./target --inspect=localhost:9229/dbg
|
||||
# Or via env var (if --inspect cannot be passed)
|
||||
BUN_INSPECT=localhost:9229/dbg ./target
|
||||
```
|
||||
|
||||
**For HTTP request tracing without an interactive debugger** (highest-value Bun-specific runtime evidence):
|
||||
|
||||
```bash
|
||||
# Print every fetch() / node:http request as a curl command + full headers/body
|
||||
BUN_CONFIG_VERBOSE_FETCH=curl ./target ...
|
||||
|
||||
# Or just print the request/response without curl-format
|
||||
BUN_CONFIG_VERBOSE_FETCH=true ./target ...
|
||||
```
|
||||
|
||||
Plus generic env-var-based debug logging if the app supports it:
|
||||
|
||||
```bash
|
||||
APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target
|
||||
```
|
||||
|
||||
### Node SEA / pkg / nexe
|
||||
```bash
|
||||
# These usually accept --inspect since they are real Node
|
||||
./target --inspect
|
||||
# Then chrome://inspect or node --inspect-brk
|
||||
```
|
||||
|
||||
### Electron
|
||||
```bash
|
||||
./target.app/Contents/MacOS/target --inspect=9229 --remote-debugging-port=9223
|
||||
# Renderer process is at chrome://inspect, main process via the inspector port
|
||||
```
|
||||
|
||||
### When you cannot make a real call
|
||||
The target's API may require credentials, network access, or paid quota you don't have. **You are not stuck** — see [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for the fallback patterns.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Gotchas — read these before extracting
|
||||
|
||||
### G1. `strings -n N` silently drops short identifier interpolations
|
||||
|
||||
`strings` outputs runs of printable characters of length **≥ N**. Default is 4 on most systems; many references (including older versions of `native-binary.md`) recommend `-n 8` for less noise.
|
||||
|
||||
**With `-n 8`, short template-literal interpolations like `${x}`, `${i}`, `${R}` are silently dropped** because they are 4 chars surrounded by non-printable bytes (newlines or section padding). The result looks like:
|
||||
|
||||
```text
|
||||
expected: <INSTRUCTIONS>\n${x}\n</INSTRUCTIONS>
|
||||
strings: <INSTRUCTIONS>\n</INSTRUCTIONS> ← ${x} is gone, no warning
|
||||
```
|
||||
|
||||
A consumer reading the strings output would conclude the template is empty.
|
||||
|
||||
**Mitigation**:
|
||||
1. Use `strings` only for **fingerprinting** (Phase 1 triage), never as the source of extracted text.
|
||||
2. For actual extraction, **read the binary as bytes** with `python3 -c "open('./target','rb').read()"` and grep / parse from there.
|
||||
3. If you must use `strings`, try `strings -n 1 -t x ./target` and post-filter — but byte-level reads are still more reliable.
|
||||
|
||||
### G2. Stale cached binary ≠ latest features
|
||||
|
||||
Bundled-app installers often check a remote version and skip download if a cached binary exists. If you reverse-engineered an old version and the user reports behavior you don't see in the source, **re-run the installer** (or fetch the version manifest manually) before assuming the source is current.
|
||||
|
||||
```bash
|
||||
# Example pattern - varies by tool
|
||||
curl -fsSL https://example.com/install.sh | head -50 # find version-fetch URL
|
||||
curl -fsSL https://static.example.com/cli/cli-version.txt
|
||||
./your-tool --version
|
||||
# Compare. If different, re-install.
|
||||
```
|
||||
|
||||
### G3. APFS / NTFS case-insensitivity silently overwrites files
|
||||
|
||||
When extracting many minified function bodies (`cVR`, `CVR`, `dpr`, `DPR`, …) and saving each to its own file, **macOS APFS and Windows NTFS treat `cVR.txt` and `CVR.txt` as the same file**. The second write silently overwrites the first.
|
||||
|
||||
**Mitigation**: prefix filenames with something case-distinguishing, e.g. `mode-cVR.txt`, `mode-CVR.txt`, or use a hash suffix.
|
||||
|
||||
### G4. Bun's runtime adds 30-50 MB of unrelated symbols
|
||||
|
||||
A 70 MB Bun-compiled binary is **mostly Bun runtime** (~50 MB) plus your app (~20 MB). When fingerprinting, you will see thousands of strings like `tree-sitter-typescript`, `react-native-stylex` etc. that the user's actual app doesn't use — these are package names baked into Bun's package-resolution data.
|
||||
|
||||
**Mitigation**: when grepping for "what does this app do?", filter out runtime noise:
|
||||
```bash
|
||||
strings -n 8 ./target | rg -v 'node_modules|@oven/bun|package-lock|tree-sitter|ffmpeg-installer' | head
|
||||
```
|
||||
|
||||
### G5. Source maps usually NOT shipped
|
||||
|
||||
Bundled apps strip source maps for production. Variable names are minified to `T`, `R`, `a`, `r`, etc. Treat the bundle like an obfuscated codebase: identify constants by tracing assignments (`var T="actual-name"`) and resolve interpolations manually.
|
||||
|
||||
### G6. The "extract" file is not legally redistributable
|
||||
|
||||
If reverse-engineering proprietary software, the extracted source is the vendor's IP. Use it for understanding behavior, **never commit it to git**, never post snippets in public issues. Cleanup your `extracted-bundle.js` files in Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns specific to bundled JS
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| Bundle includes unreachable dead code from tree-shaking failures | You read code that never runs — verify with runtime trace |
|
||||
| `process.env.X` resolved at BUILD time, not RUNTIME | Setting the env var at runtime has no effect; the value is baked in |
|
||||
| `import.meta.url` in compiled binary returns `bun://...` not a real path | File-relative resolution silently breaks |
|
||||
| Worker threads spawn from embedded code, look for sub-bundle inside main bundle | Workers may have their own copy of dependencies |
|
||||
| Minified identifiers with case variants used in same module | Easy to confuse `cVR` with `CVR` when reading fast |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics for bundled-JS work
|
||||
|
||||
```bash
|
||||
# Remove extracted bundles — they may contain proprietary source
|
||||
rm -f /tmp/extracted-bundle.js /tmp/extracted-*.js
|
||||
rm -rf /tmp/asar-extracted/
|
||||
rm -rf /tmp/_MEI*
|
||||
|
||||
# Remove strings dumps
|
||||
rm -f /tmp/*-strings.txt /tmp/*-strings-v*.txt
|
||||
|
||||
# Remove Python helper scripts created for parsing
|
||||
rm -f /tmp/extract_bundled_js.py /tmp/parse_template.py
|
||||
|
||||
# Verify the extraction directory is gone (if you used a workspace dir)
|
||||
ls /Users/$USER/local-workspaces/*-extracted/ 2>/dev/null
|
||||
# rm -rf only after journal review confirms nothing important is there
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## When to escalate back to `native-binary.md`
|
||||
|
||||
If extraction reveals the "bundle" is actually compiled to v8 cached data (pkg with `--public-packages` or PyInstaller with bytecode-only mode), and decompilation is non-trivial, **switch back to `native-binary.md` workflow** (Ghidra against the runtime + careful tracing). Bundled-JS workflow only helps when the high-level source is recoverable as readable text.
|
||||
@@ -0,0 +1,252 @@
|
||||
# Go Debugging
|
||||
|
||||
Covers goroutines, `dlv` (Delve), `pprof`, the race detector, and the fact that Go's concurrency model means most bugs are about goroutines doing something quiet and wrong.
|
||||
|
||||
---
|
||||
|
||||
## Environment detection (Phase 0)
|
||||
|
||||
```bash
|
||||
go version
|
||||
cat go.mod | head -5
|
||||
|
||||
# Delve installed?
|
||||
which dlv
|
||||
dlv version
|
||||
|
||||
# Build constraints
|
||||
grep -r '// +build\|//go:build' cmd/ internal/ pkg/ 2>/dev/null | head
|
||||
|
||||
# pprof wired up?
|
||||
grep -r 'net/http/pprof\|runtime/pprof' --include='*.go' | head -3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delve (`dlv`) — the Go debugger
|
||||
|
||||
Go's gc compiler emits DWARF, but plain gdb barely understands goroutines. **Use dlv, not gdb.** Plain gdb on a Go binary will miss goroutine state and print garbage for interface values.
|
||||
|
||||
### The five `dlv` launch modes
|
||||
|
||||
```bash
|
||||
# Build and launch under debugger (equivalent to `go run` + debug)
|
||||
dlv debug ./cmd/server -- --port=8080
|
||||
|
||||
# Debug a test binary
|
||||
dlv test ./internal/handler/ # enters the test package under debug
|
||||
|
||||
# Debug an existing binary (must be built with -gcflags="all=-N -l" for best results)
|
||||
dlv exec ./bin/myserver
|
||||
|
||||
# Attach to a running process
|
||||
dlv attach $(pgrep myserver)
|
||||
|
||||
# Headless mode (IDE / remote attach) — default port 2345
|
||||
dlv debug --headless --listen=:2345 --api-version=2 ./cmd/server
|
||||
```
|
||||
|
||||
### Building a debuggable binary
|
||||
|
||||
The compiler inlines and optimizes aggressively in normal builds, which makes stepping confusing. For serious debugging:
|
||||
|
||||
```bash
|
||||
go build -gcflags="all=-N -l" -o ./bin/server ./cmd/server
|
||||
# -N disables optimization
|
||||
# -l disables inlining
|
||||
```
|
||||
|
||||
Then `dlv exec ./bin/server`.
|
||||
|
||||
### Essential dlv commands
|
||||
|
||||
```
|
||||
(dlv) b main.main # breakpoint at function
|
||||
(dlv) b handler.go:42 # breakpoint at file:line
|
||||
(dlv) b pkg/foo.Bar # breakpoint at type method (Go path syntax)
|
||||
(dlv) c / continue # continue until next break
|
||||
(dlv) n / next # step over
|
||||
(dlv) s / step # step into
|
||||
(dlv) so / stepout # step out
|
||||
(dlv) bt / stack # stack trace of current goroutine
|
||||
(dlv) goroutines # list all goroutines
|
||||
(dlv) goroutine <id> # switch to goroutine N
|
||||
(dlv) goroutine <id> bt # stack of a specific goroutine
|
||||
(dlv) locals # all locals in frame
|
||||
(dlv) args # function args
|
||||
(dlv) p <expr> # print value (understands interfaces, maps, slices)
|
||||
(dlv) vars <regex> # package vars matching regex
|
||||
(dlv) regs # registers (rare in Go debugging)
|
||||
(dlv) on <bpid> print <expr> # auto-print on breakpoint hit (powerful!)
|
||||
(dlv) trace <location> # like breakpoint but just logs, doesn't stop
|
||||
```
|
||||
|
||||
The `trace` command is underused — it's like a logpoint, no stepping required.
|
||||
|
||||
---
|
||||
|
||||
## Goroutine-centric debugging
|
||||
|
||||
Goroutine leaks and deadlocks are the most common Go bugs. `dlv`'s `goroutines` command is the starting point.
|
||||
|
||||
```
|
||||
(dlv) goroutines -t # with truncated stack
|
||||
(dlv) goroutines -s # sorted by stack
|
||||
(dlv) goroutines -with user # filter user-spawned goroutines
|
||||
```
|
||||
|
||||
Common patterns:
|
||||
|
||||
| You see in `goroutines` | Usually means |
|
||||
|---|---|
|
||||
| 100s of goroutines stuck at `chan receive` | Producer died; consumers leak |
|
||||
| 100s stuck at `semacquire` | Lock contention; a holder probably deadlocked |
|
||||
| One stuck at `select` with no default | Missing case or closed channel scenario |
|
||||
| Stuck at `netpoll` | External I/O not responding — not a Go bug, check downstream |
|
||||
| Growing count over time | Goroutine leak — need to find who's spawning without cleanup |
|
||||
|
||||
### Panic signals in Go
|
||||
|
||||
```go
|
||||
// Without recovery, panics crash the program with a stack trace of ALL goroutines
|
||||
// With recovery, they're silent unless explicitly logged:
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
log.Printf("recovered panic: %v\n%s", r, debug.Stack()) // GOOD
|
||||
// log.Printf("recovered") // BAD — silent
|
||||
}
|
||||
}()
|
||||
```
|
||||
|
||||
**Always check for silent recovers** in Phase 8. Grep:
|
||||
```bash
|
||||
rg 'recover\(\)' --type go
|
||||
```
|
||||
|
||||
And inspect each site for whether the panic is actually surfaced.
|
||||
|
||||
---
|
||||
|
||||
## Race detector — ALWAYS run when the bug is intermittent
|
||||
|
||||
```bash
|
||||
go test -race ./...
|
||||
go run -race ./cmd/server
|
||||
go build -race ./cmd/server
|
||||
```
|
||||
|
||||
The race detector wraps memory accesses and catches concurrent read/write without synchronization. **Run this before attaching dlv** if intermittency is involved — it often finds the bug directly.
|
||||
|
||||
Output shape:
|
||||
```
|
||||
WARNING: DATA RACE
|
||||
Read at 0x00c0001a0080 by goroutine 7:
|
||||
main.(*Counter).Value()
|
||||
/path/to/counter.go:14 +0x3c
|
||||
Previous write at 0x00c0001a0080 by goroutine 6:
|
||||
main.(*Counter).Inc()
|
||||
/path/to/counter.go:10 +0x5f
|
||||
```
|
||||
|
||||
Both stacks. Both goroutines. The race is obvious from the line pair.
|
||||
|
||||
---
|
||||
|
||||
## pprof — for perf, memory, goroutine leaks
|
||||
|
||||
### Wire it up (idempotent; usually already present)
|
||||
|
||||
```go
|
||||
import _ "net/http/pprof"
|
||||
|
||||
func main() {
|
||||
go func() {
|
||||
log.Println(http.ListenAndServe("localhost:6060", nil))
|
||||
}()
|
||||
// ... rest of your server
|
||||
}
|
||||
```
|
||||
|
||||
### Queries
|
||||
|
||||
```bash
|
||||
# CPU profile (30s)
|
||||
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
|
||||
|
||||
# Heap snapshot
|
||||
go tool pprof http://localhost:6060/debug/pprof/heap
|
||||
|
||||
# Goroutine snapshot — find leaks
|
||||
go tool pprof http://localhost:6060/debug/pprof/goroutine
|
||||
|
||||
# Block profile — find blocking ops (needs runtime.SetBlockProfileRate)
|
||||
go tool pprof http://localhost:6060/debug/pprof/block
|
||||
|
||||
# Mutex profile — find lock contention (needs runtime.SetMutexProfileFraction)
|
||||
go tool pprof http://localhost:6060/debug/pprof/mutex
|
||||
```
|
||||
|
||||
Inside pprof:
|
||||
```
|
||||
(pprof) top # top functions by self time
|
||||
(pprof) list main.handler # annotated source of a function
|
||||
(pprof) web # SVG callgraph in browser (requires graphviz)
|
||||
(pprof) traces # sample traces
|
||||
```
|
||||
|
||||
For goroutine leaks, **take two snapshots 30s apart** and diff:
|
||||
```bash
|
||||
go tool pprof -base prof1.pb.gz prof2.pb.gz
|
||||
```
|
||||
|
||||
Goroutines that appear in prof2 but not prof1 are new; if they stick around, they're leaking.
|
||||
|
||||
---
|
||||
|
||||
## `GODEBUG` — runtime-level observability
|
||||
|
||||
```bash
|
||||
GODEBUG=gctrace=1 ./myserver # print GC stats
|
||||
GODEBUG=schedtrace=1000 ./myserver # scheduler trace every 1000ms
|
||||
GODEBUG=scheddetail=1,schedtrace=1000 # detailed scheduler state
|
||||
GODEBUG=allocfreetrace=1 ./myserver # every alloc/free (noisy!)
|
||||
GODEBUG=memprofilerate=1 ./myserver # profile every allocation
|
||||
```
|
||||
|
||||
Useful for diagnosing GC pressure, goroutine starvation, or memory pattern issues.
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in Go
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| `if err != nil { return err }` that returns to a caller that ignores | Error bubbles up, then gets discarded at the top |
|
||||
| `defer func() { recover() }()` — bare recover, no log | Panic swallowed, program continues with state corruption |
|
||||
| `_, _ = conn.Write(data)` | Intentionally discarded error |
|
||||
| Buffered channel send that blocks forever | Sender hangs; hard to see if no deadlock detection |
|
||||
| `time.Sleep` in a test | "Works on my machine"; test passes locally, fails in CI |
|
||||
| `go func() { ... }()` with no error path | Goroutine dies silently on panic unless recover+log |
|
||||
| Context canceled but operation continues | Ignored `ctx.Err()` check |
|
||||
| `json.Unmarshal` of zero-value struct field | Input missing the key; silently zero |
|
||||
| Closed channel read returning zero value | Consumer doesn't check `ok`; reads forever |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Kill dlv sessions
|
||||
pkill -f 'dlv' || true
|
||||
lsof -iTCP:2345 -sTCP:LISTEN -nP 2>/dev/null # dlv default
|
||||
|
||||
# Kill pprof HTTP endpoint if you started it just for this session
|
||||
lsof -iTCP:6060 -sTCP:LISTEN -nP 2>/dev/null
|
||||
|
||||
# Revert any `fmt.Println("DEBUG: ...")` or `log.Printf("DEBUG: ...")` additions
|
||||
git diff | grep -E '(fmt\.Println\("DEBUG|log\.Printf\("DEBUG|println!)'
|
||||
git checkout <file>
|
||||
|
||||
# Unset env vars
|
||||
unset GODEBUG
|
||||
```
|
||||
@@ -0,0 +1,484 @@
|
||||
# Native Binary Debugging (No Source / Reverse Engineering)
|
||||
|
||||
For binaries where you don't have trustworthy source: stripped production builds, third-party closed libs, malware, CTF challenges, firmware, vendored libs whose docs lie. The workflow is specific; doing it out of order wastes days.
|
||||
|
||||
This reference **coordinates** the triage and dynamic work. The heavy tools each have their own reference:
|
||||
- **Static decompilation** → [tools/ghidra.md](../tools/ghidra.md)
|
||||
- **Interactive debugging** → [tools/pwndbg.md](../tools/pwndbg.md)
|
||||
- **Scripted interaction / exploitation** → [tools/pwntools.md](../tools/pwntools.md)
|
||||
|
||||
Read those before using them — especially Ghidra, which has a surprising amount of workflow that's not obvious.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ STOP — is this actually a stripped C/C++ binary?
|
||||
|
||||
A growing share of "binaries" are actually **bundled high-level apps** — Bun SEA, Node SEA, Deno compile, pkg, nexe, Electron, Tauri, PyInstaller. Their workflow is completely different: the high-level source is recoverable with the right per-bundler tool (often plaintext, sometimes V8 cache / `.pyc` / eszip needing extra tooling), and Ghidra against the runtime VM wastes hours.
|
||||
|
||||
Quick check:
|
||||
|
||||
```bash
|
||||
file ./target # Mach-O / ELF / PE - inconclusive
|
||||
du -h ./target # 50 MB+ for a "simple CLI" → suspect bundled
|
||||
strings -n 12 ./target | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|tauri' | head -5
|
||||
```
|
||||
|
||||
**If any hits** → close this file, open [bundled-js-binary.md](bundled-js-binary.md) instead. Following the Ghidra/pwndbg path on a bundled-app binary wastes hours decompiling the runtime VM while the app-level bundle is recoverable with the right per-bundler tool (plaintext for Bun/pkg/nexe/Electron-asar; eszip / V8-cache / `.pyc` for Deno / Node SEA / PyInstaller).
|
||||
|
||||
If `file` says "Mach-O" or "ELF", `du` is < 20 MB, and the strings check is empty → continue here.
|
||||
|
||||
---
|
||||
|
||||
## The workflow (do these in order)
|
||||
|
||||
Every step's output is input to the next. Skipping steps means guessing later.
|
||||
|
||||
```
|
||||
[1] Triage → what kind of binary is this?
|
||||
[2] Dynamic tracing → what syscalls / libcalls does it make?
|
||||
[3] Static analysis → what does it DO, in readable form? (Ghidra)
|
||||
[4] Dynamic debug → confirm hypotheses at runtime (pwndbg)
|
||||
[5] Scripted repro → lock the bug with a pwntools script
|
||||
[6] TDD + fix / report
|
||||
```
|
||||
|
||||
Steps 1 and 2 are fast (minutes). Step 3 is slow (tens of minutes to hours depending on size). Don't skip 1-2 and go straight to Ghidra — the triage output tells you what to focus on inside Ghidra.
|
||||
|
||||
---
|
||||
|
||||
## [1] Triage — 5-minute fingerprint
|
||||
|
||||
```bash
|
||||
# Basic identity
|
||||
file ./target
|
||||
# elf, mach-o, pe? 32/64-bit? dynamically linked? stripped?
|
||||
|
||||
# Architecture details
|
||||
readelf -h ./target # ELF header: entry point, arch, type
|
||||
lipo -info ./target 2>/dev/null # macOS: universal binary?
|
||||
|
||||
# Interesting strings (often leaks function names, error messages, URLs, API keys)
|
||||
strings -n 8 ./target | head -100
|
||||
strings -n 8 ./target | grep -iE '(http|/api/|error|debug|version)'
|
||||
|
||||
# Imported symbols (what does it link against?)
|
||||
nm -D ./target 2>/dev/null # dynamic symbols
|
||||
objdump -T ./target 2>/dev/null # same, alternate tool
|
||||
readelf -d ./target # dynamic section (NEEDED libs)
|
||||
ldd ./target 2>/dev/null # resolved library paths
|
||||
|
||||
# Security posture (affects what exploits / bugs are possible)
|
||||
checksec --file=./target # requires pwntools or installing checksec
|
||||
# NX, PIE, RELRO, stack canary, FORTIFY
|
||||
|
||||
# Is it stripped?
|
||||
nm ./target 2>/dev/null | head # empty? stripped. full? not stripped.
|
||||
file ./target # will say "stripped" or "not stripped"
|
||||
```
|
||||
|
||||
### ⚠️ `strings -n N` silently drops short content
|
||||
|
||||
`strings` prints runs of printable characters of length **≥ N**. With `-n 8`, **anything shorter than 8 chars sandwiched between non-printable bytes is dropped silently**. This includes:
|
||||
|
||||
- Short identifier interpolations in templates (`${x}`, `${i}`, `${R}`)
|
||||
- Short embedded constants (`v3`, `null`, integer immediates as bytes)
|
||||
- Short error codes between binary padding
|
||||
|
||||
Real example: a JavaScript template literal `<INSTRUCTIONS>\n${x}\n</INSTRUCTIONS>` came out of `strings -n 8` as `<INSTRUCTIONS>\n</INSTRUCTIONS>` — the `${x}` (4 chars) was dropped. A consumer reading the dump would conclude the template was empty. It is not.
|
||||
|
||||
**Use `strings` only for fingerprinting (Phase 1).** For any extraction whose correctness matters, **read bytes directly**:
|
||||
|
||||
```bash
|
||||
# Count occurrences of a needle
|
||||
LC_ALL=C grep -aoc 'NEEDLE' ./target
|
||||
|
||||
# Find offsets
|
||||
LC_ALL=C grep -aob 'NEEDLE' ./target | head
|
||||
|
||||
# Or via Python for byte-precise context
|
||||
python3 -c "
|
||||
import sys
|
||||
data = open('./target','rb').read()
|
||||
needle = b'NEEDLE'
|
||||
pos = data.find(needle)
|
||||
print(repr(data[max(0,pos-100):pos+200]))
|
||||
"
|
||||
```
|
||||
|
||||
If you must keep using `strings`, lower the threshold: `strings -n 1 -t x ./target | rg ...`. The signal-to-noise drops sharply but short content is preserved.
|
||||
|
||||
Write the triage summary to the journal:
|
||||
|
||||
```markdown
|
||||
## Binary triage
|
||||
- Type: <ELF 64-bit, dynamically linked, stripped>
|
||||
- Arch: <x86_64 | arm64 | ...>
|
||||
- Libs: <libc, openssl, libcurl>
|
||||
- Security: <NX, PIE, Partial RELRO, no canary>
|
||||
- Interesting strings: <short list>
|
||||
- First hypothesis surface: <which function / area looks most relevant>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [2] Dynamic tracing — what does it actually call?
|
||||
|
||||
These are cheap — run them before Ghidra to orient yourself.
|
||||
|
||||
### Linux: strace + ltrace
|
||||
|
||||
```bash
|
||||
# System calls
|
||||
strace -f -o trace.out ./target arg1 arg2
|
||||
strace -f -e trace=network ./target # filter to network syscalls
|
||||
strace -f -e trace=file ./target # filter to file ops
|
||||
|
||||
# Library calls (less useful when stripped but still informative)
|
||||
ltrace -f -o ltrace.out ./target
|
||||
ltrace -f -e 'str*+mem*' ./target # filter to string/mem functions
|
||||
```
|
||||
|
||||
### macOS: Mach-O specifics
|
||||
|
||||
**SIP block reality check.** With System Integrity Protection enabled (default on every modern macOS), `dtruss` / `dtrace` will **silently fail** to attach to:
|
||||
- Anything in `/usr`, `/bin`, `/sbin`, `/System`
|
||||
- Apple-signed binaries (Xcode CLT, Homebrew formulae from Apple-distributed taps)
|
||||
- Notarized vendor binaries (Bun, Deno, Docker Desktop, etc.)
|
||||
|
||||
`dtruss ./target` will appear to run but produce zero events. This is not a bug; it is the SIP design. Disabling SIP requires a Recovery Mode reboot — usually not worth it. Use the alternatives below.
|
||||
|
||||
```bash
|
||||
# dtruss — works only when SIP allows it (your own unsigned binaries)
|
||||
sudo dtruss -f ./target 2>&1 | head -20 # equivalent to strace
|
||||
# If output is suspiciously empty → SIP blocked it. Switch to lldb or app-level logging.
|
||||
```
|
||||
|
||||
**Mach-O metadata inspection (no SIP issues, no debugger needed):**
|
||||
|
||||
```bash
|
||||
# Architecture and slices
|
||||
file ./target # arm64 / x86_64 / universal
|
||||
lipo -info ./target # which architectures included
|
||||
lipo -thin arm64 ./target -output ./target-arm64 # extract one slice for analysis
|
||||
|
||||
# Headers & load commands (segments, dylibs, code-signature pointer)
|
||||
otool -h ./target # Mach header (cputype, ncmds, flags)
|
||||
otool -l ./target | head -100 # load commands; entitlements live in code-signature blob, see codesign below
|
||||
|
||||
# Dynamic library dependencies (macOS equivalent of ldd)
|
||||
otool -L ./target # linked dylibs with versions
|
||||
dyld_info ./target # macOS 13+, more detailed than otool -L
|
||||
|
||||
# Disassembly
|
||||
otool -tv ./target | head -200 # quick disassembly without Ghidra
|
||||
otool -tV ./target # with symbol-resolved branches
|
||||
|
||||
# Imported / exported symbols (Apple `nm`, NOT GNU)
|
||||
nm -u ./target # undefined references = imports
|
||||
nm -gU ./target # external defined = exports
|
||||
# Note: GNU `-D`/dynamic flags are not honored on Apple `nm`; use the above forms.
|
||||
symbols -fullSourcePath -onlyWithDebugInfo ./target # if any debug info survives
|
||||
|
||||
# Code signature & entitlements (entitlements come from codesign, NOT otool)
|
||||
codesign -dv --entitlements :- ./target 2>&1 # signature info + entitlements XML on stdout
|
||||
spctl --assess --type execute -vv ./target # Gatekeeper assessment
|
||||
|
||||
# Cert chain — extract to a temp dir to avoid creating files named -0/-1 in cwd
|
||||
tmp=$(mktemp -d)
|
||||
codesign -dvv --extract-certificates="$tmp/cert" ./target 2>&1
|
||||
ls -la "$tmp"
|
||||
# rm -rf "$tmp" # journal first, clean up later
|
||||
|
||||
# Strings inside specific segments only (less noise than full-binary strings)
|
||||
otool -s __TEXT __cstring ./target # C string section
|
||||
otool -s __TEXT __const ./target # constants section
|
||||
```
|
||||
|
||||
**Interactive debugging on macOS — use `lldb`, not `gdb`.**
|
||||
|
||||
GDB on macOS requires a self-signed code-signing certificate (`codesign --entitlements gdb.entitlements --sign gdb-cert /opt/homebrew/bin/gdb`) and even then is unreliable on arm64. **Use `lldb` directly** — it ships with Xcode CLT and works without configuration.
|
||||
|
||||
```bash
|
||||
# Start lldb
|
||||
lldb ./target
|
||||
|
||||
# Set arguments
|
||||
(lldb) settings set target.run-args arg1 arg2
|
||||
|
||||
# Run with breakpoints
|
||||
(lldb) breakpoint set --name function_name # symbol-based
|
||||
(lldb) breakpoint set --address 0x1000034c0 # address-based
|
||||
(lldb) breakpoint set --regex '.*decode.*' # regex over symbols
|
||||
|
||||
# Run / step / inspect
|
||||
(lldb) run
|
||||
(lldb) bt # backtrace
|
||||
(lldb) frame variable # locals
|
||||
(lldb) register read # all registers
|
||||
(lldb) memory read --size 8 --format x --count 16 $sp # 16 qwords from stack
|
||||
(lldb) disassemble --frame # current function
|
||||
(lldb) image list # loaded modules
|
||||
(lldb) image lookup -a 0x1000034c0 # which module + symbol owns this address
|
||||
|
||||
# Process attach to running process
|
||||
(lldb) process attach --pid 12345
|
||||
(lldb) process attach --name target # attach by name
|
||||
|
||||
# Print Mach-O specific
|
||||
(lldb) image dump sections ./target
|
||||
(lldb) image dump symtab ./target
|
||||
```
|
||||
|
||||
**Function interception via `DYLD_INSERT_LIBRARIES`** (macOS equivalent of `LD_PRELOAD`):
|
||||
|
||||
```bash
|
||||
# Build a shim dylib that overrides specific functions
|
||||
# Then run target with it preloaded
|
||||
DYLD_INSERT_LIBRARIES=./shim.dylib DYLD_FORCE_FLAT_NAMESPACE=1 ./target
|
||||
```
|
||||
|
||||
DYLD_INSERT works in the unrestricted case but is blocked in three distinct scenarios — distinguish them when diagnosing why your shim didn't load:
|
||||
|
||||
1. **SIP / restricted process** (target has the `__RESTRICT,__restrict` section, is setuid/setgid, or is a platform/Apple-signed binary): dyld unconditionally strips all `DYLD_*` env vars before the process starts. Nothing you set will reach the target.
|
||||
2. **Hardened runtime + library validation** (`CS_RUNTIME` flag set, `com.apple.security.cs.disable-library-validation` entitlement absent): the process accepts `DYLD_INSERT_LIBRARIES` but **rejects** loading any dylib that isn't signed by the same Team ID or by Apple. Symptom: shim is found but not loaded; check `log show --predicate 'eventMessage CONTAINS "library validation failed"'`.
|
||||
3. **Notarization / Gatekeeper translocation**: the binary may be running from a translocated path; relative paths in `DYLD_INSERT_LIBRARIES` won't resolve. Use absolute paths.
|
||||
|
||||
Check each:
|
||||
|
||||
```bash
|
||||
# Restrict segment present? (case 1)
|
||||
otool -l ./target | grep -A2 __RESTRICT
|
||||
# Hardened runtime flag? (case 2)
|
||||
codesign -d --verbose=4 ./target 2>&1 | grep -iE 'flags=|CodeDirectory'
|
||||
# Look for "0x10000(runtime)" or similar in the flags line.
|
||||
# Disable-library-validation entitlement?
|
||||
codesign -d --entitlements :- ./target 2>&1 | grep disable-library-validation
|
||||
```
|
||||
|
||||
**App-level debug logging (always works, ignores SIP):**
|
||||
|
||||
When debugger attach is blocked, fall back to maximizing the app's own logging:
|
||||
|
||||
```bash
|
||||
# Try common patterns
|
||||
APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target
|
||||
NSDebugEnabled=YES ./target # Cocoa apps
|
||||
OS_ACTIVITY_MODE=debug ./target # os_log subsystem
|
||||
|
||||
# Then read os_log unified logging stream live
|
||||
log stream --predicate 'process == "target"' --level debug
|
||||
|
||||
# Or extract historical logs
|
||||
log show --predicate 'process == "target"' --last 1h --info --debug
|
||||
```
|
||||
|
||||
This is the **partial-runtime-evidence path** for macOS. See [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for how to combine app-level logs with static analysis when wire-level capture is blocked.
|
||||
|
||||
**Network capture on macOS (TLS-decrypted):**
|
||||
|
||||
```bash
|
||||
# 1. Find the active network service (don't assume "Wi-Fi"):
|
||||
# Map the default-route interface to the matching networksetup service name.
|
||||
networksetup -listallnetworkservices # show options
|
||||
DEFAULT_IF=$(route -n get default 2>/dev/null | awk '/interface:/ {print $2}')
|
||||
echo "Default-route interface: $DEFAULT_IF"
|
||||
# Match the interface (en0, en1, ...) back to a service name:
|
||||
SERVICE=$(networksetup -listallhardwareports | awk -v iface="$DEFAULT_IF" '
|
||||
/^Hardware Port:/ { hp = substr($0, index($0,$3)) }
|
||||
/^Device:/ { if ($2 == iface) print hp }
|
||||
')
|
||||
if [ -z "$SERVICE" ]; then
|
||||
echo "Could not auto-detect active service. Pick one from -listallnetworkservices manually." >&2
|
||||
echo "Aborting proxy setup." >&2
|
||||
false # signal failure but stay safe at top level
|
||||
else
|
||||
echo "Using service: $SERVICE"
|
||||
fi
|
||||
|
||||
# 2. JOURNAL the original proxy state before changing it (REQUIRED for safe rollback):
|
||||
networksetup -getwebproxy "$SERVICE" # save this output to journal
|
||||
networksetup -getsecurewebproxy "$SERVICE" # save this too
|
||||
|
||||
# 3. Start mitmproxy with persistent CA at ~/.mitmproxy/
|
||||
mitmproxy --listen-host 127.0.0.1 --listen-port 8888 &
|
||||
|
||||
# 4. Trust the mitmproxy CA system-wide if the target uses URLSession or any framework
|
||||
# that ignores HTTPS_PROXY/SSL_CERT_FILE (most macOS-native apps do):
|
||||
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/.mitmproxy/mitmproxy-ca-cert.pem
|
||||
|
||||
# 5. Two routing options. Try env-var first; fall back to system proxy:
|
||||
# 5a. Apps that honor env vars (most CLIs):
|
||||
HTTPS_PROXY=http://127.0.0.1:8888 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem ./target ...
|
||||
|
||||
# 5b. Apps that use URLSession / system network config (most GUI apps, Bun, some CLIs):
|
||||
networksetup -setwebproxy "$SERVICE" 127.0.0.1 8888
|
||||
networksetup -setsecurewebproxy "$SERVICE" 127.0.0.1 8888
|
||||
|
||||
# 6. Cleanup — RESTORE original state from journal, untrust CA:
|
||||
networksetup -setwebproxystate "$SERVICE" off
|
||||
networksetup -setsecurewebproxystate "$SERVICE" off
|
||||
sudo security delete-certificate -c "mitmproxy" /Library/Keychains/System.keychain
|
||||
```
|
||||
|
||||
**Critical**: forgetting step 6 leaves all your subsequent traffic mis-routed and silently MITM-able. Journal every step.
|
||||
|
||||
### What to look for
|
||||
|
||||
| Observation | Hypothesis |
|
||||
|---|---|
|
||||
| `open("/etc/secret-config", ...)` | Reads unexpected config; look at what it does with contents |
|
||||
| `connect(... 1.2.3.4:443)` | Phones home or depends on an external service |
|
||||
| `getenv("FOO")` returning NULL | Env var expected but not set |
|
||||
| Repeated `poll`/`epoll_wait` with no progress | Stuck on I/O; check downstream |
|
||||
| `SIGSEGV` caught by signal handler | Custom crash recovery — often hides the real bug |
|
||||
| `dlopen("libfoo.so.42")` | Dynamic plugin loading; check plugin path |
|
||||
|
||||
---
|
||||
|
||||
## [3] Static analysis with Ghidra
|
||||
|
||||
When triage + tracing have narrowed you to "something in function X" or "the crypto routine is weird", open Ghidra.
|
||||
|
||||
**Open [tools/ghidra.md](../tools/ghidra.md) before launching Ghidra** — the import / analyze / decompile workflow is not obvious and first-time users waste an hour figuring it out.
|
||||
|
||||
Ghidra's decompiler turns machine code into readable-ish C. That's usually what you want. Stay in the Decompiler view; drop to Listing (disassembly) only when the decompiler punts.
|
||||
|
||||
---
|
||||
|
||||
## [4] Dynamic debugging with pwndbg
|
||||
|
||||
Once static analysis gives you a hypothesis ("this branch at 0x401234 is where the validation fails"), confirm it at runtime with pwndbg.
|
||||
|
||||
**Open [tools/pwndbg.md](../tools/pwndbg.md) before launching gdb.** Pwndbg gives you the context view (registers / stack / disasm / code all visible at once) which is essential for binary debugging.
|
||||
|
||||
Typical pwndbg flow:
|
||||
|
||||
```
|
||||
$ gdb ./target # pwndbg loads automatically if installed
|
||||
pwndbg> break *0x401234 # break at the address static analysis flagged
|
||||
pwndbg> run arg1 arg2
|
||||
# At the breakpoint:
|
||||
pwndbg> context # registers + stack + disasm
|
||||
pwndbg> telescope $rdi # walk pointers at $rdi
|
||||
pwndbg> x/20xw $rsp # raw dump of stack
|
||||
pwndbg> ni / si # step next / step instruction
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [5] Scripted reproduction with pwntools
|
||||
|
||||
Once you have a hypothesis with a concrete repro input, lock it down with pwntools. This is the "failing test" equivalent for binaries.
|
||||
|
||||
**Open [tools/pwntools.md](../tools/pwntools.md)** — the Process/Remote/ELF/context APIs are the foundation.
|
||||
|
||||
```python
|
||||
from pwn import *
|
||||
|
||||
context.binary = elf = ELF('./target')
|
||||
|
||||
p = process('./target')
|
||||
p.sendlineafter(b'> ', b'<trigger input that reproduces the bug>')
|
||||
result = p.recvall(timeout=3)
|
||||
assert b'expected-output-when-fixed' in result, f'bug repro: {result}'
|
||||
```
|
||||
|
||||
This script is now your "red test". When the fix is applied, the script should pass (or the assertion should be inverted for negative tests — e.g. "the crash string should NOT appear").
|
||||
|
||||
---
|
||||
|
||||
## [6] Fixing a binary bug you can't recompile
|
||||
|
||||
Three options, in preference order:
|
||||
|
||||
### Option A: Patch at the source (if you have it)
|
||||
|
||||
If the bug is in your own code and source is available, fix it there and rebuild. Standard TDD path.
|
||||
|
||||
### Option B: Binary patch
|
||||
|
||||
For tiny fixes (one byte, one branch inversion):
|
||||
|
||||
```bash
|
||||
# Identify the exact byte offset
|
||||
# e.g. Ghidra says the bug is at 0x401234 = file offset 0x1234
|
||||
printf '\x90\x90' | dd of=./target bs=1 seek=$((0x1234)) conv=notrunc
|
||||
```
|
||||
|
||||
Journal the exact `dd` command and the original bytes so you can revert.
|
||||
|
||||
### Option C: Wrap / shim
|
||||
|
||||
If you can't patch the binary, write a shim library (LD_PRELOAD on Linux, DYLD_INSERT_LIBRARIES on macOS) that overrides the buggy function. pwntools has examples.
|
||||
|
||||
### Option D: Report upstream
|
||||
|
||||
If it's a third-party binary and none of the above are feasible, the "fix" is a high-quality bug report with:
|
||||
- Full triage summary
|
||||
- Reproducible pwntools script
|
||||
- Ghidra decompilation of the buggy function
|
||||
- Hypothesis about the root cause
|
||||
- Recommended patch sketch (in C or pseudocode)
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in native binaries
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| Ignored libc return codes (`read`, `write`, `malloc`) | Bug continues with garbage data; no check |
|
||||
| Signal handler swallows SIGSEGV | Crash converted to "something didn't work"; no log |
|
||||
| `setjmp`/`longjmp` unwinding over cleanup | Resources leak silently |
|
||||
| Thread-local error state never read (`errno`, `GetLastError`) | Error happened, nobody asked |
|
||||
| Recovered assertion failure in release build | `assert` compiled out; precondition violations silently corrupt |
|
||||
| Dangling pointer reads after free | Often looks like valid data until it doesn't |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Kill debugger sessions
|
||||
pkill -f 'gdb' || true
|
||||
pkill -f 'lldb' || true
|
||||
|
||||
# Ghidra scratch projects (if made just for this session)
|
||||
# Named something like ~/ghidra-projects/debug-<timestamp>:
|
||||
ls -la ~/ghidra-projects/ 2>/dev/null
|
||||
# rm -rf ~/ghidra-projects/debug-scratch # only if the journal says to
|
||||
|
||||
# Core dumps left from crashes
|
||||
rm -f ./core ./core.* ~/core.*
|
||||
|
||||
# strace/ltrace output files
|
||||
rm -f trace.out ltrace.out
|
||||
|
||||
# If you made a binary patch (Option B above), confirm revert
|
||||
# The journal should have the original bytes — restore them:
|
||||
# printf '<original-bytes>' | dd of=./target bs=1 seek=<offset> conv=notrunc
|
||||
|
||||
# Trace-output files
|
||||
rm -f /tmp/debug-*.bin /tmp/debug-*.strace /tmp/debug-*.ltrace
|
||||
|
||||
# macOS-specific:
|
||||
# Restore proxy settings if you set them (CRITICAL — leaves system traffic mis-routed otherwise)
|
||||
# Use the SAME $SERVICE you used when enabling the proxy (read it from the journal).
|
||||
# Do NOT hardcode "Wi-Fi" — many machines route traffic over Ethernet, USB tether, or a VPN service.
|
||||
[ -n "$SERVICE" ] && {
|
||||
networksetup -setwebproxystate "$SERVICE" off 2>/dev/null
|
||||
networksetup -setsecurewebproxystate "$SERVICE" off 2>/dev/null
|
||||
}
|
||||
# Or restore explicitly from the journaled original state — see the proxy section above.
|
||||
|
||||
# Stop mitmproxy
|
||||
pkill -f 'mitmproxy' 2>/dev/null
|
||||
|
||||
# Remove DYLD shim libraries you built
|
||||
rm -f /tmp/*-shim.dylib
|
||||
|
||||
# Clear extracted strings dumps (these can be huge and may contain secrets)
|
||||
rm -f /tmp/*-strings*.txt
|
||||
|
||||
# Verify hostname resolution returns to normal (mitmproxy can leave entries)
|
||||
scutil --dns | head -20
|
||||
```
|
||||
@@ -0,0 +1,260 @@
|
||||
# Node.js / tsx / ts-node / Bun / Deno Debugging
|
||||
|
||||
Covers Node 18+, tsx, ts-node, Bun, Deno. Launch recipes, inspector protocol usage, the `node inspect` CLI, and the **tsx source-map silent-failure** that costs people days.
|
||||
|
||||
---
|
||||
|
||||
## Environment detection (Phase 0)
|
||||
|
||||
```bash
|
||||
node --version
|
||||
cat package.json | head -40
|
||||
|
||||
# Which JS runtime launches the app? (order them; the first match wins)
|
||||
ls node_modules/.bin/tsx 2>/dev/null && echo 'has tsx'
|
||||
ls node_modules/.bin/ts-node 2>/dev/null && echo 'has ts-node'
|
||||
ls node_modules/.bin/vitest 2>/dev/null && echo 'has vitest'
|
||||
which bun 2>/dev/null && bun --version
|
||||
which deno 2>/dev/null && deno --version
|
||||
|
||||
# Source-map situation
|
||||
grep -E '"sourceMap"|"inlineSources"' tsconfig.json 2>/dev/null
|
||||
grep -l '//# sourceMappingURL' dist/*.js 2>/dev/null | head -3
|
||||
|
||||
# Debug-relevant ports
|
||||
lsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null
|
||||
lsof -iTCP:9230 -sTCP:LISTEN -nP 2>/dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚨 The tsx + `node inspect` CLI silent-failure (READ THIS)
|
||||
|
||||
`tsx` transpiles each `.ts` file on the fly and emits an inline source map. V8 Inspector registers the module with its `.ts` path (so it shows up in the debugger's `scripts` list), **but the `node inspect` CLI REPL does not resolve source-map line numbers reliably**. Setting `sb('session.ts', 285)` will show a "pending" breakpoint that **never fires even after the module loads**.
|
||||
|
||||
The breakpoint list will happily display it, so you think it's set. It isn't.
|
||||
|
||||
### Three reliable workarounds
|
||||
|
||||
| Workaround | When to use | Downside |
|
||||
|---|---|---|
|
||||
| **`debugger;` statement in source** | You can edit the source, CLI required | Requires source edit + revert |
|
||||
| **Chrome DevTools GUI** (`chrome://inspect`) | CLI not required, faster iteration | Not usable if user specifically asked for CLI |
|
||||
| **Debug the built `dist/` JS** | Source maps are working end-to-end | Requires `npm run build` on every source change |
|
||||
|
||||
The `debugger;` statement is the most reliable. Journal the edit — revert at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## Launch recipes by runtime
|
||||
|
||||
### Node (plain JS / compiled TS)
|
||||
|
||||
```bash
|
||||
# Break on first line, wait for debugger to attach
|
||||
node --inspect-brk=9229 dist/index.js
|
||||
|
||||
# Attach immediately, don't block startup — pair with debugger; statements
|
||||
node --inspect=9229 dist/index.js
|
||||
|
||||
# Wait for debugger to attach, THEN run (new in Node 20.15+)
|
||||
node --inspect-wait=9229 dist/index.js
|
||||
|
||||
# Source maps in stack traces (always a good idea in debug builds)
|
||||
node --enable-source-maps --inspect dist/index.js
|
||||
```
|
||||
|
||||
### tsx
|
||||
|
||||
```bash
|
||||
# The tsx runner is --import-compatible, so these work:
|
||||
node --inspect-brk=9229 --import tsx index.ts
|
||||
node --inspect=9229 --import tsx index.ts
|
||||
|
||||
# If user prefers invoking tsx directly, this also works but is less explicit:
|
||||
NODE_OPTIONS='--inspect-brk=9229' npx tsx index.ts
|
||||
|
||||
# ⚠️ tsx watch + inspector = inspector reloads per file change
|
||||
# Debug without watch:
|
||||
node --inspect=9229 --import tsx index.ts # (no `watch`)
|
||||
```
|
||||
|
||||
### ts-node (legacy but still encountered)
|
||||
|
||||
```bash
|
||||
node --inspect-brk -r ts-node/register src/index.ts
|
||||
# ESM (ts-node's ESM loader is fragile — if possible, migrate to tsx):
|
||||
node --inspect --loader ts-node/esm src/index.ts
|
||||
```
|
||||
|
||||
### Bun (WebKit Inspector Protocol, NOT V8)
|
||||
|
||||
```bash
|
||||
bun --inspect src/index.ts # opens debug.bun.sh URL
|
||||
bun --inspect-brk src/index.ts # break on start
|
||||
bun --inspect-wait src/index.ts # wait for attach
|
||||
bun test --inspect-brk # debug test runner
|
||||
```
|
||||
|
||||
**Critical**: Bun uses WebKit Inspector Protocol, not V8. `chrome://inspect` cannot connect directly. Use `debug.bun.sh` or the (currently buggy, per Bun docs) VS Code extension.
|
||||
|
||||
### Deno (native V8, Chrome DevTools / VS Code compatible)
|
||||
|
||||
```bash
|
||||
deno run --inspect-brk --allow-all src/main.ts
|
||||
deno test --inspect-brk --filter "auth"
|
||||
```
|
||||
|
||||
Deno is the smoothest TS debugging experience — native V8 inspector, no source-map workarounds.
|
||||
|
||||
### Vitest
|
||||
|
||||
```bash
|
||||
# Single worker required — inspector can't attach to multiple workers
|
||||
vitest --inspect-brk --no-file-parallelism
|
||||
vitest --inspect-brk --browser --no-file-parallelism # browser mode
|
||||
```
|
||||
|
||||
Without `--no-file-parallelism`, breakpoints won't fire because the process Vitest spawns workers in isn't the one listening on the inspector port.
|
||||
|
||||
---
|
||||
|
||||
## Attaching with `node inspect` CLI
|
||||
|
||||
```bash
|
||||
node inspect 127.0.0.1:9229 # attach to an existing --inspect process
|
||||
```
|
||||
|
||||
Core commands at the `debug>` prompt:
|
||||
|
||||
```
|
||||
cont, c resume until next break / debugger;
|
||||
next, n step over
|
||||
step, s step into
|
||||
out, o step out
|
||||
pause pause a running process
|
||||
bt backtrace
|
||||
scripts list all modules V8 has loaded (incl. tsx-transpiled .ts)
|
||||
sb(N) set breakpoint at line N of current file
|
||||
sb('file', N) set breakpoint at line N of matching file (⚠️ unreliable with tsx)
|
||||
sb(func) set breakpoint at function reference
|
||||
cb(N), cb('file', N) clear breakpoint
|
||||
breakpoints list breakpoints (shows pending ones, doesn't tell you they'll never fire)
|
||||
watch('expr') persistent watch expression
|
||||
watchers show watchers
|
||||
exec('expr') evaluate expression in paused frame's scope
|
||||
repl drop into full REPL with frame's scope
|
||||
restart restart the debuggee
|
||||
kill kill the debuggee
|
||||
```
|
||||
|
||||
**`exec('expr')` is the most powerful tool in this CLI** — it evaluates any JS in the paused frame and returns the value. Use it heavily.
|
||||
|
||||
---
|
||||
|
||||
## `exec()` patterns that resolve hypotheses fast
|
||||
|
||||
At a breakpoint, these queries resolve most LLM / agent / async bugs in one line each:
|
||||
|
||||
```js
|
||||
// Agent / LLM state
|
||||
exec('this.agent.state.messages.length')
|
||||
exec('this.agent.state.messages.map(m => m.role)')
|
||||
exec('JSON.stringify(this.agent.state.messages.at(-1)).substring(0, 500)')
|
||||
exec('this.agent.state.messages.at(-1).errorMessage') // silent-error sentinel
|
||||
exec('this.agent.state.messages.at(-1).stopReason')
|
||||
exec('JSON.stringify(this.agent.state.usage)') // undefined / all-zero = failed call
|
||||
exec('this.agent.state.model.baseUrl') // catch hardcoded vs env-var
|
||||
|
||||
// Env / config at runtime
|
||||
exec('process.env.RELEVANT_VAR')
|
||||
exec('Object.keys(process.env).filter(k => k.startsWith("ANTHROPIC"))')
|
||||
exec('this.config')
|
||||
|
||||
// Async / timing
|
||||
exec('Date.now() - this._turnStartedAt')
|
||||
exec('this._activePromises?.size')
|
||||
|
||||
// HTTP request/response in-flight
|
||||
exec('JSON.stringify(req.body).length')
|
||||
exec('res.statusCode')
|
||||
exec('res.headersSent')
|
||||
|
||||
// What's actually running
|
||||
exec('process.version')
|
||||
exec('process.cwd()')
|
||||
exec('process.argv')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in Node
|
||||
|
||||
These are the patterns that most commonly look like success but aren't. Always check when a response is "too fast" or "too empty":
|
||||
|
||||
| Signal | What it means |
|
||||
|---|---|
|
||||
| HTTP 200 + `content: ""` | Silent error swallowed |
|
||||
| HTTP 200 + response in <1s for an LLM call | Too fast for a real Claude/GPT call; something short-circuited |
|
||||
| `usage: { totalTokens: 0 }` | LLM SDK returned a stub without making the call |
|
||||
| `stopReason: "error" + content: []` | SDK packaged an error into a "success" message |
|
||||
| Unhandled promise rejection with no log | Caller forgot to `await`, or `.catch(() => {})` |
|
||||
| `try { await x(); } catch {}` | Error eaten, no log |
|
||||
| `void somePromise()` | Explicit opt-out of error propagation; often a bug |
|
||||
| Callback-style API where callback never fires | Error happened before callback scheduled |
|
||||
| Handler returns `res.json(...)` twice | Second call is silent on some Express versions |
|
||||
|
||||
When you find one, add a temporary `console.error('[DEBUG]', ...)` to make it loud — journal it, revert at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## tmux session layout (two sessions, one purpose each)
|
||||
|
||||
```bash
|
||||
# Long-running inspected process
|
||||
tmux new-session -d -s debug-server -c "$PWD"
|
||||
tmux send-keys -t debug-server 'node --inspect=9229 --import tsx index.ts' Enter
|
||||
|
||||
# Interactive debugger client (separate pane for readability)
|
||||
tmux new-session -d -s debug-client -c "$PWD"
|
||||
tmux send-keys -t debug-client 'node inspect 127.0.0.1:9229' Enter
|
||||
|
||||
# Non-blocking pane inspection from the outside
|
||||
tmux capture-pane -p -t debug-server -S -50
|
||||
```
|
||||
|
||||
Journal both session names. Kill both at Phase 9:
|
||||
|
||||
```bash
|
||||
tmux kill-session -t debug-server
|
||||
tmux kill-session -t debug-client
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## When to abandon the CLI and switch to Chrome DevTools
|
||||
|
||||
The user's preference for CLI is valid and should be respected. But you may recommend a switch in one short sentence if ANY of these hold:
|
||||
|
||||
- You hit source-map resolution failures (`sb('file', line)` not firing) AND the fix is time-sensitive
|
||||
- You need to watch many values simultaneously (GUI watch panel is faster to scan)
|
||||
- You're stepping through async-heavy code where CLI step semantics get murky across microtask boundaries
|
||||
|
||||
Phrase as a note, not a request: "I can push through with `debugger;` statements in CLI. If we hit three or more of these in a row, switching to `chrome://inspect` GUI would cut cycle time in half — your call."
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Revert source-level debug statements
|
||||
git diff | grep -E '(debugger;|console\.log\(.*DEBUG|\[ARBITER-DEBUG|\[DEBUG)'
|
||||
# Revert any matching files:
|
||||
git checkout <file>
|
||||
|
||||
# Kill inspector-attached processes
|
||||
pkill -f 'node --inspect' || true
|
||||
pkill -f 'bun --inspect' || true
|
||||
pkill -f 'deno.*--inspect' || true
|
||||
lsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null
|
||||
```
|
||||
@@ -0,0 +1,248 @@
|
||||
# Python Debugging
|
||||
|
||||
Covers CPython 3.9+, pytest, asyncio, Django, FastAPI. Setup commands, attach mechanisms, state-query patterns, gotchas, silent-failure signatures.
|
||||
|
||||
---
|
||||
|
||||
## Environment detection (Phase 0)
|
||||
|
||||
```bash
|
||||
# Which Python will actually run the code?
|
||||
which python; which python3
|
||||
python --version
|
||||
|
||||
# Is there a project env manager in play?
|
||||
ls poetry.lock uv.lock Pipfile.lock requirements*.txt .python-version 2>/dev/null
|
||||
|
||||
# Installed debuggers / profilers in this env?
|
||||
python -c 'import pdb, sys; print("pdb", "built-in"); print("python", sys.executable)'
|
||||
pip list 2>/dev/null | grep -iE '^(ipdb|pudb|debugpy|py-spy|memray|rich)\s'
|
||||
|
||||
# asyncio debug mode available?
|
||||
python -c 'import asyncio; print(asyncio.__version__)'
|
||||
```
|
||||
|
||||
**Wrapper gotchas** (these change how flags propagate):
|
||||
|
||||
- `poetry run python ...` — args after `python` are fine; args before `poetry run` go to poetry, not python
|
||||
- `uv run python ...` — similar; prefer `uv run -- python -X dev` if flags collide
|
||||
- `pipenv run` — same story
|
||||
- `./manage.py <cmd>` (Django) — shebang resolution; make sure it points to the right venv
|
||||
- `pytest` — loads `conftest.py` at collection; breakpoints inside collection need `pytest --pdb-trace` not `--pdb`
|
||||
|
||||
---
|
||||
|
||||
## The four ways to attach
|
||||
|
||||
| Method | When to use | Command |
|
||||
|---|---|---|
|
||||
| **`breakpoint()` inline** (Python 3.7+) | You can edit the source and restart. Most reliable. | Add `breakpoint()` to source. Run normally. It invokes `pdb` by default. |
|
||||
| **`python -m pdb <script>`** | No source edit desired. Breaks on entry. | `python -m pdb script.py arg1` |
|
||||
| **post-mortem `pdb.pm()`** | Exception already happened, you want to inspect state | In an exception-caught REPL: `import pdb; pdb.pm()` after the exception propagates |
|
||||
| **debugpy (remote / IDE)** | IDE attach, remote host, containerized process | `python -m debugpy --listen 5678 --wait-for-client script.py` then attach from VS Code / PyCharm |
|
||||
|
||||
### Prefer `ipdb` or `pudb` over plain `pdb` when available
|
||||
|
||||
- **ipdb** — drop-in replacement with tab completion, syntax highlighting. `pip install ipdb`, then `PYTHONBREAKPOINT=ipdb.set_trace` or use `import ipdb; ipdb.set_trace()`.
|
||||
- **pudb** — full-screen TUI debugger, much faster to navigate stack/locals. `pip install pudb`, then `PYTHONBREAKPOINT=pudb.set_trace`.
|
||||
|
||||
### Control `breakpoint()` globally
|
||||
|
||||
```bash
|
||||
# Use ipdb instead of pdb
|
||||
export PYTHONBREAKPOINT=ipdb.set_trace
|
||||
|
||||
# Disable all breakpoint() calls (useful to ship without removing them)
|
||||
export PYTHONBREAKPOINT=0
|
||||
```
|
||||
|
||||
**Journal this env var** — unset at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## pdb / ipdb essentials
|
||||
|
||||
At a `(Pdb)` or `ipdb>` prompt:
|
||||
|
||||
```
|
||||
l list source around current line
|
||||
ll list whole function
|
||||
s step into
|
||||
n step over (next)
|
||||
r step out (return)
|
||||
c continue
|
||||
b list breakpoints
|
||||
b <line> breakpoint at line
|
||||
b <func> breakpoint at function
|
||||
cl <n> clear breakpoint n
|
||||
w where (backtrace)
|
||||
u / d move up/down the stack
|
||||
a args of current frame
|
||||
p <expr> print expression
|
||||
pp <expr> pretty-print
|
||||
!<stmt> execute Python statement (e.g. !x = 5)
|
||||
interact drop into a full Python REPL with current frame's locals
|
||||
q quit (aborts the program)
|
||||
```
|
||||
|
||||
**`interact` is underused** — it gives you a full IPython-esque REPL with all locals available. Faster than typing `p` for 20 things.
|
||||
|
||||
---
|
||||
|
||||
## pytest-specific debugging
|
||||
|
||||
```bash
|
||||
# Enter pdb on first failure
|
||||
pytest --pdb
|
||||
|
||||
# Enter pdb at the START of each test (not on failure)
|
||||
pytest --trace
|
||||
|
||||
# Run only the failing test, with -s to show print output
|
||||
pytest --pdb -x -s path/to/test.py::test_name
|
||||
|
||||
# Collect-time debugging (for problems in conftest.py / fixture setup)
|
||||
pytest --pdb-trace
|
||||
|
||||
# Disable capture for this test (so breakpoint prompt is visible)
|
||||
pytest -s
|
||||
```
|
||||
|
||||
**Common failure**: `breakpoint()` hangs inside a pytest test — that's because pytest captures stdout/stderr by default. Always add `-s` when debugging with breakpoints inside pytest.
|
||||
|
||||
---
|
||||
|
||||
## asyncio gotchas
|
||||
|
||||
Async is where most Python debug sessions go sideways. Know these before attaching.
|
||||
|
||||
### Breakpoints inside coroutines
|
||||
|
||||
`breakpoint()` works inside an async function, but stepping into another coroutine from `pdb` is awkward. Two techniques:
|
||||
|
||||
```python
|
||||
async def handler():
|
||||
result = await some_async_fn() # add breakpoint ABOVE, not inside, when possible
|
||||
breakpoint()
|
||||
return result
|
||||
```
|
||||
|
||||
Inside the breakpoint, to inspect a coroutine without actually advancing time:
|
||||
```
|
||||
!import asyncio
|
||||
!loop = asyncio.get_event_loop()
|
||||
!task = asyncio.ensure_future(some_async_fn())
|
||||
# Now inspect task state, don't await it
|
||||
p task
|
||||
```
|
||||
|
||||
### PYTHONASYNCIODEBUG
|
||||
|
||||
Enable before running the process:
|
||||
|
||||
```bash
|
||||
PYTHONASYNCIODEBUG=1 python script.py
|
||||
```
|
||||
|
||||
Surfaces: coroutines that were never awaited, slow callbacks, unhandled task exceptions. **Always turn this on** if the bug is timing- or async-related.
|
||||
|
||||
### `asyncio.gather` swallows the first exception
|
||||
|
||||
By default, `asyncio.gather(t1, t2)` raises the first exception and cancels the rest. If you need all exceptions, use `gather(..., return_exceptions=True)`.
|
||||
|
||||
### Unhandled task exceptions are silent
|
||||
|
||||
```python
|
||||
async def main():
|
||||
task = asyncio.create_task(broken_coroutine())
|
||||
# If task raises and we never await it, the exception is eaten at gc time
|
||||
await asyncio.sleep(10)
|
||||
```
|
||||
|
||||
To catch these, set `loop.set_exception_handler(...)` or upgrade to Python 3.12+ which warns louder by default.
|
||||
|
||||
---
|
||||
|
||||
## debugpy — remote / IDE / container attach
|
||||
|
||||
Listen and wait for attach:
|
||||
|
||||
```bash
|
||||
python -m debugpy --listen 0.0.0.0:5678 --wait-for-client script.py
|
||||
```
|
||||
|
||||
Attach from VS Code:
|
||||
```json
|
||||
// .vscode/launch.json
|
||||
{
|
||||
"name": "attach",
|
||||
"type": "python",
|
||||
"request": "attach",
|
||||
"connect": { "host": "localhost", "port": 5678 }
|
||||
}
|
||||
```
|
||||
|
||||
Inside the code, programmatic attach point:
|
||||
```python
|
||||
import debugpy
|
||||
debugpy.listen(5678)
|
||||
debugpy.wait_for_client() # blocks until attached
|
||||
debugpy.breakpoint() # programmatic breakpoint
|
||||
```
|
||||
|
||||
**Journal**: the port (5678) and the listener file — unset at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## Sampling profilers for "why is it slow / stuck"
|
||||
|
||||
When the problem is performance or a hang (not a crash), don't attach pdb — it alters timing. Use a sampling profiler that attaches to the running process:
|
||||
|
||||
```bash
|
||||
# py-spy — production-safe, zero code change, works on running process
|
||||
py-spy top --pid <pid> # live top-like view
|
||||
py-spy record -o profile.svg --pid <pid> # flamegraph
|
||||
py-spy dump --pid <pid> # stack traces of all threads right now
|
||||
|
||||
# memray — memory allocation tracking
|
||||
memray run script.py
|
||||
memray flamegraph output.bin
|
||||
memray stats output.bin
|
||||
```
|
||||
|
||||
`py-spy dump` on a stuck process is often enough to find the hung call — no breakpoints needed.
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in Python
|
||||
|
||||
Add these to Phase 8's silent-failure check:
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| `except Exception: pass` or `except: pass` | Catches and discards every error including KeyboardInterrupt |
|
||||
| `logging.exception(...)` in a logger with no handlers | "Logs" but actually writes nowhere |
|
||||
| `asyncio.create_task(coro)` without storing the task | Task GC'd before completion, exception swallowed |
|
||||
| `return x.get("key")` where key is missing | Returns None silently, caller often doesn't check |
|
||||
| `subprocess.run(..., check=False)` with ignored returncode | Non-zero exit treated as success |
|
||||
| Django `transaction.atomic()` inside a broader `except` | Rolls back silently |
|
||||
| `contextlib.suppress(Exception)` | Explicit silencer; easy to leave wider than intended |
|
||||
| `queue.get(block=False)` with `except queue.Empty: pass` | Polling that silently drops the work |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Remove breakpoint() / ipdb / pudb lines from source
|
||||
git diff | grep -E '(breakpoint\(\)|import ipdb|import pudb|import pdb; pdb\.set_trace)'
|
||||
# If the above has output, revert those files:
|
||||
git checkout <file>
|
||||
|
||||
# Unset the global breakpoint override
|
||||
unset PYTHONBREAKPOINT
|
||||
|
||||
# Kill any leftover debugpy listeners
|
||||
pkill -f 'debugpy' || true
|
||||
lsof -iTCP:5678 -sTCP:LISTEN -nP 2>/dev/null # confirm free
|
||||
```
|
||||
@@ -0,0 +1,234 @@
|
||||
# Rust Debugging
|
||||
|
||||
Covers `cargo`, `tokio`, panics, and the fact that you usually don't actually need a debugger — Rust's type system, `dbg!`, and logging cover 80% of sessions faster than gdb would.
|
||||
|
||||
---
|
||||
|
||||
## Environment detection (Phase 0)
|
||||
|
||||
```bash
|
||||
cargo --version
|
||||
rustc --version
|
||||
cat rust-toolchain.toml rust-toolchain 2>/dev/null
|
||||
cat Cargo.toml | head -30
|
||||
|
||||
# Debuggers available
|
||||
which rust-gdb
|
||||
which rust-lldb
|
||||
which lldb
|
||||
|
||||
# Async infrastructure
|
||||
grep -E '"(tokio|async-std|smol)"' Cargo.toml
|
||||
|
||||
# Profile flags
|
||||
grep -E '^\[profile' Cargo.toml
|
||||
```
|
||||
|
||||
**The default `cargo run` builds with `dev` profile** which includes debug symbols. `cargo run --release` strips them. For debugging, stay in dev unless the bug only manifests under optimization.
|
||||
|
||||
---
|
||||
|
||||
## The Rust debugging hierarchy (use in this order)
|
||||
|
||||
Rust's ecosystem has a specific order that's faster than reaching for gdb first:
|
||||
|
||||
1. **`dbg!(expr)` macro** — for a single value at a specific spot. Prints file:line + value, returns the value unchanged so you can inline it. Faster than a debugger for 60% of bugs.
|
||||
2. **`RUST_LOG=trace` with `tracing` / `env_logger`** — for flow and state across an operation. Zero code change in dev-time.
|
||||
3. **`RUST_BACKTRACE=1` / `=full`** — for crashes. Almost always sufficient; you rarely need a live debugger for a panic.
|
||||
4. **`rust-gdb` / `rust-lldb`** — when you need to pause execution and inspect memory, especially for unsafe code or FFI.
|
||||
5. **`tokio-console`** — for async deadlocks, stuck tasks, hot loops.
|
||||
6. **`cargo-expand`** — when a macro is doing something weird.
|
||||
|
||||
Reach for the lightest tool that answers the hypothesis.
|
||||
|
||||
---
|
||||
|
||||
## `dbg!` — the underused macro
|
||||
|
||||
```rust
|
||||
let x = 5;
|
||||
let y = dbg!(x * 2); // prints: [src/main.rs:2] x * 2 = 10
|
||||
```
|
||||
|
||||
Inside a complex expression:
|
||||
```rust
|
||||
let total = items.iter().filter(|i| i.active).map(|i| dbg!(i.cost)).sum::<u64>();
|
||||
```
|
||||
|
||||
Multiple values at once:
|
||||
```rust
|
||||
dbg!(&user, &request, elapsed.as_millis());
|
||||
```
|
||||
|
||||
`dbg!` writes to stderr, so it won't corrupt stdout-based pipelines. **Journal each `dbg!` you add**; revert at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## `RUST_LOG` for flow-level debugging
|
||||
|
||||
If the codebase uses `tracing` or `env_logger`:
|
||||
|
||||
```bash
|
||||
RUST_LOG=debug cargo run
|
||||
RUST_LOG=trace cargo run # very verbose
|
||||
RUST_LOG=my_crate=trace,hyper=info cargo run # per-module level
|
||||
RUST_LOG=debug,tokio=off cargo run # silence noisy crates
|
||||
```
|
||||
|
||||
For `tracing`-based apps, instrument with spans:
|
||||
```rust
|
||||
#[tracing::instrument]
|
||||
fn handle_request(req: &Request) -> Response { ... }
|
||||
```
|
||||
|
||||
This gives you structured per-call entry/exit logs with args and timing, zero additional code in the body.
|
||||
|
||||
---
|
||||
|
||||
## `RUST_BACKTRACE` for panics
|
||||
|
||||
```bash
|
||||
RUST_BACKTRACE=1 cargo run # backtrace on panic
|
||||
RUST_BACKTRACE=full cargo run # include libstd/tokio frames
|
||||
```
|
||||
|
||||
The panic itself usually tells you the file:line. The backtrace tells you how it got there. Between the two, most crash bugs are solved without a debugger.
|
||||
|
||||
---
|
||||
|
||||
## rust-gdb / rust-lldb
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
# Build with debug symbols (default dev profile)
|
||||
cargo build
|
||||
|
||||
# Attach gdb wrapper (applies Rust type pretty-printers)
|
||||
rust-gdb ./target/debug/my_binary
|
||||
# Or lldb:
|
||||
rust-lldb ./target/debug/my_binary
|
||||
|
||||
# With args
|
||||
rust-gdb --args ./target/debug/my_binary arg1 arg2
|
||||
|
||||
# Attach to running process
|
||||
rust-gdb -p $(pgrep my_binary)
|
||||
```
|
||||
|
||||
### Breakpoints
|
||||
|
||||
Rust symbols are mangled. Use either:
|
||||
|
||||
```
|
||||
(gdb) b main # main function
|
||||
(gdb) b my_crate::module::function # canonical path
|
||||
(gdb) b src/handler.rs:42 # file:line
|
||||
(gdb) info functions my_function # find mangled name
|
||||
```
|
||||
|
||||
### State inspection
|
||||
|
||||
```
|
||||
(gdb) p x # print value (uses Rust pretty-printer for Vec, Option, HashMap, etc.)
|
||||
(gdb) p *ptr # deref
|
||||
(gdb) info locals # all locals in current frame
|
||||
(gdb) info args # function args
|
||||
(gdb) bt # backtrace
|
||||
(gdb) frame <n> # switch to stack frame n
|
||||
(gdb) watch my_var # stop when my_var changes
|
||||
(gdb) rbreak regex # breakpoint all functions matching regex
|
||||
```
|
||||
|
||||
**Pair with pwndbg** for better layout on native bugs — see [tools/pwndbg.md](../tools/pwndbg.md). Pwndbg works with rust-gdb too.
|
||||
|
||||
---
|
||||
|
||||
## tokio-console — async task debugging
|
||||
|
||||
For tokio-based async apps, this is essential when tasks are stuck or leaking.
|
||||
|
||||
```bash
|
||||
# Add tokio-console instrumentation to the target binary
|
||||
# In Cargo.toml:
|
||||
# [dependencies]
|
||||
# console-subscriber = "0.2"
|
||||
|
||||
# In main.rs:
|
||||
# console_subscriber::init();
|
||||
|
||||
# Build with tokio_unstable:
|
||||
RUSTFLAGS="--cfg tokio_unstable" cargo run
|
||||
|
||||
# In another terminal:
|
||||
tokio-console # connects to default port 6669
|
||||
```
|
||||
|
||||
Shows live tasks, their state, wake counts, poll durations, parent tasks. The single fastest way to find "why is my async thing stuck".
|
||||
|
||||
---
|
||||
|
||||
## cargo-expand — when a macro is suspect
|
||||
|
||||
```bash
|
||||
cargo install cargo-expand
|
||||
cargo expand # expand all macros in the crate
|
||||
cargo expand my::module::path # scope to one item
|
||||
```
|
||||
|
||||
If you suspect a macro (especially `#[derive]`, `#[tokio::main]`, `#[async_trait]`) is generating code that doesn't match your mental model, this shows you exactly what the compiler sees.
|
||||
|
||||
---
|
||||
|
||||
## Release-build gotcha
|
||||
|
||||
```bash
|
||||
cargo build --release # no debug symbols by default
|
||||
```
|
||||
|
||||
If the bug only shows up in `--release`:
|
||||
|
||||
```toml
|
||||
# Cargo.toml
|
||||
[profile.release]
|
||||
debug = true # add symbols, keep optimizations
|
||||
```
|
||||
|
||||
Now `rust-gdb ./target/release/my_binary` works on release builds. This is required when optimization-enabled codegen bugs (inlining, LLVM folding) are suspected.
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in Rust
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| `.unwrap_or_default()` | Masks errors as the zero value |
|
||||
| `.unwrap_or(fallback)` | Same, with a specific fallback |
|
||||
| `let _ = fallible_operation()` | Explicitly discards the Result, no compiler warning |
|
||||
| `if let Ok(x) = ... { use(x); } // no else` | Silent on Err |
|
||||
| `.ok()` chaining | Converts Result to Option, throwing the error away |
|
||||
| Panic inside a tokio task not `.await`ed | Task dies silently; runtime usually logs but it's quiet if logs are off |
|
||||
| `eprintln!` that goes to a redirected-null stderr | Looks like nothing happened |
|
||||
| `Drop` impl that panics under specific condition | Double-panic aborts process silently if no logging configured |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Revert dbg! macro additions
|
||||
git diff | grep -E 'dbg!\('
|
||||
# For any file with dbg! additions:
|
||||
git checkout <file>
|
||||
|
||||
# Unset env vars
|
||||
unset RUST_LOG RUST_BACKTRACE
|
||||
|
||||
# Kill any rust-gdb/lldb sessions
|
||||
pkill -f 'rust-gdb' || true
|
||||
pkill -f 'rust-lldb' || true
|
||||
pkill -f '^lldb ' || true
|
||||
|
||||
# tokio-console binds 6669 by default
|
||||
lsof -iTCP:6669 -sTCP:LISTEN -nP 2>/dev/null
|
||||
```
|
||||
Reference in New Issue
Block a user