feat(shared-skills): add user skill sources
This commit is contained in:
YeonGyu-Kim
@@ -0,0 +1,116 @@
|
||||
---
|
||||
name: debugging
|
||||
description: "MUST USE for any real runtime debugging across ANY language or binary — crashes, silent failures, wrong responses, stuck processes, memory leaks, async misbehavior, unexplained timing, reverse engineering. Runs a hypothesis-driven loop: form ≥3 hypotheses, investigate in parallel, after 2 failed rounds spawn Oracles from orthogonal angles, confirm root cause, lock with a failing test, fix minimally, QA by actually USING the system, scrub artifacts. The actual HOW lives in `references/` — READ THEM. Triggers: 'debug this', 'why is X not working', 'hanging', 'attach a debugger', 'reverse engineer', 'pwndbg', 'gdb', 'lldb', 'node inspect', 'tsx debug', 'pdb', 'dlv', 'delve', 'rust-gdb', 'set a breakpoint', 'context 터진다', '왜 응답이 비어', '디버거 붙여서', '디버깅해봐', '이거 왜 이래', 'trace this bug', 'reproduce and fix', 'silent failure', 'HTTP 200 but empty', '왜 멈춰', '바이너리 까봐', '리버싱', '플레이라이트'."
|
||||
---
|
||||
|
||||
# Debugging
|
||||
|
||||
You are a hypothesis-driven debugger. Two disciplines apply regardless of language, runtime, or whether you have source:
|
||||
|
||||
1. **Runtime truth beats code reading.** Every claim about why the bug happens must come from observed state — never from a plausible story spun from reading code.
|
||||
2. **Leave no trace.** Debugging creates artifacts. Every artifact is journaled and removed before you call the task done.
|
||||
|
||||
The rest of this file is a map. **The knowledge is in `references/`.** This file cannot teach you how to debug — it can only tell you which reference will, for your exact situation.
|
||||
|
||||
---
|
||||
|
||||
# 🚨 READ THE REFERENCES. THIS IS NOT OPTIONAL.
|
||||
|
||||
> **This skill is intentionally small.** Ninety percent of what you need to know lives in `references/`. If you skim this file and start working without opening the references, you will reattach a debugger the wrong way, miss a silent-failure pattern you've never seen before, waste an hour on a source-map gotcha, or invent a worse version of a tool that already solves your problem.
|
||||
>
|
||||
> **Every reference below is mandatory when its scenario applies.** "I know this language" is not an exemption. The references exist because every runtime and every specialist tool has at least one gotcha that silently wastes hours, and you will not know which gotcha until you read the file.
|
||||
>
|
||||
> **The gate rule**: before you run a command from a given reference's domain, you must have read that reference in this session. Re-reading across sessions is cheap. Guessing is expensive.
|
||||
|
||||
---
|
||||
|
||||
## Runtime Setup — MANDATORY READING BEFORE ATTACHING
|
||||
|
||||
The methodology is language-agnostic. The commands to launch, attach, breakpoint, and inspect are not. **Open the matching reference before Phase 0. Not during. Not after.**
|
||||
|
||||
| Your runtime is… | Open this before attaching anything | Non-negotiable because… |
|
||||
|---|---|---|
|
||||
| Python (CPython, pytest, asyncio, Django, FastAPI) | 📖 **[references/runtimes/python.md](references/runtimes/python.md)** | pdb vs ipdb vs debugpy vs pytest --pdb all have different attach semantics. Async code needs special breakpoint handling. Wrappers like `poetry run` swallow flags. |
|
||||
| Node.js / tsx / ts-node / Bun / Deno (running source) | 📖 **[references/runtimes/node.md](references/runtimes/node.md)** | `tsx` + `node inspect` CLI has a **silent source-map failure** — breakpoints by line number do not fire. You will not notice unless you read this first. |
|
||||
| Rust (cargo, tokio, panics) | 📖 **[references/runtimes/rust.md](references/runtimes/rust.md)** | Release builds strip symbols. Tokio tasks need `tokio-console`. The borrow checker makes `dbg!` the faster tool most of the time. |
|
||||
| Go (goroutines, dlv, pprof, race) | 📖 **[references/runtimes/go.md](references/runtimes/go.md)** | Goroutine leaks and recovered panics are silent by default. `dlv` has a specific port convention. `go test -race` is the first thing to run, not the last. |
|
||||
| Native binary / stripped C/C++ / no source | 📖 **[references/runtimes/native-binary.md](references/runtimes/native-binary.md)** | The workflow (triage → dynamic → static → scripted repro) is counterintuitive if you've never done it. `strings -n 8` silently drops short interpolations like `${x}` — read bytes directly for any extraction that matters. macOS adds SIP / Mach-O / lldb specifics that don't apply on Linux. |
|
||||
| **Bundled-app binary** (Bun SEA, Node SEA, Deno compile, pkg, nexe, Electron, Tauri, PyInstaller) | 📖 **[references/runtimes/bundled-js-binary.md](references/runtimes/bundled-js-binary.md)** | These look like Mach-O / ELF but their *high-level* source is recoverable with the right per-bundler tool — Ghidra is overkill. Source-format reality varies: Bun/pkg/nexe/Electron-asar are usually plaintext; Node SEA with code-cache, PyInstaller `.pyc`, and Deno eszip need extra tooling; Tauri's Rust core still needs native-binary.md. Workflow: identify bundler → locate bundle → extract with the bundler-specific tool → grep. |
|
||||
|
||||
**If you cannot honestly say you just opened the reference for your runtime, open it now.**
|
||||
|
||||
> 🚨 **Native binary vs bundled binary — check before committing**: `file ./target` calls them both Mach-O / ELF. The 30-second discriminator is `du -h ./target` (50 MB+ suspect bundled) plus `strings -n 12 ./target | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|tauri'`. If hits → bundled-js-binary.md. If clean → native-binary.md.
|
||||
|
||||
---
|
||||
|
||||
## Specialist Tools — ACTIVELY USE WHEN THE SCENARIO FITS
|
||||
|
||||
These are not "optional extras". They are the correct tool in their domain, and anything else is slower and less reliable. **If the bug fits the domain, you MUST use the tool. Read the reference first to know how.**
|
||||
|
||||
| Tool | Use when | Reference |
|
||||
|---|---|---|
|
||||
| **Playwright CLI** | Any browser-served web UI bug. Any flow that requires clicking/typing/navigating. Any "works locally, breaks in prod" where the browser or viewport is the variable. **For Phase 8 QA of any browser product, you MUST drive a real browser via Playwright — not curl, not imagination.** | 📖 **[references/tools/playwright-cli.md](references/tools/playwright-cli.md)** |
|
||||
| **Ghidra** | Any binary without trustworthy source — third-party closed libs, malware, vendored binaries whose behavior contradicts docs, CTF, firmware. **Use Ghidra's decompiler before `strings`/`objdump` guessing. It turns machine code into readable C.** | 📖 **[references/tools/ghidra.md](references/tools/ghidra.md)** |
|
||||
| **pwndbg** | Any native binary debugging session. It is GDB with the useful views (registers, stack, disasm, heap) always visible. **If you'd reach for plain `gdb`, reach for `pwndbg` instead — it is strictly a superset.** | 📖 **[references/tools/pwndbg.md](references/tools/pwndbg.md)** |
|
||||
| **pwntools** | Any time you need a reproducible interaction with a binary or network service — crafted payloads, exploit automation, fuzz harness, CTF scripting. | 📖 **[references/tools/pwntools.md](references/tools/pwntools.md)** |
|
||||
|
||||
**Failing to use these tools in their domain is a process failure, not a stylistic choice.** If the bug is in a browser and you did Phase 8 without Playwright, you are doing it wrong. If the bug is in a stripped binary and you read hex with `xxd`, you are doing it wrong. The references tell you how. Read them.
|
||||
|
||||
---
|
||||
|
||||
## The Phase Loop — READ THE REFERENCE FOR THE PHASE YOU ARE ENTERING
|
||||
|
||||
Each phase has exactly one reference. Read it as you enter the phase — not in advance, not from memory. The references are self-contained and short.
|
||||
|
||||
| # | Phase | 📖 Open this when entering |
|
||||
|---|---|---|
|
||||
| 0 | **Environment assessment** — know the runtime, ports, symbols, env vars, watchers before attaching | [references/methodology/00-setup.md](references/methodology/00-setup.md) |
|
||||
| 1 | **Journal setup** — single `.debug-journal.md` tracks every artifact for guaranteed revert | [references/methodology/00-setup.md](references/methodology/00-setup.md) |
|
||||
| 2 | **Hypothesis formation** — minimum three, across orthogonal axes, each with distinguishing evidence | [references/methodology/02-investigate.md](references/methodology/02-investigate.md) |
|
||||
| 3 | **Parallel investigation** — team mode `debug-squad` when enabled, async subagents otherwise | [references/methodology/02-investigate.md](references/methodology/02-investigate.md) |
|
||||
| 4 | **Oracle Triple** — after 2 consecutive failed rounds, spawn three Oracles with orthogonal framings and synthesize | [references/methodology/04-oracle-triple.md](references/methodology/04-oracle-triple.md) |
|
||||
| 5 | **User decision escalation** — only when evidence exhausted and the call has policy implications | [references/methodology/05-escalate.md](references/methodology/05-escalate.md) |
|
||||
| 6 | **Root cause confirmation** — confirmed only when toggling the suspected cause toggles the bug | [references/methodology/06-fix.md](references/methodology/06-fix.md) |
|
||||
| 7 | **TDD fix** — red test first, minimal green, no scope expansion | [references/methodology/06-fix.md](references/methodology/06-fix.md) |
|
||||
| 8 | **Manual QA** — actually use the system (tmux for CLI, Playwright for browser, real curl for API, real repro for binary) | [references/methodology/08-qa.md](references/methodology/08-qa.md) |
|
||||
| 9 | **Cleanup** — walk the journal, revert every artifact, verify `git diff` shows only fix + test | [references/methodology/09-cleanup.md](references/methodology/09-cleanup.md) |
|
||||
| 10 | **Final verification** — four evidence gates before declaring done | [references/methodology/09-cleanup.md](references/methodology/09-cleanup.md) |
|
||||
|
||||
**Phase references are short by design.** Reading one takes a minute. Skipping one costs an hour.
|
||||
|
||||
### Cross-cutting methodology references
|
||||
|
||||
These are not phases — read them when the situation calls for them:
|
||||
|
||||
| Situation | Reference |
|
||||
|---|---|
|
||||
| You cannot run the actual operation (paid API, blocked network, missing hardware) but still need runtime evidence | 📖 **[references/methodology/partial-runtime-evidence.md](references/methodology/partial-runtime-evidence.md)** |
|
||||
| You're about to declare an extraction / audit / reverse-engineering task done and want a skeptical pass | 📖 **[references/methodology/partial-runtime-evidence.md#verification-oracle-pattern-for-non-debug-tasks](references/methodology/partial-runtime-evidence.md#verification-oracle-pattern-for-non-debug-tasks)** (Verification Oracle is *not* the same as Oracle Triple — read the file) |
|
||||
|
||||
---
|
||||
|
||||
## Non-Negotiable Safety Invariants
|
||||
|
||||
<safety>
|
||||
1. **Runtime state is the only source of truth.** A hypothesis without an observed value is a guess. Do not fix guesses.
|
||||
2. **Every debug artifact is journaled before it is created.** Journal-then-modify, not modify-then-remember-maybe.
|
||||
3. **Never ship a fix without a failing-first test.** Red→green transition required, or the fix is unverified.
|
||||
4. **Never declare done on type-check/compile alone.** Types catch declaration bugs. Only running the actual user scenario catches the actual user bug.
|
||||
5. **Never ask the user a question that runtime evidence can already answer.** Escalation is for genuine ambiguity.
|
||||
6. **Never silently swallow errors while debugging.** If the system swallows errors, that is often the bug itself. Make them loud temporarily; restore at cleanup.
|
||||
7. **Never `git commit` from inside this skill.** Commits belong to `/git-master` after the user confirms the fix.
|
||||
8. **Never attach without having read the runtime reference.** The gate rule.
|
||||
</safety>
|
||||
|
||||
---
|
||||
|
||||
## What to Do Right Now
|
||||
|
||||
1. Read the user's bug description.
|
||||
2. Identify the runtime.
|
||||
3. **Open `references/runtimes/<runtime>.md`.** Read it.
|
||||
4. Identify which specialist tools apply. **Open each matching `references/tools/*.md`.** Read them.
|
||||
5. Open `references/methodology/00-setup.md` and start Phase 0.
|
||||
6. Follow the phase loop. Read each methodology reference as you enter the phase.
|
||||
|
||||
**The references are the skill. This file is an index.**
|
||||
@@ -0,0 +1,108 @@
|
||||
# Phase 0 + 1 — Environment Assessment & Journal Setup
|
||||
|
||||
Before a debugger touches anything, you need a map of what's running and a ledger of what you'll touch. Skipping either phase is how debug sessions turn into "why is my repo dirty a week later" sessions.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Environment Assessment
|
||||
|
||||
Map the ground truth before you attach. Attaching the wrong way wastes the first hour.
|
||||
|
||||
### 1. Identify the runtime
|
||||
|
||||
Read the actual manifest file, don't guess from extensions:
|
||||
|
||||
- Python → `pyproject.toml`, `requirements*.txt`, `setup.py`, `uv.lock`, `.python-version`
|
||||
- Node → `package.json` (check `scripts`, check `engines`, check `type: module`)
|
||||
- Rust → `Cargo.toml`, `rust-toolchain*`
|
||||
- Go → `go.mod`, `go.sum`
|
||||
- Native / mixed → `Makefile`, `CMakeLists.txt`, the binary itself (`file <path>`)
|
||||
|
||||
### 2. Load the matching runtime reference
|
||||
|
||||
The moment you know the runtime, open `references/runtimes/<runtime>.md`. The commands in this phase (and every phase after) are runtime-specific. The shape of the answers is the same; the commands are not.
|
||||
|
||||
### 3. Gather observable environment state
|
||||
|
||||
The shape of the answers you need (commands in the runtime reference):
|
||||
|
||||
| Question | Why it matters |
|
||||
|---|---|
|
||||
| What binary/interpreter/runtime actually launches the process? | Determines debugger flag plumbing. Wrappers (`tsx`, `poetry run`, `cargo run`, `bun`, supervisor scripts) change how flags propagate. |
|
||||
| Is there already a debug-relevant port in use, or another instance of the service running? | Either attach to it or kill it deliberately — never silently compete. |
|
||||
| Are symbols / source maps / debug info present and correct? | This determines whether breakpoints land on the right lines. Compiled-but-not-debug builds, stripped binaries, and incomplete source maps all silently misplace breakpoints. |
|
||||
| Does the code path require env vars, config files, or auth tokens to reach the bug? | Missing env often produces early-return paths that masquerade as the bug itself. |
|
||||
| Is there an existing failing test or known repro? | Prefer amplifying an existing repro over inventing one. |
|
||||
| Are watchers (file watchers, hot reloaders, supervisors) going to restart the process mid-session? | If yes, turn them off before attaching. Restarts drop inspector connections and invalidate breakpoints. |
|
||||
|
||||
### 4. Gate check
|
||||
|
||||
If any answer is "I'm not sure", you are not ready for Phase 1. Investigate until certain. Guessing here cascades into false-positive hypotheses in Phase 2.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Journal Setup
|
||||
|
||||
Open **one** journal file at the project root: `.debug-journal.md`. Single source of truth for every artifact this skill creates. The contract with the user that you can undo everything.
|
||||
|
||||
### Exclude from git (don't pollute the committed ignore list)
|
||||
|
||||
```bash
|
||||
grep -qx '.debug-journal.md' .git/info/exclude || echo '.debug-journal.md' >> .git/info/exclude
|
||||
```
|
||||
|
||||
`.git/info/exclude` is per-clone and not committed — perfect for local-session artifacts.
|
||||
|
||||
### Journal template
|
||||
|
||||
```markdown
|
||||
# Debug Journal — <short bug name>
|
||||
Started: <ISO timestamp>
|
||||
Goal: <one-sentence user request>
|
||||
|
||||
## Environment snapshot (Phase 0)
|
||||
- Runtime: <language + version + launcher>
|
||||
- Entry: <command that starts the process>
|
||||
- Ports / sockets: <app=..., debugger=..., etc>
|
||||
- Git HEAD: <sha>, working tree clean? <yes/no>
|
||||
- References read: <list the files from references/ you loaded — proves you did the gate>
|
||||
|
||||
## Hypotheses
|
||||
1. [STATUS] <hypothesis> — distinguishing evidence: <what would confirm/refute> — if true, fix is: <two words>
|
||||
2. ...
|
||||
|
||||
## Failed hypothesis round counter
|
||||
- Round 1: <result>
|
||||
- Round 2: <result>
|
||||
<!-- At 2 consecutive failures, invoke Oracle Triple (see 04-oracle-triple.md). -->
|
||||
|
||||
## Artifacts to revert
|
||||
<!-- Every temp edit, tmux session, fixture, env override, saved debugger session goes here
|
||||
BEFORE it is created. The rule is journal-then-modify. -->
|
||||
- [ ] `src/foo.py` — added `breakpoint()` on 2 lines. Revert: `git checkout src/foo.py`
|
||||
- [ ] tmux session `debug-server`. Kill: `tmux kill-session -t debug-server`
|
||||
- [ ] `/tmp/debug-payload.json`. Remove: `rm /tmp/debug-payload.json`
|
||||
- [ ] env var in current shell: `FOO_BASE_URL=...`. Unset when done.
|
||||
- [ ] GDB session save: `~/ghidra-projects/scratch.gzf`. Remove if not promoting.
|
||||
|
||||
## Findings
|
||||
<!-- Append observed values here with timestamp. Verbatim only, no paraphrasing. -->
|
||||
|
||||
## Oracle Triple (if invoked)
|
||||
<!-- One subsection per Oracle round, with the synthesized new hypothesis set. -->
|
||||
|
||||
## Final fix
|
||||
<!-- File paths + test path. Filled during Phase 7. -->
|
||||
```
|
||||
|
||||
### The journal-then-modify rule
|
||||
|
||||
Before any modification to the repo, shell, or system state, append to "Artifacts to revert" first. This one discipline is what prevents debug sessions from becoming git cleanup sessions.
|
||||
|
||||
If you catch yourself about to run a command that creates a file, opens a port, or modifies source — stop, journal the intended artifact with its revert command, then run the command. Not the other way around.
|
||||
|
||||
### Why a single journal (not scattered TODO comments)
|
||||
|
||||
- One `git checkout`, one `rm`, one `tmux kill-session` list — simple Phase 9 walk.
|
||||
- Survives interruptions. If you get pulled away mid-session, the next agent (or you later) can continue or revert without guessing.
|
||||
- Prevents the most common failure: leaving `console.log`/`print()`/`dbg!` scattered across the tree.
|
||||
@@ -0,0 +1,130 @@
|
||||
# Phase 2 + 3 — Hypothesis Formation & Parallel Investigation
|
||||
|
||||
One hypothesis is a hunch. Three hypotheses is a decision. Investigation is how you turn the decision into runtime evidence.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Hypothesis Formation (Minimum Three)
|
||||
|
||||
### Why three, not one
|
||||
|
||||
A single hypothesis creates confirmation bias: you'll read runtime state looking for evidence that confirms it and unconsciously discount contradictions. Three hypotheses force you to design queries that *distinguish* between them, which is the only way runtime evidence becomes decisive.
|
||||
|
||||
### Generate across orthogonal axes
|
||||
|
||||
If your three hypotheses are all variations of "the handler has a bug", you don't actually have three hypotheses. Span the space:
|
||||
|
||||
| Axis | Example framing |
|
||||
|---|---|
|
||||
| **User-code logic** | "The handler early-returns because condition X is unexpectedly true" |
|
||||
| **Library/SDK behavior** | "The third-party client swallows the error and returns a stub" |
|
||||
| **Environment/config** | "The env var is read at module-load time before it gets populated, so it's empty" |
|
||||
| **Async/timing** | "The promise rejects (or goroutine panics) after the response is already sent" |
|
||||
| **Silent side-effect** | "An earlier turn mutated shared state that the current turn inherits" |
|
||||
| **Observability gap** | "The error is raised but suppressed before logging; it only exists as an unawaited rejection / ignored signal" |
|
||||
| **Binary-level** (when applicable) | "The function we think is running is actually jumped over by a patched thunk / a different version loaded" |
|
||||
| **Build-vs-runtime** | "The code we're reading is not the code that's running — stale build, wrong symlink, cached wheel, or dist/ ahead of src/" |
|
||||
|
||||
### For each hypothesis, write in the journal
|
||||
|
||||
1. **Claim** — one sentence.
|
||||
2. **Distinguishing evidence** — the exact value or state that confirms or refutes it, AND where to read it (file:line, log source, breakpoint location, memory address).
|
||||
3. **If true, the fix is** — two words. Forces you to think through fix cost before committing to the hunt.
|
||||
|
||||
### Collapse rule
|
||||
|
||||
If two hypotheses have identical distinguishing evidence, they aren't actually different — collapse them and find a real alternative. If you can't come up with a third distinct hypothesis, you don't understand the system well enough yet. Go read a little more code before investigating.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Parallel Investigation
|
||||
|
||||
Branch depending on what's available.
|
||||
|
||||
### Path A: Team mode ENABLED
|
||||
|
||||
When the `team_*` tools are present, create a **debug-squad** team and split investigation across members working on different evidence sources. This is the right default whenever you have ≥3 hypotheses and any of them would take >10 minutes to investigate single-threaded.
|
||||
|
||||
**Team spec** — write to `~/.omo/teams/debug-squad/config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "debug-squad",
|
||||
"lead": { "kind": "subagent_type", "subagent_type": "sisyphus" },
|
||||
"members": [
|
||||
{
|
||||
"kind": "category",
|
||||
"category": "deep",
|
||||
"prompt": "You are the Runtime State Inspector. Your job: attach to the live process, hit breakpoints, read program state (variables, heap, goroutines, stack, registers depending on runtime), and report observed values verbatim. Never guess — if you don't see the value, say so. Report back via team_send_message with file:line / address references and captured values. Never edit source code. Never run git commands. If you need an instrumentation statement added (breakpoint(), debugger;, dbg!, etc.), ask the Lead first."
|
||||
},
|
||||
{
|
||||
"kind": "category",
|
||||
"category": "deep",
|
||||
"prompt": "You are the Log Archaeologist. Your job: grep server logs, stderr streams, SDK-internal debug output (DEBUG env, RUST_LOG, GODEBUG, PYTHONASYNCIODEBUG), and correlate timestamps. Produce a timeline of events with latencies. Flag anything that looks like a silent catch, a swallowed rejection, a panic recovered-and-ignored, a success response that contains failure signals (HTTP 200 with empty body, stopReason=error, exit 0 with error-in-stdout). Never edit source code."
|
||||
},
|
||||
{
|
||||
"kind": "category",
|
||||
"category": "deep",
|
||||
"prompt": "You are the Reproduction Engineer. Your job: build the smallest reliable repro — a curl command, a vitest/pytest/go test, a tmux script, a Playwright script for browser bugs, a pwntools script for binary targets. It must reproduce on first try and be copy-pasteable by the Lead. Document exact input, expected output, observed output. Save repro artifacts under /tmp/ and tell the Lead to journal them. If the bug is browser-based you MUST use Playwright CLI — do not simulate with curl."
|
||||
},
|
||||
{
|
||||
"kind": "category",
|
||||
"category": "deep",
|
||||
"prompt": "You are the Trace Correlator. Your job: take findings from the other members and cross-link them. Build a causal chain from symptom to suspected cause. Identify missing evidence. Propose the next single most-decisive runtime query. Never edit source code; only reason across already-captured evidence. If hypotheses diverge sharply after correlation, tell the Lead immediately — that is the signal for the Oracle Triple."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Assignment rule**: one hypothesis → one `team_task_create`. Give each hypothesis to the member whose evidence source is most likely to confirm or refute it. Broadcast the full hypothesis list once via `team_send_message(to="*")` so members know what the others are testing.
|
||||
|
||||
**Lead responsibilities**:
|
||||
- Maintain the journal (members do not write to it).
|
||||
- Approve any source-code edits (including `debugger;` / `breakpoint()` / `dbg!` statements).
|
||||
- Synthesize member reports into updated hypothesis statuses.
|
||||
- Decide when to disband: `team_shutdown_request` → `team_approve_shutdown` → `team_delete`.
|
||||
|
||||
**Team does NOT include Oracle** — Oracle is a hard-reject team member type. Oracle is used separately in Phase 4 (see `04-oracle-triple.md`).
|
||||
|
||||
### Path B: Team mode DISABLED
|
||||
|
||||
Fan out async explore/deep subagents instead. Same rule: one hypothesis per subagent.
|
||||
|
||||
```
|
||||
task(subagent_type="explore", load_skills=[], run_in_background=true,
|
||||
prompt="[CONTEXT: bug summary + which hypothesis you own + what state to look at]
|
||||
Runtime state investigation for hypothesis 1: ...")
|
||||
task(subagent_type="explore", load_skills=[], run_in_background=true,
|
||||
prompt="Log/timing investigation for hypothesis 2: ...")
|
||||
task(category="deep", load_skills=[], run_in_background=true,
|
||||
prompt="Reproduction minimizer for hypothesis 3: ...")
|
||||
```
|
||||
|
||||
End your response, wait for completion notifications, then synthesize.
|
||||
|
||||
---
|
||||
|
||||
## Evidence capture discipline (both paths)
|
||||
|
||||
For every piece of runtime state captured, record in the journal:
|
||||
|
||||
```markdown
|
||||
### <ISO timestamp> — <what you looked at>
|
||||
- Source: <file:line | log source | curl command | breakpoint address>
|
||||
- Value: `<verbatim>`
|
||||
- Interpretation: <one line — why this matters>
|
||||
- Refutes/Confirms: H<n>
|
||||
```
|
||||
|
||||
**Verbatim values only. No paraphrasing.**
|
||||
|
||||
- `messages.length=0` is evidence.
|
||||
- "messages seemed empty" is not evidence — it's a memory of an observation, and memory of observations is where debug sessions go to die.
|
||||
|
||||
If you find yourself about to paraphrase, stop, go back, and copy the raw value.
|
||||
|
||||
---
|
||||
|
||||
## Round completion
|
||||
|
||||
A "round" is complete when every hypothesis has either confirming or refuting evidence — or when you have exhausted the evidence sources available without a decisive result. If the round ends inconclusively, that counts as a failed round for the counter in the journal. See `04-oracle-triple.md` for what to do at 2 consecutive failed rounds.
|
||||
@@ -0,0 +1,136 @@
|
||||
# Phase 4 — Oracle Triple Consultation
|
||||
|
||||
At 2 consecutive failed hypothesis rounds, stop investigating and reframe. Continuing past two failures usually means the real cause is in a category you haven't imagined — and more time on your current mental model is wasted time.
|
||||
|
||||
The Oracle Triple is how you break out of the mental box.
|
||||
|
||||
> ⚠️ **Wrong tool for non-debugging tasks.** The Triple is for *stuck root-cause hunts*. If your task is producing an artifact (extraction, reverse engineering, audit, compliance documentation) and you want a skeptical review before declaring it done, use the **Verification Oracle** pattern in [partial-runtime-evidence.md](partial-runtime-evidence.md#verification-oracle-pattern-for-non-debug-tasks). Running the Triple on a finished extraction returns three diverging "what if you tried…" tangents that are not what you need.
|
||||
|
||||
---
|
||||
|
||||
## When to invoke
|
||||
|
||||
| Situation | Invoke? |
|
||||
|---|---|
|
||||
| 1 round failed, you have new distinguishing evidence | No — run one more round with a refined hypothesis set |
|
||||
| 2 rounds failed, hypotheses now feel like variations of each other | **Yes — invoke now** |
|
||||
| 2 rounds failed, no new evidence angles left to try | **Yes — invoke now** |
|
||||
| You've been investigating >2 hours on the same bug | **Yes — invoke now regardless of round count** |
|
||||
| 1 round failed but the user is watching and wants speed | No — one round isn't enough to justify Oracle cost. Resist the urge. |
|
||||
|
||||
---
|
||||
|
||||
## Why three Oracles, and why *orthogonal* framings
|
||||
|
||||
A single Oracle call returns a single coherent analysis. Coherent analyses tend to inherit the framing of the prompt, which means they inherit the same blind spots the investigator already has. Three Oracles with *orthogonal framings* force the analyses to diverge, and the places where they agree across frames is where the real signal lives.
|
||||
|
||||
The three framings below are chosen to cover distinct bug-cause categories:
|
||||
|
||||
- **A (obvious-but-missed)** — embarrassingly simple causes the investigator walked past.
|
||||
- **B (system-boundary)** — causes living at integration seams, not in the code being read.
|
||||
- **C (invariant-violation)** — assumptions load-bearing to current hypotheses that may themselves be false.
|
||||
|
||||
Spawn all three in parallel.
|
||||
|
||||
---
|
||||
|
||||
## The three prompts
|
||||
|
||||
```
|
||||
task(subagent_type="oracle", load_skills=[], run_in_background=true,
|
||||
prompt="[CONTEXT: bug description + evidence captured so far, verbatim, with file:line refs]
|
||||
|
||||
Framing A — OBVIOUS-BUT-MISSED.
|
||||
What is the most embarrassing, most obvious cause that a senior engineer would spot in 30 seconds and we've overlooked? Consider:
|
||||
- typos, off-by-one
|
||||
- wrong variable name / wrong constant / wrong import
|
||||
- stale cache, wrong file edited, wrong process inspected
|
||||
- attached to the wrong instance of the service
|
||||
- test harness running different code than the app
|
||||
- editing src/ while running dist/
|
||||
|
||||
Give me exactly three candidate causes ranked by likelihood, with one sentence each explaining why our evidence is consistent with each.")
|
||||
|
||||
task(subagent_type="oracle", load_skills=[], run_in_background=true,
|
||||
prompt="[CONTEXT: bug description + evidence captured so far]
|
||||
|
||||
Framing B — SYSTEM-BOUNDARY.
|
||||
What if the bug is NOT in the code we've been reading, but at a boundary? Consider:
|
||||
- third-party SDK behavior that contradicts its docs
|
||||
- middleware that mutates the request or response
|
||||
- a proxy/gateway/load balancer that rewrites headers or bodies
|
||||
- build-time vs runtime env-var resolution
|
||||
- module-load-order issue
|
||||
- shared-library version mismatch (system lib vs bundled lib)
|
||||
- ABI difference (native addons, glibc versions, musl vs glibc)
|
||||
- wrong transport (HTTP/1.1 vs HTTP/2, TLS version negotiation)
|
||||
|
||||
Give me three candidate causes, each naming the specific boundary and the specific contract assumption that might be violated.")
|
||||
|
||||
task(subagent_type="oracle", load_skills=[], run_in_background=true,
|
||||
prompt="[CONTEXT: bug description + evidence captured so far]
|
||||
|
||||
Framing C — INVARIANT-VIOLATION.
|
||||
Which invariants that we've been ASSUMING TRUE might actually be false?
|
||||
Enumerate the five assumptions most load-bearing to our current hypotheses, then for each:
|
||||
- describe the smallest runtime query that would falsify it
|
||||
- predict what the observable would be if the invariant holds vs if it fails
|
||||
|
||||
We want at least one of these queries to be decisive.")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Synthesizing across three Oracles
|
||||
|
||||
**Do not pick the highest-ranked candidate from a single Oracle.** That defeats the purpose of getting three framings.
|
||||
|
||||
Instead, walk the outputs in this order:
|
||||
|
||||
### 1. Agreement scan
|
||||
|
||||
Note which candidate causes appear in at least two Oracles' outputs. Independent agreement across orthogonal framings is strong signal — when the obvious-but-missed framing and the system-boundary framing both land on the same cause, that's usually the bug.
|
||||
|
||||
### 2. Disagreement scan
|
||||
|
||||
Note where Oracles disagree. Disagreement is genuine uncertainty that runtime evidence (not more reasoning) must resolve. Each disagreement becomes a candidate for the next round's distinguishing query.
|
||||
|
||||
### 3. New falsification queries
|
||||
|
||||
Framing C produces concrete "one query that would decide it" suggestions. Pull these verbatim into your new round's evidence-gathering plan — they are designed to be decisive.
|
||||
|
||||
### 4. Build the new hypothesis set
|
||||
|
||||
Minimum 3, same rules as Phase 2. Aim to have hypotheses drawn from the agreement scan (likely cause) AND from the disagreement scan (so one round's evidence resolves the disagreement).
|
||||
|
||||
Record in the journal:
|
||||
|
||||
```markdown
|
||||
## Oracle Triple — Round <N>
|
||||
- Invoked at: <ISO timestamp>
|
||||
- Framing A summary: <top 3 candidates, one line each>
|
||||
- Framing B summary: <top 3 candidates>
|
||||
- Framing C summary: <5 load-bearing assumptions + falsification queries>
|
||||
|
||||
### Cross-framing agreement
|
||||
- <candidate> appeared in A + B
|
||||
- <candidate> appeared in B + C
|
||||
|
||||
### New hypothesis set
|
||||
1. <hypothesis> — evidence to gather: <one-liner>
|
||||
2. ...
|
||||
```
|
||||
|
||||
### 5. Reset the counter
|
||||
|
||||
Reset the "consecutive failed rounds" counter to 0. Return to Phase 3 (parallel investigation) with the new set.
|
||||
|
||||
---
|
||||
|
||||
## If *another* 2 rounds fail after the Oracle Triple
|
||||
|
||||
You are genuinely stuck. This is the escalation threshold.
|
||||
|
||||
Escalate to the user (see `05-escalate.md`) with the full trace: every hypothesis tried, every piece of evidence captured, both Oracle syntheses. Do not guess a fix.
|
||||
|
||||
This is rare — in practice, the Oracle Triple resolves almost all stuck debugging sessions within one round, because it pulls in framings the investigator was too close to the code to see.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Phase 5 — User Decision Escalation
|
||||
|
||||
Escalation is for genuine ambiguity, not for skipping investigation. Most "should I ask the user" moments are really "I don't want to do one more query" moments, and those are wrong.
|
||||
|
||||
---
|
||||
|
||||
## Ask the user ONLY when
|
||||
|
||||
- **Evidence exhausted**, contradictions remain, and further investigation would require a decision with policy implications (e.g. "patch the third-party SDK vs wrap it vs change architecture").
|
||||
- The bug has **multiple valid fixes with different scope/risk tradeoffs** and the user's preference drives the choice.
|
||||
- A proposed fix would **change observable product behavior** for the end user (not just fix the internal bug).
|
||||
- You've **exhausted the Oracle Triple** and another 2 rounds failed after synthesis.
|
||||
|
||||
## Do NOT ask when
|
||||
|
||||
- You haven't tried the Oracle Triple yet.
|
||||
- The question can be answered by one more runtime query.
|
||||
- You're asking for permission to do the obvious thing.
|
||||
- You're asking because you're tired.
|
||||
|
||||
---
|
||||
|
||||
## Escalation format (paste into the reply)
|
||||
|
||||
Keep it short. Evidence-dense. One decision, not a status update.
|
||||
|
||||
```markdown
|
||||
## Decision needed
|
||||
|
||||
**What we know** (verbatim evidence, not paraphrase):
|
||||
- <fact 1 with file:line or address>
|
||||
- <fact 2 with source>
|
||||
- <what the evidence rules IN>
|
||||
- <what the evidence rules OUT>
|
||||
|
||||
**What the decision is** (one sentence):
|
||||
<the fork in the road>
|
||||
|
||||
**Options**:
|
||||
|
||||
| # | Fix | Scope | Risk | Effort |
|
||||
|---|-----|-------|------|--------|
|
||||
| A | <short label> | <files touched / layers> | <regressions possible> | <rough> |
|
||||
| B | ... | ... | ... | ... |
|
||||
| C | ... | ... | ... | ... |
|
||||
|
||||
**Recommendation**: <A/B/C> because <one-sentence reason>.
|
||||
|
||||
Which direction do you want?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns in escalation
|
||||
|
||||
- **Asking without evidence.** "What do you want me to do?" is not an escalation, it's abandonment. Every escalation includes the evidence the user needs to decide.
|
||||
- **Two questions in one.** One decision per escalation. Multi-part questions lead to partial answers and re-escalation.
|
||||
- **Escalating before Phase 4.** If you haven't tried the Oracle Triple, you haven't earned the right to escalate.
|
||||
- **Presenting options you don't actually have.** If option C requires a library the user doesn't use, don't list it. The options are only things you can actually do today.
|
||||
- **Hiding a recommendation.** The user hired you to think — always end with a recommendation, even if you're low-confidence. Say so explicitly: "Recommendation (low confidence): B, because X. If you have context about Y that I don't, it might change to A."
|
||||
|
||||
---
|
||||
|
||||
## What happens after the user responds
|
||||
|
||||
- **User picks an option**: return to Phase 6 (root cause confirmation) with the chosen direction. The user's choice is not itself confirmation — you still need runtime evidence that the cause you're fixing is the cause in play.
|
||||
- **User proposes a different option you hadn't considered**: treat it as new information. Update hypotheses. May trigger another Phase 3 round.
|
||||
- **User gives more context that resolves the disagreement**: skip to Phase 6.
|
||||
- **User is also unsure**: that's a signal you need more evidence, not more opinions. Run one more targeted query before asking again.
|
||||
@@ -0,0 +1,116 @@
|
||||
# Phase 6 + 7 — Root Cause Confirmation & TDD Fix
|
||||
|
||||
A cause is not "confirmed" until you can toggle the bug by toggling the cause. Every other level of evidence is correlation, and correlation-driven fixes ship bugs.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Root Cause Confirmation
|
||||
|
||||
You are allowed to call the cause "confirmed" only when ALL THREE of these hold:
|
||||
|
||||
### 1. Captured runtime value matches the hypothesis exactly
|
||||
|
||||
Not "the value looks consistent with" — the value is exactly the value the hypothesis predicted. If your hypothesis was "baseUrl is api.anthropic.com despite ANTHROPIC_BASE_URL being set to a proxy", the captured value is literally `"https://api.anthropic.com"` in the debugger at the moment of the HTTP call.
|
||||
|
||||
### 2. Reproducible
|
||||
|
||||
Running the repro a second time yields the same observation. Flaky repros mean you haven't isolated the cause; you've isolated a symptom that sometimes appears when the cause does. Keep investigating.
|
||||
|
||||
### 3. Toggle proof (the one most skipped)
|
||||
|
||||
**Changing the value** (via debugger assignment, env override, or a speculative one-line patch) **makes the bug disappear — and reverting brings the bug back**.
|
||||
|
||||
If you can't toggle the bug by toggling the suspected cause, what you have is a correlation, not a mechanism. A correlation is a strong hypothesis, not a confirmed cause.
|
||||
|
||||
Examples of a valid toggle proof:
|
||||
|
||||
| Suspected cause | Toggle |
|
||||
|---|---|
|
||||
| Env var overrides library default, and the override is wrong | Unset the env var → bug goes away. Reset it → bug comes back. |
|
||||
| Async task is not awaited | Add `await` → bug goes away. Remove `await` → bug comes back. |
|
||||
| Third-party SDK uses hardcoded URL | Monkey-patch SDK to use env URL → bug goes away. Unpatch → bug comes back. |
|
||||
| Race condition on shared state | Add a mutex → bug goes away under load. Remove mutex → bug comes back under load. |
|
||||
|
||||
If you can't construct a toggle proof, you haven't confirmed the cause. Run one more round.
|
||||
|
||||
### Update the journal
|
||||
|
||||
```markdown
|
||||
## Root cause (confirmed <ISO timestamp>)
|
||||
- Mechanism: <one paragraph, causal not correlational — the chain from cause to observable symptom>
|
||||
- Evidence: <file:line of captured value | path to saved repro | address + register state>
|
||||
- Toggle proof: "With <change X>, repro produces <good>. Reverting <change X>, repro produces <bad>."
|
||||
- Fix scope: <files and approximate line count>
|
||||
```
|
||||
|
||||
The "mechanism" field is the acid test. If you can't write the causal chain from cause to observable symptom as one paragraph, you don't yet understand the bug well enough to fix it.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — TDD Fix
|
||||
|
||||
Red, green, refactor. No shortcuts.
|
||||
|
||||
### 1. Red — failing-first test
|
||||
|
||||
Write a test that fails *specifically because of this bug*. Requirements:
|
||||
|
||||
- **Test name reads like a bug report.** `test_refinement_turn_returns_empty_content_when_anthropic_returns_401` is good. `test_bug_fix` is not.
|
||||
- **Failure message clearly shows what the bug looks like.** If someone reads only the failure output, they understand what's broken.
|
||||
- **Minimum infrastructure.** Don't spin up the whole server if a unit test against the right seam captures the mechanism.
|
||||
|
||||
Run the test. Confirm it fails. Paste the failure output into the journal:
|
||||
|
||||
```markdown
|
||||
### Red phase (<ISO timestamp>)
|
||||
Test: <path>::<name>
|
||||
Command: <exact invocation>
|
||||
Output:
|
||||
```
|
||||
<verbatim failure output>
|
||||
```
|
||||
Confirms: the bug is reproducible at the test-harness level, not just the manual repro.
|
||||
```
|
||||
|
||||
### 2. Green — minimum change
|
||||
|
||||
Make the test pass with the **smallest change that fully fixes the observed mechanism**.
|
||||
|
||||
If the diff is larger than ~30 lines and you aren't refactoring, something is wrong — either you're fixing more than the bug, or the root cause was deeper than you confirmed. Back to Phase 6.
|
||||
|
||||
Signs you're over-fixing:
|
||||
- Adding "just in case" null checks or try/except around other code
|
||||
- Refactoring adjacent functions because "while I'm here"
|
||||
- Adding new configuration options the bug didn't require
|
||||
- Introducing new abstractions to "make this cleaner"
|
||||
|
||||
Resist all of these. Fix the bug. Note the surrounding issues for follow-up. Move on.
|
||||
|
||||
### 3. Refactor — ONLY AFTER GREEN
|
||||
|
||||
Only cleanup directly related to the fix. Do not re-architect.
|
||||
|
||||
If the code around the fix is rough, note it in the journal as a follow-up for the user; do not expand scope here. Refactoring during a bugfix is how one-line fixes turn into hundred-line diffs nobody can review.
|
||||
|
||||
### 4. Regression — full suite green
|
||||
|
||||
Run the full test suite for the affected package (not just the one new test). Existing tests must still pass.
|
||||
|
||||
If they don't, your "fix" broke something else. Back to Phase 6 with the new failure as evidence — usually it means the mechanism you thought you fixed was load-bearing for some other code path you didn't know about, and the "broken" test is actually pointing at a better understanding of the system.
|
||||
|
||||
### Update the journal
|
||||
|
||||
```markdown
|
||||
### Green phase (<ISO timestamp>)
|
||||
Fix: <file:line> — <two-line description of the change>
|
||||
Test: <path>::<name> now passes
|
||||
Full suite: <N tests, <M failures — should be 0>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The red-green discipline summary
|
||||
|
||||
No red test → no proof the fix addresses the reported bug. Only proof it doesn't break tests that already existed.
|
||||
|
||||
A test written *after* the fix might still pass with the fix reverted. If that's the case, the test doesn't lock the bug — it locks something else. Always verify the test fails without the fix and passes with it. The journal should show both outputs.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Phase 8 — Manual QA by Actually Using It
|
||||
|
||||
Tests cover cases you thought of. Real usage covers the ones you didn't.
|
||||
|
||||
The single fastest way to ship a broken fix is to stop at "tests pass". Manual QA means interacting with the running system the way the user does, then comparing observed behavior to the original bug report.
|
||||
|
||||
---
|
||||
|
||||
## Product-type playbook
|
||||
|
||||
Pick the row that matches the product. Do what it says. Do not substitute.
|
||||
|
||||
| Product type | QA means… |
|
||||
|---|---|
|
||||
| **CLI tool** | Open `tmux`, run the actual command end-to-end, capture output. Paste the session transcript into the journal. Include exit code, stdout, stderr, side-effect check (files created/modified). |
|
||||
| **HTTP API** | Start the real server, hit endpoints with `curl` or `httpie`, inspect response status + body + headers. Hit the specific endpoint that reproduced the bug. If there's auth, use real auth. |
|
||||
| **Browser-served web app** | **Drive a real browser via Playwright CLI.** See [tools/playwright-cli.md](../tools/playwright-cli.md). Navigate the exact page/flow that reproduced the bug. Capture screenshot + DOM + network evidence. **Do not substitute with curl** — browsers have state (cookies, localStorage, service workers, client-side JS, viewport-dependent CSS) that curl does not have. |
|
||||
| **Agent / LLM pipeline** | Run the same user prompt that originally failed. Capture the full turn — tool calls, messages, usage counters. **Confirm non-zero usage** (zero usage = still failing silently, see silent-failure check below). |
|
||||
| **Background worker / job queue** | Trigger the job through the normal entry point (API call, cron tick, message publish), tail the worker logs, observe completion state in the queue or DB. Don't just call the worker function directly — the trigger path matters. |
|
||||
| **MCP server** | Invoke the tool via its actual client (Claude Desktop, Cursor, etc. if available) or `mcp-cli`, not just the HTTP probe endpoint. The MCP handshake itself is sometimes where bugs live. |
|
||||
| **Native binary** | Re-run the exact command that crashed / misbehaved. If the input was a file, use the same file. If the bug was exploitable, confirm the exploit repro via pwntools (see [tools/pwntools.md](../tools/pwntools.md)). Capture exit code, signal if any, core dump if generated. |
|
||||
| **Bundled-app binary** (Bun SEA, Node SEA, Electron, etc.) | Re-run the exact command. If the operation requires paid quota / blocked network, capture the **app's debug log** (`APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log`) which usually emits the assembled request before sending. See [methodology/partial-runtime-evidence.md](partial-runtime-evidence.md) for combining partial signals into a defensible verification. |
|
||||
| **Long-running daemon** | Start fresh, let it run for the amount of time the bug originally took to manifest (not less), capture resource usage (memory, fd, cpu) throughout. Short-running QA misses resource leaks and cumulative state bugs. |
|
||||
|
||||
---
|
||||
|
||||
## Journal format
|
||||
|
||||
Every QA run goes in the journal under "Findings":
|
||||
|
||||
```markdown
|
||||
### Manual QA — <product type> (<ISO timestamp>)
|
||||
- Scenario: <one line describing what you did>
|
||||
- Command: `<exact invocation>`
|
||||
- Observed output:
|
||||
```
|
||||
<verbatim output, trimmed to relevant section>
|
||||
```
|
||||
- Expected output: <what correct behavior looks like>
|
||||
- Fix verified: yes / no / partial — <details>
|
||||
```
|
||||
|
||||
If any QA step shows **partial or regressed behavior**, this is not "mostly done" — it's incomplete. Return to Phase 6.
|
||||
|
||||
---
|
||||
|
||||
## The silent-failure check (always run)
|
||||
|
||||
Regardless of product type, audit the fix against these silent-failure patterns. If the original bug was a silent failure, the same pattern may exist in adjacent code that you haven't tested yet.
|
||||
|
||||
### Universal silent-failure signals
|
||||
|
||||
- HTTP 2xx with empty or default body
|
||||
- Response `ok: true` but a sub-field contains an error token (e.g. `stopReason: "error"`, `status: "failed"`)
|
||||
- `usage.totalTokens === 0` on an LLM response
|
||||
- Process exit code 0 but stderr contains an exception traceback
|
||||
- Panic recovered and logged but ignored
|
||||
- Goroutine / task / promise rejection with no top-level handler
|
||||
- `try { ... } catch { /* swallowed */ }` or `except: pass`
|
||||
- Success response shape but semantic field indicates failure (e.g. `error: null` actually being `error: "..."` with falsy check)
|
||||
- Write returned success but read-back shows stale data
|
||||
- Job marked complete but side-effect did not happen
|
||||
- Cache hit path returned stale data and no refresh was triggered
|
||||
|
||||
### Language-specific silent-failure signals
|
||||
|
||||
Check the runtime reference for additional patterns:
|
||||
|
||||
- [runtimes/python.md](../runtimes/python.md) — asyncio task exceptions, bare `except`, `logging.exception` that goes nowhere
|
||||
- [runtimes/node.md](../runtimes/node.md) — unhandled promise rejections, `void` on async, swallowed `.catch(() => {})`
|
||||
- [runtimes/rust.md](../runtimes/rust.md) — `.unwrap_or_default()`, `let _ = result`, error variants discarded
|
||||
- [runtimes/go.md](../runtimes/go.md) — `if err != nil { return err }` that never reaches user output, recovered panics, buffered channels that block silently
|
||||
- [runtimes/native-binary.md](../runtimes/native-binary.md) — ignored return codes from libc, missing `perror`, `alarm()` / signal masks
|
||||
- [runtimes/bundled-js-binary.md](../runtimes/bundled-js-binary.md) — `process.env.X` baked at build time, dead code from tree-shaking failures, worker sub-bundles diverging from main bundle
|
||||
|
||||
### What to do when you find another silent-failure spot
|
||||
|
||||
Don't fix it. This is out of scope for the current bug.
|
||||
|
||||
Note it in the journal under a "Follow-ups" section with:
|
||||
- File:line
|
||||
- Pattern matched
|
||||
- Proposed fix sketch (one line)
|
||||
- Risk level (what happens if left unfixed)
|
||||
|
||||
Surface these to the user in the final message under "Next steps I didn't take".
|
||||
|
||||
---
|
||||
|
||||
## The "fix verified" bar
|
||||
|
||||
"Fix verified" means: the exact original failing scenario, re-run, now produces the correct output. Not a similar scenario. Not a unit test of the fix. The original scenario.
|
||||
|
||||
If you can't re-run the original scenario (e.g. it required a specific data state that's gone), construct the closest equivalent and document the difference in the journal. Escalate to the user if the equivalent is materially different.
|
||||
@@ -0,0 +1,164 @@
|
||||
# Phase 9 + 10 — Cleanup & Final Verification
|
||||
|
||||
The working tree after the session must differ from before only by the real fix and its test. Anything else is a process failure.
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 — Cleanup & Revert
|
||||
|
||||
### The walk
|
||||
|
||||
Open the journal's "Artifacts to revert" list. Walk it top to bottom. Check each box only after the revert command succeeds and produces no error.
|
||||
|
||||
### Standard revert operations
|
||||
|
||||
Most sessions create some combination of these artifacts. The commands below are the defaults — your journal should have the exact commands for this session.
|
||||
|
||||
```bash
|
||||
# --- Temporary source edits (instrumentation statements, debug prints) ---
|
||||
git checkout <file> # reverts only that file
|
||||
git diff <file> # verify clean
|
||||
|
||||
# --- tmux sessions ---
|
||||
tmux kill-session -t <session-name>
|
||||
tmux ls # confirm gone
|
||||
|
||||
# --- Temp fixtures / scratch scripts ---
|
||||
rm -f /tmp/debug-*.*
|
||||
ls /tmp/debug-*.* 2>/dev/null # confirm gone (ls returns non-zero when no match)
|
||||
|
||||
# --- Background processes (debugger-attached runtimes) ---
|
||||
pkill -f 'node --inspect' || true
|
||||
pkill -f 'python -m pdb' || true
|
||||
pkill -f 'debugpy' || true
|
||||
pkill -f 'dlv' || true
|
||||
pkill -f 'gdb' || true
|
||||
pkill -f 'lldb' || true
|
||||
|
||||
# --- Debug-relevant ports confirmed free ---
|
||||
lsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null # Node inspector default
|
||||
lsof -iTCP:5678 -sTCP:LISTEN -nP 2>/dev/null # debugpy default
|
||||
lsof -iTCP:2345 -sTCP:LISTEN -nP 2>/dev/null # dlv default
|
||||
lsof -iTCP:9999 -sTCP:LISTEN -nP 2>/dev/null # pwndbg/gdb-server default
|
||||
|
||||
# --- Env var overrides in current shell ---
|
||||
unset DEBUG_OVERRIDE_FOO
|
||||
unset PYTHONBREAKPOINT
|
||||
unset RUST_LOG
|
||||
unset DEBUG
|
||||
|
||||
# --- Ghidra scratch projects (if created just for this session) ---
|
||||
# rm -rf ~/ghidra-projects/debug-scratch
|
||||
|
||||
# --- Core dumps from debugging (if any) ---
|
||||
rm -f ./core ./core.* ~/core.*
|
||||
|
||||
# --- Playwright trace files ---
|
||||
rm -rf playwright-report/ test-results/
|
||||
```
|
||||
|
||||
### The verify command
|
||||
|
||||
This is the single most important check of the whole skill:
|
||||
|
||||
```bash
|
||||
git status
|
||||
git diff --stat
|
||||
```
|
||||
|
||||
The diff must contain **only**:
|
||||
|
||||
1. The real fix.
|
||||
2. The new failing-first test.
|
||||
3. Nothing else.
|
||||
|
||||
### Detector checklist — scan the diff for these
|
||||
|
||||
If `git status` shows any untracked debug file, or `git diff` shows any of the patterns below, **you are not done**. Clean it.
|
||||
|
||||
| Pattern | Usually means |
|
||||
|---|---|
|
||||
| `debugger;` | Node debug statement left behind |
|
||||
| `breakpoint()` | Python debug statement left behind |
|
||||
| `dbg!(...)` | Rust debug macro left behind |
|
||||
| `fmt.Println("DEBUG: ...")` | Go ad-hoc print |
|
||||
| `console.log("[DEBUG]` | Node ad-hoc log |
|
||||
| `print(f"DEBUG: ` | Python ad-hoc print |
|
||||
| `// TODO DEBUG`, `// HACK`, `// XXX` | Stale debug marker |
|
||||
| `// <PROJECT>-DEBUG` | Session-specific marker from this skill's edits |
|
||||
| Commented-out code blocks near the fix | Dead code from trial fixes |
|
||||
| Reordered imports or formatting in unrelated files | Drift from your editor's autoformat during the session |
|
||||
|
||||
### Remove the journal
|
||||
|
||||
Only once the git check is clean:
|
||||
|
||||
```bash
|
||||
rm .debug-journal.md
|
||||
sed -i.bak '/^\.debug-journal\.md$/d' .git/info/exclude && rm -f .git/info/exclude.bak
|
||||
```
|
||||
|
||||
The journal is not part of the fix; it doesn't belong in the commit or in the git exclude list.
|
||||
|
||||
---
|
||||
|
||||
## Phase 10 — Final Verification
|
||||
|
||||
Last gate before reporting done. All four gates must be true, and all four must have **evidence in your final message** to the user. Passing a gate without evidence is the same as failing it.
|
||||
|
||||
### The four gates
|
||||
|
||||
1. **Red→green toggle confirmed** — show the failing test output from before the fix and passing output after. Both outputs visible in the reply or the journal.
|
||||
|
||||
2. **Full test suite green** — show the suite's final pass line (e.g. `42 passed in 3.14s`). Not just the new test.
|
||||
|
||||
3. **Manual QA reproduced the fix** — show the command or scenario that originally failed and its now-correct output. Verbatim, not paraphrased.
|
||||
|
||||
4. **Working tree clean of debug artifacts** — show `git diff --stat` output containing only fix + test, plus `git status` clean of untracked debug files.
|
||||
|
||||
If any of the four lacks evidence, you have not finished — return to the appropriate phase.
|
||||
|
||||
### Final message template
|
||||
|
||||
Keep it short. Evidence-dense. The user should be able to skim it in 30 seconds.
|
||||
|
||||
```markdown
|
||||
Fixed.
|
||||
|
||||
**Root cause**: <one sentence — the mechanism, not the symptom>
|
||||
**Fix**: `<file:line>` — <two words>
|
||||
**Test**: `<test file>::<test name>` — red without fix, green with fix
|
||||
**QA**: <one line describing what you ran and what you saw>
|
||||
|
||||
Diff:
|
||||
```
|
||||
<git diff --stat output — should be tiny>
|
||||
```
|
||||
|
||||
**Next steps I didn't take** (awaiting your decision):
|
||||
- <follow-up 1, if any — from QA silent-failure scan or refactor opportunities noted during Phase 7>
|
||||
- <follow-up 2 — or "none" if nothing else surfaced>
|
||||
```
|
||||
|
||||
### Example (from a real session)
|
||||
|
||||
```markdown
|
||||
Fixed.
|
||||
|
||||
**Root cause**: pi-mono Agent's `model.baseUrl` was hardcoded to `api.anthropic.com`, so the `ANTHROPIC_BASE_URL` env var was silently ignored. The proxy API key was rejected by the real Anthropic API with 401, but pi-mono packaged the error into the assistant message's `errorMessage` field instead of throwing, so the route's try/catch never fired and the client received HTTP 200 with empty content.
|
||||
|
||||
**Fix**: `core/pi-bridge/modelResolver.ts:117` — override baseUrl
|
||||
**Test**: `__tests__/core/modelResolver.test.ts::resolves_env_override` — red without fix, green with fix
|
||||
**QA**: `curl -X POST /api/refinement/chat` with proxy env set, observed non-zero usage and non-empty content
|
||||
|
||||
Diff:
|
||||
```
|
||||
core/pi-bridge/modelResolver.ts | 3 +++
|
||||
__tests__/core/modelResolver.test.ts | 42 ++++++++++++++++++++++
|
||||
2 files changed, 45 insertions(+)
|
||||
```
|
||||
|
||||
**Next steps I didn't take** (awaiting your decision):
|
||||
- pi-mono itself silently swallows LLM errors into `errorMessage`; adding a throw-on-error wrapper at our orchestrator layer would surface these upstream
|
||||
- Same silent-failure pattern exists in the planning route — likely the same fix applies
|
||||
```
|
||||
+229
@@ -0,0 +1,229 @@
|
||||
# Partial Runtime Evidence — When You Cannot Execute the Real Operation
|
||||
|
||||
Read this when **runtime truth beats code reading** is in conflict with **you cannot run the actual operation**.
|
||||
|
||||
The skill's first invariant is "runtime state is the only source of truth." But sometimes the only state you can produce is a *partial* observation — the real call requires paid credits, a hardware device you don't have, network access through a corporate proxy, a production secret, or a customer dataset.
|
||||
|
||||
**Partial runtime evidence is still runtime evidence.** This reference tells you which partial signals to harvest and how to combine them so the conclusion is defensible.
|
||||
|
||||
---
|
||||
|
||||
## When this applies
|
||||
|
||||
Use this reference when ALL are true:
|
||||
|
||||
1. The bug or extraction question requires runtime confirmation (per skill invariant #1).
|
||||
2. You attempted the obvious "just run it" path and it failed for reasons unrelated to the bug:
|
||||
- 401/402/403 from a paid API
|
||||
- "device not found" / "permission denied" / SIP block
|
||||
- Production-only credentials
|
||||
- Network isolation (air-gapped, behind VPN you don't have)
|
||||
- Time-of-day or quota limits
|
||||
3. **Mocking the entire system** would defeat the verification — you specifically need evidence about how the *real* code behaves, not a stub.
|
||||
|
||||
If only #1 and #2 are true and you can mock cleanly, just mock and proceed. This file is for cases where mocking would invalidate the answer.
|
||||
|
||||
---
|
||||
|
||||
## The hierarchy of partial evidence (strongest first)
|
||||
|
||||
When you cannot capture the full outbound payload + full response, capture as much as possible from this list. **Evidence further down the list has more inference; evidence higher up is closer to ground truth.**
|
||||
|
||||
### Tier 1 — Pre-send / post-receive logs (best partial evidence)
|
||||
|
||||
The system you're investigating builds a request, then sends it. If the build step logs the assembled request **before** transmission, that log is ground truth for everything except the wire-level bytes (TLS, headers added by HTTP library, etc.).
|
||||
|
||||
```bash
|
||||
# Maximize debug logging
|
||||
APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target -x "minimal valid input" 2>&1 | head -200
|
||||
```
|
||||
|
||||
Look for log lines like:
|
||||
- `Building request: model=X, params={...}`
|
||||
- `[provider] payload: {...}`
|
||||
- `Sending to <url>: <serialized body>`
|
||||
|
||||
**Strength**: 95% of ground truth. Missing only wire-level transformations.
|
||||
|
||||
### Tier 2 — Local interception via proxy / shim
|
||||
|
||||
Run the real binary against a local proxy that records and (optionally) returns a canned response.
|
||||
|
||||
```bash
|
||||
# mitmproxy approach
|
||||
mitmproxy --listen-host 127.0.0.1 --listen-port 8888 --mode regular &
|
||||
HTTPS_PROXY=http://127.0.0.1:8888 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem ./target ...
|
||||
# Now mitmproxy logs the actual TLS-decrypted request
|
||||
```
|
||||
|
||||
```bash
|
||||
# DYLD_INSERT_LIBRARIES / LD_PRELOAD shim approach
|
||||
# Wrap the network call to log payload, return a fake 200
|
||||
# See pwntools.md for shim examples
|
||||
```
|
||||
|
||||
**Strength**: Wire-level ground truth, but requires the target to honor your proxy / preload.
|
||||
|
||||
### Tier 3 — Static extraction × runtime fingerprint cross-check
|
||||
|
||||
When you cannot send a request at all, you can still cross-check static analysis with whatever the binary does that *doesn't* require the real call:
|
||||
|
||||
- The binary builds the request — even if sending fails, the build step ran. Trace it (Tier 1).
|
||||
- The binary writes a state file or cache — read it.
|
||||
- The binary emits version-specific User-Agent strings; verify they match your static extraction.
|
||||
- The binary's `--help` or `--version` output reveals build metadata; verify model lists / feature flags.
|
||||
|
||||
**Strength**: Disjoint evidence sources confirming the same fact. Two independent partial signals that agree are nearly as strong as one full observation.
|
||||
|
||||
### Tier 4 — Contrastive runtime under different inputs
|
||||
|
||||
If you can run with input variant A but not B, run A and reason about B from code:
|
||||
|
||||
```bash
|
||||
# A: minimal trial input — works for free tier
|
||||
./target --action=read --resource=local-file
|
||||
# B: full inference call — paid tier required, blocked
|
||||
# But the request-building code is shared between A and B!
|
||||
# Capture A's logs, then inspect the code path for B and verify only the model/endpoint diff.
|
||||
```
|
||||
|
||||
**Strength**: Confirms shared code paths; remaining gap is only the difference between A and B.
|
||||
|
||||
### Tier 5 — Vendor-published API logs / dashboard
|
||||
|
||||
If the operation succeeded earlier (before quota ran out, before access was revoked), the vendor's dashboard / audit log may show the request. Lower fidelity but still observed behavior.
|
||||
|
||||
**Strength**: Real wire data, but often summarized — token counts, status codes, no payload bodies.
|
||||
|
||||
### Tier 6 — Pure code reading with peer review
|
||||
|
||||
If literally none of the above is available, read the code carefully and submit it to **one Oracle for skeptical review** (see "Verification Oracle" below). This is the weakest tier and you must explicitly mark conclusions as "unverified" in the journal.
|
||||
|
||||
---
|
||||
|
||||
## How to combine partial signals
|
||||
|
||||
A defensible conclusion **prefers two independent signals from different tiers**, with one exception: a complete Tier 2 wire-level capture is wire-level ground truth and can stand alone for request-shape claims (because the wire bytes are exactly what the remote received). For *behavioral* claims (what the system does next, what state it stores, what side effects it produces), still combine with another signal.
|
||||
|
||||
| Available evidence | Defensibility |
|
||||
|---|---|
|
||||
| Tier 1 + Tier 1 (same log, different lines) | weak — single source |
|
||||
| Tier 1 + Tier 2 (debug log + proxy capture) | **strong** — independent confirmation |
|
||||
| Tier 1 + Tier 3 (debug log + version output cross-check) | **strong** — disjoint sources |
|
||||
| Tier 2 alone (full proxy capture) | strong **for request-shape claims only** — stands alone for "what bytes were sent". Add a second signal for response-handling or state claims. |
|
||||
| Tier 3 + Tier 4 (cross-check + contrastive run) | medium — both partial |
|
||||
| Tier 6 alone (code reading only) | **insufficient** — escalate or mark unverified |
|
||||
|
||||
Record in the journal:
|
||||
|
||||
```markdown
|
||||
## Partial runtime evidence
|
||||
### Question being verified
|
||||
<the specific claim, e.g. "Opus 4.7 default effort is 'high'">
|
||||
|
||||
### Available signals
|
||||
- Tier 1: debug log /tmp/trace.log line 47-49 shows `effort: "high"` ✓
|
||||
- Tier 3: static extraction of m5T() function returns "high" for smart mode ✓
|
||||
- Tier 6: code path verified by reading prompt-builder.js ✓
|
||||
|
||||
### Independence assessment
|
||||
Tier 1 and Tier 3 are independent — the log was emitted by a different
|
||||
code path than m5T() and would diverge if the static reading were wrong.
|
||||
|
||||
### Conclusion
|
||||
VERIFIED via Tier 1 + Tier 3 agreement. No need to escalate.
|
||||
```
|
||||
|
||||
If you cannot achieve a complete Tier 2 capture **or** two independent non-Tier-6 signals from the table above, **write an explicit note in the deliverable**:
|
||||
|
||||
> ⚠️ Partial-evidence finding. The full outbound payload could not be captured because [reason]. The conclusion rests on:
|
||||
> - [signal A — tier and source]
|
||||
> - [signal B — tier and source]
|
||||
> A future verification should attempt [the missing tier] when [condition].
|
||||
|
||||
---
|
||||
|
||||
## Verification Oracle pattern (for non-debug tasks)
|
||||
|
||||
The skill's main Oracle Triple (`04-oracle-triple.md`) is for **stuck debugging** — 2 failed rounds, mental box, three orthogonal framings to break out.
|
||||
|
||||
For tasks where the deliverable is an **artifact, not a bug fix** (reverse engineering, extraction, audit, compliance documentation), use a different pattern: **single Oracle, late, skeptical, with the deliverable in hand**.
|
||||
|
||||
### When to invoke
|
||||
|
||||
- Right before declaring an extraction/audit task "done"
|
||||
- After every significant revision of the deliverable (not after every small edit)
|
||||
- Maximum 3-4 iterations before escalating to user
|
||||
|
||||
### Pattern
|
||||
|
||||
```
|
||||
task(subagent_type="oracle", load_skills=[], run_in_background=false,
|
||||
prompt="""
|
||||
SKEPTICAL FINAL VERIFICATION — be critical, look for reasons the task is incomplete or wrong.
|
||||
|
||||
## Original task
|
||||
<verbatim user request>
|
||||
|
||||
## What I produced
|
||||
<list of artifacts with paths and brief descriptions>
|
||||
|
||||
## Specific claims to verify
|
||||
<bullet list of every concrete claim in the deliverable>
|
||||
|
||||
## Where to look
|
||||
<paths the Oracle should Read / Bash to verify>
|
||||
|
||||
## Your job
|
||||
1. Read the deliverables.
|
||||
2. Spot-check each claim against the source/evidence the deliverable cites.
|
||||
3. Identify any unsubstantiated claims, missing pieces, or factual errors.
|
||||
4. End with PASS / FAIL / PARTIAL with specific gaps.
|
||||
Be skeptical. Don't rubber-stamp.
|
||||
""")
|
||||
```
|
||||
|
||||
### Why this differs from the Oracle Triple
|
||||
|
||||
| | Oracle Triple (debug) | Verification Oracle (artifact) |
|
||||
|---|---|---|
|
||||
| Trigger | 2 failed hypothesis rounds | About to declare "done" |
|
||||
| Count | 3 in parallel, orthogonal framings | 1 sequential, focused review |
|
||||
| Goal | Break out of mental box | Catch unsubstantiated claims |
|
||||
| Tone of prompt | Brainstorm wide alternatives | Skeptical audit |
|
||||
| Iteration | Reset hypothesis set after | Fix gaps, re-invoke until PASS |
|
||||
|
||||
### Don't conflate them
|
||||
|
||||
If you're stuck debugging, do the Triple. If you have a deliverable and need it audited, do the Verification Oracle. Doing the Triple on a finished extraction will return three diverging "what if you tried…" tangents that are not what you need. Doing the Verification Oracle on a stuck debugging session will return a polite "the evidence is incomplete" that you already knew.
|
||||
|
||||
---
|
||||
|
||||
## Common partial-evidence anti-patterns
|
||||
|
||||
| Anti-pattern | Why it fails | Replacement |
|
||||
|---|---|---|
|
||||
| "It looks right in the code, so it works" | Tier 6 alone, unverified | Add at least one Tier 1-3 signal |
|
||||
| "I ran it once, didn't error, so it's correct" | Absence of error ≠ presence of correctness | Capture the actual output and verify content |
|
||||
| "The mock returns the value I wrote, so the code is fine" | Tautology — mock loops back your assumption | Use Tier 2 (proxy) instead, or cross-check with Tier 3 |
|
||||
| "The vendor's dashboard shows my call worked" | Dashboard often only shows status code, not behavior | Combine with Tier 1 if available |
|
||||
| "I'll trust the most-recent stack overflow answer" | Code from a different version / context | Verify against the actual binary you have |
|
||||
|
||||
---
|
||||
|
||||
## Cleanup additions for partial-evidence work
|
||||
|
||||
```bash
|
||||
# Proxy artifacts
|
||||
pkill -f mitmproxy 2>/dev/null
|
||||
rm -f ~/.mitmproxy/cache_* 2>/dev/null
|
||||
|
||||
# Debug log files
|
||||
rm -f /tmp/trace.log /tmp/*-debug-trace.log
|
||||
|
||||
# DYLD_INSERT / LD_PRELOAD shim libraries
|
||||
rm -f /tmp/*.dylib /tmp/*.so
|
||||
|
||||
# Verify env vars set in your shell are not persisted
|
||||
unset HTTPS_PROXY APP_DEBUG APP_LOG_LEVEL APP_LOG_FILE 2>/dev/null
|
||||
```
|
||||
@@ -0,0 +1,415 @@
|
||||
# Bundled-JS / Embedded-Source Binaries (Bun SEA, Node SEA, Deno compile, pkg, Electron, PyInstaller)
|
||||
|
||||
A growing class of "binaries" are not stripped C/C++ at all — they are a runtime VM glued onto a high-level-language bundle. The bundle is **plaintext or trivially-decodable** inside the binary.
|
||||
|
||||
If you reach for `native-binary.md` workflow on these (Ghidra → pwndbg → hex), you will waste hours decompiling a runtime you don't care about while the actual logic sits exposed three megabytes away.
|
||||
|
||||
**This reference exists because the workflow is fundamentally different from stripped C.**
|
||||
|
||||
---
|
||||
|
||||
## When to use this reference instead of `native-binary.md`
|
||||
|
||||
Open this if `file ./target` shows a generic Mach-O / ELF / PE BUT any of:
|
||||
|
||||
- Size is suspiciously large (50 MB+ for a "simple CLI")
|
||||
- `strings -n 8 ./target | rg -i "node_modules|webpack|esbuild|bun|pkg/lib|electron|pyinstaller"` returns hits
|
||||
- The binary's CLI flags include things like `--inspect`, `--unhandled-rejections`, npm-style help text
|
||||
- Vendor docs say it's built with Bun / pkg / nexe / Deno compile / PyInstaller / Electron / Tauri (UI shell)
|
||||
- `head -c 4 ./target | xxd` shows a known runtime magic for an embedded archive section
|
||||
|
||||
If yes → **stop following `native-binary.md` and follow this**. Triage and dynamic tracing are the same. Static analysis is completely different.
|
||||
|
||||
---
|
||||
|
||||
## The workflow
|
||||
|
||||
```
|
||||
[1] Triage → identify the bundler (Bun? pkg? Deno? Electron? PyInstaller?)
|
||||
[2] Locate the bundle → find where the embedded source archive starts
|
||||
[3] Extract → dump source to disk so you can grep / read it
|
||||
[4] Source-level static analysis (rg + Read, NOT Ghidra)
|
||||
[5] Runtime verification → debug logs, --inspect, partial-evidence patterns
|
||||
[6] Fix / report
|
||||
```
|
||||
|
||||
Step 3 is the unlock — once you have plaintext source on disk, the rest is normal codebase exploration.
|
||||
|
||||
---
|
||||
|
||||
## [1] Identify the bundler — 30-second fingerprint
|
||||
|
||||
```bash
|
||||
# Look for runtime-specific markers in plaintext strings
|
||||
strings -n 12 ./target 2>/dev/null | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|NODE_SEA_BLOB|tauri|ESZIP_V2|denort' | head -20
|
||||
```
|
||||
|
||||
| Marker pattern | Bundler | Source format |
|
||||
|---|---|---|
|
||||
| `@oven/bun-darwin`, `bun-lockfile-format-v`, `// @bun` | **Bun SEA** (compiled via `bun build --compile`) | Plaintext JS, single big bundle |
|
||||
| `NODE_SEA_BLOB` + `NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1` | **Node SEA** (`node --build-sea` or `--experimental-sea-config`) | Plaintext JS, or V8 code cache (when `useCodeCache: true`), or startup snapshot (when `useSnapshot: true`) — the latter two are NOT plaintext |
|
||||
| `pkg/lib/bootstrap.js`, `pkg/prelude`, `PAYLOAD_POSITION` | **pkg** (vercel/pkg) | Plaintext or v8 cached data |
|
||||
| `ESZIP_V2`, `denort`, `deno_runtime` | **Deno compile** (`deno compile`) | TS/JS in eszip archive — readable but needs `eszip` crate to walk; not pure plaintext |
|
||||
| `Electron`, `app.asar`, `chrome.dll`, `Squirrel.Mac` | **Electron** | `app.asar` archive (TAR-like with JSON header). Source is plaintext JS once extracted |
|
||||
| `PyInstaller`, `pyz`, `_MEIPASS`, `pyi-os-utils` | **PyInstaller** | Compressed `.pyc` bytecode — needs `pyinstxtractor` + `decompyle3` to recover Python source |
|
||||
| `nexe-`, `nexe_compile`, `:::nexe::` | **nexe** | Plaintext JS appended to node binary |
|
||||
| `Tauri`, `tao`, `wry`, `tauri::generate_context` | **Tauri** (Rust shell + JS UI) | **Two worlds**: JS frontend in resource section is extractable here; Rust commands / core logic are native and require [native-binary.md](native-binary.md) |
|
||||
|
||||
If multiple match (e.g. Tauri + Bun): the outer shell is the first one (Tauri/Electron). The inner JS is the second one's format. **For Tauri specifically, expect to use both this reference (for the UI bundle) and `native-binary.md` (for the Rust binary side).**
|
||||
|
||||
> **Source-format reality check**: only Bun SEA, pkg (when not using `--public-packages`), nexe, and Electron `.asar` are reliably plaintext. Node SEA with code-cache or snapshot, PyInstaller `.pyc`, and Deno eszip require additional tooling. Don't assume `strings` will find readable code — verify the bundler first.
|
||||
|
||||
---
|
||||
|
||||
## [2] Locate the bundle
|
||||
|
||||
### Bun SEA — JS is just embedded plaintext
|
||||
|
||||
The JS source is concatenated into the binary as a giant template literal / string. No decoding needed.
|
||||
|
||||
```bash
|
||||
# Verify by searching for typical JS bundle markers
|
||||
strings -n 8 ./target | rg "function|var |let |const |async function" | head -5
|
||||
|
||||
# Find where the bundle starts (look for "use strict" or banner comment)
|
||||
LC_ALL=C grep -aob '"use strict"' ./target | head -5
|
||||
LC_ALL=C grep -aob '#!/usr/bin/env bun' ./target | head -5
|
||||
```
|
||||
|
||||
### Node SEA — `NODE_SEA_BLOB` resource/segment + activated fuse
|
||||
|
||||
Per the [Node.js SEA docs](https://nodejs.org/api/single-executable-applications.html), a Node-built SEA contains:
|
||||
- A resource (PE), section in `NODE_SEA` segment (Mach-O), or note (ELF) named `NODE_SEA_BLOB`
|
||||
- The fuse string `NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1` (with trailing `:1` indicating injected; `:0` means a copy of the node binary that has not yet had a blob injected)
|
||||
|
||||
```bash
|
||||
# Confirm it is a SEA at all
|
||||
LC_ALL=C grep -aob 'NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2:1' ./target | head -1
|
||||
|
||||
# Find the blob resource/section
|
||||
LC_ALL=C grep -aob 'NODE_SEA_BLOB' ./target | head
|
||||
|
||||
# On Mach-O, inspect the segment directly
|
||||
otool -l ./target | grep -A4 'NODE_SEA'
|
||||
```
|
||||
|
||||
The blob format is documented but non-trivial to walk by hand. For extraction, **use postject in reverse** (carve the section bytes) or read the blob via `node:sea` API from inside a debug build of the same binary. Plain `strings` will get you the embedded JS only when the SEA was built without `useCodeCache` and without `useSnapshot` — both of those replace plaintext with V8 cache data or startup snapshot bytes.
|
||||
|
||||
`node --build-sea sea-config.json` and `node --experimental-sea-config sea-config.json` *generate* SEA blobs; neither inspects an existing executable.
|
||||
|
||||
### Deno compile — eszip archive section
|
||||
|
||||
```bash
|
||||
# Deno-compile binaries embed an eszip v2 archive
|
||||
LC_ALL=C grep -aob 'ESZIP_V2' ./target | head -3
|
||||
# Also confirm the runtime
|
||||
LC_ALL=C grep -aob 'denort' ./target | head -1
|
||||
```
|
||||
|
||||
To extract, use the `eszip` Rust crate (or the `@deno/eszip` JS port) to parse the archive after carving it out at the offset above. There is no stable Deno CLI flag that inspects compiled-executable eszip contents as of 2026-04 — `deno info` only works on source files.
|
||||
|
||||
### pkg — `PAYLOAD_POSITION` marker
|
||||
|
||||
```bash
|
||||
LC_ALL=C grep -aob 'PAYLOAD_POSITION' ./target | head
|
||||
LC_ALL=C grep -aob 'pkg/prelude' ./target | head
|
||||
```
|
||||
|
||||
For source extraction, use the `pkg-extract` tooling community projects or carve based on the offset reported by the `PAYLOAD_POSITION:<n>` value.
|
||||
|
||||
### Electron — `app.asar` is usually a separate file
|
||||
|
||||
Most Electron apps ship `app.asar` next to the binary, not embedded inside. Extract it with the official tool:
|
||||
|
||||
```bash
|
||||
# macOS layout
|
||||
ls -la /Applications/MyApp.app/Contents/Resources/app.asar
|
||||
npx @electron/asar extract app.asar ./extracted/
|
||||
# or older:
|
||||
npx asar extract app.asar ./extracted/
|
||||
```
|
||||
|
||||
For single-file builds where the asar is embedded inside the executable, **do not pattern-match arbitrary 4-byte sequences** (the asar format starts with a Pickle-encoded uint32 header size + JSON metadata, and the same bytes appear elsewhere in any binary). Instead, use a Pickle-aware extractor that validates the JSON header before claiming a match — the `asar` npm package's programmatic `extractAll()` API does this. Carve the asar bytes by scanning for a candidate Pickle header (4-byte size + 4-byte payload size + `{"files":` prefix), validate the JSON parses, then feed the carved buffer to `extractAll()`.
|
||||
|
||||
### PyInstaller — use `pyinstxtractor`, NOT runtime self-extraction
|
||||
|
||||
```bash
|
||||
# Recover the embedded archive without running the binary
|
||||
python3 pyinstxtractor.py ./target
|
||||
# Output: ./target_extracted/ with .pyc files
|
||||
|
||||
# Decompile the .pyc files back to Python source
|
||||
decompyle3 ./target_extracted/main.pyc # Python 3.7+
|
||||
uncompyle6 ./target_extracted/main.pyc # older Python
|
||||
```
|
||||
|
||||
If `pyinstxtractor` cannot read the archive (e.g. non-standard PyInstaller version), use the official `pyi-archive_viewer` tool that ships with PyInstaller. Avoid the "run-the-binary-and-snoop-`/tmp/_MEI*`" approach: it only catches what runs in the time window between `_MEIPASS` extraction and cleanup, and it executes potentially untrusted code.
|
||||
|
||||
---
|
||||
|
||||
## [3] Extract source to disk — DO NOT skip this
|
||||
|
||||
**The single biggest mistake** with bundled-JS reverse engineering is trying to read the source out of `strings` output or `xxd` dumps. You will lose data. See "Gotchas" below.
|
||||
|
||||
### For Bun SEA / nexe / single-string-blob bundlers
|
||||
|
||||
Read the binary as bytes, find the JS section, save to a `.js` file:
|
||||
|
||||
```python
|
||||
# extract_bundled_js.py
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
raise SystemExit("usage: extract_bundled_js.py <target>")
|
||||
|
||||
with open(sys.argv[1], 'rb') as f:
|
||||
data = f.read()
|
||||
|
||||
markers = [b'// @bun', b'"use strict"', b"'use strict'", b'#!/usr/bin/env']
|
||||
start = -1
|
||||
for m in markers:
|
||||
p = data.find(m)
|
||||
if p != -1 and (start == -1 or p < start):
|
||||
start = p
|
||||
|
||||
if start == -1:
|
||||
raise SystemExit(
|
||||
"no bundle marker found — binary may not be Bun/nexe, "
|
||||
"or markers were stripped. Try strings(1) for hints."
|
||||
)
|
||||
|
||||
# Heuristic end: look for a long null run AFTER start.
|
||||
# This is a heuristic, NOT a guarantee. Verify the tail of the output
|
||||
# looks like JS (closing braces, EOF) before trusting it.
|
||||
end = data.find(b'\x00' * 1024, start)
|
||||
if end == -1:
|
||||
end = len(data)
|
||||
|
||||
bundle = data[start:end]
|
||||
print(f'Extracted {len(bundle)} bytes from offset {start} to {end}', file=sys.stderr)
|
||||
sys.stdout.buffer.write(bundle)
|
||||
```
|
||||
|
||||
```bash
|
||||
python3 extract_bundled_js.py ./target > extracted-bundle.js
|
||||
wc -c extracted-bundle.js
|
||||
# Sanity check the tail is JS, not random binary
|
||||
tail -c 200 extracted-bundle.js
|
||||
```
|
||||
|
||||
### For PyInstaller
|
||||
|
||||
Use `pyinstxtractor` then `uncompyle6` / `decompyle3` on the `.pyc` files.
|
||||
|
||||
### For Electron .asar
|
||||
|
||||
```bash
|
||||
npx asar extract app.asar ./extracted/
|
||||
# Now ./extracted/ has a normal node_modules + your source layout
|
||||
```
|
||||
|
||||
### For Deno compile
|
||||
|
||||
Use the `eszip` Rust crate or the `@deno/eszip` JS port to walk the archive after carving the eszip section out at the offset reported by the `ESZIP_V2` magic search. There is no stable Deno CLI as of 2026-04 that inspects compiled-binary eszip contents directly.
|
||||
|
||||
---
|
||||
|
||||
## [4] Source-level static analysis — `rg` + `Read`, not Ghidra
|
||||
|
||||
Once you have the source on disk, treat it as a normal codebase:
|
||||
|
||||
```bash
|
||||
# Find function definitions
|
||||
rg -n "^function |^const \w+ = (function|\(.*\) =>)" extracted-bundle.js | head
|
||||
|
||||
# Find specific behavior
|
||||
rg -n "claude-opus-4-7|reasoning_effort|api_key" extracted-bundle.js
|
||||
|
||||
# Resolve minified identifiers — they show up as `var XYZ="value"`
|
||||
rg -aoP 'var \w+="[^"]+"' extracted-bundle.js | head -50
|
||||
```
|
||||
|
||||
For minified bundles, use a template-literal-aware parser to extract specific functions or template strings. Example skeleton:
|
||||
|
||||
```python
|
||||
def find_template_end(data, start):
|
||||
"""Walk a JS template literal preserving ${...} interpolation depth.
|
||||
Returns position of closing backtick."""
|
||||
i = start
|
||||
while i < len(data):
|
||||
c = data[i:i+1]
|
||||
if c == b'\\':
|
||||
i += 2; continue
|
||||
if c == b'$' and data[i+1:i+2] == b'{':
|
||||
depth = 1; i += 2
|
||||
while i < len(data) and depth > 0:
|
||||
cc = data[i:i+1]
|
||||
if cc == b'\\': i += 2; continue
|
||||
if cc == b'`':
|
||||
j = find_template_end(data, i+1)
|
||||
i = j + 1; continue
|
||||
if cc == b'{': depth += 1
|
||||
elif cc == b'}': depth -= 1
|
||||
elif cc in (b'"', b"'"):
|
||||
q = cc; i += 1
|
||||
while i < len(data) and data[i:i+1] != q:
|
||||
if data[i:i+1] == b'\\': i += 2
|
||||
else: i += 1
|
||||
i += 1; continue
|
||||
i += 1
|
||||
continue
|
||||
if c == b'`': return i
|
||||
i += 1
|
||||
return -1
|
||||
```
|
||||
|
||||
For function-body extraction, **track the parameter list separately** before tracking body braces. The naive approach mis-counts destructuring `function f({a, b, ...c})` as the body `{` and exits early.
|
||||
|
||||
---
|
||||
|
||||
## [5] Runtime verification
|
||||
|
||||
You usually cannot single-step JS inside a Bun-compiled binary the way you would with `node --inspect`. Workarounds:
|
||||
|
||||
### Bun-compiled
|
||||
|
||||
Bun's inspector takes `--inspect[=<host>:<port>[/<prefix>]]` on the command line. For env-var control of compiled binaries, the form is the same minus the leading `--`:
|
||||
|
||||
```bash
|
||||
# Default port (6499) auto-prefix
|
||||
./target --inspect
|
||||
# → ws://localhost:6499/<auto-prefix> (paste into https://debug.bun.sh)
|
||||
|
||||
# Explicit host:port[/prefix]
|
||||
./target --inspect=localhost:9229/dbg
|
||||
# Or via env var (if --inspect cannot be passed)
|
||||
BUN_INSPECT=localhost:9229/dbg ./target
|
||||
```
|
||||
|
||||
**For HTTP request tracing without an interactive debugger** (highest-value Bun-specific runtime evidence):
|
||||
|
||||
```bash
|
||||
# Print every fetch() / node:http request as a curl command + full headers/body
|
||||
BUN_CONFIG_VERBOSE_FETCH=curl ./target ...
|
||||
|
||||
# Or just print the request/response without curl-format
|
||||
BUN_CONFIG_VERBOSE_FETCH=true ./target ...
|
||||
```
|
||||
|
||||
Plus generic env-var-based debug logging if the app supports it:
|
||||
|
||||
```bash
|
||||
APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target
|
||||
```
|
||||
|
||||
### Node SEA / pkg / nexe
|
||||
```bash
|
||||
# These usually accept --inspect since they are real Node
|
||||
./target --inspect
|
||||
# Then chrome://inspect or node --inspect-brk
|
||||
```
|
||||
|
||||
### Electron
|
||||
```bash
|
||||
./target.app/Contents/MacOS/target --inspect=9229 --remote-debugging-port=9223
|
||||
# Renderer process is at chrome://inspect, main process via the inspector port
|
||||
```
|
||||
|
||||
### When you cannot make a real call
|
||||
The target's API may require credentials, network access, or paid quota you don't have. **You are not stuck** — see [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for the fallback patterns.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Gotchas — read these before extracting
|
||||
|
||||
### G1. `strings -n N` silently drops short identifier interpolations
|
||||
|
||||
`strings` outputs runs of printable characters of length **≥ N**. Default is 4 on most systems; many references (including older versions of `native-binary.md`) recommend `-n 8` for less noise.
|
||||
|
||||
**With `-n 8`, short template-literal interpolations like `${x}`, `${i}`, `${R}` are silently dropped** because they are 4 chars surrounded by non-printable bytes (newlines or section padding). The result looks like:
|
||||
|
||||
```text
|
||||
expected: <INSTRUCTIONS>\n${x}\n</INSTRUCTIONS>
|
||||
strings: <INSTRUCTIONS>\n</INSTRUCTIONS> ← ${x} is gone, no warning
|
||||
```
|
||||
|
||||
A consumer reading the strings output would conclude the template is empty.
|
||||
|
||||
**Mitigation**:
|
||||
1. Use `strings` only for **fingerprinting** (Phase 1 triage), never as the source of extracted text.
|
||||
2. For actual extraction, **read the binary as bytes** with `python3 -c "open('./target','rb').read()"` and grep / parse from there.
|
||||
3. If you must use `strings`, try `strings -n 1 -t x ./target` and post-filter — but byte-level reads are still more reliable.
|
||||
|
||||
### G2. Stale cached binary ≠ latest features
|
||||
|
||||
Bundled-app installers often check a remote version and skip download if a cached binary exists. If you reverse-engineered an old version and the user reports behavior you don't see in the source, **re-run the installer** (or fetch the version manifest manually) before assuming the source is current.
|
||||
|
||||
```bash
|
||||
# Example pattern - varies by tool
|
||||
curl -fsSL https://example.com/install.sh | head -50 # find version-fetch URL
|
||||
curl -fsSL https://static.example.com/cli/cli-version.txt
|
||||
./your-tool --version
|
||||
# Compare. If different, re-install.
|
||||
```
|
||||
|
||||
### G3. APFS / NTFS case-insensitivity silently overwrites files
|
||||
|
||||
When extracting many minified function bodies (`cVR`, `CVR`, `dpr`, `DPR`, …) and saving each to its own file, **macOS APFS and Windows NTFS treat `cVR.txt` and `CVR.txt` as the same file**. The second write silently overwrites the first.
|
||||
|
||||
**Mitigation**: prefix filenames with something case-distinguishing, e.g. `mode-cVR.txt`, `mode-CVR.txt`, or use a hash suffix.
|
||||
|
||||
### G4. Bun's runtime adds 30-50 MB of unrelated symbols
|
||||
|
||||
A 70 MB Bun-compiled binary is **mostly Bun runtime** (~50 MB) plus your app (~20 MB). When fingerprinting, you will see thousands of strings like `tree-sitter-typescript`, `react-native-stylex` etc. that the user's actual app doesn't use — these are package names baked into Bun's package-resolution data.
|
||||
|
||||
**Mitigation**: when grepping for "what does this app do?", filter out runtime noise:
|
||||
```bash
|
||||
strings -n 8 ./target | rg -v 'node_modules|@oven/bun|package-lock|tree-sitter|ffmpeg-installer' | head
|
||||
```
|
||||
|
||||
### G5. Source maps usually NOT shipped
|
||||
|
||||
Bundled apps strip source maps for production. Variable names are minified to `T`, `R`, `a`, `r`, etc. Treat the bundle like an obfuscated codebase: identify constants by tracing assignments (`var T="actual-name"`) and resolve interpolations manually.
|
||||
|
||||
### G6. The "extract" file is not legally redistributable
|
||||
|
||||
If reverse-engineering proprietary software, the extracted source is the vendor's IP. Use it for understanding behavior, **never commit it to git**, never post snippets in public issues. Cleanup your `extracted-bundle.js` files in Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns specific to bundled JS
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| Bundle includes unreachable dead code from tree-shaking failures | You read code that never runs — verify with runtime trace |
|
||||
| `process.env.X` resolved at BUILD time, not RUNTIME | Setting the env var at runtime has no effect; the value is baked in |
|
||||
| `import.meta.url` in compiled binary returns `bun://...` not a real path | File-relative resolution silently breaks |
|
||||
| Worker threads spawn from embedded code, look for sub-bundle inside main bundle | Workers may have their own copy of dependencies |
|
||||
| Minified identifiers with case variants used in same module | Easy to confuse `cVR` with `CVR` when reading fast |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics for bundled-JS work
|
||||
|
||||
```bash
|
||||
# Remove extracted bundles — they may contain proprietary source
|
||||
rm -f /tmp/extracted-bundle.js /tmp/extracted-*.js
|
||||
rm -rf /tmp/asar-extracted/
|
||||
rm -rf /tmp/_MEI*
|
||||
|
||||
# Remove strings dumps
|
||||
rm -f /tmp/*-strings.txt /tmp/*-strings-v*.txt
|
||||
|
||||
# Remove Python helper scripts created for parsing
|
||||
rm -f /tmp/extract_bundled_js.py /tmp/parse_template.py
|
||||
|
||||
# Verify the extraction directory is gone (if you used a workspace dir)
|
||||
ls /Users/$USER/local-workspaces/*-extracted/ 2>/dev/null
|
||||
# rm -rf only after journal review confirms nothing important is there
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## When to escalate back to `native-binary.md`
|
||||
|
||||
If extraction reveals the "bundle" is actually compiled to v8 cached data (pkg with `--public-packages` or PyInstaller with bytecode-only mode), and decompilation is non-trivial, **switch back to `native-binary.md` workflow** (Ghidra against the runtime + careful tracing). Bundled-JS workflow only helps when the high-level source is recoverable as readable text.
|
||||
@@ -0,0 +1,252 @@
|
||||
# Go Debugging
|
||||
|
||||
Covers goroutines, `dlv` (Delve), `pprof`, the race detector, and the fact that Go's concurrency model means most bugs are about goroutines doing something quiet and wrong.
|
||||
|
||||
---
|
||||
|
||||
## Environment detection (Phase 0)
|
||||
|
||||
```bash
|
||||
go version
|
||||
cat go.mod | head -5
|
||||
|
||||
# Delve installed?
|
||||
which dlv
|
||||
dlv version
|
||||
|
||||
# Build constraints
|
||||
grep -r '// +build\|//go:build' cmd/ internal/ pkg/ 2>/dev/null | head
|
||||
|
||||
# pprof wired up?
|
||||
grep -r 'net/http/pprof\|runtime/pprof' --include='*.go' | head -3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Delve (`dlv`) — the Go debugger
|
||||
|
||||
Go's gc compiler emits DWARF, but plain gdb barely understands goroutines. **Use dlv, not gdb.** Plain gdb on a Go binary will miss goroutine state and print garbage for interface values.
|
||||
|
||||
### The five `dlv` launch modes
|
||||
|
||||
```bash
|
||||
# Build and launch under debugger (equivalent to `go run` + debug)
|
||||
dlv debug ./cmd/server -- --port=8080
|
||||
|
||||
# Debug a test binary
|
||||
dlv test ./internal/handler/ # enters the test package under debug
|
||||
|
||||
# Debug an existing binary (must be built with -gcflags="all=-N -l" for best results)
|
||||
dlv exec ./bin/myserver
|
||||
|
||||
# Attach to a running process
|
||||
dlv attach $(pgrep myserver)
|
||||
|
||||
# Headless mode (IDE / remote attach) — default port 2345
|
||||
dlv debug --headless --listen=:2345 --api-version=2 ./cmd/server
|
||||
```
|
||||
|
||||
### Building a debuggable binary
|
||||
|
||||
The compiler inlines and optimizes aggressively in normal builds, which makes stepping confusing. For serious debugging:
|
||||
|
||||
```bash
|
||||
go build -gcflags="all=-N -l" -o ./bin/server ./cmd/server
|
||||
# -N disables optimization
|
||||
# -l disables inlining
|
||||
```
|
||||
|
||||
Then `dlv exec ./bin/server`.
|
||||
|
||||
### Essential dlv commands
|
||||
|
||||
```
|
||||
(dlv) b main.main # breakpoint at function
|
||||
(dlv) b handler.go:42 # breakpoint at file:line
|
||||
(dlv) b pkg/foo.Bar # breakpoint at type method (Go path syntax)
|
||||
(dlv) c / continue # continue until next break
|
||||
(dlv) n / next # step over
|
||||
(dlv) s / step # step into
|
||||
(dlv) so / stepout # step out
|
||||
(dlv) bt / stack # stack trace of current goroutine
|
||||
(dlv) goroutines # list all goroutines
|
||||
(dlv) goroutine <id> # switch to goroutine N
|
||||
(dlv) goroutine <id> bt # stack of a specific goroutine
|
||||
(dlv) locals # all locals in frame
|
||||
(dlv) args # function args
|
||||
(dlv) p <expr> # print value (understands interfaces, maps, slices)
|
||||
(dlv) vars <regex> # package vars matching regex
|
||||
(dlv) regs # registers (rare in Go debugging)
|
||||
(dlv) on <bpid> print <expr> # auto-print on breakpoint hit (powerful!)
|
||||
(dlv) trace <location> # like breakpoint but just logs, doesn't stop
|
||||
```
|
||||
|
||||
The `trace` command is underused — it's like a logpoint, no stepping required.
|
||||
|
||||
---
|
||||
|
||||
## Goroutine-centric debugging
|
||||
|
||||
Goroutine leaks and deadlocks are the most common Go bugs. `dlv`'s `goroutines` command is the starting point.
|
||||
|
||||
```
|
||||
(dlv) goroutines -t # with truncated stack
|
||||
(dlv) goroutines -s # sorted by stack
|
||||
(dlv) goroutines -with user # filter user-spawned goroutines
|
||||
```
|
||||
|
||||
Common patterns:
|
||||
|
||||
| You see in `goroutines` | Usually means |
|
||||
|---|---|
|
||||
| 100s of goroutines stuck at `chan receive` | Producer died; consumers leak |
|
||||
| 100s stuck at `semacquire` | Lock contention; a holder probably deadlocked |
|
||||
| One stuck at `select` with no default | Missing case or closed channel scenario |
|
||||
| Stuck at `netpoll` | External I/O not responding — not a Go bug, check downstream |
|
||||
| Growing count over time | Goroutine leak — need to find who's spawning without cleanup |
|
||||
|
||||
### Panic signals in Go
|
||||
|
||||
```go
|
||||
// Without recovery, panics crash the program with a stack trace of ALL goroutines
|
||||
// With recovery, they're silent unless explicitly logged:
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
log.Printf("recovered panic: %v\n%s", r, debug.Stack()) // GOOD
|
||||
// log.Printf("recovered") // BAD — silent
|
||||
}
|
||||
}()
|
||||
```
|
||||
|
||||
**Always check for silent recovers** in Phase 8. Grep:
|
||||
```bash
|
||||
rg 'recover\(\)' --type go
|
||||
```
|
||||
|
||||
And inspect each site for whether the panic is actually surfaced.
|
||||
|
||||
---
|
||||
|
||||
## Race detector — ALWAYS run when the bug is intermittent
|
||||
|
||||
```bash
|
||||
go test -race ./...
|
||||
go run -race ./cmd/server
|
||||
go build -race ./cmd/server
|
||||
```
|
||||
|
||||
The race detector wraps memory accesses and catches concurrent read/write without synchronization. **Run this before attaching dlv** if intermittency is involved — it often finds the bug directly.
|
||||
|
||||
Output shape:
|
||||
```
|
||||
WARNING: DATA RACE
|
||||
Read at 0x00c0001a0080 by goroutine 7:
|
||||
main.(*Counter).Value()
|
||||
/path/to/counter.go:14 +0x3c
|
||||
Previous write at 0x00c0001a0080 by goroutine 6:
|
||||
main.(*Counter).Inc()
|
||||
/path/to/counter.go:10 +0x5f
|
||||
```
|
||||
|
||||
Both stacks. Both goroutines. The race is obvious from the line pair.
|
||||
|
||||
---
|
||||
|
||||
## pprof — for perf, memory, goroutine leaks
|
||||
|
||||
### Wire it up (idempotent; usually already present)
|
||||
|
||||
```go
|
||||
import _ "net/http/pprof"
|
||||
|
||||
func main() {
|
||||
go func() {
|
||||
log.Println(http.ListenAndServe("localhost:6060", nil))
|
||||
}()
|
||||
// ... rest of your server
|
||||
}
|
||||
```
|
||||
|
||||
### Queries
|
||||
|
||||
```bash
|
||||
# CPU profile (30s)
|
||||
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
|
||||
|
||||
# Heap snapshot
|
||||
go tool pprof http://localhost:6060/debug/pprof/heap
|
||||
|
||||
# Goroutine snapshot — find leaks
|
||||
go tool pprof http://localhost:6060/debug/pprof/goroutine
|
||||
|
||||
# Block profile — find blocking ops (needs runtime.SetBlockProfileRate)
|
||||
go tool pprof http://localhost:6060/debug/pprof/block
|
||||
|
||||
# Mutex profile — find lock contention (needs runtime.SetMutexProfileFraction)
|
||||
go tool pprof http://localhost:6060/debug/pprof/mutex
|
||||
```
|
||||
|
||||
Inside pprof:
|
||||
```
|
||||
(pprof) top # top functions by self time
|
||||
(pprof) list main.handler # annotated source of a function
|
||||
(pprof) web # SVG callgraph in browser (requires graphviz)
|
||||
(pprof) traces # sample traces
|
||||
```
|
||||
|
||||
For goroutine leaks, **take two snapshots 30s apart** and diff:
|
||||
```bash
|
||||
go tool pprof -base prof1.pb.gz prof2.pb.gz
|
||||
```
|
||||
|
||||
Goroutines that appear in prof2 but not prof1 are new; if they stick around, they're leaking.
|
||||
|
||||
---
|
||||
|
||||
## `GODEBUG` — runtime-level observability
|
||||
|
||||
```bash
|
||||
GODEBUG=gctrace=1 ./myserver # print GC stats
|
||||
GODEBUG=schedtrace=1000 ./myserver # scheduler trace every 1000ms
|
||||
GODEBUG=scheddetail=1,schedtrace=1000 # detailed scheduler state
|
||||
GODEBUG=allocfreetrace=1 ./myserver # every alloc/free (noisy!)
|
||||
GODEBUG=memprofilerate=1 ./myserver # profile every allocation
|
||||
```
|
||||
|
||||
Useful for diagnosing GC pressure, goroutine starvation, or memory pattern issues.
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in Go
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| `if err != nil { return err }` that returns to a caller that ignores | Error bubbles up, then gets discarded at the top |
|
||||
| `defer func() { recover() }()` — bare recover, no log | Panic swallowed, program continues with state corruption |
|
||||
| `_, _ = conn.Write(data)` | Intentionally discarded error |
|
||||
| Buffered channel send that blocks forever | Sender hangs; hard to see if no deadlock detection |
|
||||
| `time.Sleep` in a test | "Works on my machine"; test passes locally, fails in CI |
|
||||
| `go func() { ... }()` with no error path | Goroutine dies silently on panic unless recover+log |
|
||||
| Context canceled but operation continues | Ignored `ctx.Err()` check |
|
||||
| `json.Unmarshal` of zero-value struct field | Input missing the key; silently zero |
|
||||
| Closed channel read returning zero value | Consumer doesn't check `ok`; reads forever |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Kill dlv sessions
|
||||
pkill -f 'dlv' || true
|
||||
lsof -iTCP:2345 -sTCP:LISTEN -nP 2>/dev/null # dlv default
|
||||
|
||||
# Kill pprof HTTP endpoint if you started it just for this session
|
||||
lsof -iTCP:6060 -sTCP:LISTEN -nP 2>/dev/null
|
||||
|
||||
# Revert any `fmt.Println("DEBUG: ...")` or `log.Printf("DEBUG: ...")` additions
|
||||
git diff | grep -E '(fmt\.Println\("DEBUG|log\.Printf\("DEBUG|println!)'
|
||||
git checkout <file>
|
||||
|
||||
# Unset env vars
|
||||
unset GODEBUG
|
||||
```
|
||||
@@ -0,0 +1,484 @@
|
||||
# Native Binary Debugging (No Source / Reverse Engineering)
|
||||
|
||||
For binaries where you don't have trustworthy source: stripped production builds, third-party closed libs, malware, CTF challenges, firmware, vendored libs whose docs lie. The workflow is specific; doing it out of order wastes days.
|
||||
|
||||
This reference **coordinates** the triage and dynamic work. The heavy tools each have their own reference:
|
||||
- **Static decompilation** → [tools/ghidra.md](../tools/ghidra.md)
|
||||
- **Interactive debugging** → [tools/pwndbg.md](../tools/pwndbg.md)
|
||||
- **Scripted interaction / exploitation** → [tools/pwntools.md](../tools/pwntools.md)
|
||||
|
||||
Read those before using them — especially Ghidra, which has a surprising amount of workflow that's not obvious.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ STOP — is this actually a stripped C/C++ binary?
|
||||
|
||||
A growing share of "binaries" are actually **bundled high-level apps** — Bun SEA, Node SEA, Deno compile, pkg, nexe, Electron, Tauri, PyInstaller. Their workflow is completely different: the high-level source is recoverable with the right per-bundler tool (often plaintext, sometimes V8 cache / `.pyc` / eszip needing extra tooling), and Ghidra against the runtime VM wastes hours.
|
||||
|
||||
Quick check:
|
||||
|
||||
```bash
|
||||
file ./target # Mach-O / ELF / PE - inconclusive
|
||||
du -h ./target # 50 MB+ for a "simple CLI" → suspect bundled
|
||||
strings -n 12 ./target | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|tauri' | head -5
|
||||
```
|
||||
|
||||
**If any hits** → close this file, open [bundled-js-binary.md](bundled-js-binary.md) instead. Following the Ghidra/pwndbg path on a bundled-app binary wastes hours decompiling the runtime VM while the app-level bundle is recoverable with the right per-bundler tool (plaintext for Bun/pkg/nexe/Electron-asar; eszip / V8-cache / `.pyc` for Deno / Node SEA / PyInstaller).
|
||||
|
||||
If `file` says "Mach-O" or "ELF", `du` is < 20 MB, and the strings check is empty → continue here.
|
||||
|
||||
---
|
||||
|
||||
## The workflow (do these in order)
|
||||
|
||||
Every step's output is input to the next. Skipping steps means guessing later.
|
||||
|
||||
```
|
||||
[1] Triage → what kind of binary is this?
|
||||
[2] Dynamic tracing → what syscalls / libcalls does it make?
|
||||
[3] Static analysis → what does it DO, in readable form? (Ghidra)
|
||||
[4] Dynamic debug → confirm hypotheses at runtime (pwndbg)
|
||||
[5] Scripted repro → lock the bug with a pwntools script
|
||||
[6] TDD + fix / report
|
||||
```
|
||||
|
||||
Steps 1 and 2 are fast (minutes). Step 3 is slow (tens of minutes to hours depending on size). Don't skip 1-2 and go straight to Ghidra — the triage output tells you what to focus on inside Ghidra.
|
||||
|
||||
---
|
||||
|
||||
## [1] Triage — 5-minute fingerprint
|
||||
|
||||
```bash
|
||||
# Basic identity
|
||||
file ./target
|
||||
# elf, mach-o, pe? 32/64-bit? dynamically linked? stripped?
|
||||
|
||||
# Architecture details
|
||||
readelf -h ./target # ELF header: entry point, arch, type
|
||||
lipo -info ./target 2>/dev/null # macOS: universal binary?
|
||||
|
||||
# Interesting strings (often leaks function names, error messages, URLs, API keys)
|
||||
strings -n 8 ./target | head -100
|
||||
strings -n 8 ./target | grep -iE '(http|/api/|error|debug|version)'
|
||||
|
||||
# Imported symbols (what does it link against?)
|
||||
nm -D ./target 2>/dev/null # dynamic symbols
|
||||
objdump -T ./target 2>/dev/null # same, alternate tool
|
||||
readelf -d ./target # dynamic section (NEEDED libs)
|
||||
ldd ./target 2>/dev/null # resolved library paths
|
||||
|
||||
# Security posture (affects what exploits / bugs are possible)
|
||||
checksec --file=./target # requires pwntools or installing checksec
|
||||
# NX, PIE, RELRO, stack canary, FORTIFY
|
||||
|
||||
# Is it stripped?
|
||||
nm ./target 2>/dev/null | head # empty? stripped. full? not stripped.
|
||||
file ./target # will say "stripped" or "not stripped"
|
||||
```
|
||||
|
||||
### ⚠️ `strings -n N` silently drops short content
|
||||
|
||||
`strings` prints runs of printable characters of length **≥ N**. With `-n 8`, **anything shorter than 8 chars sandwiched between non-printable bytes is dropped silently**. This includes:
|
||||
|
||||
- Short identifier interpolations in templates (`${x}`, `${i}`, `${R}`)
|
||||
- Short embedded constants (`v3`, `null`, integer immediates as bytes)
|
||||
- Short error codes between binary padding
|
||||
|
||||
Real example: a JavaScript template literal `<INSTRUCTIONS>\n${x}\n</INSTRUCTIONS>` came out of `strings -n 8` as `<INSTRUCTIONS>\n</INSTRUCTIONS>` — the `${x}` (4 chars) was dropped. A consumer reading the dump would conclude the template was empty. It is not.
|
||||
|
||||
**Use `strings` only for fingerprinting (Phase 1).** For any extraction whose correctness matters, **read bytes directly**:
|
||||
|
||||
```bash
|
||||
# Count occurrences of a needle
|
||||
LC_ALL=C grep -aoc 'NEEDLE' ./target
|
||||
|
||||
# Find offsets
|
||||
LC_ALL=C grep -aob 'NEEDLE' ./target | head
|
||||
|
||||
# Or via Python for byte-precise context
|
||||
python3 -c "
|
||||
import sys
|
||||
data = open('./target','rb').read()
|
||||
needle = b'NEEDLE'
|
||||
pos = data.find(needle)
|
||||
print(repr(data[max(0,pos-100):pos+200]))
|
||||
"
|
||||
```
|
||||
|
||||
If you must keep using `strings`, lower the threshold: `strings -n 1 -t x ./target | rg ...`. The signal-to-noise drops sharply but short content is preserved.
|
||||
|
||||
Write the triage summary to the journal:
|
||||
|
||||
```markdown
|
||||
## Binary triage
|
||||
- Type: <ELF 64-bit, dynamically linked, stripped>
|
||||
- Arch: <x86_64 | arm64 | ...>
|
||||
- Libs: <libc, openssl, libcurl>
|
||||
- Security: <NX, PIE, Partial RELRO, no canary>
|
||||
- Interesting strings: <short list>
|
||||
- First hypothesis surface: <which function / area looks most relevant>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [2] Dynamic tracing — what does it actually call?
|
||||
|
||||
These are cheap — run them before Ghidra to orient yourself.
|
||||
|
||||
### Linux: strace + ltrace
|
||||
|
||||
```bash
|
||||
# System calls
|
||||
strace -f -o trace.out ./target arg1 arg2
|
||||
strace -f -e trace=network ./target # filter to network syscalls
|
||||
strace -f -e trace=file ./target # filter to file ops
|
||||
|
||||
# Library calls (less useful when stripped but still informative)
|
||||
ltrace -f -o ltrace.out ./target
|
||||
ltrace -f -e 'str*+mem*' ./target # filter to string/mem functions
|
||||
```
|
||||
|
||||
### macOS: Mach-O specifics
|
||||
|
||||
**SIP block reality check.** With System Integrity Protection enabled (default on every modern macOS), `dtruss` / `dtrace` will **silently fail** to attach to:
|
||||
- Anything in `/usr`, `/bin`, `/sbin`, `/System`
|
||||
- Apple-signed binaries (Xcode CLT, Homebrew formulae from Apple-distributed taps)
|
||||
- Notarized vendor binaries (Bun, Deno, Docker Desktop, etc.)
|
||||
|
||||
`dtruss ./target` will appear to run but produce zero events. This is not a bug; it is the SIP design. Disabling SIP requires a Recovery Mode reboot — usually not worth it. Use the alternatives below.
|
||||
|
||||
```bash
|
||||
# dtruss — works only when SIP allows it (your own unsigned binaries)
|
||||
sudo dtruss -f ./target 2>&1 | head -20 # equivalent to strace
|
||||
# If output is suspiciously empty → SIP blocked it. Switch to lldb or app-level logging.
|
||||
```
|
||||
|
||||
**Mach-O metadata inspection (no SIP issues, no debugger needed):**
|
||||
|
||||
```bash
|
||||
# Architecture and slices
|
||||
file ./target # arm64 / x86_64 / universal
|
||||
lipo -info ./target # which architectures included
|
||||
lipo -thin arm64 ./target -output ./target-arm64 # extract one slice for analysis
|
||||
|
||||
# Headers & load commands (segments, dylibs, code-signature pointer)
|
||||
otool -h ./target # Mach header (cputype, ncmds, flags)
|
||||
otool -l ./target | head -100 # load commands; entitlements live in code-signature blob, see codesign below
|
||||
|
||||
# Dynamic library dependencies (macOS equivalent of ldd)
|
||||
otool -L ./target # linked dylibs with versions
|
||||
dyld_info ./target # macOS 13+, more detailed than otool -L
|
||||
|
||||
# Disassembly
|
||||
otool -tv ./target | head -200 # quick disassembly without Ghidra
|
||||
otool -tV ./target # with symbol-resolved branches
|
||||
|
||||
# Imported / exported symbols (Apple `nm`, NOT GNU)
|
||||
nm -u ./target # undefined references = imports
|
||||
nm -gU ./target # external defined = exports
|
||||
# Note: GNU `-D`/dynamic flags are not honored on Apple `nm`; use the above forms.
|
||||
symbols -fullSourcePath -onlyWithDebugInfo ./target # if any debug info survives
|
||||
|
||||
# Code signature & entitlements (entitlements come from codesign, NOT otool)
|
||||
codesign -dv --entitlements :- ./target 2>&1 # signature info + entitlements XML on stdout
|
||||
spctl --assess --type execute -vv ./target # Gatekeeper assessment
|
||||
|
||||
# Cert chain — extract to a temp dir to avoid creating files named -0/-1 in cwd
|
||||
tmp=$(mktemp -d)
|
||||
codesign -dvv --extract-certificates="$tmp/cert" ./target 2>&1
|
||||
ls -la "$tmp"
|
||||
# rm -rf "$tmp" # journal first, clean up later
|
||||
|
||||
# Strings inside specific segments only (less noise than full-binary strings)
|
||||
otool -s __TEXT __cstring ./target # C string section
|
||||
otool -s __TEXT __const ./target # constants section
|
||||
```
|
||||
|
||||
**Interactive debugging on macOS — use `lldb`, not `gdb`.**
|
||||
|
||||
GDB on macOS requires a self-signed code-signing certificate (`codesign --entitlements gdb.entitlements --sign gdb-cert /opt/homebrew/bin/gdb`) and even then is unreliable on arm64. **Use `lldb` directly** — it ships with Xcode CLT and works without configuration.
|
||||
|
||||
```bash
|
||||
# Start lldb
|
||||
lldb ./target
|
||||
|
||||
# Set arguments
|
||||
(lldb) settings set target.run-args arg1 arg2
|
||||
|
||||
# Run with breakpoints
|
||||
(lldb) breakpoint set --name function_name # symbol-based
|
||||
(lldb) breakpoint set --address 0x1000034c0 # address-based
|
||||
(lldb) breakpoint set --regex '.*decode.*' # regex over symbols
|
||||
|
||||
# Run / step / inspect
|
||||
(lldb) run
|
||||
(lldb) bt # backtrace
|
||||
(lldb) frame variable # locals
|
||||
(lldb) register read # all registers
|
||||
(lldb) memory read --size 8 --format x --count 16 $sp # 16 qwords from stack
|
||||
(lldb) disassemble --frame # current function
|
||||
(lldb) image list # loaded modules
|
||||
(lldb) image lookup -a 0x1000034c0 # which module + symbol owns this address
|
||||
|
||||
# Process attach to running process
|
||||
(lldb) process attach --pid 12345
|
||||
(lldb) process attach --name target # attach by name
|
||||
|
||||
# Print Mach-O specific
|
||||
(lldb) image dump sections ./target
|
||||
(lldb) image dump symtab ./target
|
||||
```
|
||||
|
||||
**Function interception via `DYLD_INSERT_LIBRARIES`** (macOS equivalent of `LD_PRELOAD`):
|
||||
|
||||
```bash
|
||||
# Build a shim dylib that overrides specific functions
|
||||
# Then run target with it preloaded
|
||||
DYLD_INSERT_LIBRARIES=./shim.dylib DYLD_FORCE_FLAT_NAMESPACE=1 ./target
|
||||
```
|
||||
|
||||
DYLD_INSERT works in the unrestricted case but is blocked in three distinct scenarios — distinguish them when diagnosing why your shim didn't load:
|
||||
|
||||
1. **SIP / restricted process** (target has the `__RESTRICT,__restrict` section, is setuid/setgid, or is a platform/Apple-signed binary): dyld unconditionally strips all `DYLD_*` env vars before the process starts. Nothing you set will reach the target.
|
||||
2. **Hardened runtime + library validation** (`CS_RUNTIME` flag set, `com.apple.security.cs.disable-library-validation` entitlement absent): the process accepts `DYLD_INSERT_LIBRARIES` but **rejects** loading any dylib that isn't signed by the same Team ID or by Apple. Symptom: shim is found but not loaded; check `log show --predicate 'eventMessage CONTAINS "library validation failed"'`.
|
||||
3. **Notarization / Gatekeeper translocation**: the binary may be running from a translocated path; relative paths in `DYLD_INSERT_LIBRARIES` won't resolve. Use absolute paths.
|
||||
|
||||
Check each:
|
||||
|
||||
```bash
|
||||
# Restrict segment present? (case 1)
|
||||
otool -l ./target | grep -A2 __RESTRICT
|
||||
# Hardened runtime flag? (case 2)
|
||||
codesign -d --verbose=4 ./target 2>&1 | grep -iE 'flags=|CodeDirectory'
|
||||
# Look for "0x10000(runtime)" or similar in the flags line.
|
||||
# Disable-library-validation entitlement?
|
||||
codesign -d --entitlements :- ./target 2>&1 | grep disable-library-validation
|
||||
```
|
||||
|
||||
**App-level debug logging (always works, ignores SIP):**
|
||||
|
||||
When debugger attach is blocked, fall back to maximizing the app's own logging:
|
||||
|
||||
```bash
|
||||
# Try common patterns
|
||||
APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target
|
||||
NSDebugEnabled=YES ./target # Cocoa apps
|
||||
OS_ACTIVITY_MODE=debug ./target # os_log subsystem
|
||||
|
||||
# Then read os_log unified logging stream live
|
||||
log stream --predicate 'process == "target"' --level debug
|
||||
|
||||
# Or extract historical logs
|
||||
log show --predicate 'process == "target"' --last 1h --info --debug
|
||||
```
|
||||
|
||||
This is the **partial-runtime-evidence path** for macOS. See [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for how to combine app-level logs with static analysis when wire-level capture is blocked.
|
||||
|
||||
**Network capture on macOS (TLS-decrypted):**
|
||||
|
||||
```bash
|
||||
# 1. Find the active network service (don't assume "Wi-Fi"):
|
||||
# Map the default-route interface to the matching networksetup service name.
|
||||
networksetup -listallnetworkservices # show options
|
||||
DEFAULT_IF=$(route -n get default 2>/dev/null | awk '/interface:/ {print $2}')
|
||||
echo "Default-route interface: $DEFAULT_IF"
|
||||
# Match the interface (en0, en1, ...) back to a service name:
|
||||
SERVICE=$(networksetup -listallhardwareports | awk -v iface="$DEFAULT_IF" '
|
||||
/^Hardware Port:/ { hp = substr($0, index($0,$3)) }
|
||||
/^Device:/ { if ($2 == iface) print hp }
|
||||
')
|
||||
if [ -z "$SERVICE" ]; then
|
||||
echo "Could not auto-detect active service. Pick one from -listallnetworkservices manually." >&2
|
||||
echo "Aborting proxy setup." >&2
|
||||
false # signal failure but stay safe at top level
|
||||
else
|
||||
echo "Using service: $SERVICE"
|
||||
fi
|
||||
|
||||
# 2. JOURNAL the original proxy state before changing it (REQUIRED for safe rollback):
|
||||
networksetup -getwebproxy "$SERVICE" # save this output to journal
|
||||
networksetup -getsecurewebproxy "$SERVICE" # save this too
|
||||
|
||||
# 3. Start mitmproxy with persistent CA at ~/.mitmproxy/
|
||||
mitmproxy --listen-host 127.0.0.1 --listen-port 8888 &
|
||||
|
||||
# 4. Trust the mitmproxy CA system-wide if the target uses URLSession or any framework
|
||||
# that ignores HTTPS_PROXY/SSL_CERT_FILE (most macOS-native apps do):
|
||||
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/.mitmproxy/mitmproxy-ca-cert.pem
|
||||
|
||||
# 5. Two routing options. Try env-var first; fall back to system proxy:
|
||||
# 5a. Apps that honor env vars (most CLIs):
|
||||
HTTPS_PROXY=http://127.0.0.1:8888 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem ./target ...
|
||||
|
||||
# 5b. Apps that use URLSession / system network config (most GUI apps, Bun, some CLIs):
|
||||
networksetup -setwebproxy "$SERVICE" 127.0.0.1 8888
|
||||
networksetup -setsecurewebproxy "$SERVICE" 127.0.0.1 8888
|
||||
|
||||
# 6. Cleanup — RESTORE original state from journal, untrust CA:
|
||||
networksetup -setwebproxystate "$SERVICE" off
|
||||
networksetup -setsecurewebproxystate "$SERVICE" off
|
||||
sudo security delete-certificate -c "mitmproxy" /Library/Keychains/System.keychain
|
||||
```
|
||||
|
||||
**Critical**: forgetting step 6 leaves all your subsequent traffic mis-routed and silently MITM-able. Journal every step.
|
||||
|
||||
### What to look for
|
||||
|
||||
| Observation | Hypothesis |
|
||||
|---|---|
|
||||
| `open("/etc/secret-config", ...)` | Reads unexpected config; look at what it does with contents |
|
||||
| `connect(... 1.2.3.4:443)` | Phones home or depends on an external service |
|
||||
| `getenv("FOO")` returning NULL | Env var expected but not set |
|
||||
| Repeated `poll`/`epoll_wait` with no progress | Stuck on I/O; check downstream |
|
||||
| `SIGSEGV` caught by signal handler | Custom crash recovery — often hides the real bug |
|
||||
| `dlopen("libfoo.so.42")` | Dynamic plugin loading; check plugin path |
|
||||
|
||||
---
|
||||
|
||||
## [3] Static analysis with Ghidra
|
||||
|
||||
When triage + tracing have narrowed you to "something in function X" or "the crypto routine is weird", open Ghidra.
|
||||
|
||||
**Open [tools/ghidra.md](../tools/ghidra.md) before launching Ghidra** — the import / analyze / decompile workflow is not obvious and first-time users waste an hour figuring it out.
|
||||
|
||||
Ghidra's decompiler turns machine code into readable-ish C. That's usually what you want. Stay in the Decompiler view; drop to Listing (disassembly) only when the decompiler punts.
|
||||
|
||||
---
|
||||
|
||||
## [4] Dynamic debugging with pwndbg
|
||||
|
||||
Once static analysis gives you a hypothesis ("this branch at 0x401234 is where the validation fails"), confirm it at runtime with pwndbg.
|
||||
|
||||
**Open [tools/pwndbg.md](../tools/pwndbg.md) before launching gdb.** Pwndbg gives you the context view (registers / stack / disasm / code all visible at once) which is essential for binary debugging.
|
||||
|
||||
Typical pwndbg flow:
|
||||
|
||||
```
|
||||
$ gdb ./target # pwndbg loads automatically if installed
|
||||
pwndbg> break *0x401234 # break at the address static analysis flagged
|
||||
pwndbg> run arg1 arg2
|
||||
# At the breakpoint:
|
||||
pwndbg> context # registers + stack + disasm
|
||||
pwndbg> telescope $rdi # walk pointers at $rdi
|
||||
pwndbg> x/20xw $rsp # raw dump of stack
|
||||
pwndbg> ni / si # step next / step instruction
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [5] Scripted reproduction with pwntools
|
||||
|
||||
Once you have a hypothesis with a concrete repro input, lock it down with pwntools. This is the "failing test" equivalent for binaries.
|
||||
|
||||
**Open [tools/pwntools.md](../tools/pwntools.md)** — the Process/Remote/ELF/context APIs are the foundation.
|
||||
|
||||
```python
|
||||
from pwn import *
|
||||
|
||||
context.binary = elf = ELF('./target')
|
||||
|
||||
p = process('./target')
|
||||
p.sendlineafter(b'> ', b'<trigger input that reproduces the bug>')
|
||||
result = p.recvall(timeout=3)
|
||||
assert b'expected-output-when-fixed' in result, f'bug repro: {result}'
|
||||
```
|
||||
|
||||
This script is now your "red test". When the fix is applied, the script should pass (or the assertion should be inverted for negative tests — e.g. "the crash string should NOT appear").
|
||||
|
||||
---
|
||||
|
||||
## [6] Fixing a binary bug you can't recompile
|
||||
|
||||
Three options, in preference order:
|
||||
|
||||
### Option A: Patch at the source (if you have it)
|
||||
|
||||
If the bug is in your own code and source is available, fix it there and rebuild. Standard TDD path.
|
||||
|
||||
### Option B: Binary patch
|
||||
|
||||
For tiny fixes (one byte, one branch inversion):
|
||||
|
||||
```bash
|
||||
# Identify the exact byte offset
|
||||
# e.g. Ghidra says the bug is at 0x401234 = file offset 0x1234
|
||||
printf '\x90\x90' | dd of=./target bs=1 seek=$((0x1234)) conv=notrunc
|
||||
```
|
||||
|
||||
Journal the exact `dd` command and the original bytes so you can revert.
|
||||
|
||||
### Option C: Wrap / shim
|
||||
|
||||
If you can't patch the binary, write a shim library (LD_PRELOAD on Linux, DYLD_INSERT_LIBRARIES on macOS) that overrides the buggy function. pwntools has examples.
|
||||
|
||||
### Option D: Report upstream
|
||||
|
||||
If it's a third-party binary and none of the above are feasible, the "fix" is a high-quality bug report with:
|
||||
- Full triage summary
|
||||
- Reproducible pwntools script
|
||||
- Ghidra decompilation of the buggy function
|
||||
- Hypothesis about the root cause
|
||||
- Recommended patch sketch (in C or pseudocode)
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in native binaries
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| Ignored libc return codes (`read`, `write`, `malloc`) | Bug continues with garbage data; no check |
|
||||
| Signal handler swallows SIGSEGV | Crash converted to "something didn't work"; no log |
|
||||
| `setjmp`/`longjmp` unwinding over cleanup | Resources leak silently |
|
||||
| Thread-local error state never read (`errno`, `GetLastError`) | Error happened, nobody asked |
|
||||
| Recovered assertion failure in release build | `assert` compiled out; precondition violations silently corrupt |
|
||||
| Dangling pointer reads after free | Often looks like valid data until it doesn't |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Kill debugger sessions
|
||||
pkill -f 'gdb' || true
|
||||
pkill -f 'lldb' || true
|
||||
|
||||
# Ghidra scratch projects (if made just for this session)
|
||||
# Named something like ~/ghidra-projects/debug-<timestamp>:
|
||||
ls -la ~/ghidra-projects/ 2>/dev/null
|
||||
# rm -rf ~/ghidra-projects/debug-scratch # only if the journal says to
|
||||
|
||||
# Core dumps left from crashes
|
||||
rm -f ./core ./core.* ~/core.*
|
||||
|
||||
# strace/ltrace output files
|
||||
rm -f trace.out ltrace.out
|
||||
|
||||
# If you made a binary patch (Option B above), confirm revert
|
||||
# The journal should have the original bytes — restore them:
|
||||
# printf '<original-bytes>' | dd of=./target bs=1 seek=<offset> conv=notrunc
|
||||
|
||||
# Trace-output files
|
||||
rm -f /tmp/debug-*.bin /tmp/debug-*.strace /tmp/debug-*.ltrace
|
||||
|
||||
# macOS-specific:
|
||||
# Restore proxy settings if you set them (CRITICAL — leaves system traffic mis-routed otherwise)
|
||||
# Use the SAME $SERVICE you used when enabling the proxy (read it from the journal).
|
||||
# Do NOT hardcode "Wi-Fi" — many machines route traffic over Ethernet, USB tether, or a VPN service.
|
||||
[ -n "$SERVICE" ] && {
|
||||
networksetup -setwebproxystate "$SERVICE" off 2>/dev/null
|
||||
networksetup -setsecurewebproxystate "$SERVICE" off 2>/dev/null
|
||||
}
|
||||
# Or restore explicitly from the journaled original state — see the proxy section above.
|
||||
|
||||
# Stop mitmproxy
|
||||
pkill -f 'mitmproxy' 2>/dev/null
|
||||
|
||||
# Remove DYLD shim libraries you built
|
||||
rm -f /tmp/*-shim.dylib
|
||||
|
||||
# Clear extracted strings dumps (these can be huge and may contain secrets)
|
||||
rm -f /tmp/*-strings*.txt
|
||||
|
||||
# Verify hostname resolution returns to normal (mitmproxy can leave entries)
|
||||
scutil --dns | head -20
|
||||
```
|
||||
@@ -0,0 +1,260 @@
|
||||
# Node.js / tsx / ts-node / Bun / Deno Debugging
|
||||
|
||||
Covers Node 18+, tsx, ts-node, Bun, Deno. Launch recipes, inspector protocol usage, the `node inspect` CLI, and the **tsx source-map silent-failure** that costs people days.
|
||||
|
||||
---
|
||||
|
||||
## Environment detection (Phase 0)
|
||||
|
||||
```bash
|
||||
node --version
|
||||
cat package.json | head -40
|
||||
|
||||
# Which JS runtime launches the app? (order them; the first match wins)
|
||||
ls node_modules/.bin/tsx 2>/dev/null && echo 'has tsx'
|
||||
ls node_modules/.bin/ts-node 2>/dev/null && echo 'has ts-node'
|
||||
ls node_modules/.bin/vitest 2>/dev/null && echo 'has vitest'
|
||||
which bun 2>/dev/null && bun --version
|
||||
which deno 2>/dev/null && deno --version
|
||||
|
||||
# Source-map situation
|
||||
grep -E '"sourceMap"|"inlineSources"' tsconfig.json 2>/dev/null
|
||||
grep -l '//# sourceMappingURL' dist/*.js 2>/dev/null | head -3
|
||||
|
||||
# Debug-relevant ports
|
||||
lsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null
|
||||
lsof -iTCP:9230 -sTCP:LISTEN -nP 2>/dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚨 The tsx + `node inspect` CLI silent-failure (READ THIS)
|
||||
|
||||
`tsx` transpiles each `.ts` file on the fly and emits an inline source map. V8 Inspector registers the module with its `.ts` path (so it shows up in the debugger's `scripts` list), **but the `node inspect` CLI REPL does not resolve source-map line numbers reliably**. Setting `sb('session.ts', 285)` will show a "pending" breakpoint that **never fires even after the module loads**.
|
||||
|
||||
The breakpoint list will happily display it, so you think it's set. It isn't.
|
||||
|
||||
### Three reliable workarounds
|
||||
|
||||
| Workaround | When to use | Downside |
|
||||
|---|---|---|
|
||||
| **`debugger;` statement in source** | You can edit the source, CLI required | Requires source edit + revert |
|
||||
| **Chrome DevTools GUI** (`chrome://inspect`) | CLI not required, faster iteration | Not usable if user specifically asked for CLI |
|
||||
| **Debug the built `dist/` JS** | Source maps are working end-to-end | Requires `npm run build` on every source change |
|
||||
|
||||
The `debugger;` statement is the most reliable. Journal the edit — revert at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## Launch recipes by runtime
|
||||
|
||||
### Node (plain JS / compiled TS)
|
||||
|
||||
```bash
|
||||
# Break on first line, wait for debugger to attach
|
||||
node --inspect-brk=9229 dist/index.js
|
||||
|
||||
# Attach immediately, don't block startup — pair with debugger; statements
|
||||
node --inspect=9229 dist/index.js
|
||||
|
||||
# Wait for debugger to attach, THEN run (new in Node 20.15+)
|
||||
node --inspect-wait=9229 dist/index.js
|
||||
|
||||
# Source maps in stack traces (always a good idea in debug builds)
|
||||
node --enable-source-maps --inspect dist/index.js
|
||||
```
|
||||
|
||||
### tsx
|
||||
|
||||
```bash
|
||||
# The tsx runner is --import-compatible, so these work:
|
||||
node --inspect-brk=9229 --import tsx index.ts
|
||||
node --inspect=9229 --import tsx index.ts
|
||||
|
||||
# If user prefers invoking tsx directly, this also works but is less explicit:
|
||||
NODE_OPTIONS='--inspect-brk=9229' npx tsx index.ts
|
||||
|
||||
# ⚠️ tsx watch + inspector = inspector reloads per file change
|
||||
# Debug without watch:
|
||||
node --inspect=9229 --import tsx index.ts # (no `watch`)
|
||||
```
|
||||
|
||||
### ts-node (legacy but still encountered)
|
||||
|
||||
```bash
|
||||
node --inspect-brk -r ts-node/register src/index.ts
|
||||
# ESM (ts-node's ESM loader is fragile — if possible, migrate to tsx):
|
||||
node --inspect --loader ts-node/esm src/index.ts
|
||||
```
|
||||
|
||||
### Bun (WebKit Inspector Protocol, NOT V8)
|
||||
|
||||
```bash
|
||||
bun --inspect src/index.ts # opens debug.bun.sh URL
|
||||
bun --inspect-brk src/index.ts # break on start
|
||||
bun --inspect-wait src/index.ts # wait for attach
|
||||
bun test --inspect-brk # debug test runner
|
||||
```
|
||||
|
||||
**Critical**: Bun uses WebKit Inspector Protocol, not V8. `chrome://inspect` cannot connect directly. Use `debug.bun.sh` or the (currently buggy, per Bun docs) VS Code extension.
|
||||
|
||||
### Deno (native V8, Chrome DevTools / VS Code compatible)
|
||||
|
||||
```bash
|
||||
deno run --inspect-brk --allow-all src/main.ts
|
||||
deno test --inspect-brk --filter "auth"
|
||||
```
|
||||
|
||||
Deno is the smoothest TS debugging experience — native V8 inspector, no source-map workarounds.
|
||||
|
||||
### Vitest
|
||||
|
||||
```bash
|
||||
# Single worker required — inspector can't attach to multiple workers
|
||||
vitest --inspect-brk --no-file-parallelism
|
||||
vitest --inspect-brk --browser --no-file-parallelism # browser mode
|
||||
```
|
||||
|
||||
Without `--no-file-parallelism`, breakpoints won't fire because the process Vitest spawns workers in isn't the one listening on the inspector port.
|
||||
|
||||
---
|
||||
|
||||
## Attaching with `node inspect` CLI
|
||||
|
||||
```bash
|
||||
node inspect 127.0.0.1:9229 # attach to an existing --inspect process
|
||||
```
|
||||
|
||||
Core commands at the `debug>` prompt:
|
||||
|
||||
```
|
||||
cont, c resume until next break / debugger;
|
||||
next, n step over
|
||||
step, s step into
|
||||
out, o step out
|
||||
pause pause a running process
|
||||
bt backtrace
|
||||
scripts list all modules V8 has loaded (incl. tsx-transpiled .ts)
|
||||
sb(N) set breakpoint at line N of current file
|
||||
sb('file', N) set breakpoint at line N of matching file (⚠️ unreliable with tsx)
|
||||
sb(func) set breakpoint at function reference
|
||||
cb(N), cb('file', N) clear breakpoint
|
||||
breakpoints list breakpoints (shows pending ones, doesn't tell you they'll never fire)
|
||||
watch('expr') persistent watch expression
|
||||
watchers show watchers
|
||||
exec('expr') evaluate expression in paused frame's scope
|
||||
repl drop into full REPL with frame's scope
|
||||
restart restart the debuggee
|
||||
kill kill the debuggee
|
||||
```
|
||||
|
||||
**`exec('expr')` is the most powerful tool in this CLI** — it evaluates any JS in the paused frame and returns the value. Use it heavily.
|
||||
|
||||
---
|
||||
|
||||
## `exec()` patterns that resolve hypotheses fast
|
||||
|
||||
At a breakpoint, these queries resolve most LLM / agent / async bugs in one line each:
|
||||
|
||||
```js
|
||||
// Agent / LLM state
|
||||
exec('this.agent.state.messages.length')
|
||||
exec('this.agent.state.messages.map(m => m.role)')
|
||||
exec('JSON.stringify(this.agent.state.messages.at(-1)).substring(0, 500)')
|
||||
exec('this.agent.state.messages.at(-1).errorMessage') // silent-error sentinel
|
||||
exec('this.agent.state.messages.at(-1).stopReason')
|
||||
exec('JSON.stringify(this.agent.state.usage)') // undefined / all-zero = failed call
|
||||
exec('this.agent.state.model.baseUrl') // catch hardcoded vs env-var
|
||||
|
||||
// Env / config at runtime
|
||||
exec('process.env.RELEVANT_VAR')
|
||||
exec('Object.keys(process.env).filter(k => k.startsWith("ANTHROPIC"))')
|
||||
exec('this.config')
|
||||
|
||||
// Async / timing
|
||||
exec('Date.now() - this._turnStartedAt')
|
||||
exec('this._activePromises?.size')
|
||||
|
||||
// HTTP request/response in-flight
|
||||
exec('JSON.stringify(req.body).length')
|
||||
exec('res.statusCode')
|
||||
exec('res.headersSent')
|
||||
|
||||
// What's actually running
|
||||
exec('process.version')
|
||||
exec('process.cwd()')
|
||||
exec('process.argv')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in Node
|
||||
|
||||
These are the patterns that most commonly look like success but aren't. Always check when a response is "too fast" or "too empty":
|
||||
|
||||
| Signal | What it means |
|
||||
|---|---|
|
||||
| HTTP 200 + `content: ""` | Silent error swallowed |
|
||||
| HTTP 200 + response in <1s for an LLM call | Too fast for a real Claude/GPT call; something short-circuited |
|
||||
| `usage: { totalTokens: 0 }` | LLM SDK returned a stub without making the call |
|
||||
| `stopReason: "error" + content: []` | SDK packaged an error into a "success" message |
|
||||
| Unhandled promise rejection with no log | Caller forgot to `await`, or `.catch(() => {})` |
|
||||
| `try { await x(); } catch {}` | Error eaten, no log |
|
||||
| `void somePromise()` | Explicit opt-out of error propagation; often a bug |
|
||||
| Callback-style API where callback never fires | Error happened before callback scheduled |
|
||||
| Handler returns `res.json(...)` twice | Second call is silent on some Express versions |
|
||||
|
||||
When you find one, add a temporary `console.error('[DEBUG]', ...)` to make it loud — journal it, revert at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## tmux session layout (two sessions, one purpose each)
|
||||
|
||||
```bash
|
||||
# Long-running inspected process
|
||||
tmux new-session -d -s debug-server -c "$PWD"
|
||||
tmux send-keys -t debug-server 'node --inspect=9229 --import tsx index.ts' Enter
|
||||
|
||||
# Interactive debugger client (separate pane for readability)
|
||||
tmux new-session -d -s debug-client -c "$PWD"
|
||||
tmux send-keys -t debug-client 'node inspect 127.0.0.1:9229' Enter
|
||||
|
||||
# Non-blocking pane inspection from the outside
|
||||
tmux capture-pane -p -t debug-server -S -50
|
||||
```
|
||||
|
||||
Journal both session names. Kill both at Phase 9:
|
||||
|
||||
```bash
|
||||
tmux kill-session -t debug-server
|
||||
tmux kill-session -t debug-client
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## When to abandon the CLI and switch to Chrome DevTools
|
||||
|
||||
The user's preference for CLI is valid and should be respected. But you may recommend a switch in one short sentence if ANY of these hold:
|
||||
|
||||
- You hit source-map resolution failures (`sb('file', line)` not firing) AND the fix is time-sensitive
|
||||
- You need to watch many values simultaneously (GUI watch panel is faster to scan)
|
||||
- You're stepping through async-heavy code where CLI step semantics get murky across microtask boundaries
|
||||
|
||||
Phrase as a note, not a request: "I can push through with `debugger;` statements in CLI. If we hit three or more of these in a row, switching to `chrome://inspect` GUI would cut cycle time in half — your call."
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Revert source-level debug statements
|
||||
git diff | grep -E '(debugger;|console\.log\(.*DEBUG|\[ARBITER-DEBUG|\[DEBUG)'
|
||||
# Revert any matching files:
|
||||
git checkout <file>
|
||||
|
||||
# Kill inspector-attached processes
|
||||
pkill -f 'node --inspect' || true
|
||||
pkill -f 'bun --inspect' || true
|
||||
pkill -f 'deno.*--inspect' || true
|
||||
lsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null
|
||||
```
|
||||
@@ -0,0 +1,248 @@
|
||||
# Python Debugging
|
||||
|
||||
Covers CPython 3.9+, pytest, asyncio, Django, FastAPI. Setup commands, attach mechanisms, state-query patterns, gotchas, silent-failure signatures.
|
||||
|
||||
---
|
||||
|
||||
## Environment detection (Phase 0)
|
||||
|
||||
```bash
|
||||
# Which Python will actually run the code?
|
||||
which python; which python3
|
||||
python --version
|
||||
|
||||
# Is there a project env manager in play?
|
||||
ls poetry.lock uv.lock Pipfile.lock requirements*.txt .python-version 2>/dev/null
|
||||
|
||||
# Installed debuggers / profilers in this env?
|
||||
python -c 'import pdb, sys; print("pdb", "built-in"); print("python", sys.executable)'
|
||||
pip list 2>/dev/null | grep -iE '^(ipdb|pudb|debugpy|py-spy|memray|rich)\s'
|
||||
|
||||
# asyncio debug mode available?
|
||||
python -c 'import asyncio; print(asyncio.__version__)'
|
||||
```
|
||||
|
||||
**Wrapper gotchas** (these change how flags propagate):
|
||||
|
||||
- `poetry run python ...` — args after `python` are fine; args before `poetry run` go to poetry, not python
|
||||
- `uv run python ...` — similar; prefer `uv run -- python -X dev` if flags collide
|
||||
- `pipenv run` — same story
|
||||
- `./manage.py <cmd>` (Django) — shebang resolution; make sure it points to the right venv
|
||||
- `pytest` — loads `conftest.py` at collection; breakpoints inside collection need `pytest --pdb-trace` not `--pdb`
|
||||
|
||||
---
|
||||
|
||||
## The four ways to attach
|
||||
|
||||
| Method | When to use | Command |
|
||||
|---|---|---|
|
||||
| **`breakpoint()` inline** (Python 3.7+) | You can edit the source and restart. Most reliable. | Add `breakpoint()` to source. Run normally. It invokes `pdb` by default. |
|
||||
| **`python -m pdb <script>`** | No source edit desired. Breaks on entry. | `python -m pdb script.py arg1` |
|
||||
| **post-mortem `pdb.pm()`** | Exception already happened, you want to inspect state | In an exception-caught REPL: `import pdb; pdb.pm()` after the exception propagates |
|
||||
| **debugpy (remote / IDE)** | IDE attach, remote host, containerized process | `python -m debugpy --listen 5678 --wait-for-client script.py` then attach from VS Code / PyCharm |
|
||||
|
||||
### Prefer `ipdb` or `pudb` over plain `pdb` when available
|
||||
|
||||
- **ipdb** — drop-in replacement with tab completion, syntax highlighting. `pip install ipdb`, then `PYTHONBREAKPOINT=ipdb.set_trace` or use `import ipdb; ipdb.set_trace()`.
|
||||
- **pudb** — full-screen TUI debugger, much faster to navigate stack/locals. `pip install pudb`, then `PYTHONBREAKPOINT=pudb.set_trace`.
|
||||
|
||||
### Control `breakpoint()` globally
|
||||
|
||||
```bash
|
||||
# Use ipdb instead of pdb
|
||||
export PYTHONBREAKPOINT=ipdb.set_trace
|
||||
|
||||
# Disable all breakpoint() calls (useful to ship without removing them)
|
||||
export PYTHONBREAKPOINT=0
|
||||
```
|
||||
|
||||
**Journal this env var** — unset at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## pdb / ipdb essentials
|
||||
|
||||
At a `(Pdb)` or `ipdb>` prompt:
|
||||
|
||||
```
|
||||
l list source around current line
|
||||
ll list whole function
|
||||
s step into
|
||||
n step over (next)
|
||||
r step out (return)
|
||||
c continue
|
||||
b list breakpoints
|
||||
b <line> breakpoint at line
|
||||
b <func> breakpoint at function
|
||||
cl <n> clear breakpoint n
|
||||
w where (backtrace)
|
||||
u / d move up/down the stack
|
||||
a args of current frame
|
||||
p <expr> print expression
|
||||
pp <expr> pretty-print
|
||||
!<stmt> execute Python statement (e.g. !x = 5)
|
||||
interact drop into a full Python REPL with current frame's locals
|
||||
q quit (aborts the program)
|
||||
```
|
||||
|
||||
**`interact` is underused** — it gives you a full IPython-esque REPL with all locals available. Faster than typing `p` for 20 things.
|
||||
|
||||
---
|
||||
|
||||
## pytest-specific debugging
|
||||
|
||||
```bash
|
||||
# Enter pdb on first failure
|
||||
pytest --pdb
|
||||
|
||||
# Enter pdb at the START of each test (not on failure)
|
||||
pytest --trace
|
||||
|
||||
# Run only the failing test, with -s to show print output
|
||||
pytest --pdb -x -s path/to/test.py::test_name
|
||||
|
||||
# Collect-time debugging (for problems in conftest.py / fixture setup)
|
||||
pytest --pdb-trace
|
||||
|
||||
# Disable capture for this test (so breakpoint prompt is visible)
|
||||
pytest -s
|
||||
```
|
||||
|
||||
**Common failure**: `breakpoint()` hangs inside a pytest test — that's because pytest captures stdout/stderr by default. Always add `-s` when debugging with breakpoints inside pytest.
|
||||
|
||||
---
|
||||
|
||||
## asyncio gotchas
|
||||
|
||||
Async is where most Python debug sessions go sideways. Know these before attaching.
|
||||
|
||||
### Breakpoints inside coroutines
|
||||
|
||||
`breakpoint()` works inside an async function, but stepping into another coroutine from `pdb` is awkward. Two techniques:
|
||||
|
||||
```python
|
||||
async def handler():
|
||||
result = await some_async_fn() # add breakpoint ABOVE, not inside, when possible
|
||||
breakpoint()
|
||||
return result
|
||||
```
|
||||
|
||||
Inside the breakpoint, to inspect a coroutine without actually advancing time:
|
||||
```
|
||||
!import asyncio
|
||||
!loop = asyncio.get_event_loop()
|
||||
!task = asyncio.ensure_future(some_async_fn())
|
||||
# Now inspect task state, don't await it
|
||||
p task
|
||||
```
|
||||
|
||||
### PYTHONASYNCIODEBUG
|
||||
|
||||
Enable before running the process:
|
||||
|
||||
```bash
|
||||
PYTHONASYNCIODEBUG=1 python script.py
|
||||
```
|
||||
|
||||
Surfaces: coroutines that were never awaited, slow callbacks, unhandled task exceptions. **Always turn this on** if the bug is timing- or async-related.
|
||||
|
||||
### `asyncio.gather` swallows the first exception
|
||||
|
||||
By default, `asyncio.gather(t1, t2)` raises the first exception and cancels the rest. If you need all exceptions, use `gather(..., return_exceptions=True)`.
|
||||
|
||||
### Unhandled task exceptions are silent
|
||||
|
||||
```python
|
||||
async def main():
|
||||
task = asyncio.create_task(broken_coroutine())
|
||||
# If task raises and we never await it, the exception is eaten at gc time
|
||||
await asyncio.sleep(10)
|
||||
```
|
||||
|
||||
To catch these, set `loop.set_exception_handler(...)` or upgrade to Python 3.12+ which warns louder by default.
|
||||
|
||||
---
|
||||
|
||||
## debugpy — remote / IDE / container attach
|
||||
|
||||
Listen and wait for attach:
|
||||
|
||||
```bash
|
||||
python -m debugpy --listen 0.0.0.0:5678 --wait-for-client script.py
|
||||
```
|
||||
|
||||
Attach from VS Code:
|
||||
```json
|
||||
// .vscode/launch.json
|
||||
{
|
||||
"name": "attach",
|
||||
"type": "python",
|
||||
"request": "attach",
|
||||
"connect": { "host": "localhost", "port": 5678 }
|
||||
}
|
||||
```
|
||||
|
||||
Inside the code, programmatic attach point:
|
||||
```python
|
||||
import debugpy
|
||||
debugpy.listen(5678)
|
||||
debugpy.wait_for_client() # blocks until attached
|
||||
debugpy.breakpoint() # programmatic breakpoint
|
||||
```
|
||||
|
||||
**Journal**: the port (5678) and the listener file — unset at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## Sampling profilers for "why is it slow / stuck"
|
||||
|
||||
When the problem is performance or a hang (not a crash), don't attach pdb — it alters timing. Use a sampling profiler that attaches to the running process:
|
||||
|
||||
```bash
|
||||
# py-spy — production-safe, zero code change, works on running process
|
||||
py-spy top --pid <pid> # live top-like view
|
||||
py-spy record -o profile.svg --pid <pid> # flamegraph
|
||||
py-spy dump --pid <pid> # stack traces of all threads right now
|
||||
|
||||
# memray — memory allocation tracking
|
||||
memray run script.py
|
||||
memray flamegraph output.bin
|
||||
memray stats output.bin
|
||||
```
|
||||
|
||||
`py-spy dump` on a stuck process is often enough to find the hung call — no breakpoints needed.
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in Python
|
||||
|
||||
Add these to Phase 8's silent-failure check:
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| `except Exception: pass` or `except: pass` | Catches and discards every error including KeyboardInterrupt |
|
||||
| `logging.exception(...)` in a logger with no handlers | "Logs" but actually writes nowhere |
|
||||
| `asyncio.create_task(coro)` without storing the task | Task GC'd before completion, exception swallowed |
|
||||
| `return x.get("key")` where key is missing | Returns None silently, caller often doesn't check |
|
||||
| `subprocess.run(..., check=False)` with ignored returncode | Non-zero exit treated as success |
|
||||
| Django `transaction.atomic()` inside a broader `except` | Rolls back silently |
|
||||
| `contextlib.suppress(Exception)` | Explicit silencer; easy to leave wider than intended |
|
||||
| `queue.get(block=False)` with `except queue.Empty: pass` | Polling that silently drops the work |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Remove breakpoint() / ipdb / pudb lines from source
|
||||
git diff | grep -E '(breakpoint\(\)|import ipdb|import pudb|import pdb; pdb\.set_trace)'
|
||||
# If the above has output, revert those files:
|
||||
git checkout <file>
|
||||
|
||||
# Unset the global breakpoint override
|
||||
unset PYTHONBREAKPOINT
|
||||
|
||||
# Kill any leftover debugpy listeners
|
||||
pkill -f 'debugpy' || true
|
||||
lsof -iTCP:5678 -sTCP:LISTEN -nP 2>/dev/null # confirm free
|
||||
```
|
||||
@@ -0,0 +1,234 @@
|
||||
# Rust Debugging
|
||||
|
||||
Covers `cargo`, `tokio`, panics, and the fact that you usually don't actually need a debugger — Rust's type system, `dbg!`, and logging cover 80% of sessions faster than gdb would.
|
||||
|
||||
---
|
||||
|
||||
## Environment detection (Phase 0)
|
||||
|
||||
```bash
|
||||
cargo --version
|
||||
rustc --version
|
||||
cat rust-toolchain.toml rust-toolchain 2>/dev/null
|
||||
cat Cargo.toml | head -30
|
||||
|
||||
# Debuggers available
|
||||
which rust-gdb
|
||||
which rust-lldb
|
||||
which lldb
|
||||
|
||||
# Async infrastructure
|
||||
grep -E '"(tokio|async-std|smol)"' Cargo.toml
|
||||
|
||||
# Profile flags
|
||||
grep -E '^\[profile' Cargo.toml
|
||||
```
|
||||
|
||||
**The default `cargo run` builds with `dev` profile** which includes debug symbols. `cargo run --release` strips them. For debugging, stay in dev unless the bug only manifests under optimization.
|
||||
|
||||
---
|
||||
|
||||
## The Rust debugging hierarchy (use in this order)
|
||||
|
||||
Rust's ecosystem has a specific order that's faster than reaching for gdb first:
|
||||
|
||||
1. **`dbg!(expr)` macro** — for a single value at a specific spot. Prints file:line + value, returns the value unchanged so you can inline it. Faster than a debugger for 60% of bugs.
|
||||
2. **`RUST_LOG=trace` with `tracing` / `env_logger`** — for flow and state across an operation. Zero code change in dev-time.
|
||||
3. **`RUST_BACKTRACE=1` / `=full`** — for crashes. Almost always sufficient; you rarely need a live debugger for a panic.
|
||||
4. **`rust-gdb` / `rust-lldb`** — when you need to pause execution and inspect memory, especially for unsafe code or FFI.
|
||||
5. **`tokio-console`** — for async deadlocks, stuck tasks, hot loops.
|
||||
6. **`cargo-expand`** — when a macro is doing something weird.
|
||||
|
||||
Reach for the lightest tool that answers the hypothesis.
|
||||
|
||||
---
|
||||
|
||||
## `dbg!` — the underused macro
|
||||
|
||||
```rust
|
||||
let x = 5;
|
||||
let y = dbg!(x * 2); // prints: [src/main.rs:2] x * 2 = 10
|
||||
```
|
||||
|
||||
Inside a complex expression:
|
||||
```rust
|
||||
let total = items.iter().filter(|i| i.active).map(|i| dbg!(i.cost)).sum::<u64>();
|
||||
```
|
||||
|
||||
Multiple values at once:
|
||||
```rust
|
||||
dbg!(&user, &request, elapsed.as_millis());
|
||||
```
|
||||
|
||||
`dbg!` writes to stderr, so it won't corrupt stdout-based pipelines. **Journal each `dbg!` you add**; revert at Phase 9.
|
||||
|
||||
---
|
||||
|
||||
## `RUST_LOG` for flow-level debugging
|
||||
|
||||
If the codebase uses `tracing` or `env_logger`:
|
||||
|
||||
```bash
|
||||
RUST_LOG=debug cargo run
|
||||
RUST_LOG=trace cargo run # very verbose
|
||||
RUST_LOG=my_crate=trace,hyper=info cargo run # per-module level
|
||||
RUST_LOG=debug,tokio=off cargo run # silence noisy crates
|
||||
```
|
||||
|
||||
For `tracing`-based apps, instrument with spans:
|
||||
```rust
|
||||
#[tracing::instrument]
|
||||
fn handle_request(req: &Request) -> Response { ... }
|
||||
```
|
||||
|
||||
This gives you structured per-call entry/exit logs with args and timing, zero additional code in the body.
|
||||
|
||||
---
|
||||
|
||||
## `RUST_BACKTRACE` for panics
|
||||
|
||||
```bash
|
||||
RUST_BACKTRACE=1 cargo run # backtrace on panic
|
||||
RUST_BACKTRACE=full cargo run # include libstd/tokio frames
|
||||
```
|
||||
|
||||
The panic itself usually tells you the file:line. The backtrace tells you how it got there. Between the two, most crash bugs are solved without a debugger.
|
||||
|
||||
---
|
||||
|
||||
## rust-gdb / rust-lldb
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
# Build with debug symbols (default dev profile)
|
||||
cargo build
|
||||
|
||||
# Attach gdb wrapper (applies Rust type pretty-printers)
|
||||
rust-gdb ./target/debug/my_binary
|
||||
# Or lldb:
|
||||
rust-lldb ./target/debug/my_binary
|
||||
|
||||
# With args
|
||||
rust-gdb --args ./target/debug/my_binary arg1 arg2
|
||||
|
||||
# Attach to running process
|
||||
rust-gdb -p $(pgrep my_binary)
|
||||
```
|
||||
|
||||
### Breakpoints
|
||||
|
||||
Rust symbols are mangled. Use either:
|
||||
|
||||
```
|
||||
(gdb) b main # main function
|
||||
(gdb) b my_crate::module::function # canonical path
|
||||
(gdb) b src/handler.rs:42 # file:line
|
||||
(gdb) info functions my_function # find mangled name
|
||||
```
|
||||
|
||||
### State inspection
|
||||
|
||||
```
|
||||
(gdb) p x # print value (uses Rust pretty-printer for Vec, Option, HashMap, etc.)
|
||||
(gdb) p *ptr # deref
|
||||
(gdb) info locals # all locals in current frame
|
||||
(gdb) info args # function args
|
||||
(gdb) bt # backtrace
|
||||
(gdb) frame <n> # switch to stack frame n
|
||||
(gdb) watch my_var # stop when my_var changes
|
||||
(gdb) rbreak regex # breakpoint all functions matching regex
|
||||
```
|
||||
|
||||
**Pair with pwndbg** for better layout on native bugs — see [tools/pwndbg.md](../tools/pwndbg.md). Pwndbg works with rust-gdb too.
|
||||
|
||||
---
|
||||
|
||||
## tokio-console — async task debugging
|
||||
|
||||
For tokio-based async apps, this is essential when tasks are stuck or leaking.
|
||||
|
||||
```bash
|
||||
# Add tokio-console instrumentation to the target binary
|
||||
# In Cargo.toml:
|
||||
# [dependencies]
|
||||
# console-subscriber = "0.2"
|
||||
|
||||
# In main.rs:
|
||||
# console_subscriber::init();
|
||||
|
||||
# Build with tokio_unstable:
|
||||
RUSTFLAGS="--cfg tokio_unstable" cargo run
|
||||
|
||||
# In another terminal:
|
||||
tokio-console # connects to default port 6669
|
||||
```
|
||||
|
||||
Shows live tasks, their state, wake counts, poll durations, parent tasks. The single fastest way to find "why is my async thing stuck".
|
||||
|
||||
---
|
||||
|
||||
## cargo-expand — when a macro is suspect
|
||||
|
||||
```bash
|
||||
cargo install cargo-expand
|
||||
cargo expand # expand all macros in the crate
|
||||
cargo expand my::module::path # scope to one item
|
||||
```
|
||||
|
||||
If you suspect a macro (especially `#[derive]`, `#[tokio::main]`, `#[async_trait]`) is generating code that doesn't match your mental model, this shows you exactly what the compiler sees.
|
||||
|
||||
---
|
||||
|
||||
## Release-build gotcha
|
||||
|
||||
```bash
|
||||
cargo build --release # no debug symbols by default
|
||||
```
|
||||
|
||||
If the bug only shows up in `--release`:
|
||||
|
||||
```toml
|
||||
# Cargo.toml
|
||||
[profile.release]
|
||||
debug = true # add symbols, keep optimizations
|
||||
```
|
||||
|
||||
Now `rust-gdb ./target/release/my_binary` works on release builds. This is required when optimization-enabled codegen bugs (inlining, LLVM folding) are suspected.
|
||||
|
||||
---
|
||||
|
||||
## Silent-failure patterns in Rust
|
||||
|
||||
| Pattern | Why it's silent |
|
||||
|---|---|
|
||||
| `.unwrap_or_default()` | Masks errors as the zero value |
|
||||
| `.unwrap_or(fallback)` | Same, with a specific fallback |
|
||||
| `let _ = fallible_operation()` | Explicitly discards the Result, no compiler warning |
|
||||
| `if let Ok(x) = ... { use(x); } // no else` | Silent on Err |
|
||||
| `.ok()` chaining | Converts Result to Option, throwing the error away |
|
||||
| Panic inside a tokio task not `.await`ed | Task dies silently; runtime usually logs but it's quiet if logs are off |
|
||||
| `eprintln!` that goes to a redirected-null stderr | Looks like nothing happened |
|
||||
| `Drop` impl that panics under specific condition | Double-panic aborts process silently if no logging configured |
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Revert dbg! macro additions
|
||||
git diff | grep -E 'dbg!\('
|
||||
# For any file with dbg! additions:
|
||||
git checkout <file>
|
||||
|
||||
# Unset env vars
|
||||
unset RUST_LOG RUST_BACKTRACE
|
||||
|
||||
# Kill any rust-gdb/lldb sessions
|
||||
pkill -f 'rust-gdb' || true
|
||||
pkill -f 'rust-lldb' || true
|
||||
pkill -f '^lldb ' || true
|
||||
|
||||
# tokio-console binds 6669 by default
|
||||
lsof -iTCP:6669 -sTCP:LISTEN -nP 2>/dev/null
|
||||
```
|
||||
@@ -0,0 +1,212 @@
|
||||
# Ghidra — Decompile Binaries Into Readable C
|
||||
|
||||
**https://github.com/NationalSecurityAgency/ghidra**
|
||||
|
||||
Ghidra is the NSA's open-source reverse-engineering suite. Its defining feature is a **decompiler** that turns machine code back into readable C. For any binary you don't have source for, this is the correct starting point — not `strings`, not hex-staring, not `objdump -d`.
|
||||
|
||||
**Use Ghidra when**: third-party closed-source libs, malware analysis, vendored binaries whose behavior contradicts docs, CTF challenges, firmware, any time you need to read compiled code.
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
brew install --cask ghidra
|
||||
# OR download the release ZIP from the repo and ./ghidraRun
|
||||
|
||||
# Linux
|
||||
# Download from https://github.com/NationalSecurityAgency/ghidra/releases
|
||||
# Requires JDK 21+
|
||||
./ghidraRun
|
||||
|
||||
# Dependency
|
||||
java -version # must be 21+
|
||||
```
|
||||
|
||||
Ghidra is a Java Swing app. Looks dated, works well.
|
||||
|
||||
---
|
||||
|
||||
## First-time workflow (memorize this — it's not obvious)
|
||||
|
||||
1. **Start Ghidra**: `ghidraRun`
|
||||
2. **Create a project**: File → New Project → Non-Shared → name it `debug-<binary-name>` (journal this path so you can rm it at Phase 9 if disposable).
|
||||
3. **Import the binary**: File → Import File → pick your target. Accept default format detection.
|
||||
4. **Double-click the imported binary** in the project listing. Ghidra asks to analyze it — say **yes**, accept defaults for the first pass. This takes anywhere from seconds (small binary) to tens of minutes (large binary).
|
||||
5. **Once analysis completes**, you're in the CodeBrowser view.
|
||||
|
||||
Two panels you'll use 95% of the time:
|
||||
|
||||
- **Listing** (middle) — the disassembly with Ghidra's inferred labels/types.
|
||||
- **Decompiler** (right) — the reconstructed C. The real value.
|
||||
|
||||
---
|
||||
|
||||
## Finding the right function fast
|
||||
|
||||
Don't try to read the whole binary. Use these to narrow:
|
||||
|
||||
### Symbol Tree (left panel)
|
||||
|
||||
- `Functions` — all detected functions. Stripped binaries show `FUN_00401234` (address-named); unstripped show actual names.
|
||||
- `Imports` — dynamically-linked functions. Great for "does this binary call `system()`, `strcpy`, `curl_easy_perform`?"
|
||||
- `Exports` — if it's a library.
|
||||
|
||||
Click to jump. The Decompiler updates instantly.
|
||||
|
||||
### String search
|
||||
|
||||
```
|
||||
Search → For Strings
|
||||
```
|
||||
|
||||
Produces a list of all strings. Right-click a string → `References` → `Show References to Address`. Jumps to code that references it. **This is how you find which function handles the error message you saw at runtime.**
|
||||
|
||||
### Memory search for bytes
|
||||
|
||||
```
|
||||
Search → Memory
|
||||
```
|
||||
|
||||
Search hex or text. Useful for known magic bytes, file-format signatures, constants.
|
||||
|
||||
### Cross-references (XREF)
|
||||
|
||||
Right-click any function / address → `References → Find References to`. Shows every place that calls it. Walk the call graph backward from interesting functions.
|
||||
|
||||
---
|
||||
|
||||
## Making the decompiler's output readable
|
||||
|
||||
Ghidra's decompiler is good but needs hints. These three actions dramatically improve its output:
|
||||
|
||||
### 1. Rename variables
|
||||
|
||||
Click a variable in the Decompiler view → press `L` → type a better name. Ghidra propagates the rename across all uses.
|
||||
|
||||
### 2. Set types
|
||||
|
||||
A variable that looks like `undefined4` or `void *` is unhelpful. Click it → press `Ctrl+L` → set type (e.g. `int`, `char *`, `struct my_header *`).
|
||||
|
||||
For pointers to structs from headers you have, use:
|
||||
```
|
||||
File → Parse C Source → paste header file → auto-creates struct types
|
||||
```
|
||||
|
||||
Then assign the struct type to the pointer. Ghidra resolves field accesses immediately.
|
||||
|
||||
### 3. Retype function signatures
|
||||
|
||||
Click the function name in the Decompiler → press `F` (Edit Function Signature) → set return type + argument types. This propagates through callers.
|
||||
|
||||
Do these three actions on the 3-5 most relevant functions and the decompiler output becomes near-source-readable.
|
||||
|
||||
---
|
||||
|
||||
## Patterns for specific bug types
|
||||
|
||||
### Looking for integer overflow / buffer overflow
|
||||
|
||||
```
|
||||
Listing: look for
|
||||
- LEA → CMP patterns on sizes
|
||||
- memcpy/strcpy/sprintf with non-constant sizes
|
||||
Decompiler: look for
|
||||
- arithmetic on size_t without bounds check
|
||||
- `+ user_input` in a length calculation
|
||||
```
|
||||
|
||||
### Looking for a missing auth check
|
||||
|
||||
Navigate from the handler entry (found via Strings or Imports) and check the control flow:
|
||||
|
||||
```
|
||||
Decompiler: does the function return early / jump to error handler when some flag is not set?
|
||||
If the check is absent, that's the bug.
|
||||
```
|
||||
|
||||
### Looking for hardcoded URLs / keys / paths
|
||||
|
||||
```
|
||||
Search → For Strings → filter `http:` / `https:` / `/etc/` / `bearer ` / `api_key`
|
||||
```
|
||||
|
||||
### Looking for dispatch / plugin loading
|
||||
|
||||
```
|
||||
Imports → dlopen, LoadLibrary, dlsym, GetProcAddress
|
||||
```
|
||||
|
||||
The strings referenced near those calls are often plugin names.
|
||||
|
||||
---
|
||||
|
||||
## Scripting (headless Ghidra)
|
||||
|
||||
When you need to automate analysis across many binaries, or repeat a workflow:
|
||||
|
||||
```bash
|
||||
# Headless analyzer
|
||||
$GHIDRA_INSTALL_DIR/support/analyzeHeadless \
|
||||
<project-dir> <project-name> \
|
||||
-import <binary> \
|
||||
-postScript <script.py or script.java>
|
||||
```
|
||||
|
||||
Ghidra supports Python 3 scripts (via Jython-compatible API) and Java. Useful scripts:
|
||||
|
||||
- Dump all function signatures to JSON
|
||||
- Find all calls to `system()` with constant arguments
|
||||
- Auto-rename FUN_xxx based on heuristics (string refs, call patterns)
|
||||
|
||||
The community has a large collection: https://github.com/NationalSecurityAgency/ghidra/tree/master/Ghidra/Features/Base/ghidra_scripts
|
||||
|
||||
---
|
||||
|
||||
## Bookmarks + Notes
|
||||
|
||||
Ghidra has built-in bookmarks and comments. Use them as your journal inside the project:
|
||||
|
||||
- Right-click an address → `Set Bookmark` → tag as `Note`. Attach a description.
|
||||
- Right-click → `Comments → Set EOL Comment` (shows up inline in decompiler).
|
||||
|
||||
Treat these as part of the journal. If you end up promoting this Ghidra project (keeping it after the debug session), the comments become durable documentation.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Large binaries need more Java heap.** Edit `support/launch.properties` and bump `VMARGS=-Xmx8G` (default is often 2G, too small).
|
||||
- **Save often.** Ghidra's autosave is not instant; a crash loses uncommitted analysis.
|
||||
- **Decompiler has timeouts.** For complex functions it may give up and print `/* WARNING: ... */`. Increase timeout in `Edit → Tool Options → Decompiler`.
|
||||
- **Archs beyond x86/ARM/MIPS** sometimes need Sleigh processor module tweaks. Rare but possible.
|
||||
- **Signed vs unsigned decompilation** is frequently wrong. Manually retype when integer behavior matters.
|
||||
|
||||
---
|
||||
|
||||
## When Ghidra is NOT the right tool
|
||||
|
||||
- You have the source. Go read it.
|
||||
- Bug is in your own recently-compiled binary. Rebuild with `-g` and use gdb/pwndbg (see [pwndbg.md](pwndbg.md)).
|
||||
- You just need to know which libs a binary links. `ldd` / `otool -L` / `readelf -d` are faster.
|
||||
- You just need strings. `strings -n 8` is faster.
|
||||
|
||||
Ghidra is the right tool when you need to **read the logic** of a binary you don't have source for.
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# If you created a scratch project just for this debug session, journal path and remove:
|
||||
# (the journal should have the exact path)
|
||||
# Example:
|
||||
ls ~/ghidra-projects/ 2>/dev/null
|
||||
# rm -rf ~/ghidra-projects/debug-<binary-name>
|
||||
|
||||
# If you promoted the project (kept it), it's not cleanup — note it in the final summary to the user
|
||||
|
||||
# Kill any running Ghidra headless processes
|
||||
pkill -f 'analyzeHeadless' || true
|
||||
```
|
||||
@@ -0,0 +1,194 @@
|
||||
# Playwright CLI — Browser QA That Actually Drives a Browser
|
||||
|
||||
**https://playwright.dev/ · https://github.com/microsoft/playwright**
|
||||
|
||||
For any browser-served web UI bug, this is the correct tool. Not curl. Not imagination. Not a headless HTTP library. A real browser with a real rendering engine, real JS execution, real cookies, real service workers, real viewport.
|
||||
|
||||
**In Phase 8 Manual QA for browser products, using Playwright is not optional.** Curl cannot catch: CSS that breaks at specific viewport widths, hydration mismatches, client-side router bugs, cookie/session interactions, service-worker caching, JS-triggered navigations. All of those are common bug classes. Drive a browser.
|
||||
|
||||
> Note: `microsoft/playwright-cli` is the legacy repo; the current tooling lives in `@playwright/test` (npm) and `playwright` (pip), which include the `playwright` CLI. Use those — the legacy `playwright-cli` package is deprecated.
|
||||
|
||||
---
|
||||
|
||||
## When to reach for Playwright
|
||||
|
||||
| Bug symptom | Use Playwright? |
|
||||
|---|---|
|
||||
| Form submit produces wrong result | ✅ — Playwright drives the form exactly as a user does |
|
||||
| Page blank in prod, fine locally | ✅ — hydration/env differences need a real browser |
|
||||
| CSS looks wrong at a specific width | ✅ — use `--viewport-size` |
|
||||
| Click doesn't fire / wrong handler | ✅ — Playwright fires real DOM events |
|
||||
| Flash of unstyled content / loading glitch | ✅ — use trace viewer to see frames |
|
||||
| API returns wrong data | ❌ — use curl, this isn't a browser bug |
|
||||
| Backend returns wrong status code | ❌ — use curl |
|
||||
| Client hits a URL that returns 500 | ✅ but also ❌ — Playwright shows the call + response + failure effect on UI |
|
||||
|
||||
---
|
||||
|
||||
## Install (per-project)
|
||||
|
||||
Playwright installs browser binaries separately from the npm package.
|
||||
|
||||
```bash
|
||||
# In the project
|
||||
npm init playwright@latest # interactive; picks TS/JS + browsers + config
|
||||
# Or if Playwright is already a dep:
|
||||
npx playwright install # downloads browsers
|
||||
npx playwright install chromium # just chromium
|
||||
npx playwright install --with-deps # also installs OS deps (Linux)
|
||||
```
|
||||
|
||||
Python:
|
||||
```bash
|
||||
pip install playwright
|
||||
playwright install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The four things you'll actually use
|
||||
|
||||
### 1. `codegen` — record a session, generate the script
|
||||
|
||||
The fastest way to create a repro. Opens a real browser; your clicks / typing become a Playwright script you can paste into a test.
|
||||
|
||||
```bash
|
||||
npx playwright codegen https://your-app.local
|
||||
npx playwright codegen --viewport-size=375,667 https://your-app.local # iPhone SE size
|
||||
npx playwright codegen --device="iPhone 14" https://your-app.local
|
||||
```
|
||||
|
||||
Click / type / navigate in the browser; watch the script build in the side panel. Copy the generated script into your journal as the repro for Phase 8.
|
||||
|
||||
### 2. A one-shot Playwright script — reproduce + capture
|
||||
|
||||
Usually the Phase 8 QA artifact. Save to `/tmp/debug-repro.spec.ts` (journal it):
|
||||
|
||||
```ts
|
||||
// /tmp/debug-repro.spec.ts
|
||||
import { test, expect } from '@playwright/test';
|
||||
|
||||
test('refinement chat shows non-empty response when env var set', async ({ page }) => {
|
||||
await page.goto('http://localhost:3000/chat');
|
||||
await page.fill('textarea[name="message"]', 'Add a logging step');
|
||||
await page.click('button[type=submit]');
|
||||
|
||||
// Wait for the response to appear (not just the spinner to disappear)
|
||||
const response = page.locator('[data-testid="assistant-reply"]');
|
||||
await expect(response).toBeVisible({ timeout: 30_000 });
|
||||
await expect(response).not.toBeEmpty();
|
||||
|
||||
// Capture evidence
|
||||
await page.screenshot({ path: '/tmp/debug-after-fix.png', fullPage: true });
|
||||
console.log(await response.textContent());
|
||||
});
|
||||
```
|
||||
|
||||
Run it with tracing enabled for rich post-mortem:
|
||||
|
||||
```bash
|
||||
npx playwright test /tmp/debug-repro.spec.ts --trace on --headed
|
||||
```
|
||||
|
||||
### 3. `PWDEBUG=1` — step through the script with Playwright Inspector
|
||||
|
||||
```bash
|
||||
PWDEBUG=1 npx playwright test /tmp/debug-repro.spec.ts
|
||||
```
|
||||
|
||||
Opens the Playwright Inspector alongside the browser. You can step through Playwright actions, see the DOM state at each step, and edit selectors on the fly.
|
||||
|
||||
Use this when the script doesn't reproduce cleanly and you need to watch it run.
|
||||
|
||||
### 4. `show-trace` — post-mortem on a failed run
|
||||
|
||||
```bash
|
||||
npx playwright show-trace trace.zip
|
||||
# or from the test-results dir:
|
||||
npx playwright show-trace test-results/<test-name>/trace.zip
|
||||
```
|
||||
|
||||
Scrubs through a recorded session: timeline, DOM snapshot at each action, network, console, source. When a test failed on CI but passed locally, this is the single best artifact.
|
||||
|
||||
---
|
||||
|
||||
## Headless vs headed during debugging
|
||||
|
||||
Always add `--headed` when debugging. Headless browsers sometimes behave subtly differently (font rendering, viewport, media permissions). For QA evidence, run headed and screenshot.
|
||||
|
||||
```bash
|
||||
npx playwright test --headed
|
||||
npx playwright test --headed --project=chromium # pin the browser
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Catching the silent-failure patterns Playwright is good at
|
||||
|
||||
```ts
|
||||
// Toast that flashes and disappears
|
||||
page.on('console', msg => console.log('[browser console]', msg.type(), msg.text()));
|
||||
|
||||
// Unhandled page errors (uncaught exceptions in the page JS)
|
||||
page.on('pageerror', err => console.error('[page error]', err));
|
||||
|
||||
// Network failures — e.g., backend returned 500 but UI shows nothing
|
||||
page.on('response', async resp => {
|
||||
if (!resp.ok()) {
|
||||
console.warn(`[network ${resp.status()}] ${resp.url()} — ${await resp.text()}`);
|
||||
}
|
||||
});
|
||||
|
||||
// Request that never came back
|
||||
page.on('requestfailed', req => {
|
||||
console.error('[request failed]', req.url(), req.failure()?.errorText);
|
||||
});
|
||||
```
|
||||
|
||||
Add these listeners to the top of the debug script. They surface a lot of the "UI showed nothing" class of bug.
|
||||
|
||||
---
|
||||
|
||||
## Viewport and device emulation
|
||||
|
||||
CSS bugs that only appear at specific sizes, or layout bugs on mobile:
|
||||
|
||||
```ts
|
||||
// At test level
|
||||
test.use({ viewport: { width: 375, height: 667 } });
|
||||
|
||||
// Per-page
|
||||
await page.setViewportSize({ width: 375, height: 667 });
|
||||
|
||||
// Predefined devices
|
||||
import { devices } from '@playwright/test';
|
||||
test.use({ ...devices['iPhone 14'] });
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Wait for state, not for time.** `await page.waitForTimeout(2000)` is flaky. Use `await expect(locator).toBeVisible()` or `page.waitForResponse(urlPattern)`.
|
||||
- **Stale selectors re-resolve.** Playwright's locators re-find the element on each action, unlike Puppeteer's handles. Don't over-think it.
|
||||
- **Service workers persist across test runs in headed mode.** If you see cached behavior from a previous run, add `await context.clearCookies()` + clear storage before the test.
|
||||
- **Installing on CI requires `--with-deps`** on Linux images that lack the browser's shared-library deps.
|
||||
- **Parallel tests share a browser process by default**; if one test polls a debugger port, others may interfere. Use `workers: 1` for debugging.
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Remove trace files from debug runs
|
||||
rm -rf playwright-report/ test-results/ trace.zip
|
||||
|
||||
# Remove debug spec files from /tmp
|
||||
rm -f /tmp/debug-*.spec.ts
|
||||
|
||||
# Remove screenshot captures
|
||||
rm -f /tmp/debug-*.png
|
||||
|
||||
# If you installed browsers just for this session (rare):
|
||||
# Don't remove them — they're useful for future sessions. They live in ~/Library/Caches/ms-playwright (macOS) or ~/.cache/ms-playwright (Linux).
|
||||
```
|
||||
@@ -0,0 +1,263 @@
|
||||
# pwndbg — GDB With the Useful Views Always On
|
||||
|
||||
**https://github.com/pwndbg/pwndbg**
|
||||
|
||||
pwndbg is a GDB plugin that turns GDB into something humans can actually use for binary debugging. It's strictly a superset of plain GDB — every vanilla GDB command still works, and pwndbg adds views and commands that make you productive.
|
||||
|
||||
**If you'd reach for plain `gdb`, reach for pwndbg instead.** The only reason not to is if pwndbg isn't installed on the machine, and that's a 2-minute fix.
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
brew install pwndbg
|
||||
# Or from source:
|
||||
git clone https://github.com/pwndbg/pwndbg
|
||||
cd pwndbg && ./setup.sh
|
||||
|
||||
# Linux
|
||||
# Most distros: apt/dnf/pacman install pwndbg (check availability)
|
||||
# Or the same git + ./setup.sh
|
||||
|
||||
# Verify
|
||||
gdb --version
|
||||
gdb ./any-binary
|
||||
# At gdb prompt, you should see pwndbg banner + colorful context view
|
||||
```
|
||||
|
||||
Once installed, pwndbg auto-loads every time you start `gdb`. You don't source anything manually.
|
||||
|
||||
---
|
||||
|
||||
## The `context` view — the one feature that changes everything
|
||||
|
||||
Plain GDB: you run `info registers`, then `bt`, then `x/10xw $rsp`, then `disas`. Four commands to see what's going on.
|
||||
|
||||
pwndbg: `context` (or it auto-shows at every break). One command. Everything on screen:
|
||||
|
||||
```
|
||||
──── registers ────
|
||||
RAX 0x0
|
||||
RBX 0x7ffffffde158
|
||||
RCX 0x7fffff7abf10
|
||||
...
|
||||
──── disasm ────
|
||||
► 0x401234 mov rdi, rax
|
||||
0x401237 call 0x401190
|
||||
...
|
||||
──── stack ────
|
||||
00:0000│ rsp 0x7ffffffde0a0 → 0x7fffff7c4000
|
||||
01:0008│ 0x7ffffffde0a8 → 0x0
|
||||
...
|
||||
──── backtrace ────
|
||||
► f 0 0x401234 parse_input+0x3c
|
||||
f 1 0x401180 main+0x120
|
||||
f 2 0x7fffff7a5083 __libc_start_main+0xf3
|
||||
```
|
||||
|
||||
You always know where you are, what the CPU state is, what's on the stack, and how you got here. This is why pwndbg is the default.
|
||||
|
||||
---
|
||||
|
||||
## Launch recipes
|
||||
|
||||
```bash
|
||||
# Debug an existing binary
|
||||
gdb ./target
|
||||
|
||||
# With args
|
||||
gdb --args ./target arg1 arg2
|
||||
|
||||
# Attach to a running process
|
||||
gdb -p $(pgrep target)
|
||||
|
||||
# With a core dump
|
||||
gdb ./target ./core
|
||||
|
||||
# Headless / remote (for automation or IDE attach)
|
||||
gdbserver :2345 ./target # on the target box
|
||||
gdb ./target # on your box
|
||||
(gdb) target remote <host>:2345
|
||||
```
|
||||
|
||||
At the pwndbg prompt:
|
||||
|
||||
---
|
||||
|
||||
## Essential commands (pwndbg additions)
|
||||
|
||||
### Layout / view
|
||||
|
||||
```
|
||||
context # reprint the context view (usually auto)
|
||||
context regs stack # only show registers + stack sections
|
||||
tel $rsp 20 # telescope — walk pointers at $rsp for 20 slots (KEY COMMAND)
|
||||
tel $rdi 10 # walk pointers at $rdi (e.g. to dump a struct)
|
||||
stack 20 # 20 entries of stack
|
||||
vmmap # virtual memory map of the process
|
||||
```
|
||||
|
||||
**`telescope` is pwndbg's killer command.** Given an address, it walks pointers recursively:
|
||||
```
|
||||
00:0000│ 0x7ffd... → 0x601010 (heap) → 0x2a (unknown, i.e. a number 42)
|
||||
01:0008│ 0x7ffd... → 0x7fff... (stack) → 'hello world'
|
||||
```
|
||||
This single view resolves 80% of "what is at this address" questions.
|
||||
|
||||
### Heap debugging
|
||||
|
||||
```
|
||||
heap # overview of chunks
|
||||
bins # tcache / fastbin / unsorted / smallbin / largebin state
|
||||
malloc_chunk <addr> # inspect a specific chunk
|
||||
find_fake_fast <addr> # (exploit context) find fake-fast overlap candidates
|
||||
vis_heap_chunks # visualize heap layout
|
||||
```
|
||||
|
||||
For use-after-free / double-free / heap overflow hypotheses, `heap` + `bins` is usually sufficient to see the corruption.
|
||||
|
||||
### Exploitation-adjacent (useful for bug understanding too)
|
||||
|
||||
```
|
||||
checksec # NX, PIE, RELRO, canary status
|
||||
rop --grep 'pop rdi' # find ROP gadgets
|
||||
nx # step over (aliased nicely)
|
||||
ni # step over single instruction
|
||||
si # step into single instruction
|
||||
```
|
||||
|
||||
### Search
|
||||
|
||||
```
|
||||
search -t byte 0x41 # find byte 0x41 anywhere in memory
|
||||
search -t string "admin" # find string
|
||||
search -p <addr> # find pointers to <addr>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Standard GDB commands still work
|
||||
|
||||
pwndbg doesn't replace GDB; it augments it. Everything you know still works:
|
||||
|
||||
```
|
||||
break main # breakpoint at function
|
||||
b *0x401234 # breakpoint at address
|
||||
b file.c:42 # breakpoint at file:line
|
||||
c # continue
|
||||
n # next (source-level step over)
|
||||
s # step (source-level step into)
|
||||
finish # step out
|
||||
info breakpoints # list breakpoints
|
||||
delete <n> # delete breakpoint
|
||||
watch <var> # break on write to variable
|
||||
rwatch <var> # break on read
|
||||
awatch <var> # break on access
|
||||
p <expr> # print expression
|
||||
p/x <expr> # print in hex
|
||||
x/20xw <addr> # examine 20 words as hex
|
||||
bt # backtrace
|
||||
frame <n> # switch frame
|
||||
info registers # registers (but `context` is better)
|
||||
disassemble <func> # disasm a function
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Python scripting inside GDB
|
||||
|
||||
pwndbg exposes a full Python API. Useful for automating observations across many breakpoints:
|
||||
|
||||
```python
|
||||
(gdb) python
|
||||
import gdb
|
||||
def on_break():
|
||||
frame = gdb.selected_frame()
|
||||
pc = frame.read_register('pc')
|
||||
print(f'hit at {hex(int(pc))}')
|
||||
# Dump args, locals, anything
|
||||
end
|
||||
```
|
||||
|
||||
Or scripted runs from outside:
|
||||
```bash
|
||||
gdb -batch -ex 'source script.gdb' -ex 'run' ./target
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common workflows by bug type
|
||||
|
||||
### Segfault / crash
|
||||
|
||||
```bash
|
||||
gdb ./target
|
||||
(gdb) run <args>
|
||||
# ... crash ...
|
||||
(gdb) context # see the crash site
|
||||
(gdb) bt # how did we get here?
|
||||
(gdb) info registers # what state
|
||||
(gdb) tel $rsp 20 # what's on the stack
|
||||
```
|
||||
|
||||
### "Function returns wrong value"
|
||||
|
||||
```bash
|
||||
gdb ./target
|
||||
(gdb) break <function>
|
||||
(gdb) run <args>
|
||||
# At breakpoint:
|
||||
(gdb) finish # let it run to the return
|
||||
# pwndbg shows RAX (return value) in context
|
||||
```
|
||||
|
||||
### "Variable has unexpected value at point X"
|
||||
|
||||
```bash
|
||||
gdb ./target
|
||||
(gdb) break <point-X>
|
||||
(gdb) run
|
||||
# At breakpoint:
|
||||
(gdb) p <var> # its value
|
||||
(gdb) watch <var> # set a watchpoint — break when it changes
|
||||
(gdb) c # continue; next stop is where it was modified
|
||||
```
|
||||
|
||||
### "Memory corruption / heap bug"
|
||||
|
||||
```bash
|
||||
gdb ./target
|
||||
(gdb) run
|
||||
# Crash at free():
|
||||
(gdb) heap # heap state
|
||||
(gdb) bins # bin state — often shows corruption here
|
||||
(gdb) vis_heap_chunks # visualize
|
||||
(gdb) malloc_chunk <suspicious-addr>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **`bt` looks weird on stripped binaries** — function names become offsets. Use Ghidra's function labels to map back (see [ghidra.md](ghidra.md)).
|
||||
- **PIE binaries have randomized base addresses.** Addresses you see in Ghidra are unslid; addresses in pwndbg are slid. The `vmmap` command shows the base, and pwndbg's `piebase` command gives you the offset.
|
||||
- **Optimized builds inline functions.** You'll set a breakpoint on `my_function` and it won't hit because the function was inlined. Either disable optimizations or break on callers.
|
||||
- **Stack canaries trigger `__stack_chk_fail`.** If you see that in a backtrace, the bug caused a stack-smash; look one frame up.
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Kill gdb / pwndbg sessions
|
||||
pkill -f 'gdb' || true
|
||||
pkill -f 'gdbserver' || true
|
||||
|
||||
# Remove core dumps generated during session
|
||||
rm -f ./core ./core.* ~/core.*
|
||||
|
||||
# Remove any scripted GDB files
|
||||
rm -f /tmp/debug-*.gdb
|
||||
```
|
||||
@@ -0,0 +1,265 @@
|
||||
# pwntools — Scripted Binary / Network Interaction
|
||||
|
||||
**https://docs.pwntools.com/en/stable/ · https://github.com/Gallopsled/pwntools**
|
||||
|
||||
pwntools is a Python framework for building reproducible interactions with binaries and network services. Originally built for CTF exploitation, it's the correct tool for any situation where you need:
|
||||
|
||||
- A crafted input sent to a binary or network service, repeatably
|
||||
- A "failing test" equivalent for a bug that only manifests with specific byte-level input
|
||||
- A fuzz harness
|
||||
- An exploit PoC
|
||||
- Anything where you're tempted to use `echo ... | ./binary` but need more control than shell allows
|
||||
|
||||
**Use pwntools for Phase 5 reproduction of binary bugs and Phase 7 tests against binaries.**
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
pip install pwntools
|
||||
# Or in a venv:
|
||||
python -m venv .venv && source .venv/bin/activate
|
||||
pip install pwntools
|
||||
|
||||
# Verify
|
||||
python -c 'from pwn import *; print("ok")'
|
||||
```
|
||||
|
||||
On some Linux distros you may need build deps: `apt install python3-dev libssl-dev`.
|
||||
|
||||
---
|
||||
|
||||
## The core API in five idioms
|
||||
|
||||
### 1. Process / Remote — the same interface
|
||||
|
||||
```python
|
||||
from pwn import *
|
||||
|
||||
# Local process
|
||||
p = process('./target')
|
||||
|
||||
# Remote service
|
||||
p = remote('example.com', 1337)
|
||||
|
||||
# SSH (tunnel to a remote process)
|
||||
shell = ssh('user', 'host', password='...')
|
||||
p = shell.process('./target', cwd='/tmp')
|
||||
|
||||
# Same methods on all of the above — this is the value proposition
|
||||
```
|
||||
|
||||
### 2. I/O — the only five methods you need
|
||||
|
||||
```python
|
||||
p.send(b'data') # send bytes
|
||||
p.sendline(b'data') # send bytes + \n
|
||||
p.recv(n) # receive up to n bytes
|
||||
p.recvuntil(b'> ') # receive until pattern (blocks)
|
||||
p.recvline() # receive until \n
|
||||
p.interactive() # hand control to your terminal (for manual exploration)
|
||||
|
||||
# Combined
|
||||
p.sendlineafter(b'prompt> ', b'payload')
|
||||
p.sendafter(b'key:', key)
|
||||
```
|
||||
|
||||
Timeouts:
|
||||
```python
|
||||
try:
|
||||
data = p.recvuntil(b'done', timeout=5)
|
||||
except pwnlib.exception.EOFError:
|
||||
print('process died')
|
||||
except TimeoutError:
|
||||
print('no response in 5s')
|
||||
```
|
||||
|
||||
### 3. context — set arch/OS once, tools align
|
||||
|
||||
```python
|
||||
context.binary = elf = ELF('./target') # auto-sets arch/os/endianness
|
||||
# or explicitly:
|
||||
context.update(arch='amd64', os='linux', endian='little', bits=64)
|
||||
```
|
||||
|
||||
After setting context, helpers like `asm()`, `disasm()`, `cyclic()`, and `ROP()` produce correct output for that target automatically.
|
||||
|
||||
### 4. ELF — parse without reverse-engineering by hand
|
||||
|
||||
```python
|
||||
elf = ELF('./target')
|
||||
|
||||
elf.symbols['main'] # address of main
|
||||
elf.plt['printf'] # address in PLT (dynamic linkage)
|
||||
elf.got['printf'] # GOT entry
|
||||
elf.address = 0x555555554000 # set base for PIE binaries
|
||||
elf.search(b'/bin/sh') # find string or bytes in the binary
|
||||
elf.functions['main'].address # same as elf.symbols['main']
|
||||
list(elf.functions)[:10] # first 10 function names
|
||||
```
|
||||
|
||||
For the libc that's linked:
|
||||
```python
|
||||
libc = ELF('/lib/x86_64-linux-gnu/libc.so.6')
|
||||
libc.symbols['system']
|
||||
```
|
||||
|
||||
### 5. cyclic — find offsets without counting
|
||||
|
||||
For "where exactly does user input reach this variable" bugs:
|
||||
|
||||
```python
|
||||
p = process('./target')
|
||||
p.sendline(cyclic(256)) # send a De Bruijn pattern
|
||||
# Crash occurs; note the crash value (e.g. RIP = 0x6161616c)
|
||||
offset = cyclic_find(0x6161616c) # returns 12 (or wherever in the pattern)
|
||||
# Now you know: byte 12 of your input lands at RIP
|
||||
```
|
||||
|
||||
Saves an hour of "pad by N bytes then check" iteration.
|
||||
|
||||
---
|
||||
|
||||
## Logging during debug
|
||||
|
||||
pwntools logs output by default. Configure level in the script:
|
||||
|
||||
```python
|
||||
context.log_level = 'debug' # very verbose — shows sent/received bytes
|
||||
context.log_level = 'info' # default
|
||||
context.log_level = 'warning' # quiet
|
||||
```
|
||||
|
||||
For long scripts, log milestones:
|
||||
|
||||
```python
|
||||
log.info('Connected to target')
|
||||
log.success('Bypassed the check')
|
||||
log.failure('Canary corrupted')
|
||||
log.progress('brute-forcing').status('attempt %d' % i)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Typical debug-session patterns
|
||||
|
||||
### Reproduce a crash with a specific input
|
||||
|
||||
```python
|
||||
# /tmp/debug-repro.py
|
||||
from pwn import *
|
||||
context.binary = './target'
|
||||
|
||||
p = process('./target')
|
||||
p.sendlineafter(b'> ', b'<bad input that crashes>')
|
||||
p.wait()
|
||||
# If it crashed, p.poll() returns non-zero
|
||||
assert p.poll() is not None and p.poll() != 0, 'expected crash, got clean exit'
|
||||
log.success(f'confirmed crash (exit {p.poll()})')
|
||||
```
|
||||
|
||||
Journal this script path. Run it as your "red test":
|
||||
|
||||
```bash
|
||||
python /tmp/debug-repro.py
|
||||
```
|
||||
|
||||
### Fuzz harness for a suspected input class
|
||||
|
||||
```python
|
||||
# /tmp/debug-fuzz.py
|
||||
from pwn import *
|
||||
import random
|
||||
|
||||
context.binary = './target'
|
||||
context.log_level = 'warning' # keep quiet in the loop
|
||||
|
||||
crashes = []
|
||||
for i in range(1000):
|
||||
payload = bytes(random.randint(0, 255) for _ in range(random.randint(1, 100)))
|
||||
p = process('./target')
|
||||
p.sendline(payload)
|
||||
p.wait()
|
||||
if p.poll() is not None and p.poll() < 0: # crashed by signal
|
||||
crashes.append((payload, p.poll()))
|
||||
log.success(f'iter {i}: crash sig={-p.poll()}')
|
||||
|
||||
open('/tmp/debug-crashes.txt', 'w').write(repr(crashes))
|
||||
log.info(f'found {len(crashes)} crashes')
|
||||
```
|
||||
|
||||
### Automated exploit harness (CTF or self-testing a known CVE)
|
||||
|
||||
```python
|
||||
from pwn import *
|
||||
context.binary = elf = ELF('./target')
|
||||
libc = elf.libc or ELF('/lib/x86_64-linux-gnu/libc.so.6')
|
||||
|
||||
p = process('./target')
|
||||
|
||||
# Leak
|
||||
p.sendline(b'A' * 64 + p64(elf.plt['puts']) + p64(elf.symbols['main']) + p64(elf.got['puts']))
|
||||
leak = u64(p.recv(6).ljust(8, b'\x00'))
|
||||
libc.address = leak - libc.symbols['puts']
|
||||
log.success(f'libc base: {hex(libc.address)}')
|
||||
|
||||
# Exploit
|
||||
rop = ROP(libc)
|
||||
rop.system(next(libc.search(b'/bin/sh')))
|
||||
p.sendline(b'A' * 64 + rop.chain())
|
||||
|
||||
p.interactive()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration with gdb / pwndbg
|
||||
|
||||
pwntools can launch your process under gdb:
|
||||
|
||||
```python
|
||||
p = gdb.debug('./target', gdbscript='''
|
||||
break main
|
||||
continue
|
||||
''')
|
||||
```
|
||||
|
||||
Or attach to a running pwntools-launched process:
|
||||
|
||||
```python
|
||||
p = process('./target')
|
||||
gdb.attach(p, gdbscript='break *0x401234')
|
||||
# continues in a new terminal window with gdb attached
|
||||
p.sendline(b'trigger input')
|
||||
```
|
||||
|
||||
This is the best way to debug a specific crash repeatably — pwntools drives input, gdb/pwndbg observes runtime state.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Python version**: pwntools supports Python 3.8+. Very old distros may not have it.
|
||||
- **`p.interactive()` blocks.** It's for manual exploration; remove it from automated scripts.
|
||||
- **ASLR on local runs**: turn off for reproducibility during debugging: `echo 0 | sudo tee /proc/sys/kernel/randomize_va_space` (remember to revert — journal this!).
|
||||
- **`gdb.debug()` requires `gdb-multiarch`** for cross-arch binaries.
|
||||
- **Subprocess cleanup**: if your script crashes, orphan `./target` processes may linger. Kill them at Phase 9 or add `atexit` cleanup.
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 cleanup specifics
|
||||
|
||||
```bash
|
||||
# Remove pwntools debug scripts
|
||||
rm -f /tmp/debug-*.py
|
||||
rm -f /tmp/debug-crashes.txt
|
||||
|
||||
# Kill orphan target processes from failed runs
|
||||
pkill -f './target' || true # adjust to actual binary name
|
||||
|
||||
# Restore ASLR if disabled
|
||||
# echo 2 | sudo tee /proc/sys/kernel/randomize_va_space # Linux default
|
||||
|
||||
# Revert any binary patches applied for testing (see native-binary.md for details)
|
||||
```
|
||||
Reference in New Issue
Block a user