luna/oh-my-opencode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(shared-skills): add user skill sources

Browse Source
This commit is contained in:
YeonGyu-Kim
2026-05-26 20:28:24 +09:00
parent 74ece8181a
commit 3ccbe2fd4f
94 changed files with 24640 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/shared-skills/skills/programming/SKILL.md
+463
View File
@@ -0,0 +1,463 @@
---
name: programming
description: "MUST USE for ANY work on .py .pyi .rs .ts .tsx .mts .cts .go files. One philosophy: strict types, modern stacks (Pydantic v2 / serde+thiserror / Zod / gin+sqlc+pgx+slog), modern toolchains (uv+basedpyright+ruff / cargo+clippy+miri / Bun+Biome+tsc / gofumpt+golangci-lint v2+nilaway+go-race), parse-don't-validate, exhaustive match, typed errors, no any/unwrap/panic, 250 LOC ceiling, TDD. Routes to references/{python,rust,typescript,rust-ub,go}/. Triggers: write/edit Python/Rust/TypeScript/Go code, new project, gin server, bubbletea TUI, CJK IME, connect-go RPC, sqlc pgx, branded ids, exhaustive match, unsafe Rust, miri, oversized file, refactor, TDD, e2e test, arena, allocator, bumpalo, const fn, const generics, comptime, zero-alloc, bitfield, repr, scopeguard, errdefer, Zig-like, zerocopy, packed struct."
---
# Programming
You are a senior engineer who writes Python, Rust, and TypeScript with one shared discipline. **Type-strict. Stack-first. Async-correct. Architecturally honest about file size.**
This skill is an index. The hard per-language rules live under `references/`. Load the language-specific reference **before** writing a single line of code.
---
## PHASE 0 — LANGUAGE GATE (RUN THIS FIRST, EVERY TIME)
**DO NOT WRITE OR EDIT A SINGLE LINE OF CODE BEFORE COMPLETING THIS GATE.**
1. **Identify the language** from the file extension or the user's request.
2. **STOP** and read the matching reference set:
| File / Language | MANDATORY reading (load `Read` tool on every file below) |
|---|---|
| `.py`, `.pyi`, "Python" | `references/python/README.md` + every file under `references/python/` that the README tells you to load on demand |
| `.rs`, `Cargo.toml`, "Rust" | `references/rust/README.md` + every file under `references/rust/` that the README tells you to load on demand. **IF the change touches `unsafe`, `*mut`, `*const`, `MaybeUninit`, FFI, `unsafe impl Send/Sync`, or a custom lock-free primitive: ALSO load `references/rust-ub/README.md` plus every file under `references/rust-ub/`.** |
| `.ts`, `.tsx`, `.mts`, `.cts`, "TypeScript" | `references/typescript/README.md` + every file under `references/typescript/` that the README tells you to load on demand |
| `.go`, `go.mod`, `go.sum`, `.golangci.yml`, `*.proto` next to a Go module, "Go" / "Golang" | `references/go/README.md` + every file under `references/go/` that the README tells you to load on demand |
3. Only after the references are loaded, apply the **shared philosophy** below plus the per-language iron list from the reference.
**No exceptions for "small" or "one-off" code.** The whole point of the modern toolchain (uv + PEP 723, `rust-script`, Bun) is that disposable scripts cost nothing to write with full discipline.
---
## Shared philosophy (all three languages)
These are not style preferences. They are the six axioms every recipe in `references/` derives from.
1. **The type system is your proof system.** Make illegal states unrepresentable. The compiler / type checker is the cheapest test you will ever run. If a bug can be expressed as a type error, it is *required* to be expressed as a type error.
2. **Parse, don't validate.** Untrusted input crosses a boundary exactly once - at the boundary it is parsed into a typed value (Pydantic v2 in Python, `serde` + `#[derive]` in Rust, Zod in TypeScript). Inside the boundary, code receives typed values and never re-validates. The boundary owns trust; the interior owns logic.
3. **One name = one concept.** A `UserId` is not a `string`. A `Seconds` is not a `Milliseconds`. Use `NewType` (Python), newtype tuple structs (Rust), or branded types (TypeScript) for every distinct semantic primitive. The compiler refuses to let two semantic units mix.
4. **Exhaustive variant matching, always.** Discriminated unions and enums are matched exhaustively. Python: `match` + `case unreachable: assert_never(unreachable)`. Rust: `match` (the compiler enforces). TypeScript: `switch` + `assertNever`. **`if`/`elif`/`else` is forbidden for discriminating on a tagged variant** - it silently swallows new variants.
5. **Trust framework guarantees. Validate only at boundaries.** No null checks for values the type system already proves non-null. No `try/except` around code that cannot raise. No `unwrap`/`!`/`as` to paper over a contract you should have encoded in types. No defensive layer for a scenario you cannot name.
6. **Test-driven, with the right shape of test.** No production line ships without a failing test that proves it was needed. Behavior is locked by tests, not by hope. See the TDD discipline below.
---
## TDD DISCIPLINE — NON-NEGOTIABLE
**Every change follows the red → green → refactor loop.** The order is mandatory; reverse it and you have written speculative code.
### The order
1. **Red.** Write a failing test that names the behavior in `Given / When / Then`. Run it. *Confirm it fails for the right reason* — not a typo, not an import error. A test that fails because the function does not exist yet is the right reason. A test that fails because of a missing import is not.
2. **Green.** Write the minimum code to make the test pass. Resist adding the second case until the first passes. The second case is the next red.
3. **Refactor.** With the test green, restructure ruthlessly. The test is your safety net. If the test is hard to refactor against, the test is bad — fix the test before the code.
### The shape of the test pyramid
Every feature ships with all three rungs, sized in this proportion:
| Rung | Count | Purpose | Speed budget |
|---|---|---|---|
| **Unit** | many | Pure-function correctness for every meaningful input class (happy + edges + boundaries + error paths) | < 10 ms each |
| **Integration** | some | The real adapter against the real downstream (DB, queue, HTTP) — via `testcontainers`, `httptest`, or equivalent. NEVER a unit test pretending to be integration. | < 1 s each |
| **E2E scenario** | few | One narrative per user-visible outcome. Spins the binary or the full app; drives it through its real surface (HTTP route, CLI invocation, TUI keystroke). Asserts the *observable outcome*, not internal state. | seconds, run on CI |
If a feature has zero E2E coverage, it is undone — even if every unit test passes.
### Given / When / Then is mandatory
Every test — unit, integration, E2E — is structured by these three blocks. Names follow `Test_<Behavior>_when_<Condition>` or the language idiom (`it("<does X> when <Y>")`, `#[test] fn behavior_when_condition`).
```
Given: the preconditions and fixtures
When: the single action under test
Then: the observable outcome AND only that outcome
```
One `When` per test. Multiple `When`s = multiple tests. The `Then` asserts only what changed because of the `When` — not unrelated invariants.
### Less mock, the better
Mocks are a last resort, not a default. The priority order:
1. **Real object.** Use it when constructable in <1 ms (most domain types, pure functions, value objects).
2. **In-memory fake.** A real implementation of the interface backed by a map/slice — for stores, caches, queues. The fake has its OWN test that proves it behaves like the real one.
3. **Testcontainer / sandbox.** Real Postgres, real Redis, real S3-compatible (MinIO), via `testcontainers`. Slow but truthful.
4. **HTTP-level fake.** `httptest.Server` (Go), `respx` (Python), `msw` (TS) — fake at the wire, not at the SDK.
5. **Mock.** Only when 1–4 are genuinely infeasible (clock, randomness, external SaaS with no sandbox). Then mock the **narrowest** seam — never an entire service. A mock that returns whatever the test wants is a tautology and proves nothing.
**The rule**: if your test fails when the production code's *implementation* changes but its *behavior* did not, the test is over-mocked. Delete the mock; assert on observable outputs.
### Efficient AND accurate — both, not either
- **Accurate**: the test fails for the bug it names, and only that bug. No incidental coupling to format, ordering, whitespace, or unrelated fields. Assert on the *contract*, not on the dump.
- **Efficient**: the whole unit suite runs in < 30 seconds on a developer laptop. The whole integration suite in < 5 minutes. If you cross those budgets, profile and split — fast tests run on every save, slow ones run on push.
- **Deterministic**: no `sleep`, no wall-clock dependence, no order dependence (`-shuffle=on`, pytest-randomly, vitest random seed). Inject a `Clock`. Subscribe to the event, do not poll for it. Time-based flake is a bug, not a test issue.
- **Isolated**: every test starts from a known fixture and tears down. `t.TempDir()`, `t.Setenv()`, transactional rollback for DB tests. Two tests passing individually but failing together is a fixture leak — fix it immediately.
### Prompt tests follow the same rule
When tests cover LLM prompts or agent outputs, assert on **parsed structure, decisions, or rule data**, never on exact prompt strings. Pinning a sentence is brittle pretend-coverage; asserting that the prompt instructs the model to refuse on category X is real coverage.
### Anti-patterns the skill rejects
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Writing code first, tests "to add later" | Tests-after rationalize the existing design, even when wrong. | Red first. Always. |
| One mega-test asserting 12 things | First failure hides the next 11. | Split by `Then` clause — one assertion class per test. |
| Mocking every collaborator | Test passes regardless of real behavior. | Use a fake or the real thing. Mock only true unmockables. |
| `time.sleep(0.1)` to "let it finish" | Flake guaranteed. | Subscribe to the completion signal; bounded await. |
| Snapshot tests for everything | Locks formatting, not behavior. | Snapshots for *structure* (CLI help, JSON shape). Assertions for *behavior*. |
| Removing a failing test to "unblock CI" | You just deleted a bug report. | Fix the code or fix the test — never delete to silence. |
| `assert result is not None` and stopping there | Passes when result is garbage. | Assert the *value*, not its existence. |
| Single happy-path E2E, no edges | Most bugs live on edges. | Edges are unit-test territory — but include at least one E2E that exercises an error path. |
---
## Cross-language iron list
Apply unless the per-language reference overrides with something stricter.
| Rule | Python | Rust | TypeScript | Go |
|---|---|---|---|---|
| Immutable by default | `@dataclass(frozen=True, slots=True)` / Pydantic `frozen=True` | every binding is `let` (not `let mut`) unless mutation is the documented purpose | every field is `readonly`; arrays are `readonly T[]` | value types, unexported fields, no mutation methods unless mutation is the purpose |
| Branded primitives | `UserId = NewType("UserId", int)` | `struct UserId(u64);` (newtype tuple) | `type UserId = Brand<string, "UserId">` | `type UserID string` + smart constructor with unexported field |
| Exhaustive variant matching | `match` + `assert_never` | `match` (compiler-enforced) | `switch` + `assertNever` | sealed interface + type switch + **`exhaustive` linter** (the compiler will not help) |
| No untyped escape hatches | no `Any` in public sigs, no `cast`, no `# type: ignore` | no `unwrap`/`expect` outside `main`/tests, no `as` for narrowing, no `#[allow]` to silence real warnings | no `any`, no `as` (except `as const`, `satisfies`), no `!`, no `@ts-ignore`, no `@ts-expect-error` | no `interface{}` / bare `any` in domain sigs; no `_ = err`; no `//nolint` without reason |
| No bare error strings | typed exception dataclass with `__str__` | `thiserror` enum (lib) or `anyhow` with `.context(...)` (app) | `Error` subclass with typed fields | sentinel `errors.New` + typed `*XError` struct; wrap with `%w`; check via `errors.Is/As` |
| Boundary catch only | catch the exact exception you expect; broad `except Exception` only in `main()`, with logging + re-raise | `?` everywhere; never `panic!` in library code | `catch` must narrow with `instanceof` and re-throw or convert; no empty catch | every `(T, error)` checked; `panic` only in `main`/tests; one `httperr.Write` funnel in handlers |
| Resources via RAII | `with` (sync) / `async with` (async) | `Drop` impl or RAII guard | `using`/`await using` (TC39 explicit resource management) | `defer x.Close()` immediately after acquisition; `bodyclose`/`sqlclosecheck` linters enforce |
| Async runtime is mandatory | `anyio` (NEVER bare `asyncio`) | `tokio` (`async-std` is unmaintained) | platform-native async (Bun/Node) with structured cancellation via `AbortSignal` | `context.Context` as first param + `errgroup` for structured concurrency; `-race` on every test |
| Modern HTTP client | [`httpx2`](https://github.com/pydantic/httpx2) with HTTP/2 + brotli + zstd | `reqwest` with rustls | `ky` (default) / `undici` direct API (Node perf) - NEVER bare `fetch` in prod | stdlib `net/http.Client` with tuned `Transport` + `go-retryablehttp` for retry/backoff |
| No parameter mutation | params are inputs; produce a new value | `&mut` only when mutation is the documented purpose | parameters never reassigned (`noParameterAssign`) | value receivers when not mutating; pointer receivers only for genuine mutation; `copylocks` vet enforces |
| No helpers for one-off | inline a 3-line operation; do not abstract until the second caller | same | same | same |
---
## Modern ecosystem - canonical libraries (2026)
Use these unless the project's manifest explicitly picks something else.
| Domain | Python | Rust | TypeScript | Go |
|---|---|---|---|---|
| Data validation / boundary parse | **Pydantic v2** | **serde** + `#[derive(Deserialize)]` + `validator` | **Zod v4** (Standard Schema) | `validator/v10` (HTTP) + `protovalidate` (proto) + smart constructors (domain) |
| Internal value object | `@dataclass(frozen=True, slots=True)` | newtype tuple struct or plain `struct` | `type` alias with `readonly` | struct with unexported fields + `NewX(...)` constructor |
| Error types | typed exception dataclass | `thiserror` (lib) + `anyhow` (app) | `Error` subclass + Result pattern | sentinel `errors.New` + typed `*XError` struct + `%w` wrap |
| HTTP client | [`httpx2`](https://github.com/pydantic/httpx2) | `reqwest` | `ky` / `undici` | stdlib `net/http` + `go-retryablehttp` |
| Web framework | **FastAPI** | **axum** | **Hono** + `hono-openapi` | **gin** (de facto, ~48%) / `chi` (minimalist) / `connect-go` (RPC) |
| ORM / DB | SQLAlchemy 2.x async + `asyncpg` | `sqlx` (compile-time checked) | **Drizzle** | **sqlc** (codegen from `.sql`) + `pgx/v5` + `goose` migrations |
| CLI | **typer** + `rich` | **clap** (derive) + `color-eyre` + `indicatif` | `@clack/prompts` + `commander` | **cobra** + `huh` (prompts) + `slog` |
| Logging / observability | `structlog` (prod) or `rich.logging` (dev) | **tracing** + `tracing-subscriber` | `pino` (structured JSON) | stdlib **`log/slog`** (NEVER logrus/zap/zerolog for new code) |
| Testing | `pytest` | `cargo nextest` + `proptest` + `insta` | `bun test` / `vitest` | stdlib `testing` + `testify/require` + `goleak` + `autogold` + `rapid` + `testcontainers` |
| Data / analytics | **polars** + **duckdb** + `numpy` (NEVER pandas) | `polars-rs` or `arrow` | (defer to backend service) | `arrow-go` + DuckDB-Go bindings + `gonum` |
| LLM / agent | **pydantic-ai** | (call out to Python via subprocess) | **Vercel AI SDK** | direct `net/http` + Connect (langchaingo not recommended) |
| TUI | **textual** | `ratatui` | `@clack/prompts` or ink | **bubbletea v2 RC** + `bubbles/v2` + `lipgloss/v2` (v2 mandatory for CJK IME) |
| Config from env | **pydantic-settings** | `figment` or `config` | `zod` + `process.env` | `caarlos0/env/v11` (struct-tag env) |
A bare default constructor for any of these (no timeouts, no pool tuning, no schema) is a bug. See the per-language reference for the canonical production defaults.
---
## Modern toolchain - the only acceptable setup
| Tool category | Python | Rust | TypeScript | Go |
|---|---|---|---|---|
| Package / project manager | **uv** (NEVER pip/poetry/conda) | **cargo** + `cargo-nextest` + `cargo-machete` + `cargo-deny` | **Bun** (runtime + package manager); pnpm if Node is forced | **`go modules`** + `go work` for monorepos |
| Type checker | **basedpyright** with `typeCheckingMode = "all"` | the Rust compiler with `-D warnings` + clippy `pedantic` + `nursery` + `cargo` groups | `tsc --noEmit` (or `tsgo` when available) with `strict` + `noUncheckedIndexedAccess` + `exactOptionalPropertyTypes` + `verbatimModuleSyntax` | the Go compiler + **`golangci-lint v2`** with the strict bundle + **`nilaway`** (nil-deref static analysis) |
| Linter + formatter | **ruff** with `select = ["ALL"]` | `clippy` + `rustfmt` | **Biome** (single binary - replaces ESLint + Prettier) | **`gofumpt`** (stricter gofmt) + `goimports -local` + `golangci-lint v2` |
| Test runner | **pytest** | **cargo-nextest** | `bun test` / `vitest` | stdlib `go test -race -shuffle=on -count=1` + `goleak` |
| UB / soundness gate | (n/a) | **nightly miri** with strict provenance + Tree Borrows pass | (n/a) | **`nilaway`** + `-race` detector + `goleak` are the equivalent gate |
| Disposable scripts | **PEP 723** inline metadata + `uv run script.py` | **rust-script** with inline `Cargo.toml` block | `bun run script.ts` | `//go:build ignore` + `go run script.go` |
| Bootstrap a new project | `scripts/python/new-project.py` | `scripts/rust/new-project.py` | `scripts/typescript/new-project.ts` | `scripts/go/new-project.py` |
| Pre-commit / CI gate | `ruff check . && basedpyright && pytest` | `cargo +nightly clippy -- -D warnings && cargo nextest run && cargo +nightly miri test` | `bunx biome check . && bunx tsc --noEmit && bun test` | `gofumpt -l . && golangci-lint run ./... && nilaway ./... && go test -race -shuffle=on -count=1 ./...` |
A `tsconfig.json` with `"strict": true` alone is **not** strict. The reference enumerates the additional flags. Same for `pyproject.toml` and `Cargo.toml` - the references contain the canonical full configuration.
---
## THE 250 PURE LOC CEILING (NON-NEGOTIABLE)
**A source file whose pure LOC (non-blank, non-comment lines) exceeds 250 is architecturally broken.** Not a style preference. Not a soft suggestion. **A defect.**
A file past this line is telling you, loudly:
- The module is doing more than one thing.
- Multiple cohesive units got merged "to save a file".
- Re-exports, barrels, and orchestrators got fused into pure-logic units.
- Every future reader pays a tax to find what they need.
### Why 250 and not 500 or 1000
At 250 pure LOC a file still fits in one screen on a 32-inch monitor with a 14pt font. A reviewer can hold the whole thing in working memory and spot a cross-cutting bug. At 500 LOC they cannot. At 1000 LOC they stop trying. The number is **the cognitive ceiling of a single human reviewer who has not memorized the file.**
### Measuring pure LOC
```bash
# Quick (line-comment + blank exclusion - good enough for Python, Rust, TypeScript):
awk '!/^[[:space:]]*$/ && !/^[[:space:]]*(\/\/|#|--)/' <file> | wc -l
# Authoritative (handles block comments correctly):
cloc --by-file <file> # the "code" column is the number that matters
```
### Required behavior
**Creating a file that will exceed 250 pure LOC.** STOP. Split it **before the first commit**. Carve by responsibility (single-responsibility principle), one cohesive unit per file. Use a barrel (`__init__.py`, `mod.rs`, `index.ts`) for re-exports ONLY. **Never** for logic.
**Editing a file that already exceeds 250 pure LOC and your edit adds lines.** STOP. Refactor the unit you are touching into its own file BEFORE adding the new lines. The split is part of THIS task, not a follow-up someone will never do.
**Reading a file that exceeds 250 pure LOC while implementing a feature.** Surface the smell explicitly in your reply, propose a concrete split (which functions go where, in 1-2 lines each), and ask the user whether to split now or carry the smell into the feature work. Do not silently keep going.
### Forbidden escapes
- Counting comments and blank lines toward the budget. **Pure LOC means code lines.** Period.
- Splitting by token count (`foo_1.py`, `module_part_A.rs`, `service-2.ts`). **REJECT.** Split by what each file DOES. Name each file after the concept it owns.
- Catch-all dump files: `utils.py`, `helpers.ts`, `lib.rs` (as a logic dump), `common.py`, `shared.ts`. **REJECT.** These just relocate the smell.
- "It's generated, so it's fine." Only true if the file lives in `dist/`, `target/`, `__generated__/`, or wherever the build authoritatively rewrites. Hand-edited "I will regenerate it later" files do NOT qualify.
- "It's a test file with many cases." Split by SUT or by behavior cluster. One file per cohesive `describe` group.
- "230 pure LOC, close enough." A 230-LOC file about to grow is already over the line. Split now. **Do not race to the ceiling.**
### Acceptable exceptions (rare, require justification)
A file may legitimately exceed 250 pure LOC if **and only if** it is:
- A **truly indivisible single-responsibility unit** (e.g., a generated parser table, a state machine whose states share a single closure, a `derive` macro implementation). Mark the first 5 lines with a comment such as `# noqa: SIZE_OK - generated parser table, 612 states share branch tables` (Python) / `// allow: SIZE_OK - state machine, removing any state breaks the transition matrix` (Rust/TS), and explain WHY no split is possible.
- A **pure data table** (translation strings, error code lookup, brand color palette). Tables of data are not logic.
**`# noqa: SIZE_OK` without a justifying comment is itself slop** and must be rejected by the next person to touch the file.
### Concrete split examples
#### Python - BEFORE (`user_service.py`, 412 pure LOC, broken)
```python
# user_service.py - DOES TOO MUCH
class UserRepository: ... # 90 LOC of SQLAlchemy
class UserValidator: ... # 60 LOC of Pydantic + business rules
class PasswordHasher: ... # 40 LOC of bcrypt wrapper
class EmailSender: ... # 50 LOC of httpx2 client
class UserService: ... # 130 LOC orchestrating the four above
def _build_query(...): ... # 25 LOC helper
def _format_email(...): ... # 17 LOC helper
```
#### Python - AFTER (split by responsibility)
```
src/myapp/users/
├── __init__.py # barrel: re-exports UserService only (5 LOC)
├── repository.py # UserRepository (~95 LOC)
├── validator.py # UserValidator (~65 LOC)
├── password.py # PasswordHasher (~45 LOC)
├── notifier.py # EmailSender (renamed - the role, not the verb)
├── service.py # UserService (orchestrator) (~135 LOC)
└── _queries.py # _build_query (private) (~30 LOC)
```
Every file is < 250 pure LOC. Each owns one concept. The barrel exposes the only public name. The reviewer never has to scroll through password hashing to understand SMTP retry policy.
#### Rust - BEFORE (`auth.rs`, 380 pure LOC)
```rust
// auth.rs - DOES TOO MUCH
pub struct Session { ... } // 40 LOC
impl Session { ... } // 90 LOC of methods
pub struct TokenIssuer { ... } // 30 LOC
impl TokenIssuer { ... } // 70 LOC
pub struct RateLimiter { ... } // 50 LOC
impl RateLimiter { ... } // 70 LOC
fn parse_authorization_header(...) { ... } // 30 LOC
```
#### Rust - AFTER
```
src/auth/
├── mod.rs # re-exports Session, TokenIssuer, RateLimiter (8 LOC)
├── session.rs # Session + impl (~130 LOC)
├── token.rs # TokenIssuer + impl (~100 LOC)
├── rate_limit.rs # RateLimiter + impl (~120 LOC)
└── header.rs # parse_authorization_header (~35 LOC)
```
#### TypeScript - BEFORE (`api/orders.ts`, 510 pure LOC)
```typescript
// api/orders.ts - DOES TOO MUCH
export const OrderSchema = z.object({ ... }) // 30 LOC
type Order = z.infer<typeof OrderSchema>
export class OrderRepository { ... } // 110 LOC
export class PricingEngine { ... } // 130 LOC
export class TaxCalculator { ... } // 90 LOC
export class OrderService { ... } // 150 LOC
```
#### TypeScript - AFTER
```
src/orders/
├── index.ts # barrel (6 LOC)
├── schema.ts # OrderSchema + Order type (~35 LOC)
├── repository.ts # OrderRepository (~115 LOC)
├── pricing.ts # PricingEngine (~135 LOC)
├── tax.ts # TaxCalculator (~95 LOC)
└── service.ts # OrderService (orchestrator) (~155 LOC)
```
---
## MANDATORY POST-WRITE REVIEW LOOP
**This runs EVERY time you finish writing or substantively editing code, before you claim the task is done.** No exceptions.
### Step 1 — measure
For every file you created or modified:
```bash
awk '!/^[[:space:]]*$/ && !/^[[:space:]]*(\/\/|#|--)/' <file> | wc -l
```
Or run the per-language checker the skill ships:
```bash
# Python
uv run scripts/python/check-no-excuse-rules.py <changed paths>
# Rust
bash scripts/rust/check-no-excuse-rules.sh <changed paths>
# TypeScript
bun run scripts/typescript/check-no-excuse-rules.ts <changed paths>
```
### Step 2 — interpret
| Pure LOC | Verdict | Required action |
|---|---|---|
| ≤ 200 | Healthy | continue |
| 200 - 250 | **Warning band** - the file is approaching the ceiling. State that fact explicitly in the next message and propose a split if the next planned edit will add lines. |
| > 250 | **DEFECT** - the architecture is wrong. Do NOT commit. Refactor into smaller cohesive units **now**, in this same task. |
### Step 3 — architectural self-review (always, even at 80 LOC)
After every code-writing session, answer these out loud (in your reply) before declaring done:
1. **Single responsibility?** Can I name what this file owns in one short noun phrase? If the answer needs the word "and", split.
2. **Boundary purity?** Did I parse untrusted input into a typed value at the boundary, or did I pass `dict[str, Any]` / `serde_json::Value` / `unknown` past the boundary? If the latter, fix it.
3. **Variant discrimination?** Did I use `if`/`elif`/`else` (or `switch` without `assertNever`, or `match` without `assert_never`) anywhere to discriminate on a tagged type or enum? If yes, rewrite as exhaustive match.
4. **Escape hatches?** Any `Any`, `# type: ignore`, `unwrap`, `expect` outside `main`/tests, `as` numeric cast, `!`, `@ts-ignore`, `@ts-expect-error`, `#[allow]` on a real warning? If yes, fix the type or document why with a comment.
5. **Defensive layer?** Any null check, try/except, or `isinstance` guarding a value the type system already proves? If yes, delete.
6. **Helpers for one-off?** Any function, class, or trait introduced for a single caller that will never get a second caller? If yes, inline.
7. **Tests?** Is the behavior I just introduced locked by a test that would fail if I revert this commit?
**If any answer fails, fix it before declaring done.** This loop is the difference between "the code compiles" and "the code is correct."
### Step 4 — if you need to refactor right now, invoke the right skill
- The file you just wrote (or an adjacent one) is over 250 pure LOC, or step 3 surfaced more than two issues: **load the `refactor` skill** and execute its safe-refactor protocol (codemap, plan, LSP-driven edits, test after each step). Do not improvise a refactor under time pressure - the refactor skill exists precisely so you do not corrupt behavior while reshaping structure.
- You inherited a branch with AI-generated patterns (broad `except`, redundant null checks, vague TODOs, oversized modules, dead helpers): **load the `remove-ai-slops` skill** to do a categorized branch-scope cleanup with regression tests pinned first.
These two skills are not optional cosmetics. They are the recovery path for the defects this loop is designed to catch.
---
## Companion skills - explicit invocation triggers
| Trigger | Skill to load | Why |
|---|---|---|
| File exceeds 250 pure LOC, OR the post-write loop surfaces 2+ issues, OR the user says "reshape this", "extract this", "clean this up" | `refactor` | Safe codemap-driven multi-step refactor with LSP + tests after each step. Never improvise a structural change. |
| Recent branch contains AI-authored code that smells (broad except, dead helpers, vague comments, oversized files), OR the user says "remove slop", "clean AI code", "deslop" | `remove-ai-slops` | Tests pinned FIRST, then categorized parallel cleanup, then quality gates. Behavior-preserving. |
| Rust code touches `unsafe`, `*mut`, `*const`, `MaybeUninit`, FFI, `unsafe impl Send/Sync`, or a custom lock-free primitive | `references/rust-ub/` | Full UB taxonomy + Miri strictness escalation. Every `unsafe` block must survive Miri Level 3 (strict provenance + symbolic alignment + preemption) before it ships. |
---
## Per-language jump table
**Stop. Read the matching reference fully before writing code.**
### Python (`.py`, `.pyi`)
**READ `references/python/README.md` FIRST.** Then load on demand:
| Need | Load |
|---|---|
| Strict pyproject.toml / basedpyright / ruff config | `references/python/pyproject-strict.md` |
| Type patterns (`NewType`, `Final`, `TypeGuard`, `Protocol`) | `references/python/type-patterns.md` |
| Data modeling (Pydantic vs dataclass vs TypedDict vs StrEnum) | `references/python/data-modeling.md` |
| Error handling (typed exceptions, exhaustive match, union returns) | `references/python/error-handling.md` |
| Async with anyio (task groups, cancel scopes, channels) | `references/python/async-anyio.md` |
| httpx2 production defaults (HTTP/2, brotli+zstd, pool tuning) | `references/python/httpx2-optimization.md` |
| **orjson** in hot paths (FastAPI integration, Pydantic v2 `model_dump_json` vs orjson, Redis/queue/log) | `references/python/orjson-stack.md` |
| Data processing with polars + duckdb (NEVER pandas) | `references/python/data-processing.md` |
| FastAPI + SQLAlchemy 2.x async stack | `references/python/fastapi-stack.md` |
| pydantic-ai agents | `references/python/pydantic-ai.md` |
| Textual TUI | `references/python/textual-tui.md` |
| Disposable PEP 723 scripts | `references/python/one-liners.md` |
| Canonical library defaults | `references/python/libraries.md` |
### Rust (`.rs`, `Cargo.toml`)
**READ `references/rust/README.md` FIRST.** It defines the five pillars (explicit allocation, compile-time proof, zero hidden cost, type-encoded invariants, deterministic cleanup) and the post-write review checklist. Then load on demand:
| Need | Load |
|---|---|
| **Arena allocation, const fn, zero-alloc APIs, bitfield, scopeguard, errdefer, Zig-like patterns** | **`references/rust/zero-cost-safety.md`** |
| Strict `Cargo.toml` lints + profile + workspace config | `references/rust/cargo-strict.md` |
| Type-state and newtype patterns (Chris Allen's `Point<Screen>` rule) | `references/rust/type-state.md` |
| `unsafe` discipline (safe wrapper + SAFETY comment + miri proof) | `references/rust/unsafe-discipline.md` |
| Async with tokio (JoinSet, cancellation, select, blocking work) | `references/rust/async-tokio.md` |
| Concurrency primitives (locks, atomics, channels, loom) | `references/rust/concurrency.md` |
| axum + sqlx + tracing + tower HTTP stack | `references/rust/axum-stack.md` |
| clap + color-eyre + tracing + indicatif CLI stack | `references/rust/clap-stack.md` |
| Property tests (proptest) + snapshot tests (insta) | `references/rust/proptest-insta.md` |
| Disposable `rust-script` scripts | `references/rust/one-liners.md` |
| Canonical library defaults | `references/rust/libraries.md` |
| **ANY `unsafe` / FFI / `MaybeUninit` / lock-free work** | **`references/rust-ub/` (full directory)** |
### TypeScript (`.ts`, `.tsx`, `.mts`, `.cts`)
**READ `references/typescript/README.md` FIRST.** Then load on demand:
| Need | Load |
|---|---|
| Strict tsconfig + Biome config | `references/typescript/tsconfig-strict.md` |
| Type patterns (branded types, `as const`, `satisfies`, narrowing, `assertNever`) | `references/typescript/type-patterns.md` |
| Data modeling (type vs interface vs Zod, readonly, parse-don't-validate) | `references/typescript/data-modeling.md` |
| Error handling (Result, typed errors, union vs throw, AbortSignal timeouts) | `references/typescript/error-handling.md` |
| Bootstrapping a new project (Bun, pnpm, Hono, Vite) | `references/typescript/bootstrap.md` |
| Hono backend stack (hono-openapi, Scalar, Swagger, Zod v4) | `references/typescript/backend-hono.md` |
### Go (`.go`, `go.mod`, `go.sum`, `.golangci.yml`, `*.proto`)
**READ `references/go/README.md` FIRST.** Then load on demand:
| Need | Load |
|---|---|
| Library defaults (gin vs chi, sqlc, slog, the 2026 stack reasoning) | `references/go/libraries.md` |
| Canonical strict `.golangci.yml` (v2) with per-linter rationale | `references/go/golangci-strict.md` |
| Project layout, Taskfile, CI, `go.mod` template | `references/go/bootstrap.md` |
| Type patterns (named types, smart constructors, sealed interfaces, generics) | `references/go/type-patterns.md` |
| Data modeling — the three layers of validation (validator/v10 → smart ctor → sqlc) | `references/go/data-modeling.md` |
| Error handling (`errors.Is/As`, typed errors, `%w` wrapping, no panic) | `references/go/error-handling.md` |
| Concurrency (`context.Context`, `errgroup`, channels, locks, `-race`, `goleak`) | `references/go/concurrency.md` |
| HTTP backend stack (gin + slog + validator + pgx, middleware ordering, SSE, WS) | `references/go/backend-stack.md` |
| RPC stack (Connect-Go default, grpc-go fallback, protovalidate, Buf) | `references/go/grpc-connect.md` |
| CLI stack (cobra + slog + huh) | `references/go/cobra-stack.md` |
| Database stack (sqlc + pgx + goose + testcontainers) | `references/go/sqlc-pgx.md` |
| TUI stack (bubbletea v2 + bubbles v2 + lipgloss v2; **CJK / IME support**) | `references/go/bubbletea-v2.md` |
| Testing (Given/When/Then, table-driven, fakes-over-mocks, autogold, rapid) | `references/go/testing.md` |
| Disposable `go run` scripts | `references/go/one-liners.md` |
---
## Activation
This skill activates whenever you are writing or modifying any `.py`, `.pyi`, `.rs`, `.ts`, `.tsx`, `.mts`, `.cts`, `.go` file, or any project manifest (`pyproject.toml`, `Cargo.toml`, `package.json`, `tsconfig.json`, `biome.json`, `go.mod`, `go.sum`, `.golangci.yml`, `Taskfile.yml`, `buf.yaml`, `sqlc.yaml`). **Even one-off scripts get the full treatment** - that is the whole point of `uv run` + PEP 723, `rust-script`, `bun run`, and `go run` + `//go:build ignore`: production hygiene with throwaway ergonomics.
The references contain the recipes. **Read them before writing code. Re-read them when the model drifts.** The post-write review loop is non-negotiable.
packages/shared-skills/skills/programming/references/go/README.md
+90
View File
@@ -0,0 +1,90 @@
# Go Programmer
Production Go in 2026. **Boring on purpose, strict by tooling, illegal states unrepresentable by convention.**
## Philosophy
Go gives you fewer type-system tools than Python, TypeScript, or Rust:
- No sum types — only `interface{}` with type-switch.
- No exhaustiveness check from the compiler — only the `exhaustive` linter.
- No `Option<T>` — only `nil` and the eternal trap of "is this nil interface or nil concrete?".
- No `Result<T, E>` — only `(T, error)`, no compiler enforcement of unwrapping.
- No newtype that prevents primitive coercion — `type UserID string` is still implicitly convertible from a literal when used carelessly.
**This is the whole point of the skill.** Where the language is weak, the linter bundle becomes the type checker, and code patterns become the type system. Treat `golangci-lint v2` with the configuration in `golangci-strict.md` as if it were `tsc --strict` or `basedpyright`. Treat `nilaway` and `go test -race` as if they were Miri.
The skill enforces five non-negotiables:
1. **Parse-don't-validate at every boundary.** HTTP/RPC/CLI/config gets parsed into a domain struct constructed only via `New*(...)` smart constructors. Once inside the domain, no further validation. See `data-modeling.md`.
2. **`(T, error)` everywhere.** No panics in library code. No bare `_ = err`. Errors are wrapped with `%w` and asserted with `errors.Is` / `errors.As`. Typed error structs for anything a caller can branch on. See `error-handling.md`.
3. **Sealed interfaces for variants.** Sum types via a sealed unexported method, dispatched through a `type switch`, with the `exhaustive` linter checking completeness. See `type-patterns.md`.
4. **`context.Context` is the first parameter.** Always. No `context.Background()` inside leaf functions. No goroutine without context-driven shutdown. No `time.Now()` in domain code — inject a clock. See `concurrency.md`.
5. **Generated, not hand-written, for external contracts.** `sqlc` for DB, `oapi-codegen` for OpenAPI servers and clients, `protoc-gen-go` + `protoc-gen-connect-go` for RPC. Hand-rolled marshalling is a regression. See `sqlc-pgx.md`, `grpc-connect.md`.
## Hard rules — tooling
| Category | Use | Never |
|---|---|---|
| Go version | **1.23+** (range-over-func, iter package, slog stable) | <1.22 |
| Module | `go modules` + `go work` for monorepos | dep, GOPATH layouts |
| Format | **`gofumpt`** (stricter gofmt) + `goimports -local <module>` | bare `gofmt` |
| Linter | **`golangci-lint v2`** with the strict bundle in `golangci-strict.md` | bare `go vet` |
| Nil checker | **`nilaway`** (Uber, stable since 2024) in CI | hope |
| Vet bundle | `go vet` + `fieldalignment` + `shadow` | "tests cover it" |
| Tests | `go test -race -shuffle=on -count=1` | `-count` cache, no race |
| Goroutine leaks | `go.uber.org/goleak` in `TestMain` | "looks fine" |
| Mock | `go.uber.org/mock` (gomock successor) | hand-written stubs |
| DB | `sqlc` + `jackc/pgx/v5` | `database/sql` + `gorm` |
| HTTP framework | **`gin-gonic/gin`** (de facto, ~48% of Go API repos) — `go-chi/chi` for minimalist, `connectrpc/connect-go` for RPC | `echo` (smaller eco), `fiber` (fasthttp = non-stdlib), `gorilla/mux` (in maintenance mode) |
| RPC | **`connectrpc/connect-go`** (gRPC-compatible, HTTP/1.1-friendly, browser-friendly) | hand-rolled `grpc-go` unless you specifically need bidi streaming features Connect lacks |
| Validation | `go-playground/validator/v10` for HTTP boundary + `bufbuild/protovalidate-go` for proto + smart constructors for domain | ad-hoc `if len(s) == 0` chains |
| Config | `caarlos0/env/v11` (struct-tag env) | `viper` unless you actually need file+env+flag merging |
| Logging | **`log/slog`** (stdlib, Go 1.21+) | logrus, zap, zerolog (all superseded) |
| CLI | `spf13/cobra` | hand-rolled `os.Args` parsing past 2 flags |
| TUI | `charm.land/bubbletea/v2` + `bubbles/v2` + `lipgloss/v2` — see `bubbletea-v2.md` for CJK/IME | bubbletea v1 if you need IME |
A single CI command should be the gate:
```bash
gofumpt -l . && \
golangci-lint run ./... && \
nilaway ./... && \
go test -race -shuffle=on -count=1 ./...
```
If any of these fails, the change is not done. Period. The bundle is set up so a clean run actually means clean — see `golangci-strict.md` for the per-linter rationale and the deliberate `nolint:` policy.
## Hard rules — code
Read these per-file references for the canonical patterns:
- **Types & data** → `type-patterns.md`, `data-modeling.md` — branded named types, smart constructors with unexported fields, sealed interfaces as sum types.
- **Errors** → `error-handling.md` — sentinel vs typed struct, `errors.Is/As`, `%w` wrapping, no panic in libraries, the `errorlint` ruleset.
- **Concurrency** → `concurrency.md` — `context.Context` discipline, `errgroup`, `sync.OnceValue`, `goleak`, `-race`, channel selection rules.
- **HTTP backend** → `backend-stack.md` — `gin` server skeleton, middleware ordering, SSE/streaming with `http.Flusher`, structured slog logging, graceful shutdown — distilled from the CLIProxyAPI codebase (a real proxy serving OpenAI/Gemini/Claude APIs).
- **RPC** → `grpc-connect.md` — when to pick Connect vs grpc-go, codegen pipeline, protovalidate, streaming.
- **DB** → `sqlc-pgx.md` — compile-time-safe SQL via sqlc + pgx connection pool + migrations via goose + testcontainers in CI.
- **CLI** → `cobra-stack.md` — cobra layout, slog integration, graceful shutdown on signals, fang-style colored help.
- **TUI** → `bubbletea-v2.md` — v2 model, `SetVirtualCursor(false)` + `tea.View{Cursor}` for CJK IME, why v1 was broken for Korean/Japanese/Chinese input.
- **Testing** → `testing.md` — table-driven tests, `require` vs `assert`, `autogold` snapshots, `gopter` property tests, `testcontainers` for integration, `goleak` for goroutine leaks.
- **Bootstrap** → `bootstrap.md` — `new-project.go` invocation, project layout (`cmd/`, `internal/`, `pkg/`), Taskfile, CI.
- **Strict config** → `golangci-strict.md` — the canonical `.golangci.yml` with the full linter whitelist and per-linter rationale.
- **One-liners** → `one-liners.md` — `go run` scripts with `//go:build ignore`, `gorun`-style invocation.
## The 250 pure LOC ceiling
Same rule as Python/Rust/TS: a `.go` file whose pure LOC (non-blank, non-comment) exceeds 250 is architecturally broken. Go encourages many small files in a single package, so this is *more* natural here than elsewhere — split by responsibility, keep one cohesive type and its methods per file.
The `cmd/server/main.go` is the most common violator. Refactor it: `main.go` only wires `os.Args` → `cmd.Execute()`. Anything else lives in `internal/`.
## Existing codebases — non-strict project
When editing an existing `.go` file that doesn't follow these rules: **write new code in strict style, don't refactor existing code in the same change.** Use the `remove-ai-slops` skill for branch-scope cleanup.
## Activation
This skill activates whenever you write or modify any `.go` file, `go.mod`, `go.sum`, `.golangci.yml`, `Taskfile.yml`, or any of the codegen specs (`*.proto`, `*.sql` next to `sqlc.yaml`, `openapi.yaml` next to `oapi-codegen.yaml`). Even one-off scripts get the strict treatment — that is what `//go:build ignore` + `go run` is for: production hygiene with throwaway ergonomics.
The references contain the recipes. **Read them before writing code. Re-read them when the model drifts.** The post-write architectural review loop is non-negotiable.
packages/shared-skills/skills/programming/references/go/backend-stack.md
+641
View File
@@ -0,0 +1,641 @@
# HTTP Backend Stack — gin + slog + validator + pgx
The canonical production HTTP service skeleton. Distilled from the [CLIProxyAPI](https://github.com/router-for-me/CLIProxyAPI) codebase — a real proxy serving OpenAI / Gemini / Claude / Codex APIs in production, with SSE streaming, WebSocket upgrades, request logging, and hot-reload config.
If you are tempted to pick echo or chi instead, see `libraries.md` — gin wins on ecosystem, not technical merit, and the win is large enough to matter.
---
## `go.mod`
```go
module github.com/your-org/myservice
go 1.23
require (
github.com/gin-gonic/gin v1.10.1
github.com/go-playground/validator/v10 v10.22.1
github.com/caarlos0/env/v11 v11.2.2
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.7.6
golang.org/x/sync v0.18.0
)
```
---
## Project structure
```
cmd/server/main.go # ≤ 50 LOC; flags → run.Execute(ctx)
internal/
cmd/run.go # ~150 LOC; signal handling, config load, server.Run
config/config.go # env-driven Config struct
api/
server.go # gin.Engine setup, route mounting, http.Server
middleware/
request_id.go
request_logging.go
auth.go
recovery.go
cors.go
handlers/
users.go # one file per resource
streams.go # SSE / WebSocket endpoints
domain/ # smart-constructor types (Email, UserID, ...)
service/ # business logic
store/ # pgx + sqlc
obs/
logger.go # slog setup
```
---
## `cmd/server/main.go`
```go
package main
import (
"context"
"log/slog"
"os"
"os/signal"
"syscall"
"github.com/your-org/myservice/internal/cmd"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGINT, syscall.SIGTERM)
defer stop()
if err := cmd.Execute(ctx); err != nil {
slog.Error("fatal", slog.Any("err", err))
os.Exit(1)
}
}
```
That is the entire `main`. Anything more is a smell.
---
## `internal/config/config.go`
```go
package config
import (
"time"
"github.com/caarlos0/env/v11"
)
type Config struct {
Host string `env:"HOST" envDefault:"0.0.0.0"`
Port int `env:"PORT" envDefault:"8080"`
DatabaseURL string `env:"DATABASE_URL,required"`
ReadTimeout time.Duration `env:"READ_TIMEOUT" envDefault:"15s"`
WriteTimeout time.Duration `env:"WRITE_TIMEOUT" envDefault:"30s"`
ShutdownTimeout time.Duration `env:"SHUTDOWN_TIMEOUT" envDefault:"20s"`
LogLevel string `env:"LOG_LEVEL" envDefault:"info"`
LogFormat string `env:"LOG_FORMAT" envDefault:"json"`
Env string `env:"ENV" envDefault:"development"`
}
func Load() (Config, error) {
var cfg Config
if err := env.Parse(&cfg); err != nil {
return Config{}, err
}
return cfg, nil
}
```
---
## `internal/obs/logger.go`
```go
package obs
import (
"context"
"log/slog"
"os"
)
type ctxKey struct{ name string }
var requestIDKey = ctxKey{"request_id"}
func NewLogger(level, format string) *slog.Logger {
var lvl slog.Level
_ = lvl.UnmarshalText([]byte(level))
opts := &slog.HandlerOptions{Level: lvl, AddSource: true}
var h slog.Handler
switch format {
case "text":
h = slog.NewTextHandler(os.Stdout, opts)
default:
h = slog.NewJSONHandler(os.Stdout, opts)
}
return slog.New(&ctxHandler{Handler: h})
}
// ctxHandler pulls request_id from ctx into every log line.
type ctxHandler struct{ slog.Handler }
func (h *ctxHandler) Handle(ctx context.Context, r slog.Record) error {
if id, ok := ctx.Value(requestIDKey).(string); ok && id != "" {
r.AddAttrs(slog.String("request_id", id))
}
return h.Handler.Handle(ctx, r)
}
func WithRequestID(ctx context.Context, id string) context.Context {
return context.WithValue(ctx, requestIDKey, id)
}
```
---
## `internal/api/server.go`
```go
package api
import (
"context"
"fmt"
"log/slog"
"net/http"
"github.com/gin-gonic/gin"
"github.com/your-org/myservice/internal/api/handlers"
"github.com/your-org/myservice/internal/api/middleware"
"github.com/your-org/myservice/internal/config"
)
type Server struct {
cfg config.Config
srv *http.Server
logger *slog.Logger
}
func New(cfg config.Config, logger *slog.Logger, h *handlers.Handler) *Server {
gin.SetMode(gin.ReleaseMode)
r := gin.New()
// Middleware order matters — see "Middleware ordering" below.
r.Use(
middleware.RequestID(), // 1. assign request_id first
middleware.Recovery(logger), // 2. recovery wraps everything
middleware.RequestLogger(logger),
middleware.CORS(),
)
h.Mount(r)
return &Server{
cfg: cfg,
logger: logger,
srv: &http.Server{
Addr: fmt.Sprintf("%s:%d", cfg.Host, cfg.Port),
Handler: r,
ReadTimeout: cfg.ReadTimeout,
WriteTimeout: cfg.WriteTimeout,
},
}
}
func (s *Server) Run(ctx context.Context) error {
errCh := make(chan error, 1)
go func() {
s.logger.InfoContext(ctx, "server starting",
slog.String("addr", s.srv.Addr))
if err := s.srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
errCh <- err
}
close(errCh)
}()
select {
case <-ctx.Done():
s.logger.InfoContext(ctx, "shutdown signal received")
shutdownCtx, cancel := context.WithTimeout(
context.Background(), s.cfg.ShutdownTimeout)
defer cancel()
return s.srv.Shutdown(shutdownCtx)
case err := <-errCh:
return err
}
}
```
Notes:
- `gin.New()` not `gin.Default()` — `Default()` adds `Logger()` (text format, not slog) and `Recovery()` (no logger injection). We replace both.
- `gin.SetMode(gin.ReleaseMode)` silences debug output. Production assumed.
- `http.Server` with explicit timeouts. The default `nil` timeouts are a DoS waiting to happen.
- Graceful shutdown: SIGINT/SIGTERM cancels the ctx → `Shutdown(shutdownCtx)` gives in-flight requests up to `ShutdownTimeout` to finish.
---
## Middleware ordering — the rule that actually matters
```
RequestID → Recovery → Logger → CORS → Auth → Handler
(1) (2) (3) (4) (5)
```
1. **RequestID** is first so every subsequent middleware sees it.
2. **Recovery** wraps everything after it. Order: a panic in CORS still gets caught.
3. **Logger** sees the request_id and the recovered panic.
4. **CORS** before Auth — OPTIONS preflight must return without auth.
5. **Auth** is the last cross-cutting middleware. Per-route auth (admin-only) is mounted on a sub-router with extra middleware.
```go
// Public routes — no auth
api := r.Group("/api/v1")
{
api.POST("/auth/login", h.Login)
api.GET("/healthz", h.Healthz)
}
// Authenticated routes
authed := r.Group("/api/v1", middleware.Auth(authSvc))
{
authed.GET("/users/:id", h.GetUser)
authed.POST("/users", h.CreateUser)
}
// Admin-only routes
admin := r.Group("/api/v1/admin",
middleware.Auth(authSvc),
middleware.RequireRole("admin"))
{
admin.GET("/users", h.ListAllUsers)
}
```
---
## Middleware examples
### `middleware/request_id.go`
```go
package middleware
import (
"github.com/gin-gonic/gin"
"github.com/google/uuid"
"github.com/your-org/myservice/internal/obs"
)
func RequestID() gin.HandlerFunc {
return func(c *gin.Context) {
id := c.GetHeader("X-Request-ID")
if id == "" {
id = uuid.Must(uuid.NewV7()).String()
}
c.Request = c.Request.WithContext(obs.WithRequestID(c.Request.Context(), id))
c.Header("X-Request-ID", id)
c.Next()
}
}
```
### `middleware/recovery.go`
```go
package middleware
import (
"log/slog"
"net/http"
"runtime/debug"
"github.com/gin-gonic/gin"
)
func Recovery(logger *slog.Logger) gin.HandlerFunc {
return func(c *gin.Context) {
defer func() {
if r := recover(); r != nil {
logger.ErrorContext(c.Request.Context(), "panic recovered",
slog.Any("panic", r),
slog.String("stack", string(debug.Stack())),
)
if !c.Writer.Written() {
c.JSON(http.StatusInternalServerError,
gin.H{"error": "internal_error"})
}
c.Abort()
}
}()
c.Next()
}
}
```
### `middleware/request_logging.go`
```go
func RequestLogger(logger *slog.Logger) gin.HandlerFunc {
return func(c *gin.Context) {
start := time.Now()
c.Next()
logger.InfoContext(c.Request.Context(), "http request",
slog.String("method", c.Request.Method),
slog.String("path", c.Request.URL.Path),
slog.Int("status", c.Writer.Status()),
slog.Int("bytes", c.Writer.Size()),
slog.Duration("elapsed", time.Since(start)),
slog.String("ip", c.ClientIP()),
)
}
}
```
The `sloglint` linter enforces typed attrs (`slog.String(...)`) over `slog.Any("path", ...)`. Keep the form.
### `middleware/cors.go`
```go
func CORS() gin.HandlerFunc {
return func(c *gin.Context) {
c.Header("Access-Control-Allow-Origin", "*")
c.Header("Access-Control-Allow-Methods", "GET, POST, PUT, PATCH, DELETE, OPTIONS")
c.Header("Access-Control-Allow-Headers", "*")
if c.Request.Method == http.MethodOptions {
c.AbortWithStatus(http.StatusNoContent)
return
}
c.Next()
}
}
```
Note the explicit OPTIONS short-circuit — preflight must NOT go through Auth.
---
## Handlers — the canonical shape
```go
package handlers
import (
"errors"
"net/http"
"github.com/gin-gonic/gin"
"github.com/go-playground/validator/v10"
"github.com/your-org/myservice/internal/domain"
"github.com/your-org/myservice/internal/httperr"
"github.com/your-org/myservice/internal/service"
)
type Handler struct {
Users *service.UserService
}
func (h *Handler) Mount(r gin.IRouter) {
api := r.Group("/api/v1")
api.POST("/users", h.CreateUser)
api.GET("/users/:id", h.GetUser)
}
type createUserReq struct {
Email string `json:"email" binding:"required,email"`
Username string `json:"username" binding:"required,alphanum,min=3,max=32"`
}
func (h *Handler) CreateUser(c *gin.Context) {
var req createUserReq
if err := c.ShouldBindJSON(&req); err != nil {
writeBindingError(c, err)
return
}
email, err := domain.NewEmail(req.Email)
if err != nil {
httperr.Write(c, err)
return
}
username, err := domain.NewUsername(req.Username)
if err != nil {
httperr.Write(c, err)
return
}
user, err := h.Users.Create(c.Request.Context(), email, username)
if err != nil {
httperr.Write(c, err)
return
}
c.JSON(http.StatusCreated, user)
}
func writeBindingError(c *gin.Context, err error) {
var vErr validator.ValidationErrors
if errors.As(err, &vErr) {
out := make(map[string]string, len(vErr))
for _, fe := range vErr {
out[fe.Field()] = fe.Tag()
}
c.JSON(http.StatusBadRequest, gin.H{"errors": out})
return
}
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid_json"})
}
```
See `data-modeling.md` for the validator tag reference; see `error-handling.md` for the `httperr.Write` funnel.
---
## SSE streaming — the production pattern
CLIProxyAPI streams OpenAI-compatible SSE for hundreds of concurrent clients. The pattern:
```go
func (h *Handler) StreamChat(c *gin.Context) {
ctx, cancel := context.WithCancel(c.Request.Context())
defer cancel()
// 1. Set SSE headers BEFORE writing any body
c.Header("Content-Type", "text/event-stream")
c.Header("Cache-Control", "no-cache")
c.Header("Connection", "keep-alive")
c.Header("X-Accel-Buffering", "no") // disable nginx buffering
// 2. Obtain the flusher — REQUIRED for streaming
flusher, ok := c.Writer.(http.Flusher)
if !ok {
httperr.Write(c, errors.New("streaming unsupported"))
return
}
// 3. Pull chunks from upstream
chunks, errs := h.svc.StreamCompletions(ctx, req)
for {
select {
case <-ctx.Done():
return // client disconnected, ctx cancelled
case chunk, ok := <-chunks:
if !ok {
fmt.Fprint(c.Writer, "data: [DONE]\n\n")
flusher.Flush()
return
}
fmt.Fprintf(c.Writer, "data: %s\n\n", chunk)
flusher.Flush()
case err := <-errs:
// Error mid-stream — emit as SSE event and bail
fmt.Fprintf(c.Writer, "event: error\ndata: %s\n\n", err.Error())
flusher.Flush()
return
}
}
}
```
Key facts:
- **Headers MUST be set before the first `Write`.** Otherwise gin auto-sets `Content-Type: text/plain`.
- **`c.Writer.(http.Flusher)` is the streaming primitive.** Without `flusher.Flush()`, the response is buffered and arrives as one blob at the end.
- **Always respond to `<-ctx.Done()`.** A disconnected client must stop upstream work — otherwise you generate tokens for nothing.
- **The trailing `\n\n` per event is wire-mandatory** for SSE parsing. Missing it = the client never sees the event.
---
## WebSocket upgrade
```go
import "github.com/gorilla/websocket" // still the canonical WS lib in 2026
var upgrader = websocket.Upgrader{
ReadBufferSize: 4096,
WriteBufferSize: 4096,
CheckOrigin: func(r *http.Request) bool {
// tighten in production
return true
},
}
func (h *Handler) WebSocketEcho(c *gin.Context) {
conn, err := upgrader.Upgrade(c.Writer, c.Request, nil)
if err != nil {
slog.ErrorContext(c.Request.Context(), "ws upgrade failed", slog.Any("err", err))
return
}
defer conn.Close()
for {
mt, msg, err := conn.ReadMessage()
if err != nil { return }
if err := conn.WriteMessage(mt, msg); err != nil { return }
}
}
```
For long-lived connections, use `conn.SetReadDeadline` + `SetPongHandler` for keepalive. CLIProxyAPI's `wsrelay` package is a reference implementation.
---
## Database wiring — pgx pool, injected, never global
```go
package store
import (
"context"
"fmt"
"github.com/jackc/pgx/v5/pgxpool"
)
func NewPool(ctx context.Context, dsn string) (*pgxpool.Pool, error) {
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil {
return nil, fmt.Errorf("parse dsn: %w", err)
}
cfg.MaxConns = 25
cfg.MinConns = 5
cfg.MaxConnLifetime = time.Hour
cfg.MaxConnIdleTime = 30 * time.Minute
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
return nil, fmt.Errorf("connect: %w", err)
}
if err := pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("ping: %w", err)
}
return pool, nil
}
```
See `sqlc-pgx.md` for queries.
---
## Healthcheck
```go
func (h *Handler) Healthz(c *gin.Context) {
if err := h.pool.Ping(c.Request.Context()); err != nil {
c.JSON(503, gin.H{"db": "down", "error": err.Error()})
return
}
c.JSON(200, gin.H{"ok": true})
}
```
Mount BEFORE auth. Health checks must be unauthenticated.
---
## Testing the server
```go
func TestCreateUser_returns_201_for_valid_input(t *testing.T) {
// Given
h := newTestHandler(t)
r := gin.New()
h.Mount(r)
body := `{"email":"a@b.com","username":"alice"}`
req := httptest.NewRequest("POST", "/api/v1/users", strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
rec := httptest.NewRecorder()
// When
r.ServeHTTP(rec, req)
// Then
require.Equal(t, http.StatusCreated, rec.Code)
var got struct{ ID string `json:"id"` }
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))
require.NotEmpty(t, got.ID)
}
```
See `testing.md` for full patterns (testcontainers integration, table-driven, goleak).
---
## Sources
- gin docs: https://gin-gonic.com/docs/
- CLIProxyAPI (reference impl): https://github.com/router-for-me/CLIProxyAPI
- pgx pool: https://pkg.go.dev/github.com/jackc/pgx/v5/pgxpool
- SSE spec: https://html.spec.whatwg.org/multipage/server-sent-events.html
- Go's `http.Server` graceful shutdown: https://pkg.go.dev/net/http#Server.Shutdown
packages/shared-skills/skills/programming/references/go/bootstrap.md
+328
View File
@@ -0,0 +1,328 @@
# Bootstrap — Project Layout, Toolchain, Taskfile, CI
What every new Go project gets in the first 60 seconds. Drop the script in `scripts/go/new-project.go` does all of this — this document explains *what* it produces and *why*.
## Toolchain pin
`go.work` (monorepo) or just rely on `go.mod`'s `go 1.23` directive (single module). Go 1.21+ auto-downloads matching toolchain when the local `go` binary is older. **No `.tool-versions` / `asdf` / `mise` indirection required** unless your shop standardizes on it.
```bash
# Confirm a working toolchain
go env GOTOOLCHAIN # should be "auto" or your pinned version
go version # ≥ 1.23
```
## Required global installs
These are CLI tools, installed once per machine via `go install`:
```bash
go install mvdan.cc/gofumpt@latest
go install golang.org/x/tools/cmd/goimports@latest
go install github.com/golangci/golangci-lint/cmd/golangci-lint@v2.0.0
go install go.uber.org/nilaway/cmd/nilaway@latest
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
go install github.com/pressly/goose/v3/cmd/goose@latest
go install go.uber.org/mock/mockgen@latest
go install github.com/go-task/task/v3/cmd/task@latest
```
For Connect/protobuf projects, additionally:
```bash
go install github.com/bufbuild/buf/cmd/buf@latest
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install connectrpc.com/connect/cmd/protoc-gen-connect-go@latest
```
## Project layout — the canonical tree
```
myservice/
├── go.mod
├── go.sum
├── Taskfile.yml # task runner
├── .golangci.yml # see golangci-strict.md
├── .editorconfig
├── .gitignore
├── README.md
├── AGENTS.md # agent-readable project facts
├── cmd/
│ └── server/
│ └── main.go # ONLY: parse flags, call cmd.Execute(); ≤ 50 LOC
├── internal/ # NEVER importable from outside this module
│ ├── api/ # transport layer (gin/connect routers)
│ │ ├── server.go # gin engine setup, route registration
│ │ ├── middleware/
│ │ │ ├── request_id.go
│ │ │ ├── logging.go
│ │ │ └── auth.go
│ │ └── handlers/
│ │ ├── users.go
│ │ └── users_test.go
│ ├── domain/ # parse-don't-validate types, smart constructors
│ │ ├── user.go
│ │ └── email.go
│ ├── service/ # business logic, depends on domain only
│ │ └── user_service.go
│ ├── store/ # persistence; sqlc-generated code lives here
│ │ ├── sqlc/ # sqlc-generated, do not hand-edit
│ │ ├── queries/ # *.sql files sqlc reads
│ │ └── migrations/ # goose migrations
│ ├── config/ # env-driven config (caarlos0/env)
│ │ └── config.go
│ └── obs/ # observability: slog setup, otel, healthz
│ └── logger.go
├── pkg/ # exportable libraries — only if you publish
│ └── …
├── proto/ # *.proto definitions (Connect/gRPC projects)
│ └── service.proto
├── gen/ # generated code (Connect, OpenAPI)
│ └── service/v1/
│ ├── service.pb.go
│ └── servicev1connect/
├── test/ # cross-cutting test helpers, fixtures
└── .github/workflows/ci.yml
```
**Rules**:
- `cmd/<binary>/main.go` is ≤ 50 LOC. Anything more lives in `internal/cmd/`.
- `internal/` is **the** business code. Other modules cannot import it (Go compiler-enforced).
- `pkg/` is for things you genuinely want third parties to import. Empty until proven otherwise.
- No `utils/`, `helpers/`, `common/`, `shared/`. **REJECT.** Files are named after the concept they own.
- One package per directory. One responsibility per package.
## `Taskfile.yml` — the entry point for every action
`go-task/task` is the modern Make replacement. Cross-platform, YAML, fast.
```yaml
version: '3'
vars:
BINARY: server
PKG: ./cmd/server
tasks:
default:
deps: [fmt, lint, test]
fmt:
desc: Format all Go files
cmds:
- gofumpt -w .
- goimports -w -local "$(go list -m)" .
lint:
desc: Run all linters
cmds:
- golangci-lint run --timeout 5m ./...
- nilaway -include-pkgs "$(go list -m)/..." ./...
test:
desc: Run tests with race detector
cmds:
- go test -race -shuffle=on -count=1 ./...
test-cover:
desc: Coverage report
cmds:
- go test -race -shuffle=on -count=1 -coverprofile=coverage.out ./...
- go tool cover -html=coverage.out -o coverage.html
build:
desc: Build the binary
cmds:
- go build -trimpath -ldflags="-s -w" -o bin/{{.BINARY}} {{.PKG}}
run:
desc: Run the server locally
deps: [build]
cmds:
- ./bin/{{.BINARY}}
gen:
desc: Run all code generators
cmds:
- task: gen:sqlc
- task: gen:mocks
- task: gen:proto
gen:sqlc:
cmds:
- sqlc generate
sources:
- internal/store/queries/*.sql
- internal/store/sqlc.yaml
generates:
- internal/store/sqlc/*.go
gen:mocks:
cmds:
- go generate ./...
gen:proto:
cmds:
- buf generate
sources:
- proto/**/*.proto
- buf.yaml
- buf.gen.yaml
migrate:up:
cmds:
- goose -dir internal/store/migrations postgres "$DATABASE_URL" up
migrate:down:
cmds:
- goose -dir internal/store/migrations postgres "$DATABASE_URL" down
ci:
desc: Everything CI does, locally
deps: [fmt, lint, test, build]
```
`task` (no args) runs format + lint + test in parallel where possible. `task ci` runs the full pipeline.
## `go.mod` template
```go
module github.com/your-org/myservice
go 1.23
require (
github.com/caarlos0/env/v11 v11.2.2
github.com/gin-gonic/gin v1.10.1
github.com/go-playground/validator/v10 v10.22.1
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.7.6
golang.org/x/sync v0.18.0
)
```
Only direct deps listed; `go mod tidy` populates indirects.
## `.editorconfig`
```ini
root = true
[*]
indent_style = tab
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[*.{yml,yaml,json,md}]
indent_style = space
indent_size = 2
```
## `.gitignore`
```gitignore
bin/
coverage.out
coverage.html
*.test
*.prof
# IDE
.idea/
.vscode/
*.swp
# Local env
.env
.env.local
# Secrets
*.pem
*.key
```
## CI — minimal GitHub Actions
`.github/workflows/ci.yml`:
```yaml
name: ci
on:
pull_request:
push:
branches: [main]
jobs:
ci:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.23'
cache: true
- name: Install tools
run: |
go install mvdan.cc/gofumpt@latest
go install github.com/golangci/golangci-lint/cmd/golangci-lint@v2.0.0
go install go.uber.org/nilaway/cmd/nilaway@latest
go install github.com/go-task/task/v3/cmd/task@latest
- name: Format check
run: gofumpt -l . | (! grep .)
- name: Lint
run: golangci-lint run --timeout 5m ./...
- name: Nilaway
run: nilaway ./...
- name: Test
run: go test -race -shuffle=on -count=1 ./...
- name: Build
run: go build -trimpath ./...
```
The order matters: format → lint → nilaway → test → build. Fail fast on the cheap checks.
## `AGENTS.md` — agent-readable project facts
Every new project gets an `AGENTS.md` at the root. The content is **machine-friendly**: short, declarative, no marketing prose. Example:
```markdown
# AGENTS.md
Go 1.23+ HTTP service for {one-line purpose}.
## Commands
- `task` — fmt + lint + test
- `task build` — produce ./bin/server
- `task gen` — regenerate sqlc + mocks + proto
## Architecture
- `cmd/server/main.go` — entrypoint, ≤50 LOC
- `internal/api/` — gin handlers + middleware
- `internal/domain/` — smart-constructor types, no I/O
- `internal/store/sqlc/` — generated; never hand-edit
## Conventions
- `slog` for all logs; never `log.*`, never `fmt.Println`
- `context.Context` first arg for every public function
- Errors wrapped with `%w`; check with `errors.Is/As`
- 250 pure LOC ceiling per file — split before adding lines
```
The skill's `cmd/new-project.go` writes this file with project-specific values filled in.
## Sources
- Go modules reference: https://go.dev/ref/mod
- go-task: https://taskfile.dev
- golangci-lint v2: https://golangci-lint.run/docs/configuration/
- Standard project layout debate: https://go.dev/doc/modules/layout (NOT `golang-standards/project-layout` — that repo is community, not official)
packages/shared-skills/skills/programming/references/go/bubbletea-v2.md
+360
View File
@@ -0,0 +1,360 @@
# Bubbletea v2 — TUI with First-Class CJK / IME Support
The TUI stack for 2026. Use **v2 RC**, not v1. If your users include Korean, Japanese, or Chinese speakers, v1 is broken — IME composition lands in the wrong cells. v2 fixes this. This document is the canonical setup.
The reference implementation this document is distilled from: [`code-yeongyu/bubbletea-wm`](https://github.com/code-yeongyu/bubbletea-wm) — a floating window manager built specifically to nail down v2 + IME.
---
## Why v2 (not v1) — the IME story
Bubbletea v1 manages cursor positioning in software ("virtual cursor"). It draws a `█` at the cursor position. The terminal's *real* cursor stays at `(0, 0)`.
This breaks every CJK input method. IME candidate windows (the popup showing 가/각/간 for Korean Hangul composition, kana → kanji for Japanese, pinyin lookup for Chinese) anchor to the terminal's **real** cursor position. With v1, the candidate window appears at top-left while you are typing somewhere in the middle of the screen.
Bubbletea v2 fixes this with two changes:
1. **`tea.View{Cursor: *tea.Cursor}`** — your `View()` method returns a view that *includes* the desired cursor position. The framework moves the terminal's real cursor there.
2. **`textarea.SetVirtualCursor(false)`** — textareas no longer draw their own `█`. They expose `.Cursor()` so you can read where they want the real cursor.
Together: IME popups appear where the user is typing. As they should.
### Other v2 wins (incidental)
- `tea.MouseClickMsg` / `MouseMotionMsg` / `MouseReleaseMsg` instead of one coarse `MouseMsg`.
- Cleaner `View` struct with `AltScreen`, `MouseMode` fields instead of `tea.Cmd` setters.
- Pluggable rendering pipeline; better performance under high message volume.
---
## `go.mod`
```go
module github.com/your-org/mytui
go 1.23
require (
charm.land/bubbletea/v2 v2.0.0-rc.2
charm.land/bubbles/v2 v2.0.0-rc.1
charm.land/lipgloss/v2 v2.0.0-beta.3
github.com/mattn/go-runewidth v0.0.19
)
```
The packages live under `charm.land/` (NOT `github.com/charmbracelet/...`) for v2. This is the Charm team's deliberate import-path break to keep v2 separate from v1 until stable.
---
## Minimal app — the IME-correct skeleton
```go
package main
import (
"fmt"
"log"
tea "charm.land/bubbletea/v2"
"charm.land/bubbles/v2/textarea"
)
type model struct {
width, height int
ta textarea.Model
}
func initial() model {
ta := textarea.New()
ta.Placeholder = "Type Korean / Japanese / Chinese here..."
ta.SetWidth(60)
ta.SetHeight(10)
ta.SetVirtualCursor(false) // ← THE LINE. Without this, IME breaks.
ta.Focus()
return model{ta: ta}
}
func (m model) Init() tea.Cmd { return textarea.Blink }
func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.WindowSizeMsg:
m.width, m.height = msg.Width, msg.Height
case tea.KeyPressMsg:
if msg.String() == "ctrl+c" {
return m, tea.Quit
}
}
var cmd tea.Cmd
m.ta, cmd = m.ta.Update(msg)
return m, cmd
}
func (m model) View() tea.View {
var view tea.View
view.AltScreen = true
view.SetContent(m.ta.View())
// ── THE OTHER LINE. Position the REAL cursor for IME. ──
if cursor := m.ta.Cursor(); cursor != nil {
view.Cursor = cursor
}
return view
}
func main() {
if _, err := tea.NewProgram(initial(), tea.WithAltScreen()).Run(); err != nil {
log.Fatal(err)
}
fmt.Println("bye")
}
```
The two lines that matter:
1. `ta.SetVirtualCursor(false)` — disables the virtual `█`.
2. `view.Cursor = cursor` (where `cursor = m.ta.Cursor()`) — exports the real cursor position to the framework.
Without **both**, IME breaks.
---
## CJK width — go-runewidth, not `len()`
Korean, Japanese, Chinese characters render as **two terminal cells** (wide characters per Unicode East Asian Width). Naive `len(string)` returns byte count, not display width. `utf8.RuneCountInString` returns rune count, also not display width.
Use `github.com/mattn/go-runewidth`:
```go
import "github.com/mattn/go-runewidth"
func displayWidth(s string) int {
return runewidth.StringWidth(s)
}
// Wide character occupies two cells; pad accordingly
for _, r := range s {
cell := string(r)
w := runewidth.RuneWidth(r)
canvas = append(canvas, cell)
if w == 2 {
canvas = append(canvas, "") // placeholder for second cell
}
}
```
`lipgloss/v2` uses `go-runewidth` internally — `lipgloss.Width("안녕")` returns 4, not 2. **If you measure outside lipgloss, you must call runewidth directly.**
---
## Mouse — v2 has typed events
```go
case tea.MouseClickMsg:
// msg.X, msg.Y, msg.Button
return m.handleClick(msg.X, msg.Y, msg.Button)
case tea.MouseMotionMsg:
return m.handleHover(msg.X, msg.Y)
case tea.MouseReleaseMsg:
return m.handleRelease(msg.X, msg.Y)
```
Enable mouse via the `View`:
```go
view.MouseMode = tea.MouseModeCellMotion // or MouseModeAll
```
`CellMotion` reports clicks + motion-while-button-pressed (drag). `MouseModeAll` reports motion always — heavier, only when you need hover.
---
## Components from `bubbles/v2`
```go
import (
"charm.land/bubbles/v2/textarea"
"charm.land/bubbles/v2/textinput"
"charm.land/bubbles/v2/spinner"
"charm.land/bubbles/v2/viewport"
"charm.land/bubbles/v2/list"
"charm.land/bubbles/v2/table"
"charm.land/bubbles/v2/help"
"charm.land/bubbles/v2/key"
)
```
All v2 components support `SetVirtualCursor(false)` where they accept text input. Use it for every text input that users might type CJK into — and "might" should be assumed *yes*.
---
## Styling — `lipgloss/v2`
```go
import "charm.land/lipgloss/v2"
titleStyle := lipgloss.NewStyle().
Bold(true).
Foreground(lipgloss.Color("230")).
Background(lipgloss.Color("62")).
Padding(0, 1).
Border(lipgloss.RoundedBorder()).
BorderForeground(lipgloss.Color("63"))
rendered := titleStyle.Render("안녕하세요")
```
`lipgloss/v2` width and padding correctly account for CJK display width. v1 did too — this is not a v2-specific fix, just a reminder.
---
## Architecture pattern — Model–Update–View
```
+--------------------------------------------+
| tea.Program runs the event loop |
| |
| loop: |
| msg <- queue |
| model, cmd = model.Update(msg) |
| view = model.View() |
| render(view) |
| if cmd != nil: go run(cmd) -> queue |
+--------------------------------------------+
```
Rules:
- **Model is a value type, not a pointer.** Bubbletea calls `Update` with a value receiver and expects a new value returned. Pointer receivers cause subtle bugs where state mutation leaks across draws.
- **`Update` is pure.** No I/O. No goroutines started inline. Any I/O returns a `tea.Cmd` — Bubbletea runs it in a goroutine and feeds the result back as a message.
- **`View` is read-only.** It returns a `tea.View` without modifying state.
- **`tea.Cmd` is `func() tea.Msg`.** It runs once, returns a message, exits. For repeating work, use `tea.Tick` or a self-resending command.
```go
// One-shot command
func loadData() tea.Cmd {
return func() tea.Msg {
data, err := fetch()
if err != nil { return errMsg{err} }
return dataLoadedMsg{data}
}
}
// Periodic
func tickEvery() tea.Cmd {
return tea.Tick(time.Second, func(t time.Time) tea.Msg {
return tickMsg{t}
})
}
```
---
## Splitting the model — sub-models
```go
type model struct {
list list.Model
input textinput.Model
spinner spinner.Model
}
func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
var cmds []tea.Cmd
var cmd tea.Cmd
m.list, cmd = m.list.Update(msg)
cmds = append(cmds, cmd)
m.input, cmd = m.input.Update(msg)
cmds = append(cmds, cmd)
m.spinner, cmd = m.spinner.Update(msg)
cmds = append(cmds, cmd)
return m, tea.Batch(cmds...)
}
```
`tea.Batch` runs commands concurrently. The framework collects their results in the order they arrive.
When the model exceeds 250 LOC, split by sub-model into separate files:
```
internal/ui/
├── model.go # root model orchestration
├── list.go # list sub-model state + update + view
├── input.go # input sub-model
└── spinner.go # spinner sub-model
```
---
## Testing TUI code — `teatest`
```go
import "charm.land/bubbletea/v2/teatest"
func TestModel_typing_korean_keeps_cursor_in_position(t *testing.T) {
// Given
m := initial()
tm := teatest.NewTestModel(t, m, teatest.WithInitialTermSize(80, 24))
// When — simulate typing "안녕"
tm.Send(tea.KeyPressMsg{Code: '안'})
tm.Send(tea.KeyPressMsg{Code: '녕'})
// Then
out := tm.FinalOutput(t)
require.Contains(t, string(out), "안녕")
// Cursor should be at column 4 (two wide chars = 4 cells)
// ...
}
```
`teatest` lets you drive the model through synthetic messages and inspect the rendered output. Pair with `autogold` snapshots for full-view regression tests.
---
## Common antipatterns
| Bad | Why | Good |
|---|---|---|
| `tea.Program` with `tea.WithoutSignals()` | Ctrl-C does not work | Default signal handling |
| Pointer receivers on Model | Bubbletea expects value semantics | Value receivers, return new model |
| `time.Sleep` inside `Update` | Blocks the event loop | `tea.Tick` or async `tea.Cmd` |
| `fmt.Println` for debug | Corrupts the rendered output | `tea.Printf` for logging, or write to a file |
| `len(s)` for CJK width | Off by 2x | `runewidth.StringWidth(s)` |
| `Bubbletea v1` for an app with text input | Korean/Japanese IME breaks | v2 + `SetVirtualCursor(false)` |
| Drawing your own `█` block cursor in v2 | Conflicts with `view.Cursor` | Let the terminal handle it |
---
## Performance — when v2 starts to crawl
- **Reduce View frequency.** If the model changes 60 times/sec but the rendered view changes once/sec, gate redraws on a "dirty" flag.
- **`viewport.Model` for scrollable content.** Avoid re-rendering thousands of lines on every keystroke.
- **`Batch` your commands.** A series of synchronous `tea.Cmd` returns serializes; `tea.Batch` parallelizes.
- **Profile with `tea.WithFPS(N)`** to cap repaint rate during development.
---
## When NOT to use Bubbletea
- The app is one prompt + one answer. Use `huh` (also from Charm) — simpler, no Model–Update–View ceremony.
- The app is a long-running daemon with occasional status output. Use `slog` to stderr and `tea.Program` only if interactivity becomes necessary.
- The app must run as a non-tty subprocess (CI, redirected stdin). `tea.Program` requires a tty for input. Detect via `term.IsTerminal(int(os.Stdin.Fd()))` and fall back to a non-interactive path.
---
## Sources
- bubbletea v2 RC: https://github.com/charmbracelet/bubbletea/tree/v2
- bubbles v2: https://github.com/charmbracelet/bubbles/tree/v2
- lipgloss v2: https://github.com/charmbracelet/lipgloss/tree/v2
- bubbletea-wm (IME reference): https://github.com/code-yeongyu/bubbletea-wm
- crush CLI (production IME impl): https://github.com/charmbracelet/crush
- go-runewidth: https://github.com/mattn/go-runewidth
- Unicode East Asian Width: https://www.unicode.org/reports/tr11/
packages/shared-skills/skills/programming/references/go/cobra-stack.md
+468
View File
@@ -0,0 +1,468 @@
# CLI Stack — cobra + slog + caarlos0/env + signal handling
The canonical Go CLI skeleton. `cobra` is the de facto framework — Kubernetes, Docker CLI, Helm, GitHub CLI, gh, Hugo all use it. Use it.
---
## Toolchain
```bash
go install github.com/spf13/cobra-cli@latest
cobra-cli init mytool
cobra-cli add server
cobra-cli add migrate
```
`cobra-cli` scaffolds the `cmd/` package. Edit the result; do not regenerate.
---
## Layout
```
mytool/
├── go.mod
├── main.go # ≤ 30 LOC, calls cmd.Execute
├── cmd/
│ ├── root.go # rootCmd, persistent flags, slog setup
│ ├── server.go # `mytool server` subcommand
│ ├── migrate.go # `mytool migrate` subcommand
│ └── version.go # `mytool version` — auto-injected version
├── internal/
│ ├── config/
│ └── server/
└── Taskfile.yml
```
---
## `main.go`
```go
package main
import (
"context"
"log/slog"
"os"
"os/signal"
"syscall"
"github.com/your-org/mytool/cmd"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGINT, syscall.SIGTERM)
defer stop()
if err := cmd.Execute(ctx); err != nil {
slog.Error("fatal", slog.Any("err", err))
os.Exit(1)
}
}
```
`signal.NotifyContext` (Go 1.16+) gives every subcommand a ctx that cancels on Ctrl-C. Subcommands plumb the ctx into their workers.
---
## `cmd/root.go`
```go
package cmd
import (
"context"
"log/slog"
"os"
"github.com/spf13/cobra"
)
var (
verbose bool
logFormat string
configPath string
)
var rootCmd = &cobra.Command{
Use: "mytool",
Short: "Short description of mytool",
Long: `Long description, prose; cobra wraps it for --help.`,
PersistentPreRunE: func(c *cobra.Command, args []string) error {
return setupLogger()
},
SilenceUsage: true, // don't print --help on every error
SilenceErrors: true, // we log them ourselves in Execute
}
func init() {
rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false,
"enable debug logging")
rootCmd.PersistentFlags().StringVar(&logFormat, "log-format", "text",
"log format: text or json")
rootCmd.PersistentFlags().StringVarP(&configPath, "config", "c", "",
"path to config file (optional)")
}
func Execute(ctx context.Context) error {
return rootCmd.ExecuteContext(ctx)
}
func setupLogger() error {
level := slog.LevelInfo
if verbose { level = slog.LevelDebug }
opts := &slog.HandlerOptions{Level: level}
var h slog.Handler
switch logFormat {
case "json":
h = slog.NewJSONHandler(os.Stderr, opts)
case "text":
h = slog.NewTextHandler(os.Stderr, opts)
default:
return fmt.Errorf("invalid log-format %q", logFormat)
}
slog.SetDefault(slog.New(h))
return nil
}
```
Notes:
- `RunE` / `PersistentPreRunE` (the `E` variants) return errors. Use these; never use `Run` (no error return, encourages `log.Fatal`).
- `SilenceUsage: true` + `SilenceErrors: true` together: cobra stops printing the full `--help` on every command failure (the default behavior is rude in production scripts).
- `ExecuteContext` (cobra 1.8+) plumbs the ctx into every subcommand's `cmd.Context()`.
---
## `cmd/server.go`
```go
package cmd
import (
"log/slog"
"github.com/spf13/cobra"
"github.com/your-org/mytool/internal/server"
)
var (
serverAddr string
)
var serverCmd = &cobra.Command{
Use: "server",
Short: "Run the HTTP server",
RunE: func(c *cobra.Command, args []string) error {
ctx := c.Context()
slog.InfoContext(ctx, "starting", slog.String("addr", serverAddr))
return server.Run(ctx, serverAddr)
},
}
func init() {
serverCmd.Flags().StringVar(&serverAddr, "addr", ":8080",
"listen address")
rootCmd.AddCommand(serverCmd)
}
```
The subcommand is a thin shim — flags + log line + delegate to `internal/server`. Anything bigger violates the 250-LOC ceiling and belongs in `internal/`.
---
## Subcommands with arguments
```go
var migrateUpCmd = &cobra.Command{
Use: "up [N]",
Short: "Apply N migrations (default: all)",
Args: cobra.MaximumNArgs(1),
RunE: func(c *cobra.Command, args []string) error {
n := -1 // all
if len(args) == 1 {
var err error
n, err = strconv.Atoi(args[0])
if err != nil {
return fmt.Errorf("invalid N: %w", err)
}
}
return migrate.Up(c.Context(), n)
},
}
```
Use cobra's argument validators (`cobra.ExactArgs`, `cobra.MaximumNArgs`, `cobra.OnlyValidArgs`). They produce clean help text.
---
## Flag types — typed, not strings
```go
// GOOD
serverCmd.Flags().DurationVar(&timeout, "timeout", 30*time.Second, "request timeout")
serverCmd.Flags().IntVar(&port, "port", 8080, "port")
serverCmd.Flags().StringSliceVar(&hosts, "host", nil, "allowed hosts (repeatable)")
// BAD — manual parsing
serverCmd.Flags().StringVar(&timeoutStr, "timeout", "30s", "")
// ...then later: time.ParseDuration(timeoutStr)
```
`pflag` (cobra's flag lib) has typed variants for every common type. Use them; the parsing and error messages are free.
---
## Bind flags to env vars
cobra + viper is overkill for env binding. Use `caarlos0/env/v11`:
```go
type ServerOpts struct {
Addr string `env:"ADDR" envDefault:":8080"`
Timeout time.Duration `env:"TIMEOUT" envDefault:"30s"`
}
var opts ServerOpts
var serverCmd = &cobra.Command{
Use: "server",
PersistentPreRunE: func(c *cobra.Command, args []string) error {
// 1. Parse env first.
if err := env.Parse(&opts); err != nil { return err }
// 2. Flags override env if explicitly set.
if c.Flags().Changed("addr") {
opts.Addr, _ = c.Flags().GetString("addr")
}
return nil
},
RunE: func(c *cobra.Command, args []string) error {
return server.Run(c.Context(), opts)
},
}
func init() {
serverCmd.Flags().String("addr", "", "listen address (env: ADDR)")
serverCmd.Flags().Duration("timeout", 0, "request timeout (env: TIMEOUT)")
rootCmd.AddCommand(serverCmd)
}
```
Precedence: **flag (if set) > env > default**. Document the env var in the flag usage string.
---
## Version subcommand — build-injected
```go
// cmd/version.go
package cmd
import (
"fmt"
"runtime/debug"
"github.com/spf13/cobra"
)
// Set by -ldflags at build time, falls back to debug.BuildInfo.
var (
version = ""
commit = ""
date = ""
)
var versionCmd = &cobra.Command{
Use: "version",
Short: "Print version",
Run: func(c *cobra.Command, args []string) {
v, c2, d := resolveVersion()
fmt.Printf("mytool %s (commit %s, built %s)\n", v, c2, d)
},
}
func resolveVersion() (string, string, string) {
if version != "" { return version, commit, date }
info, ok := debug.ReadBuildInfo()
if !ok { return "dev", "unknown", "unknown" }
var vcs, hash, time string
for _, s := range info.Settings {
switch s.Key {
case "vcs.revision": hash = s.Value
case "vcs.time": time = s.Value
case "vcs": vcs = s.Value
}
}
return info.Main.Version, hash, time + " (" + vcs + ")"
}
func init() { rootCmd.AddCommand(versionCmd) }
```
Build with version injection:
```bash
go build \
-ldflags="-X 'github.com/your-org/mytool/cmd.version=v1.2.3' -X 'github.com/your-org/mytool/cmd.commit=$(git rev-parse --short HEAD)' -X 'github.com/your-org/mytool/cmd.date=$(date -u +%Y-%m-%dT%H:%M:%SZ)'" \
-o bin/mytool ./
```
The `debug.BuildInfo` fallback means a `go install`'d binary also has version info — no manual `-ldflags` needed.
---
## Shell completions
```go
var completionCmd = &cobra.Command{
Use: "completion [bash|zsh|fish|powershell]",
Short: "Generate shell completion",
Args: cobra.ExactValidArgs(1),
ValidArgs: []string{"bash", "zsh", "fish", "powershell"},
DisableFlagsInUseLine: true,
RunE: func(c *cobra.Command, args []string) error {
switch args[0] {
case "bash": return rootCmd.GenBashCompletionV2(os.Stdout, true)
case "zsh": return rootCmd.GenZshCompletion(os.Stdout)
case "fish": return rootCmd.GenFishCompletion(os.Stdout, true)
case "powershell": return rootCmd.GenPowerShellCompletion(os.Stdout)
}
return nil
},
}
func init() { rootCmd.AddCommand(completionCmd) }
```
User:
```bash
mytool completion zsh > "${fpath[1]}/_mytool"
```
---
## Interactive prompts — `huh` from charm
For prompts/forms (`Are you sure?`, "Pick an environment", multi-field forms):
```go
import "github.com/charmbracelet/huh"
var confirm bool
err := huh.NewConfirm().
Title("Apply migrations to PRODUCTION?").
Affirmative("Yes, do it").
Negative("Abort").
Value(&confirm).
Run()
```
`huh` replaces `survey` (which is no longer maintained). It composes with `lipgloss` for styling.
---
## Progress / spinners
```go
import "github.com/charmbracelet/huh/spinner"
err := spinner.New().Title("Fetching...").Action(func() {
// long-running work
}).Run()
```
For determinate progress (downloads, batch processing), use `vbauerster/mpb/v8`:
```go
import "github.com/vbauerster/mpb/v8"
p := mpb.New(mpb.WithWidth(60))
bar := p.AddBar(int64(total), /* decorators */)
for i := 0; i < total; i++ {
work()
bar.Increment()
}
p.Wait()
```
---
## Output — JSON vs text
Honor `--output json` for any CLI that scripts will parse:
```go
var outputFmt string
rootCmd.PersistentFlags().StringVar(&outputFmt, "output", "text",
"output format: text or json")
func render(v any) error {
switch outputFmt {
case "json":
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
return enc.Encode(v)
case "text":
return renderText(v)
default:
return fmt.Errorf("invalid --output %q", outputFmt)
}
}
```
The `text` format uses `lipgloss` tables or `aquasecurity/table` for nicely-aligned columns. The `json` format is for `jq`-style piping.
---
## Error semantics
- Return errors from `RunE`. Cobra catches them and the `Execute` wrapper logs + exits non-zero.
- `os.Exit(1)` should appear **only in `main.go`**. Anywhere else means a subcommand cannot be tested.
- For graceful early termination ("user cancelled"), return a sentinel and check it in `Execute`:
```go
var ErrCancelled = errors.New("cancelled by user")
// ... return ErrCancelled
// in main:
if errors.Is(err, cmd.ErrCancelled) { os.Exit(130) } // 128 + SIGINT
```
---
## Testing CLI commands
```go
func TestServerCmd_runs_with_default_addr(t *testing.T) {
// Given
buf := &bytes.Buffer{}
rootCmd.SetOut(buf)
rootCmd.SetErr(buf)
rootCmd.SetArgs([]string{"server", "--addr", ":0"})
// When
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
err := rootCmd.ExecuteContext(ctx)
// Then
require.NoError(t, err)
require.Contains(t, buf.String(), "starting")
}
```
`SetArgs` + `ExecuteContext` is the canonical pattern. Bind a ctx with a short deadline for tests that would otherwise block.
---
## Sources
- cobra docs: https://github.com/spf13/cobra/blob/main/site/content/user_guide.md
- pflag: https://github.com/spf13/pflag
- huh: https://github.com/charmbracelet/huh
- caarlos0/env: https://github.com/caarlos0/env
- signal.NotifyContext: https://pkg.go.dev/os/signal#NotifyContext
packages/shared-skills/skills/programming/references/go/concurrency.md
+362
View File
@@ -0,0 +1,362 @@
# Concurrency
Goroutines, context, errgroup, channels, locks, and the discipline that keeps them from leaking. Go makes concurrency *easy to start* and *easy to get wrong*. This document is the boring rule set.
---
## The four non-negotiables
1. **`ctx context.Context` is the first parameter of every public function that does I/O or can be cancelled.**
2. **No goroutine without a shutdown path.** Every `go` keyword must answer "how does this stop?".
3. **`-race` on every test run.** The `Taskfile.yml` and CI both enforce it.
4. **`goleak` in `TestMain`** for every package that spawns goroutines. Catches leaks the race detector cannot.
---
## `context.Context` — the cancellation backbone
```go
// GOOD — ctx as first param, propagated through
func (s *UserService) Create(ctx context.Context, email Email) (User, error) {
user, err := s.store.Insert(ctx, email)
if err != nil {
return User{}, fmt.Errorf("insert: %w", err)
}
if err := s.notifier.Welcome(ctx, user); err != nil {
return User{}, fmt.Errorf("notify: %w", err)
}
return user, nil
}
// BAD — creates a fresh ctx, breaks request cancellation
func (s *UserService) Create(email Email) (User, error) {
ctx := context.Background() // ← contextcheck linter rejects this
// ...
}
```
The `contextcheck` linter (enabled in `golangci-strict.md`) refuses any function that has `ctx context.Context` available but uses `context.Background()` instead.
### `context.Value` — use sparingly
```go
// Typed key — never use a bare string
type ctxKey struct{ name string }
var requestIDKey = ctxKey{"request_id"}
func WithRequestID(ctx context.Context, id string) context.Context {
return context.WithValue(ctx, requestIDKey, id)
}
func RequestID(ctx context.Context) string {
v, _ := ctx.Value(requestIDKey).(string)
return v
}
```
**Rules**:
- Keys are unexported struct types, not strings. Prevents collisions across packages.
- `context.Value` is for *request-scoped metadata* (request ID, auth subject, trace span), NEVER for application-scoped dependencies.
- Dependencies (loggers, DB pools, config) go in your service struct, not in `context.Value`.
### `WithTimeout` / `WithCancel` — always pair with `defer cancel()`
```go
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel() // ← MUST be deferred. fatcontext linter catches misses.
if err := slow(ctx); err != nil { ... }
```
Forgetting `defer cancel()` leaks a context goroutine until the parent expires — the `lostcancel` vet check catches it.
---
## `errgroup` — the structured concurrency primitive
`golang.org/x/sync/errgroup` is Go's answer to Python's `asyncio.TaskGroup` or Rust's `JoinSet`. Use it instead of raw `go` for any group of related goroutines.
```go
import "golang.org/x/sync/errgroup"
func FetchAll(ctx context.Context, urls []string) ([][]byte, error) {
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8) // concurrency cap — leave unbounded = production outage
results := make([][]byte, len(urls))
for i, u := range urls {
g.Go(func() error {
body, err := fetch(ctx, u)
if err != nil {
return fmt.Errorf("fetch %s: %w", u, err)
}
results[i] = body
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
}
```
Properties:
- `WithContext(parent)` returns a child ctx that gets cancelled on **first non-nil error**. All in-flight goroutines see `ctx.Done()` and bail.
- `SetLimit(n)` blocks `g.Go(...)` when the in-flight count hits `n`. **Always set this.** Unbounded fan-out is how services die.
- `g.Wait()` returns the **first** non-nil error. Others are dropped. If you need all errors, accumulate them manually:
```go
var mu sync.Mutex
var errs []error
// inside g.Go:
// mu.Lock(); errs = append(errs, err); mu.Unlock()
// after Wait, errors.Join(errs...)
```
---
## Goroutine leaks — `goleak`
```go
package store_test
import (
"testing"
"go.uber.org/goleak"
)
func TestMain(m *testing.M) {
goleak.VerifyTestMain(m)
}
```
This single line at the top of `*_test.go` runs goleak's check after every test in the package. If a test leaks a goroutine, the run fails — pointing at which goroutine.
**The bug it catches**: starting a goroutine in `setUp` and never joining it. Common in DB connection pools, background workers, ticker loops. The race detector does NOT catch this.
If you have a known long-lived goroutine (a singleton background worker, a metrics exporter), use `goleak.IgnoreTopFunction`:
```go
goleak.VerifyTestMain(m,
goleak.IgnoreTopFunction("github.com/prometheus/client_golang/prometheus.(*Registry).Push"),
)
```
---
## Channels — the rules that hold
### Direction
```go
// GOOD — direction in signatures
func produce(out chan<- Item)
func consume(in <-chan Item)
func pipeline(in <-chan Item, out chan<- Item)
```
Direction restricts misuse. A consumer cannot close the producer's channel.
### Closing
- **The sender closes.** Always. Never the receiver, never multiple senders.
- **Multiple senders → use a `sync.WaitGroup` + one closer.**
- **Closing a closed channel panics.** Closing a `nil` channel panics. Sending on a closed channel panics. Receiving from a closed channel returns zero value with `ok = false`.
```go
// Canonical fan-in: multiple producers, one closer
func fanIn(ctx context.Context, sources ...<-chan Item) <-chan Item {
out := make(chan Item)
var wg sync.WaitGroup
wg.Add(len(sources))
for _, src := range sources {
go func() {
defer wg.Done()
for item := range src {
select {
case out <- item:
case <-ctx.Done():
return
}
}
}()
}
go func() { wg.Wait(); close(out) }()
return out
}
```
### Selecting
```go
select {
case msg := <-incoming:
handle(msg)
case <-ctx.Done():
return ctx.Err()
case <-time.After(5 * time.Second):
return ErrTimeout
}
```
- `time.After` allocates a timer each call — fine for occasional selects, **NOT for hot loops**. Use `time.NewTimer` + `timer.Reset` for repeat selects.
- A `default:` case makes `select` non-blocking. Use deliberately, not by accident.
### Buffered vs unbuffered
- **Unbuffered** (`make(chan T)`) = synchronous handoff. Sender blocks until receiver is ready. Use for *coordination*.
- **Buffered** (`make(chan T, n)`) = asynchronous up to `n`. Use for *decoupling producer rate from consumer rate*.
A buffered channel of size 1 acts as a **non-blocking signal**:
```go
ready := make(chan struct{}, 1)
// Producer
select {
case ready <- struct{}{}: // signal once, non-blocking
default: // already signaled, skip
}
// Consumer
<-ready
```
---
## Locks — the pyramid
```
Highest level (preferred)
channels (message passing — "share memory by communicating")
errgroup / wait group
sync.RWMutex (many readers, occasional writer)
sync.Mutex (mutual exclusion)
atomic.Int64 / atomic.Pointer (single-word lock-free)
Lowest level (rare)
unsafe.Pointer + barriers (custom lock-free; needs -race AND review)
```
### `sync.Mutex` — embed, don't expose
```go
type Cache struct {
mu sync.RWMutex
items map[string]Entry
}
func (c *Cache) Get(key string) (Entry, bool) {
c.mu.RLock()
defer c.mu.RUnlock()
e, ok := c.items[key]
return e, ok
}
func (c *Cache) Set(key string, e Entry) {
c.mu.Lock()
defer c.mu.Unlock()
c.items[key] = e
}
```
- `sync.Mutex` is **not** copyable. The `copylocks` vet check catches `var c2 = c1` where `c1` has a mutex.
- Always `defer mu.Unlock()` immediately after `Lock()`. Forgetting is the #1 deadlock cause.
- Never call user code (callbacks, listener notifications) while holding the lock. Drop the lock, snapshot the data, release, then call out.
### `sync.OnceValue` / `sync.OnceFunc` (Go 1.21+)
Replacement for `sync.Once` for typed lazy init:
```go
var loadConfig = sync.OnceValue(func() Config {
var cfg Config
if err := env.Parse(&cfg); err != nil { panic(err) }
return cfg
})
func handler() { cfg := loadConfig(); ... }
```
Type-safe, no `sync.Once` + global variable boilerplate.
### Atomics — the typed API only
```go
// Go 1.19+ — use the typed atomic.* family
var counter atomic.Int64
counter.Add(1)
n := counter.Load()
// NEVER — the old function-style is type-unsafe
atomic.AddInt64(&counter, 1) // ← rejected
```
---
## Time — inject a clock for testability
```go
type Clock interface {
Now() time.Time
}
type realClock struct{}
func (realClock) Now() time.Time { return time.Now() }
type Service struct {
clock Clock
}
// Tests
import "github.com/benbjohnson/clock"
fake := clock.NewMock()
fake.Set(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC))
svc := &Service{clock: fake}
```
**Never call `time.Now()` in domain or service code.** The `time` package becomes a hidden dependency — tests become flaky, retries become time-of-day-dependent, expirations cannot be tested.
`time.Sleep` in production code is a code smell. Use:
- `time.NewTicker` for periodic work (and a `<-ctx.Done()` exit).
- `time.NewTimer` for one-shot delays.
- `time.After` ONLY in select statements, ONLY in non-hot paths.
---
## Race detector — non-negotiable in CI
```bash
go test -race -shuffle=on -count=1 ./...
```
- `-race` instruments memory accesses; catches data races at runtime. ~10x slow-down — acceptable for tests, not production.
- `-shuffle=on` randomizes test order; catches hidden ordering dependencies.
- `-count=1` defeats the test cache. Without it, "passing" might mean "ran 3 weeks ago".
If a test ONLY fails under `-race`, the bug is real. Don't disable the test; fix the race.
---
## Common antipatterns
| Bad | Why | Good |
|---|---|---|
| `go func() { ... }()` with no `ctx` plumbing | Leaks on shutdown | `errgroup.WithContext` or pass ctx |
| Bare `time.Sleep(d)` in production | Untestable, blocks | `time.NewTimer` + select with `ctx.Done()` |
| Channel of `interface{}` | Loses type | Typed channel; use sealed interface if variants needed |
| `sync.Mutex` in a struct passed by value | Locked copies, undefined behavior | Embed in pointer-receiver type; copylocks catches it |
| Locking around an entire request handler | Serializes the whole API | Lock only the smallest critical section |
| `for { select { ... } }` without `<-ctx.Done()` | Cannot stop | Add ctx case in every long-lived select |
| `sync.WaitGroup.Add(1)` inside the goroutine | Race: Wait can return before Add | Add **before** `go` |
---
## Sources
- Go memory model: https://go.dev/ref/mem
- `errgroup` package: https://pkg.go.dev/golang.org/x/sync/errgroup
- `goleak`: https://github.com/uber-go/goleak
- "Go concurrency patterns" (Pike): https://go.dev/blog/pipelines
- Sync.OnceValue blog: https://go.dev/blog/synctest (1.24+ note: `testing/synctest` for time-controlled tests is now experimental)
packages/shared-skills/skills/programming/references/go/data-modeling.md
+329
View File
@@ -0,0 +1,329 @@
# Data Modeling — Three Layers of Validation
Go has no Pydantic. Go has no Zod. **You do not need them**, but only if you wire three layers correctly. This document is the canonical pattern.
## The three layers
```
┌─────────────────────────────────────────────────────────────┐
│ HTTP / RPC / CLI │
│ Raw bytes, strings, untrusted input │
│ │
│ Layer 1: validator/v10 (struct tags) ◄── parse-once │
│ OR protovalidate (proto) │
│ │
└──────────────────────────┬──────────────────────────────────┘
│ raw req → domain.X
▼
┌─────────────────────────────────────────────────────────────┐
│ Domain (internal/domain) │
│ │
│ Layer 2: Smart constructors + unexported fields │
│ NewEmail(s) → (Email, error) │
│ NewUserID(s) → (UserID, error) │
│ │
│ Once inside this layer, NO further validation. │
│ The types prove correctness. │
└──────────────────────────┬──────────────────────────────────┘
│ domain.X (proven valid)
▼
┌─────────────────────────────────────────────────────────────┐
│ Storage (internal/store) │
│ │
│ Layer 3: sqlc-generated row structs ↔ domain types │
│ Hand-written mappers, NOT struct tags │
└─────────────────────────────────────────────────────────────┘
```
Each layer parses once, into the next layer's types. **A function in the domain layer should never receive a raw string and validate it.** If it does, the boundary above failed.
---
## Layer 1: HTTP boundary — `go-playground/validator/v10`
```go
package handlers
import (
"github.com/gin-gonic/gin"
"github.com/go-playground/validator/v10"
)
// CreateUserRequest is the wire format. Tags drive validation.
type CreateUserRequest struct {
Email string `json:"email" binding:"required,email"`
Username string `json:"username" binding:"required,alphanum,min=3,max=32"`
Age int `json:"age" binding:"required,gte=13,lte=130"`
Country string `json:"country" binding:"required,iso3166_1_alpha2"`
}
func (h *Handler) CreateUser(c *gin.Context) {
var req CreateUserRequest
if err := c.ShouldBindJSON(&req); err != nil {
// validator returns ValidationErrors with field-by-field detail
var vErr validator.ValidationErrors
if errors.As(err, &vErr) {
c.JSON(400, gin.H{"errors": fieldErrors(vErr)})
return
}
c.JSON(400, gin.H{"error": "invalid json"})
return
}
// Cross into domain — single point of failure
email, err := domain.NewEmail(req.Email)
if err != nil {
c.JSON(400, gin.H{"error": err.Error()})
return
}
username, err := domain.NewUsername(req.Username)
if err != nil {
c.JSON(400, gin.H{"error": err.Error()})
return
}
user, err := h.svc.Create(c.Request.Context(), email, username, req.Age)
if err != nil {
h.writeServiceError(c, err)
return
}
c.JSON(201, user)
}
func fieldErrors(vErr validator.ValidationErrors) map[string]string {
out := make(map[string]string, len(vErr))
for _, fe := range vErr {
out[fe.Field()] = fe.Tag() + "(" + fe.Param() + ")"
}
return out
}
```
**Tag reference — the tags you actually use**:
| Tag | Meaning |
|---|---|
| `required` | Non-zero value |
| `omitempty` (json) | Skip if zero |
| `min=N` / `max=N` | Length (strings/slices) or value (numbers) |
| `gte=N` / `lte=N` / `gt=N` / `lt=N` | Numeric comparison |
| `email` | RFC 5322-ish email |
| `url` | Valid URL |
| `uuid` / `uuid4` / `uuid7` | UUID format |
| `alphanum` / `alpha` / `numeric` | Character class |
| `iso3166_1_alpha2` | Country code (US, KR, JP) |
| `iso4217` | Currency code (USD, KRW) |
| `oneof=a b c` | Enum of literal values |
| `dive` | Apply rules to each element of slice/map |
| `eqfield=Field` | Cross-field equality (e.g., password confirm) |
### Custom validators — register at startup
```go
func init() {
if v, ok := binding.Validator.Engine().(*validator.Validate); ok {
_ = v.RegisterValidation("strongpassword", validateStrongPassword)
}
}
func validateStrongPassword(fl validator.FieldLevel) bool {
s := fl.Field().String()
return len(s) >= 12 && hasUpper(s) && hasDigit(s) && hasSymbol(s)
}
```
Use sparingly. Most domain rules belong in smart constructors, not validators.
---
## Layer 2: Domain — smart constructors
Covered in detail in `type-patterns.md`. Recap:
```go
package domain
type Username struct{ raw string }
func NewUsername(s string) (Username, error) {
s = strings.TrimSpace(s)
if len(s) < 3 || len(s) > 32 {
return Username{}, ErrInvalidUsername
}
if !isAlphanum(s) {
return Username{}, ErrInvalidUsername
}
return Username{raw: s}, nil
}
func (u Username) String() string { return u.raw }
```
**Rule**: every domain type that has invariants has:
1. An unexported field holding the raw form.
2. A `New<Type>(raw) (<Type>, error)` constructor as the sole entry point.
3. A `String() string` for printing.
4. `MarshalJSON` / `UnmarshalJSON` if it crosses a JSON boundary outside HTTP handlers (e.g., logging payloads, queue messages).
5. Optionally: `Scan` and `Value` for `database/sql` interop (rare with sqlc).
---
## Layer 3: Storage — sqlc rows ↔ domain types
sqlc generates row structs from `.sql` files. **Do not put validation tags on them.** Map between sqlc rows and domain types explicitly:
```go
// internal/store/user_store.go
package store
import "myservice/internal/domain"
func (s *UserStore) Get(ctx context.Context, id domain.UserID) (domain.User, error) {
row, err := s.q.GetUser(ctx, string(id))
if err != nil {
return domain.User{}, err
}
return rowToUser(row)
}
func rowToUser(r sqlc.UserRow) (domain.User, error) {
email, err := domain.NewEmail(r.Email)
if err != nil {
// DB invariant broken — this is a programmer error, not a user error
return domain.User{}, fmt.Errorf("db invariant: invalid email for user %s: %w", r.ID, err)
}
username, err := domain.NewUsername(r.Username)
if err != nil {
return domain.User{}, fmt.Errorf("db invariant: invalid username: %w", err)
}
return domain.User{
ID: domain.UserID(r.ID),
Email: email,
Username: username,
Created: r.CreatedAt,
}, nil
}
```
The mapping is verbose. **That is the point.** Each field is a deliberate choice; refactors flag every site.
---
## Discriminated unions (sum types) at the boundary
When a wire payload has variants (e.g., `{"type": "user.created", ...}` vs `{"type": "user.deleted", ...}`):
```go
// Wire DTO with raw discriminator
type EventDTO struct {
Type string `json:"type" binding:"required,oneof=created deleted updated"`
Payload json.RawMessage `json:"payload" binding:"required"`
}
// Parse into the sealed domain type
func ParseEvent(dto EventDTO) (event.Event, error) {
switch dto.Type {
case "created":
var c event.Created
if err := json.Unmarshal(dto.Payload, &c); err != nil {
return nil, fmt.Errorf("decode created: %w", err)
}
return c, nil
case "deleted":
var d event.Deleted
if err := json.Unmarshal(dto.Payload, &d); err != nil {
return nil, fmt.Errorf("decode deleted: %w", err)
}
return d, nil
case "updated":
var u event.Updated
if err := json.Unmarshal(dto.Payload, &u); err != nil {
return nil, fmt.Errorf("decode updated: %w", err)
}
return u, nil
default:
return nil, fmt.Errorf("unknown event type %q", dto.Type)
}
}
```
The `exhaustive` linter on the switch + the `oneof` validation tag together cover both "unknown type" and "unhandled variant".
---
## Enums — typed string consts, not iota
```go
// GOOD — string-based, JSON-serializes correctly, debuggable
type Status string
const (
StatusPending Status = "pending"
StatusActive Status = "active"
StatusClosed Status = "closed"
)
func (s Status) IsValid() bool {
switch s {
case StatusPending, StatusActive, StatusClosed:
return true
}
return false
}
func (s *Status) UnmarshalJSON(data []byte) error {
var raw string
if err := json.Unmarshal(data, &raw); err != nil { return err }
parsed := Status(raw)
if !parsed.IsValid() { return fmt.Errorf("invalid status %q", raw) }
*s = parsed
return nil
}
```
**Never use `iota` enums for anything that crosses a wire boundary.** They serialize as integers, which (a) breaks debuggability, (b) makes reordering enum values a silent breaking change.
Use the validator tag `binding:"oneof=pending active closed"` to enforce at the HTTP boundary.
---
## Nullable fields — `*T` vs sentinel
Three choices, in order of preference:
1. **Sentinel zero value**: `Age int` with `0` meaning "unknown". Works when zero is genuinely unreachable as a valid value.
2. **`sql.Null<T>`** for DB columns: `sql.NullString`, `sql.NullInt64`, `sql.NullTime`. sqlc generates these for nullable columns.
3. **`*T`**: only when you need to distinguish "not provided" from "set to zero" in a JSON payload (PATCH semantics).
```go
// PATCH payload — `*string` discriminates absent vs empty
type UpdateUserRequest struct {
Email *string `json:"email,omitempty"`
Username *string `json:"username,omitempty"`
}
```
Avoid `*T` in domain types — it bloats every consumer with nil checks. Keep `*T` at the boundary, unwrap on the way in.
---
## Common AI-generated antipatterns this rejects
| Bad | Why | Good |
|---|---|---|
| `func handle(req map[string]any)` | No types, no validation | Define a struct, parse with `validator` |
| `if email != "" { ... }` inside domain | Validation in the wrong layer | Make `email Email`, no check needed |
| `type Status int` with `iota` for wire field | Silent breaking on reorder | `type Status string` with const literals |
| Struct tags `json:"email,string"` (the `,string` coercion) | Magic coercion hides bad input | Strict parsing, fail-fast |
| `json.Unmarshal` then range-check after | Two-step "validate after parse" | Use `validator` tags or custom `UnmarshalJSON` |
| Reusing handler DTO as the domain type | Couples wire format to business logic | Two distinct types, explicit mapping |
---
## Sources
- go-playground/validator: https://github.com/go-playground/validator
- gin binding internals: https://github.com/gin-gonic/gin/blob/master/binding/json.go
- Parse, don't validate: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
- sqlc with custom types: https://docs.sqlc.dev/en/latest/howto/overrides.html
packages/shared-skills/skills/programming/references/go/error-handling.md
+359
View File
@@ -0,0 +1,359 @@
# Error Handling
Typed errors, wrap chains, `errors.Is` / `errors.As`, no panic in libraries, resource cleanup. Go errors look simple and are full of footguns. This document is the canonical set of moves.
---
## The five rules
1. **Every error is wrapped on the way up, with `%w`, with context.** Never `return err` from a non-trivial site.
2. **Compare with `errors.Is`, not `==`.** Wrap chains break `==`. The `errorlint` linter forbids `==` on errors.
3. **Cast with `errors.As`, not type assertion.** Same reason.
4. **`panic` is reserved for programmer errors.** Library code never panics on user input or environment failures. Use `(T, error)`.
5. **Resources released via `defer` immediately after acquisition.** No "I'll add it later".
---
## Sentinel errors — for invariant programmatic checks
```go
package domain
import "errors"
var (
ErrInvalidEmail = errors.New("domain: invalid email")
ErrInvalidPhone = errors.New("domain: invalid phone")
ErrInvalidAge = errors.New("domain: invalid age")
)
func NewEmail(s string) (Email, error) {
if !emailRe.MatchString(s) {
return Email{}, fmt.Errorf("email %q: %w", s, ErrInvalidEmail)
}
return Email{raw: strings.ToLower(s)}, nil
}
```
Caller branches on identity:
```go
email, err := domain.NewEmail(input)
if errors.Is(err, domain.ErrInvalidEmail) {
return c.JSON(400, gin.H{"error": "email format"})
}
```
`errors.Is` walks the wrap chain. `err == domain.ErrInvalidEmail` would have failed because `fmt.Errorf` wrapped it.
---
## Typed errors — when you need structured data
When callers need fields off the error (the offending value, the failing field name, the upstream HTTP status):
```go
type ValidationError struct {
Field string
Value string
Rule string
}
func (e *ValidationError) Error() string {
return fmt.Sprintf("validation: %s=%q failed %s", e.Field, e.Value, e.Rule)
}
// Optional: identity sentinel for errors.Is comparisons
var ErrValidation = errors.New("validation")
func (e *ValidationError) Is(target error) bool {
return target == ErrValidation
}
```
Caller:
```go
err := svc.Save(ctx, user)
var vErr *ValidationError
if errors.As(err, &vErr) {
// vErr.Field, vErr.Rule are available
c.JSON(400, gin.H{"field": vErr.Field, "rule": vErr.Rule})
return
}
```
**`errors.As` requires a non-nil pointer-to-pointer.** Almost always the type is `*ConcreteError`. Forgetting the leading `*` is the most common bug here.
---
## Wrapping — `%w` is mandatory
```go
// BAD — drops context
return err
// BAD — drops the error chain (errors.Is/As stops working)
return fmt.Errorf("failed to save user: %v", err)
// GOOD — preserves chain via %w
return fmt.Errorf("save user %s: %w", userID, err)
```
The `errorlint` linter catches `%v` where `%w` was meant. **Wrap once per layer**, with the minimum useful context:
```
api/handler: "create user request: %w"
service: "validate inputs: %w"
domain: "email %q: %w"
```
Each frame adds one fact, not a duplicate. The top-level error message reads as a path: `create user request: validate inputs: email "foo": domain: invalid email`.
### `errors.Join` — multiple errors at once
```go
// Validate all fields, collect all errors
var errs []error
if _, err := NewEmail(req.Email); err != nil {
errs = append(errs, fmt.Errorf("email: %w", err))
}
if _, err := NewUsername(req.Username); err != nil {
errs = append(errs, fmt.Errorf("username: %w", err))
}
if len(errs) > 0 {
return errors.Join(errs...)
}
```
`errors.Is` still walks each joined error. Use when reporting batch validation, not for "wrap two unrelated errors".
---
## Panics — when allowed, when banned
**Banned**:
- Anywhere a `(T, error)` could be returned.
- Inside HTTP handlers (gin's `Recovery` middleware catches them, but you've already lost the error context).
- Inside any goroutine that survives request lifetime.
**Allowed** (with documentation):
- Map literal init at package level: `var statusNames = map[Status]string{...}` followed by a `func init()` that panics if a const has no name. Catches the bug at startup, not runtime.
- The `must*` convention for genuinely unrecoverable startup:
```go
func MustParseURL(s string) *url.URL {
u, err := url.Parse(s)
if err != nil { panic(err) }
return u
}
// Use only with literals known at compile time:
var defaultAPI = MustParseURL("https://api.example.com")
```
- `default:` case of an exhaustive sealed-interface switch — see `type-patterns.md`.
The `revive` linter rule `error-return` will flag suspect panic sites; treat them as bugs.
---
## `defer` for resources — the only safe pattern
```go
func writeReport(path string) (err error) {
f, err := os.Create(path)
if err != nil {
return fmt.Errorf("create %s: %w", path, err)
}
defer func() {
if cerr := f.Close(); cerr != nil && err == nil {
err = fmt.Errorf("close %s: %w", path, cerr)
}
}()
if _, err := f.Write(data); err != nil {
return fmt.Errorf("write %s: %w", path, err)
}
return nil
}
```
Key points:
- `defer f.Close()` immediately after `os.Create` — never further down.
- Named return `(err error)` so the deferred close can mutate it on close failure.
- `bodyclose` linter catches missed `defer resp.Body.Close()` for HTTP responses.
- `sqlclosecheck` linter catches missed `defer rows.Close()` for SQL.
### `errors.Join` for multi-stage cleanup
```go
func process(path string) (err error) {
f, err := os.Open(path)
if err != nil { return err }
defer func() {
err = errors.Join(err, f.Close())
}()
// ... use f ...
return nil
}
```
When both the main operation AND `Close` can fail, `errors.Join` reports both without dropping either.
---
## HTTP error responses — a single funnel
Build one helper, route all handler errors through it:
```go
package httperr
type APIError struct {
Status int `json:"-"`
Code string `json:"code"`
Message string `json:"message"`
}
func (e *APIError) Error() string { return e.Code + ": " + e.Message }
var (
NotFound = &APIError{Status: 404, Code: "not_found", Message: "resource not found"}
Unauthorized = &APIError{Status: 401, Code: "unauthorized", Message: "unauthorized"}
BadRequest = &APIError{Status: 400, Code: "bad_request", Message: "bad request"}
Internal = &APIError{Status: 500, Code: "internal", Message: "internal error"}
)
// Wrap a domain error into an API error.
func From(err error) *APIError {
if err == nil { return nil }
var apiErr *APIError
if errors.As(err, &apiErr) { return apiErr }
switch {
case errors.Is(err, domain.ErrInvalidEmail),
errors.Is(err, domain.ErrInvalidUsername):
return &APIError{Status: 400, Code: "validation", Message: err.Error()}
case errors.Is(err, ErrNotFound):
return NotFound
case errors.Is(err, ErrUnauthorized):
return Unauthorized
default:
// unknown — log full chain, return generic
slog.Error("unmapped error", slog.Any("err", err))
return Internal
}
}
func Write(c *gin.Context, err error) {
apiErr := From(err)
c.JSON(apiErr.Status, apiErr)
}
```
Handlers become trivial:
```go
func (h *Handler) Create(c *gin.Context) {
user, err := h.svc.Create(c.Request.Context(), req)
if err != nil {
httperr.Write(c, err)
return
}
c.JSON(201, user)
}
```
---
## errgroup — error propagation across goroutines
```go
import "golang.org/x/sync/errgroup"
func fetchAll(ctx context.Context, urls []string) ([][]byte, error) {
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8) // concurrency cap
results := make([][]byte, len(urls))
for i, u := range urls {
g.Go(func() error {
body, err := fetch(ctx, u)
if err != nil {
return fmt.Errorf("fetch %s: %w", u, err)
}
results[i] = body
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
}
```
- `errgroup.WithContext` cancels remaining tasks on first error.
- `SetLimit` bounds concurrency.
- First non-nil error is returned; others are discarded — by design.
See `concurrency.md` for the full pattern.
---
## Logging errors — structured, once
```go
slog.ErrorContext(ctx, "save user failed",
slog.String("user_id", string(id)),
slog.Any("err", err), // %w chain is fully rendered
)
```
**Log once, at the outermost frame.** Logging at every wrap site produces five log lines for one error.
The `sloglint` linter enforces `slog.Any("err", err)` over `slog.String("err", err.Error())` — the former preserves the chain when handlers walk the value.
---
## Antipatterns
| Bad | Why | Good |
|---|---|---|
| `_ = err` | Silent ignore | Handle, log, or wrap |
| `if err != nil { return err }` chained 10 deep without wrap | No path info | Add one fact per layer: `fmt.Errorf("step: %w", err)` |
| `panic(err)` in HTTP handlers | Loses error chain, hits gin Recovery | `httperr.Write(c, err)` |
| `err.Error() == "some string"` | Brittle, breaks on wrap | Define a sentinel, use `errors.Is` |
| `if err == sql.ErrNoRows` | Breaks under wrap | `errors.Is(err, sql.ErrNoRows)` |
| `catch-all log.Fatal(err)` in library code | Crashes the caller's process | Return error, let main decide |
| Returning a typed nil pointer wrapped in error interface | Classic "nil != nil" bug | Return explicit `nil` for the error |
The last bug deserves its own example:
```go
// BUG — returns a non-nil error interface containing a nil concrete type
func bad() error {
var e *MyError = nil
return e // interface wraps nil pointer; errors == nil is FALSE
}
// Caller
if err := bad(); err != nil {
// ← entered, but err.(*MyError) is nil — surprise panic
}
```
Fix: return explicit `nil`, not a typed nil. The `nilnil` linter catches this in `(T, error)` returns.
---
## Sources
- Go blog "Working with Errors in Go 1.13+": https://go.dev/blog/go1.13-errors
- `errors.Join` (Go 1.20+): https://pkg.go.dev/errors#Join
- errorlint: https://github.com/polyfloyd/go-errorlint
- nilaway nil-interface check: https://github.com/uber-go/nilaway
packages/shared-skills/skills/programming/references/go/golangci-strict.md
+236
View File
@@ -0,0 +1,236 @@
# Strict `.golangci.yml` (golangci-lint v2)
The single source of truth for "is this Go code acceptable". Drop this in unmodified. **Every linter below is enabled deliberately — read the rationale before disabling one.**
`golangci-lint` v2 changed config schema (top-level `version: "2"`). All v1 configs are incompatible. The block below is v2.
## `.golangci.yml`
```yaml
version: "2"
run:
timeout: 5m
tests: true
modules-download-mode: readonly
linters:
default: none
enable:
# ── Correctness — bug catchers ───────────────────────────────
- govet # stdlib vet, includes shadow, fieldalignment, nilness
- staticcheck # SA1*-SA9* — the de facto Go correctness linter
- errcheck # unhandled errors. ZERO tolerance.
- errorlint # %w wrapping, errors.As vs type-assertion, errors.Is vs ==
- nilerr # `return nil` after `err != nil` — classic bug
- nilnil # returning `(nil, nil)` from a (*T, error) function
- bodyclose # http.Response.Body not closed
- rowserrcheck # sql.Rows.Err() not checked
- sqlclosecheck # sql.Rows / sql.Stmt not closed
- contextcheck # functions taking context.Context don't get context.Background()
- fatcontext # context.WithValue() in a loop — leaks
- copyloopvar # Go 1.22 loop-var capture — should now use the new semantics
- intrange # use `for i := range N` (Go 1.22+) instead of `for i := 0; i < N; i++`
- usetesting # use t.TempDir/t.Setenv over os.* in tests
- testifylint # require vs assert correctness, ObjectsAreEqual misuse
# ── Style / readability — kept narrow to avoid bikeshedding ─
- gofumpt # stricter gofmt
- goimports # import grouping + local prefix
- whitespace # leading/trailing whitespace
- misspell # typos in comments and strings
- unconvert # redundant type conversions
- unparam # unused function parameters
- ineffassign # ineffective assignments
- dupword # duplicate words ("the the")
# ── Architecture — file size, complexity, dead code ─────────
- gocognit # cognitive complexity per function (threshold 25)
- gocyclo # cyclomatic complexity per function (threshold 15)
- funlen # function length (90 lines, 60 statements)
- lll # line length 120
- nestif # excessive nesting depth (>4)
- dupl # duplicate code blocks
- revive # extensible replacement for golint; selected rules below
- unused # unused vars/funcs/types
# ── Exhaustiveness — Go's weakest spot ──────────────────────
- exhaustive # type switch and enum-like const groups completeness
# ── Security ────────────────────────────────────────────────
- gosec # CWE-aware security scanner
# ── Logging ─────────────────────────────────────────────────
- sloglint # slog attr style + no slog.Any(); enforce structured logs
# ── Performance ─────────────────────────────────────────────
- perfsprint # fmt.Sprintf where strconv suffices
- prealloc # slice prealloc when length is known
- makezero # make([]T, n) with non-zero n then append (the classic bug)
linters-settings:
errcheck:
check-type-assertions: true
check-blank: true # `_ = err` is a violation
govet:
enable-all: true
settings:
shadow:
strict: true
fieldalignment:
# On by default; this catches struct layouts wasting memory.
# Disable per-file with //nolint:fieldalignment ONLY for boundary types
# whose JSON tag order matters for OpenAPI doc stability.
errorlint:
errorf: true # %w mandatory for wrapping
asserts: true # errors.As over type-assertion on `error`
comparison: true # errors.Is over ==
gocognit:
min-complexity: 25
gocyclo:
min-complexity: 15
funlen:
lines: 90
statements: 60
ignore-comments: true
lll:
line-length: 120
tab-width: 4
nestif:
min-complexity: 4
exhaustive:
default-signifies-exhaustive: false
check:
- switch
- map
gosec:
excludes:
- G104 # handled by errcheck/errorlint
- G304 # file path provided as input — too noisy for CLIs
sloglint:
no-mixed-args: true # all attr or all key-value, never mixed
kv-only: false
attr-only: true # force slog.String(...) form
no-global: all # disallow slog.Info; force a logger receiver
context: scope # require *Context variants where ctx is in scope
static-msg: true # msg must be a string literal (not fmt.Sprintf)
no-raw-keys: true # use slog.String("key", ...) not raw "key", "val"
key-naming-case: snake
testifylint:
enable-all: true
disable:
- require-error # We DO use assert.Error in table-driven loops
revive:
severity: warning
rules:
- name: var-naming
- name: package-comments
- name: exported
- name: error-return
- name: error-naming
- name: errorf # use fmt.Errorf instead of errors.New(fmt.Sprintf)
- name: if-return
- name: indent-error-flow
- name: range-val-in-closure
- name: redefines-builtin-id
- name: superfluous-else
- name: unhandled-error
arguments:
- "fmt.Print.*"
- "fmt.Fprint.*"
perfsprint:
integer-format: true
error-format: true
bool-format: true
string-format: true
goimports:
local-prefixes:
- github.com/your-org
issues:
max-issues-per-linter: 0
max-same-issues: 0
exclude-rules:
# Tests get a longer leash on funlen + lll
- path: _test\.go
linters:
- funlen
- lll
- dupl
- gosec
# Generated code never lints
- path: \.pb\.go$
linters: [all]
- path: \.connect\.go$
linters: [all]
- path: ^.*sqlc/.*\.sql\.go$
linters: [all]
formatters:
enable:
- gofumpt
- goimports
```
## Per-linter rationale (why each is on)
| Linter | What it catches | Why no compromise |
|---|---|---|
| `errcheck` (incl. `check-blank: true`) | `_ = err`, ignored errors from `Close()`, `Write()`, `json.Marshal()` | Silent error ignore is the #1 Go bug class. Banning `_ = err` forces a decision at every site. |
| `errorlint` | `err == io.EOF` instead of `errors.Is(err, io.EOF)`; missing `%w` in `fmt.Errorf` | Once you wrap in middleware, `==` checks silently break. `errors.Is/As` is the only safe form. |
| `nilerr` / `nilnil` | `return nil` after `err != nil`; `return nil, nil` from `(*T, error)` | Classic AI-generated bugs. Linter catches them mechanically. |
| `bodyclose` | `defer resp.Body.Close()` missed | Single most common Go memory leak. |
| `contextcheck` | `ctx := context.Background()` inside a function that received `ctx` | Breaks cancellation propagation — the entire reason ctx exists. |
| `exhaustive` | `switch x.(type)` missing a sealed-interface variant | **Go's weakest type-system spot.** This linter is the closest thing to compiler-enforced exhaustiveness. |
| `sloglint` | `slog.Info(...)` (global), mixed `Any`/typed attrs | Without this, structured logging silently degrades into string concatenation. |
| `govet/shadow` strict | `err := ... ; if ... { err := ...; ... }` shadowing | Hides the real error from outer scope — extremely common. |
| `govet/fieldalignment` | Struct field order wasting memory | Cheap correctness signal. Disable per-file when JSON tag order matters for OpenAPI. |
| `copyloopvar` + `intrange` | Pre-1.22 loop-var capture and old `for i := 0; i < N; i++` | The language modernized; the lint enforces it. |
| `usetesting` | `os.Setenv` / `os.Mkdir` in tests instead of `t.Setenv` / `t.TempDir` | Avoids test isolation bugs. |
| `gocognit` / `gocyclo` / `funlen` | Functions exceeding cognitive thresholds | Direct architectural signal — same purpose as the 250 LOC ceiling, at function granularity. |
| `gosec` | CWE patterns — SQL injection, weak crypto, path traversal | Production must pass this. |
| `testifylint` | `assert.Equal` where `require.Equal` was meant; `ObjectsAreEqual` misuse | Subtle test-correctness bugs. |
| `perfsprint` | `fmt.Sprintf("%d", n)` instead of `strconv.Itoa(n)` | 5–10x faster in tight loops, lints catch the lazy form. |
## `nolint` policy
`//nolint:linter1,linter2 // <reason>` is permitted with **two hard rules**:
1. **One linter at a time per directive.** No `//nolint:all`. No omitting the linter name.
2. **A reason after `//` is mandatory.** "Generated code", "false positive — protobuf imports", "OpenAPI field order" are acceptable. "Ignore" is not.
The skill auto-rejects `//nolint` without a reason. So does `revive` if you enable its `nolint` rule.
## CI gate
```bash
gofumpt -l . | (! grep .) # format
golangci-lint run --timeout 5m ./... # everything above
go vet -vettool=$(which fieldalignment) ./... # extra check (also in govet)
nilaway ./... # nil-deref static analysis
go test -race -shuffle=on -count=1 ./... # races + ordering
```
Any non-zero exit = the change does not ship.
## Sources
- golangci-lint v2 docs: https://golangci-lint.run/docs/configuration/
- staticcheck rules: https://staticcheck.dev/docs/checks
- sloglint: https://github.com/go-simpler/sloglint
- exhaustive: https://github.com/nishanths/exhaustive
- nilaway: https://github.com/uber-go/nilaway
packages/shared-skills/skills/programming/references/go/grpc-connect.md
+375
View File
@@ -0,0 +1,375 @@
# RPC — Connect-Go (default) + grpc-go (fallback) + protovalidate
`connectrpc/connect-go` is the default. It is wire-compatible with gRPC, also speaks Connect protocol + gRPC-Web from browsers, and uses ordinary `net/http` so middleware (logging, auth, tracing) composes the same way as REST. Reach for raw `grpc-go` only when you need a gRPC-specific feature Connect lacks.
---
## When Connect vs grpc-go
| Need | Use |
|---|---|
| Standard unary + server-streaming + client-streaming | **Connect** |
| Browser client without `grpc-web` proxy | **Connect** (native gRPC-Web support) |
| HTTP/1.1 fallback for hostile networks | **Connect** (gRPC requires HTTP/2 end-to-end) |
| Server reflection for `grpcurl` | grpc-go (Connect has reflection too, but ecosystem smaller) |
| Bidirectional streaming with frame-level control | grpc-go |
| Strict gRPC environment (Envoy with gRPC filters, Istio strict mode) | grpc-go |
**Default**: Connect. The default has been correct since 2024.
---
## Toolchain — Buf, not protoc
```bash
go install github.com/bufbuild/buf/cmd/buf@latest
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install connectrpc.com/connect/cmd/protoc-gen-connect-go@latest
go install github.com/bufbuild/protovalidate/cmd/protoc-gen-go-vtproto@latest
```
Buf replaces `protoc` for everything: linting, breaking-change detection, codegen, formatting. The `protoc` toolchain is dead-letter walking — every modern proto project uses Buf.
---
## Project layout
```
proto/
buf.yaml
buf.gen.yaml
buf.lock
myservice/v1/
user.proto
auth.proto
gen/
myservice/v1/
user.pb.go # protoc-gen-go output
auth.pb.go
myservicev1connect/ # protoc-gen-connect-go output
user.connect.go
auth.connect.go
```
**`gen/` is committed.** Generated code is part of the API contract; CI proves it is up-to-date.
---
## `buf.yaml`
```yaml
version: v2
modules:
- path: proto
lint:
use:
- STANDARD
breaking:
use:
- FILE
```
## `buf.gen.yaml`
```yaml
version: v2
managed:
enabled: true
override:
- file_option: go_package_prefix
value: github.com/your-org/myservice/gen
plugins:
- remote: buf.build/protocolbuffers/go
out: gen
opt:
- paths=source_relative
- remote: buf.build/connectrpc/go
out: gen
opt:
- paths=source_relative
- remote: buf.build/bufbuild/validate-go
out: gen
opt:
- paths=source_relative
```
The `buf.build/...` plugin URIs use Buf's hosted remote registry — no local plugin installation needed.
## Taskfile target
```yaml
gen:proto:
cmds:
- buf lint
- buf format -w
- buf generate
sources:
- proto/**/*.proto
- buf.yaml
- buf.gen.yaml
```
Run `task gen:proto` after editing any `.proto`. CI runs `buf generate` then `git diff --exit-code` to catch stale generated code.
---
## A `.proto` with validation
```proto
syntax = "proto3";
package myservice.v1;
import "buf/validate/validate.proto";
option go_package = "github.com/your-org/myservice/gen/myservice/v1;myservicev1";
service UserService {
rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
rpc GetUser(GetUserRequest) returns (GetUserResponse);
rpc StreamEvents(StreamEventsRequest) returns (stream Event);
}
message CreateUserRequest {
string email = 1 [(buf.validate.field).string.email = true];
string username = 2 [
(buf.validate.field).string.min_len = 3,
(buf.validate.field).string.max_len = 32,
(buf.validate.field).string.pattern = "^[a-zA-Z0-9_]+$"
];
int32 age = 3 [
(buf.validate.field).int32.gte = 13,
(buf.validate.field).int32.lte = 130
];
}
message CreateUserResponse {
User user = 1;
}
message User {
string id = 1;
string email = 2;
string username = 3;
google.protobuf.Timestamp created_at = 4;
}
```
`protovalidate` replaces the abandoned `protoc-gen-validate` — it is the official Buf-backed successor as of 2024, supported by Connect's interceptor pipeline.
---
## Server — Connect
```go
package main
import (
"context"
"log/slog"
"net/http"
"connectrpc.com/connect"
"buf.build/go/protovalidate"
validateinterceptor "connectrpc.com/validate"
"golang.org/x/net/http2"
"golang.org/x/net/http2/h2c"
myservicev1 "github.com/your-org/myservice/gen/myservice/v1"
"github.com/your-org/myservice/gen/myservice/v1/myservicev1connect"
)
type UserServer struct {
svc *UserService
}
func (s *UserServer) CreateUser(
ctx context.Context,
req *connect.Request[myservicev1.CreateUserRequest],
) (*connect.Response[myservicev1.CreateUserResponse], error) {
// protovalidate already ran via the interceptor below.
// req.Msg is guaranteed to satisfy the .proto constraints.
user, err := s.svc.Create(ctx, req.Msg.Email, req.Msg.Username, req.Msg.Age)
if err != nil {
return nil, mapError(err)
}
return connect.NewResponse(&myservicev1.CreateUserResponse{
User: userToProto(user),
}), nil
}
func main() {
validator, _ := protovalidate.New()
interceptors := connect.WithInterceptors(
loggingInterceptor(),
validateinterceptor.NewInterceptor(validator),
)
mux := http.NewServeMux()
mux.Handle(myservicev1connect.NewUserServiceHandler(
&UserServer{svc: newUserService()},
interceptors,
))
// h2c lets the server speak HTTP/2 cleartext for gRPC clients.
srv := &http.Server{
Addr: ":8080",
Handler: h2c.NewHandler(mux, &http2.Server{}),
}
slog.Info("rpc server listening", slog.String("addr", srv.Addr))
if err := srv.ListenAndServe(); err != nil { slog.Error("rpc", slog.Any("err", err)) }
}
```
The handler is **just an `http.Handler`** — mount it in the same `http.ServeMux` as your REST routes if you want one binary serving both.
---
## Error mapping — Connect codes
```go
func mapError(err error) error {
if err == nil { return nil }
switch {
case errors.Is(err, domain.ErrInvalidEmail),
errors.Is(err, domain.ErrInvalidUsername):
return connect.NewError(connect.CodeInvalidArgument, err)
case errors.Is(err, ErrNotFound):
return connect.NewError(connect.CodeNotFound, err)
case errors.Is(err, ErrUnauthorized):
return connect.NewError(connect.CodeUnauthenticated, err)
case errors.Is(err, ErrConflict):
return connect.NewError(connect.CodeAlreadyExists, err)
default:
slog.Error("unmapped rpc error", slog.Any("err", err))
return connect.NewError(connect.CodeInternal, errors.New("internal"))
}
}
```
Connect codes map 1:1 to gRPC codes. Clients see canonical error semantics.
---
## Logging interceptor
```go
func loggingInterceptor() connect.UnaryInterceptorFunc {
return func(next connect.UnaryFunc) connect.UnaryFunc {
return func(ctx context.Context, req connect.AnyRequest) (connect.AnyResponse, error) {
start := time.Now()
res, err := next(ctx, req)
attrs := []slog.Attr{
slog.String("proc", req.Spec().Procedure),
slog.Duration("elapsed", time.Since(start)),
}
if err != nil {
attrs = append(attrs, slog.Any("err", err))
slog.LogAttrs(ctx, slog.LevelWarn, "rpc failed", attrs...)
} else {
slog.LogAttrs(ctx, slog.LevelInfo, "rpc ok", attrs...)
}
return res, err
}
}
}
```
For streaming, implement the full `connect.Interceptor` (`WrapStreamingClient`, `WrapStreamingHandler`). Pattern is identical.
---
## Server streaming
```go
func (s *UserServer) StreamEvents(
ctx context.Context,
req *connect.Request[myservicev1.StreamEventsRequest],
stream *connect.ServerStream[myservicev1.Event],
) error {
events, errs := s.svc.Subscribe(ctx, req.Msg.UserId)
for {
select {
case <-ctx.Done():
return ctx.Err()
case e, ok := <-events:
if !ok { return nil }
if err := stream.Send(eventToProto(e)); err != nil {
return err
}
case err := <-errs:
return connect.NewError(connect.CodeInternal, err)
}
}
}
```
Same shape as SSE in `backend-stack.md`. Connect handles HTTP/2 framing.
---
## Client
```go
client := myservicev1connect.NewUserServiceClient(
http.DefaultClient,
"https://api.example.com",
// Use connect.WithGRPC() if the server is grpc-go and you want strict gRPC framing.
// Default is Connect protocol — works with Connect or gRPC servers transparently.
)
res, err := client.CreateUser(ctx, connect.NewRequest(&myservicev1.CreateUserRequest{
Email: "a@b.com",
Username: "alice",
Age: 30,
}))
if err != nil {
var connectErr *connect.Error
if errors.As(err, &connectErr) {
slog.Error("rpc failed",
slog.String("code", connectErr.Code().String()),
slog.String("msg", connectErr.Message()))
}
return err
}
slog.Info("created", slog.String("id", res.Msg.User.Id))
```
---
## When you genuinely need raw grpc-go
```go
import "google.golang.org/grpc"
lis, _ := net.Listen("tcp", ":8080")
srv := grpc.NewServer(
grpc.UnaryInterceptor(loggingUnaryInterceptor),
)
myservicev1.RegisterUserServiceServer(srv, &userServer{})
_ = srv.Serve(lis)
```
The codegen is from `protoc-gen-go-grpc` (different binary from `protoc-gen-connect-go`). You can codegen **both** in the same `buf.gen.yaml` and switch by importing the right package. Most teams pick one.
---
## When NOT to use RPC at all
If your callers are all browsers, mobile apps, third-party developers, or the long tail of "things humans curl": **stay with REST + OpenAPI**. RPC's overhead is justified for service-to-service inside a single org. Outside that boundary, JSON over HTTP wins on debuggability.
`oapi-codegen/oapi-codegen/v2` generates Go server stubs and clients from OpenAPI 3 — the REST equivalent of what Connect does for proto. Same parse-don't-validate boundary discipline, different wire format.
---
## Sources
- Connect docs: https://connectrpc.com/docs/go/getting-started
- Buf: https://buf.build/docs
- protovalidate: https://github.com/bufbuild/protovalidate
- "Why we replaced protoc with buf" (Buf blog): https://buf.build/blog
- gRPC vs Connect comparison: https://connectrpc.com/docs/introduction
packages/shared-skills/skills/programming/references/go/libraries.md
+337
View File
@@ -0,0 +1,337 @@
# Library Defaults — Full Decision Tree (Go 2026)
The opinionated, in-production stack for 2026 Go. Every entry has a one-line rationale and a canonical snippet so the agent does not relearn each library's idioms.
The biggest difference from Python/Rust/TypeScript: **Go has fewer "best" choices and more "boring" choices.** The standard library is the default; reach outside it only when the rationale below applies.
---
## HTTP framework — `gin` (default) or `chi` (minimalist) or `net/http` (no deps)
The reality of 2026 Go: **`gin` runs ~48% of new Go API projects** (Go Developer Survey 2024 + crawls of new repos), with `gorilla/mux` (~17%, in maintenance), `echo` (~16%), and `fiber` (~11%) the remaining quarter. The skill picks gin not because it is technically superior — it is not — but because:
1. The ecosystem (middleware, examples, SO answers) is largest.
2. The CLIProxyAPI codebase, which this skill's `backend-stack.md` is distilled from, uses gin in production for OpenAI/Gemini/Claude proxying including SSE streaming and WebSocket upgrades. That is real reference code, not a toy.
3. Gin's `Context` API is the closest thing Go has to a framework-blessed "request-scoped object", which makes middleware composition straightforward.
```go
import "github.com/gin-gonic/gin"
func main() {
r := gin.New()
r.Use(gin.Recovery(), middleware.RequestLogger(), middleware.RequestID())
r.GET("/healthz", func(c *gin.Context) { c.JSON(200, gin.H{"ok": true}) })
_ = r.Run(":8080")
}
```
**Pick `chi` instead** when:
- You want `net/http`-compatible handlers (you do, eventually — chi is closer to stdlib).
- The service is small and you do not need gin's binding helpers.
**Pick `net/http` (stdlib) directly** when:
- The service has fewer than 10 routes and zero auth complexity. Go 1.22's enhanced `ServeMux` (method+path patterns) eliminated 80% of the historical reason to use a framework.
**Never use** `gorilla/mux` (effectively in maintenance), `fiber` (uses `fasthttp` which is **not stdlib-compatible**, so middleware ecosystem is split), or `echo` (smaller eco than gin, no real advantage today).
See `backend-stack.md` for the gin canonical layout, middleware ordering, SSE, graceful shutdown, structured logging integration.
---
## RPC — `connectrpc/connect-go`
The default RPC layer. **Use Connect, not raw grpc-go**, unless you have a measured reason.
- Connect is wire-compatible with gRPC AND speaks HTTP/1.1 + HTTP/2 + Connect protocol. One server, three clients (gRPC, gRPC-Web, Connect-Web from browsers).
- No `grpcurl` needed for debugging — `curl -H "Content-Type: application/json" -d ...` works.
- Streaming, interceptors, deadlines, errors are first-class.
- Buf toolchain (`buf generate`, `buf lint`, `buf breaking`) for codegen is dramatically nicer than `protoc`.
```go
// Server
mux := http.NewServeMux()
mux.Handle(elizav1connect.NewElizaServiceHandler(&elizaServer{}))
_ = http.ListenAndServe(":8080", h2c.NewHandler(mux, &http2.Server{}))
// Client
client := elizav1connect.NewElizaServiceClient(
http.DefaultClient,
"http://localhost:8080",
)
res, err := client.Say(ctx, connect.NewRequest(&elizav1.SayRequest{Sentence: "hi"}))
```
**Use raw `grpc-go`** only when:
- You need server-streaming-from-multiple-services with a single gRPC mux.
- You are integrating with a strict gRPC-only environment (Envoy proxy with gRPC reflection, Istio strict-gRPC).
See `grpc-connect.md`.
---
## Database — `pgx/v5` + `sqlc` + `goose`
```bash
go get github.com/jackc/pgx/v5
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
go install github.com/pressly/goose/v3/cmd/goose@latest
```
- **`pgx/v5`** is faster, more type-safe, and has better PostgreSQL feature coverage than `database/sql + lib/pq`. Use the `pgxpool` package for connection pooling. Avoid `database/sql` driver mode — it loses pgx's batch, COPY, listen/notify.
- **`sqlc`** generates type-safe Go from `.sql` files. Hand-written SQL with hand-written struct mapping is the #1 source of subtle DB bugs. sqlc eliminates the class.
- **`goose`** for migrations — small, command-line first, no global state.
**Never use** `gorm` (active record, slow, brings runtime reflection into hot paths, encourages N+1 queries). **Never use** `ent` (heavy, opinionated graph layer) unless you specifically want a graph-shaped data model.
See `sqlc-pgx.md`.
---
## Validation — three layers, three tools
Go has no Pydantic / Zod equivalent and **does not need one** — but only because you wire three layers properly:
| Layer | Tool | Pattern |
|---|---|---|
| HTTP boundary (gin/chi/net/http) | `go-playground/validator/v10` via struct tags | `binding:"required,email,min=3"` |
| RPC boundary (protobuf) | `bufbuild/protovalidate-go` | `(buf.validate.field).string.min_len = 3` in `.proto` |
| Domain core | **Smart constructor + unexported fields** | `NewEmail(s) (Email, error)` returns a type whose fields cannot be set from outside |
```go
// HTTP boundary
type CreateUserReq struct {
Email string `json:"email" binding:"required,email"`
Username string `json:"username" binding:"required,alphanum,min=3,max=32"`
}
// Domain — once a value is of type Email it is provably valid
type Email struct{ raw string }
func NewEmail(s string) (Email, error) {
if !emailRegex.MatchString(s) { return Email{}, ErrInvalidEmail }
return Email{raw: strings.ToLower(s)}, nil
}
func (e Email) String() string { return e.raw }
```
The boundary parses raw input into the domain type **once**. Inside the domain, no further validation is permitted — the types prove it. This is parse-don't-validate adapted to Go.
See `data-modeling.md` for the full pattern.
---
## Logging — `log/slog` (stdlib)
```go
import "log/slog"
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
AddSource: true,
}))
slog.SetDefault(logger)
slog.InfoContext(ctx, "request handled",
slog.String("path", r.URL.Path),
slog.Int("status", 200),
slog.Duration("elapsed", elapsed),
)
```
- **stdlib since 1.21**, stable since 1.23. Performance is on par with zerolog for structured output, and faster than logrus by a wide margin.
- The `slog.Handler` interface is implemented by all major exporters (OpenTelemetry, Datadog, Honeycomb).
- The skill bans `logrus`, `zap`, `zerolog` for new code. They are not bad — they are simply superseded. Existing projects on those keep them; new files use slog.
Use the `sloglint` linter from `golangci-strict.md` to enforce attr style (`slog.String(...)` instead of `slog.Any(...)`).
---
## CLI — `cobra` + `pflag` + slog
```bash
go install github.com/spf13/cobra-cli@latest
cobra-cli init mytool
cobra-cli add server
```
`cobra` is the de facto Go CLI framework — Kubernetes, Docker CLI, Helm, GitHub CLI all use it. The companion `viper` for config-file-+-env-+-flag merging is **optional**: prefer `caarlos0/env/v11` for env-only configs (12-factor apps), reach for viper only when you genuinely need file-based config.
See `cobra-stack.md`.
---
## TUI — `bubbletea v2` + `bubbles v2` + `lipgloss v2`
Use **v2 RC** (`charm.land/bubbletea/v2`), not v1. The v2 model adds:
- `tea.View{Cursor: *tea.Cursor, ...}` for real-cursor positioning.
- `SetVirtualCursor(false)` on textareas — lets the terminal own the cursor, which is **required** for CJK IME (Korean Hangul composition, Japanese kana→kanji conversion, Chinese pinyin lookup).
- Granular mouse events (`MouseClickMsg`, `MouseMotionMsg`, `MouseReleaseMsg`) instead of v1's coarse `MouseMsg`.
This is not a preference. v1 has no way to position the IME candidate window correctly — Korean input shows up two cells to the left of where you typed, every time. **If your TUI accepts text input AND your users include CJK speakers, v1 is broken.**
See `bubbletea-v2.md` for the full IME-correct skeleton.
---
## HTTP client — stdlib + `hashicorp/go-retryablehttp`
Default: `net/http.Client` with a tuned `http.Transport`. The stdlib client is **already excellent** in 2026 — HTTP/2 by default, connection pooling, sane timeouts when configured.
```go
client := &http.Client{
Timeout: 30 * time.Second,
Transport: &http.Transport{
MaxIdleConns: 200,
MaxIdleConnsPerHost: 40,
IdleConnTimeout: 90 * time.Second,
DisableCompression: false,
ForceAttemptHTTP2: true,
},
}
```
For retry/backoff, add `github.com/hashicorp/go-retryablehttp` — small, single-purpose, integrates as a wrapper.
**Never use** `resty` (too much magic, hides headers, encourages wrong defaults). `req` is fine but adds dependency surface for marginal benefit over the stdlib + retry wrapper.
---
## JSON — stdlib (default), `goccy/go-json` (perf), `bytedance/sonic` (extreme perf)
Stdlib `encoding/json` improved dramatically in Go 1.21+. **Use it.**
Reach for `goccy/go-json` (~3x faster) only when you have measured a hot-path bottleneck:
```go
import json "github.com/goccy/go-json"
// drop-in replacement — same API
```
Reach for `bytedance/sonic` (~5x faster, requires amd64/arm64) for production proxies with thousands of RPS of JSON traversal. CLIProxyAPI uses `tidwall/gjson` + `tidwall/sjson` for **partial-tree mutation without full unmarshal** — a different optimization, useful when you transform large payloads. See `backend-stack.md`.
---
## Concurrency primitives — stdlib only
| Need | Use |
|---|---|
| Goroutine group with error propagation | `golang.org/x/sync/errgroup` |
| Semaphore | `golang.org/x/sync/semaphore` |
| Single-flight dedup | `golang.org/x/sync/singleflight` |
| Lazy init | **`sync.OnceValue` / `sync.OnceFunc`** (Go 1.21+, replaces `sync.Once` for typed values) |
| Atomic counter | `atomic.Int64` (Go 1.19+, typed atomics — don't use the old func-style) |
| Channel-based fanout | `chan T` with `errgroup` for shutdown |
The `x/sync` packages are stdlib-quality but live outside `std`. See `concurrency.md` for the discipline.
---
## Time — stdlib + `benbjohnson/clock` for tests
```go
type Clock interface { Now() time.Time }
// Production
var realClock Clock = clockImpl{}
// Test
fake := clock.NewMock()
fake.Set(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC))
```
**Never call `time.Now()` directly inside domain code.** Inject a `Clock`. Tests become deterministic, no `time.Sleep` flakiness.
---
## IDs — `google/uuid` (UUID v4/v7) or `xid` (sortable short ID)
```go
import "github.com/google/uuid"
id := uuid.Must(uuid.NewV7()) // sortable, time-ordered, 128-bit
```
UUID v7 is the modern default — sortable like v6, random like v4. Use v4 only when leaking creation time is a privacy concern.
For short, URL-safe IDs (~12 bytes, sortable) use `rs/xid` — Kubernetes-style.
---
## Crypto — stdlib + `alecthomas/argon2id` for passwords
Stdlib `crypto/*` for everything. For password hashing, **argon2id is the 2026 standard** — bcrypt is acceptable but argon2 is OWASP's recommendation since 2023.
```go
import "github.com/alecthomas/argon2id"
hash, err := argon2id.CreateHash("password", argon2id.DefaultParams)
```
---
## Data — `apache/arrow-go/v18` + `marcboeker/go-duckdb` + `gonum`
Same philosophy as Python's "never pandas":
| Need | Use |
|---|---|
| Tabular over CSV/Parquet/JSON | DuckDB-Go bindings — zero-copy Arrow integration |
| In-memory frame | Arrow + custom code (Go has no pandas-equivalent and that's fine) |
| Numerical | `gonum.org/v1/gonum` |
| Stats | `gonum/stat` |
Go's data-science story is intentionally thin. For heavy data work, write the pipeline in Polars/DuckDB (see `python/data-processing.md`), expose the result via Parquet or Arrow, consume from Go.
---
## Testing — stdlib + selective additions
| Need | Use |
|---|---|
| Assertions | `stretchr/testify/require` (fail-fast) — `assert` only in table-driven loops |
| Snapshots / golden | `hexops/autogold/v2` (auto-updates with `-update`) |
| Property-based | `pgregory.net/rapid` (modern) or stdlib `testing/quick` |
| Mocks | `go.uber.org/mock` (gomock successor) |
| HTTP mocks | `h2non/gock` for outbound, stdlib `httptest` for inbound |
| Integration containers | `testcontainers/testcontainers-go` |
| Goroutine leak | `go.uber.org/goleak` |
| Benchmarks | stdlib `testing.B` + `perf.dev/benchstat` |
See `testing.md` for canonical patterns.
---
## Config — `caarlos0/env/v11`
```go
type Config struct {
Port int `env:"PORT" envDefault:"8080"`
DatabaseURL string `env:"DATABASE_URL,required"`
Timeout time.Duration `env:"TIMEOUT" envDefault:"30s"`
}
var cfg Config
if err := env.Parse(&cfg); err != nil { log.Fatal(err) }
```
Pure 12-factor. Defaults via struct tag, required marker, parsing for `time.Duration`, slices, maps. **Use viper only if you also need file-based config** — most services do not.
---
## Choosing an unfamiliar dependency — the checklist
Before `go get`-ing anything new:
1. Is it maintained? Latest tag within 12 months? Owner active?
2. Does it expose stdlib-compatible types (`io.Reader`, `context.Context`, `http.Handler`)? If it invents its own `Connection` or `Request` type, that's a yellow flag.
3. Does it use `init()` for side effects? **REJECT.** `init()` ruins testability.
4. Does it call `log.Fatal` / `panic` outside of true programmer-error paths? **REJECT.**
5. Does it have a `context.Context` first-arg convention? If not, **REJECT** — cancellation is non-negotiable.
6. Does adding it overlap with something already in your `go.mod`? Pick one.
---
## Sources
- 2024 Go Developer Survey: https://go.dev/blog/survey2024-h1-results
- Connect-Go docs: https://connectrpc.com/docs/go/getting-started
- sqlc: https://docs.sqlc.dev
- bubbletea v2 IME: https://github.com/code-yeongyu/bubbletea-wm (reference for `SetVirtualCursor(false)` pattern)
- CLIProxyAPI (gin + SSE + WebSocket in production): https://github.com/router-for-me/CLIProxyAPI
- slog blog: https://go.dev/blog/slog
packages/shared-skills/skills/programming/references/go/one-liners.md
+202
View File
@@ -0,0 +1,202 @@
# One-Liners and Disposable Scripts
Production hygiene with throwaway ergonomics. Go scripts get the same strict lints, the same type discipline, the same 250 LOC ceiling. The difference: they live as single `.go` files invoked via `go run`, not as full modules.
Python has PEP 723 + `uv run`. Rust has `rust-script`. **Go has `go run` directly** — no extra tooling needed.
---
## Pattern 1: Single-file `go run`
A `.go` file with a `main` package, run directly:
```go
//go:build ignore
// fetch.go — fetch a URL and print body length.
//
// Usage:
// go run fetch.go <url>
package main
import (
"fmt"
"io"
"log"
"net/http"
"os"
)
func main() {
if len(os.Args) < 2 {
log.Fatal("usage: go run fetch.go <url>")
}
resp, err := http.Get(os.Args[1])
if err != nil { log.Fatal(err) }
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil { log.Fatal(err) }
fmt.Printf("%d bytes\n", len(body))
}
```
Run: `go run fetch.go https://example.com`.
The `//go:build ignore` directive keeps this file out of `go build ./...` — it is a script, not part of the module. Without that line, every `.go` file in the package gets compiled into your binary.
---
## Pattern 2: Throwaway directory under `scripts/`
```
myproject/
├── go.mod
├── internal/...
└── scripts/
├── seed/
│ └── main.go # `go run ./scripts/seed`
├── migrate/
│ └── main.go
└── one-time-fix/
└── main.go
```
Each `scripts/<name>/main.go` is its own `main` package. Invoke as `go run ./scripts/seed/`. Dependencies are shared with the parent module — no separate `go.mod`.
This is the right pattern when:
- You need module deps (sqlc, pgx, your own internal packages).
- You want IDE support, type-checking, test coverage.
- The script lives alongside the project, runs in CI.
---
## Pattern 3: Inline `go run` from shell
```bash
go run -mod=mod <(cat <<'EOF'
package main
import "fmt"
func main() { fmt.Println("hello") }
EOF
)
```
Rare, but useful for one-shot terminal experiments. The `<(...)` is process substitution; `go run -mod=mod` reads from stdin.
---
## Hard rules for scripts
Even a 30-line script follows the philosophy:
1. **Typed flags via `flag` or `pflag`**, not `os.Args` string parsing past 2 args.
```go
var (
url = flag.String("url", "", "URL to fetch")
limit = flag.Int("limit", 100, "max bytes")
)
flag.Parse()
if *url == "" { log.Fatal("--url required") }
```
2. **`context.Context` propagation** wherever I/O happens.
```go
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer cancel()
req, _ := http.NewRequestWithContext(ctx, "GET", *url, nil)
```
3. **`log.Fatal` is fine in `main()`** of a script (programmer error / fatal path), but **never inside any function the script imports.** Library code returns errors.
4. **Errors get wrapped.** Same rule as production code:
```go
if err != nil { return fmt.Errorf("fetch %s: %w", *url, err) }
```
5. **Resources released via `defer`.** No "I'll fix it later".
6. **slog for output if it must be parseable.** `fmt.Println` for one-shot terminal output is fine.
7. **No more than 250 pure LOC.** If it grows, it stops being a script and becomes a subcommand of your CLI tool.
---
## Pattern 4: Standalone tool with deps — temporary module
Some scripts need deps the parent module does not have. Two options:
### Option A — script in its own tiny module
```bash
mkdir /tmp/migrate-tool && cd $_
go mod init scratch.local/migrate-tool
go get github.com/pressly/goose/v3
cat > main.go <<'EOF'
package main
import ... // use goose
func main() { ... }
EOF
go run .
```
Run, then delete `/tmp/migrate-tool`. Throwaway.
### Option B — `gorun` (community tool)
```bash
go install github.com/erning/gorun@latest
cat > script.go <<'EOF'
//usr/bin/env gorun "$0" "$@"; exit
// /// go.mod
// module scratch
// go 1.23
// require github.com/spf13/cobra v1.8.0
// ///
package main
...
EOF
chmod +x script.go
./script.go
```
`gorun` parses the inline `go.mod` block, materializes a temp module, runs the script. Niche tool — only if you want the executable-script experience.
---
## When a script becomes a CLI
If your script needs:
- More than one subcommand
- Long-term storage of state
- Help text more than a paragraph
- Repeated invocations from CI
... promote it to a real CLI tool via `cobra` — see `cobra-stack.md`. The boundary is fuzzy; trust your judgment, but **a 500-line "script" is not a script.**
---
## Antipatterns
| Bad | Why | Good |
|---|---|---|
| `os.Args[1]` indexing without length check | Panics on missing arg | `flag.Parse()` with explicit checks |
| `log.Fatal` inside a function the script imports | Crashes caller's process | Return error |
| `panic(err)` for expected failures | Same as above | `log.Fatal` in `main`, error return elsewhere |
| Skipping `defer resp.Body.Close()` because "it's a script" | Leaks fd | Always close |
| One 800-LOC `main.go` "to keep it simple" | Now harder to read than a real CLI | Promote to `cmd/<name>/` with subcommands |
| `// TODO: handle error` | Production-grade hygiene means production-grade hygiene | Handle now or document why ignored |
---
## Sources
- `go run` docs: https://pkg.go.dev/cmd/go#hdr-Compile_and_run_Go_program
- `//go:build` constraints: https://pkg.go.dev/cmd/go#hdr-Build_constraints
- `signal.NotifyContext`: https://pkg.go.dev/os/signal#NotifyContext
- gorun: https://github.com/erning/gorun
packages/shared-skills/skills/programming/references/go/sqlc-pgx.md
+471
View File
@@ -0,0 +1,471 @@
# Database Stack — sqlc + pgx + goose + testcontainers
The canonical 2026 PostgreSQL stack. **Type-safe SQL with zero runtime reflection**, hot-path-friendly connection pooling, sane migrations, real Postgres in tests.
If you came here from a `gorm` project: gorm is rejected. See "Why not gorm" at the end.
---
## Toolchain
```bash
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
go install github.com/pressly/goose/v3/cmd/goose@latest
```
---
## Layout
```
internal/store/
├── sqlc.yaml # sqlc config
├── schema.sql # the cumulative DDL sqlc parses
├── queries/ # one *.sql per resource
│ ├── users.sql
│ ├── orders.sql
│ └── sessions.sql
├── sqlc/ # GENERATED — do not hand-edit
│ ├── db.go
│ ├── models.go
│ ├── users.sql.go
│ ├── orders.sql.go
│ └── sessions.sql.go
├── migrations/ # goose migrations, ordered
│ ├── 20260101000001_create_users.sql
│ └── 20260102000001_add_orders.sql
├── pool.go # pgxpool factory
├── user_store.go # domain-facing wrapper around sqlc
└── user_store_test.go # testcontainers integration test
```
---
## `sqlc.yaml`
```yaml
version: "2"
sql:
- engine: "postgresql"
schema: "schema.sql"
queries: "queries"
gen:
go:
package: "sqlc"
out: "sqlc"
sql_package: "pgx/v5"
emit_json_tags: false
emit_prepared_queries: false
emit_interface: true # generates a Querier interface
emit_exact_table_names: false
emit_pointers_for_null_types: true
emit_empty_slices: true
overrides:
- db_type: "uuid"
go_type:
import: "github.com/google/uuid"
type: "UUID"
- db_type: "timestamptz"
go_type:
import: "time"
type: "Time"
```
Key choices:
- `sql_package: "pgx/v5"` — generated code uses pgx directly, not `database/sql`. Faster, type-safer.
- `emit_interface: true` — generates a `Querier` interface. Lets stores accept either `*pgxpool.Pool` or `pgx.Tx` for transaction support.
- `emit_pointers_for_null_types: true` — nullable columns become `*T`, not `sql.NullString`. Cleaner mapping to domain types.
- `overrides` for `uuid` → `google/uuid.UUID` and `timestamptz` → `time.Time`.
---
## `schema.sql`
```sql
-- internal/store/schema.sql
-- The CUMULATIVE schema sqlc parses. Not migrations — the end state.
-- Regenerate from a fresh DB via `pg_dump --schema-only`, or hand-maintain.
CREATE TABLE users (
id UUID PRIMARY KEY,
email TEXT NOT NULL UNIQUE,
username TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_users_created_at ON users(created_at DESC);
```
---
## `queries/users.sql`
```sql
-- name: GetUser :one
SELECT id, email, username, created_at
FROM users
WHERE id = $1;
-- name: ListUsers :many
SELECT id, email, username, created_at
FROM users
ORDER BY created_at DESC
LIMIT $1 OFFSET $2;
-- name: CreateUser :one
INSERT INTO users (id, email, username)
VALUES ($1, $2, $3)
RETURNING id, email, username, created_at;
-- name: UpdateUserEmail :exec
UPDATE users
SET email = $2
WHERE id = $1;
-- name: DeleteUser :exec
DELETE FROM users WHERE id = $1;
```
sqlc directives:
- `:one` — exactly one row; returns `(T, error)`. Returns `pgx.ErrNoRows` on miss.
- `:many` — zero or more rows; returns `([]T, error)`.
- `:exec` — no rows returned; returns `error`.
- `:execrows` — returns `(int64, error)` with affected row count.
- `:batchone` / `:batchmany` / `:batchexec` — pgx batch mode for bulk operations.
Run `task gen:sqlc` (or `sqlc generate`). The generated file is committed; CI checks it is up-to-date.
---
## Generated code shape (`sqlc/users.sql.go`)
```go
// GENERATED — do not edit
type User struct {
ID uuid.UUID
Email string
Username string
CreatedAt time.Time
}
const getUser = `-- name: GetUser :one
SELECT id, email, username, created_at FROM users WHERE id = $1`
func (q *Queries) GetUser(ctx context.Context, id uuid.UUID) (User, error) {
row := q.db.QueryRow(ctx, getUser, id)
var u User
err := row.Scan(&u.ID, &u.Email, &u.Username, &u.CreatedAt)
return u, err
}
```
Type-safe inputs, type-safe outputs, compile-time-checked column-to-field mapping. **A schema change that drops a column breaks compilation.** Hand-rolled SQL would have failed at runtime.
---
## `store/pool.go`
```go
package store
import (
"context"
"fmt"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
func NewPool(ctx context.Context, dsn string) (*pgxpool.Pool, error) {
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil { return nil, fmt.Errorf("parse dsn: %w", err) }
cfg.MaxConns = 25
cfg.MinConns = 5
cfg.MaxConnLifetime = time.Hour
cfg.MaxConnIdleTime = 30 * time.Minute
cfg.HealthCheckPeriod = 1 * time.Minute
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil { return nil, fmt.Errorf("connect: %w", err) }
if err := pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("ping: %w", err)
}
return pool, nil
}
```
`pgxpool.Pool` is `Querier`-compatible (implements the interface sqlc generates). Same pool flows into sqlc queries unchanged.
---
## `store/user_store.go` — domain ↔ sqlc
```go
package store
import (
"context"
"errors"
"fmt"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/your-org/myservice/internal/domain"
"github.com/your-org/myservice/internal/store/sqlc"
)
type UserStore struct {
q *sqlc.Queries
}
func NewUserStore(pool *pgxpool.Pool) *UserStore {
return &UserStore{q: sqlc.New(pool)}
}
func (s *UserStore) Get(ctx context.Context, id domain.UserID) (domain.User, error) {
row, err := s.q.GetUser(ctx, uuid.UUID(id))
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return domain.User{}, domain.ErrUserNotFound
}
return domain.User{}, fmt.Errorf("get user %s: %w", id, err)
}
return rowToDomain(row)
}
func (s *UserStore) Create(ctx context.Context, u domain.User) (domain.User, error) {
row, err := s.q.CreateUser(ctx, sqlc.CreateUserParams{
ID: uuid.UUID(u.ID),
Email: u.Email.String(),
Username: u.Username.String(),
})
if err != nil {
return domain.User{}, fmt.Errorf("create user: %w", err)
}
return rowToDomain(row)
}
func rowToDomain(r sqlc.User) (domain.User, error) {
email, err := domain.NewEmail(r.Email)
if err != nil {
return domain.User{}, fmt.Errorf("db invariant: email %q: %w", r.Email, err)
}
username, err := domain.NewUsername(r.Username)
if err != nil {
return domain.User{}, fmt.Errorf("db invariant: username %q: %w", r.Username, err)
}
return domain.User{
ID: domain.UserID(r.ID),
Email: email,
Username: username,
CreatedAt: r.CreatedAt,
}, nil
}
```
The wrapping is verbose. **That is the point.** sqlc rows are storage representations; domain types are business representations. Mapping them explicitly is where invariants are enforced.
`pgx.ErrNoRows` becomes `domain.ErrUserNotFound` — callers never see storage-level errors.
---
## Transactions — pgx.Tx satisfies the Querier interface
```go
func (s *UserStore) CreateWithProfile(
ctx context.Context,
pool *pgxpool.Pool,
u domain.User,
p domain.Profile,
) error {
tx, err := pool.Begin(ctx)
if err != nil { return fmt.Errorf("begin: %w", err) }
defer tx.Rollback(ctx) // no-op if Commit succeeded
q := s.q.WithTx(tx) // sqlc.Queries bound to the tx
if _, err := q.CreateUser(ctx, /* ... */); err != nil {
return fmt.Errorf("create user: %w", err)
}
if _, err := q.CreateProfile(ctx, /* ... */); err != nil {
return fmt.Errorf("create profile: %w", err)
}
return tx.Commit(ctx)
}
```
Pattern:
- `defer tx.Rollback(ctx)` immediately after `Begin` — safe even after Commit (returns "tx closed", which we ignore via the unhandled return).
- `q.WithTx(tx)` returns a `*Queries` bound to the tx.
- Last line: `tx.Commit(ctx)`.
For nested transactions across multiple stores, accept a `Querier` parameter:
```go
func (s *UserStore) CreateTx(ctx context.Context, q sqlc.Querier, u domain.User) (domain.User, error) {
// uses q instead of s.q — caller decides if it's pool or tx
}
```
---
## Migrations — goose
```bash
goose -dir internal/store/migrations create create_users sql
```
```sql
-- migrations/20260101000001_create_users.sql
-- +goose Up
CREATE TABLE users (
id UUID PRIMARY KEY,
email TEXT NOT NULL UNIQUE,
username TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- +goose Down
DROP TABLE users;
```
Run:
```bash
goose -dir internal/store/migrations postgres "$DATABASE_URL" up
goose -dir internal/store/migrations postgres "$DATABASE_URL" status
goose -dir internal/store/migrations postgres "$DATABASE_URL" down
```
Rules:
- One DDL change per migration. Never combine schema + data migrations in one file.
- `Down` is real, not a stub. CI runs `up` → `down` → `up` on a fresh container to prove reversibility.
- Migrations are append-only. Never edit a merged migration; add a new one.
`goose` can run programmatically as well:
```go
import "github.com/pressly/goose/v3"
if err := goose.UpContext(ctx, db, "migrations"); err != nil { ... }
```
Useful for tools that own their schema (CI runner, integration test setup).
---
## Integration tests — testcontainers
```go
package store_test
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"github.com/testcontainers/testcontainers-go/modules/postgres"
)
func newTestDB(t *testing.T) *pgxpool.Pool {
t.Helper()
ctx := context.Background()
pgC, err := postgres.Run(ctx,
"postgres:16-alpine",
postgres.WithDatabase("test"),
postgres.WithUsername("test"),
postgres.WithPassword("test"),
postgres.BasicWaitStrategies(),
)
require.NoError(t, err)
t.Cleanup(func() { _ = pgC.Terminate(ctx) })
dsn, err := pgC.ConnectionString(ctx, "sslmode=disable")
require.NoError(t, err)
pool, err := store.NewPool(ctx, dsn)
require.NoError(t, err)
t.Cleanup(pool.Close)
require.NoError(t, goose.UpContext(ctx, /* sql.DB from pool */, "../migrations"))
return pool
}
func TestUserStore_Create_returns_new_user(t *testing.T) {
// Given
pool := newTestDB(t)
s := store.NewUserStore(pool)
ctx := context.Background()
// When
user, err := s.Create(ctx, domain.User{
ID: domain.UserID(uuid.Must(uuid.NewV7())),
Email: mustEmail("a@b.com"),
Username: mustUsername("alice"),
})
// Then
require.NoError(t, err)
require.NotEmpty(t, user.ID)
fetched, err := s.Get(ctx, user.ID)
require.NoError(t, err)
require.Equal(t, user.Email, fetched.Email)
}
```
testcontainers spins a real Postgres in Docker, runs migrations, hands you a pool. Tests are slow (~2s startup) but **real** — no fake that diverges from production.
For test suites with many cases, share one container across tests in the same package via `TestMain`:
```go
var testPool *pgxpool.Pool
func TestMain(m *testing.M) {
ctx := context.Background()
pgC, _ := postgres.Run(ctx, "postgres:16-alpine", /* ... */)
defer pgC.Terminate(ctx)
dsn, _ := pgC.ConnectionString(ctx, "sslmode=disable")
testPool, _ = store.NewPool(ctx, dsn)
// run migrations once
os.Exit(m.Run())
}
```
Each test then uses a transaction it rolls back at the end — fast and isolated.
---
## Why NOT gorm
| Concern | gorm | sqlc + pgx |
|---|---|---|
| Type safety | runtime reflection; column-to-field via tags | compile-time-checked from SQL |
| Performance | 2–5x slower than pgx | pgx is the fastest Go pg driver |
| N+1 queries | encouraged by `Preload` API | explicit JOIN in `.sql` |
| Migrations | AutoMigrate (unsafe in prod) | goose, explicit |
| Debugging | "what query did it run?" requires logging | the query IS the source |
| Cancellation | spotty ctx support | first-class |
| Active development | Yes but with churn and breaking changes | sqlc is stable |
Existing gorm projects: leave them. New code: sqlc + pgx.
---
## Sources
- sqlc docs: https://docs.sqlc.dev
- pgx: https://github.com/jackc/pgx
- goose: https://github.com/pressly/goose
- testcontainers-go: https://golang.testcontainers.org
- pgx pool config: https://pkg.go.dev/github.com/jackc/pgx/v5/pgxpool#Config
packages/shared-skills/skills/programming/references/go/testing.md
+467
View File
@@ -0,0 +1,467 @@
# Testing
TDD shape, table-driven tests, `require` vs `assert`, snapshot tests, property-based tests, integration tests with testcontainers, goroutine-leak detection. The discipline in `programming/SKILL.md` (Given/When/Then, less mock the better, efficient AND accurate) — this document gives the Go-specific recipes.
---
## Tools
| Need | Use |
|---|---|
| Assertions | `stretchr/testify/require` (and `assert` only inside table loops) |
| Mocks | `go.uber.org/mock` (gomock successor) |
| Goroutine leaks | `go.uber.org/goleak` |
| Snapshots / golden | `hexops/autogold/v2` |
| Property-based | `pgregory.net/rapid` |
| HTTP mocks (outbound) | `h2non/gock` |
| HTTP test server (inbound) | stdlib `net/http/httptest` |
| Integration containers | `testcontainers/testcontainers-go` |
| TUI | `charm.land/bubbletea/v2/teatest` |
| Bench tooling | stdlib `testing.B` + `perf.dev/benchstat` |
---
## Test naming — Given / When / Then in the name
```go
// ──── PATTERN ────
// Test_<Subject>_<Outcome>_when_<Condition>
// OR
// Test_<Subject>_<Action>_<ExpectedOutcome>
func Test_Email_NewEmail_lowercases_input(t *testing.T)
func Test_Email_NewEmail_rejects_input_without_at_sign(t *testing.T)
func Test_UserService_Create_persists_user_when_inputs_valid(t *testing.T)
func Test_UserService_Create_returns_validation_error_when_email_invalid(t *testing.T)
```
A test name should answer "what behavior is this asserting?" without reading the body. Names that need a comment to explain them are misnamed.
---
## Single test — explicit Given/When/Then
```go
func Test_Email_NewEmail_rejects_input_without_at_sign(t *testing.T) {
// Given
raw := "not-an-email"
// When
_, err := domain.NewEmail(raw)
// Then
require.Error(t, err)
require.ErrorIs(t, err, domain.ErrInvalidEmail)
}
```
`require.*` fails the test immediately on miss. Use `require` for preconditions and primary assertions. Use `assert.*` only inside table-driven loops where you want all cases to report.
---
## Table-driven tests
```go
func Test_Email_NewEmail(t *testing.T) {
tests := []struct {
name string
input string
want string
wantErr error
}{
{"lowercases", "ALICE@example.com", "alice@example.com", nil},
{"trims whitespace", " bob@example.com ", "bob@example.com", nil},
{"rejects missing @", "no-at-sign", "", domain.ErrInvalidEmail},
{"rejects empty", "", "", domain.ErrInvalidEmail},
{"rejects too long", strings.Repeat("a", 256) + "@e.com", "", domain.ErrInvalidEmail},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// When
got, err := domain.NewEmail(tt.input)
// Then
if tt.wantErr != nil {
require.ErrorIs(t, err, tt.wantErr)
return
}
require.NoError(t, err)
assert.Equal(t, tt.want, got.String())
})
}
}
```
Rules:
- One **scenario** per row, not one **assertion** per row.
- Subtest names are sentences in lowercase; `t.Run(tt.name, ...)` makes them filterable: `go test -run Test_Email_NewEmail/rejects_missing_@`.
- The loop body itself is Given/When/Then in shape.
- For Go 1.22+, the loop var capture works correctly without the `tt := tt` shadow line — the `copyloopvar` linter enforces the new style.
---
## Less mocks — the priority order
In Go specifically:
1. **Real implementation.** Domain types, pure functions, value objects — instantiate them. They are fast.
2. **In-memory fake** that satisfies the interface. Has its own test suite proving behavioral parity with the real impl.
3. **`httptest.Server`** for HTTP collaborators (real wire, no internet).
4. **`testcontainers`** for stateful collaborators (Postgres, Redis, S3-compatible, Kafka).
5. **gomock** ONLY for: clocks, randomness, third-party SaaS with no sandbox.
### Example: an in-memory fake
```go
// Real interface
type UserRepo interface {
Save(ctx context.Context, u domain.User) error
Get(ctx context.Context, id domain.UserID) (domain.User, error)
}
// In-memory fake — production-quality, tested separately
type FakeUserRepo struct {
mu sync.RWMutex
users map[domain.UserID]domain.User
}
func NewFakeUserRepo() *FakeUserRepo {
return &FakeUserRepo{users: map[domain.UserID]domain.User{}}
}
func (r *FakeUserRepo) Save(ctx context.Context, u domain.User) error {
r.mu.Lock(); defer r.mu.Unlock()
r.users[u.ID] = u
return nil
}
func (r *FakeUserRepo) Get(ctx context.Context, id domain.UserID) (domain.User, error) {
r.mu.RLock(); defer r.mu.RUnlock()
u, ok := r.users[id]
if !ok { return domain.User{}, domain.ErrUserNotFound }
return u, nil
}
```
The fake has the same observable behavior as the real one. Tests against `FakeUserRepo` survive when the production repo's internals change. Tests against a gomock stub of `UserRepo` break.
**A test passing against a fake AND a test passing against the real impl is the gold standard.** Run the same test suite twice — once with the fake, once with testcontainers. The fakes earn their keep when the suites diverge.
### Example: gomock for the unmockable
```go
//go:generate mockgen -source=clock.go -destination=mocks/clock_mock.go -package=mocks
type Clock interface {
Now() time.Time
}
// In a test:
ctrl := gomock.NewController(t)
clock := mocks.NewMockClock(ctrl)
clock.EXPECT().Now().Return(fixedTime).AnyTimes()
```
Mock the narrowest seam. Never mock `UserRepo` if a fake suffices.
---
## E2E scenario tests
```go
//go:build e2e
func Test_E2E_user_can_signup_then_login(t *testing.T) {
// Given — full server in a goroutine
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
pool := newTestDB(t) // testcontainers Postgres
server := startServer(t, pool) // real gin engine on a random port
defer server.Close()
client := server.Client()
// When — sign up
resp, err := client.Post(server.URL+"/api/v1/users",
"application/json",
strings.NewReader(`{"email":"a@b.com","username":"alice","password":"PassWord!23"}`),
)
require.NoError(t, err)
require.Equal(t, 201, resp.StatusCode)
// When — log in
resp, err = client.Post(server.URL+"/api/v1/auth/login",
"application/json",
strings.NewReader(`{"email":"a@b.com","password":"PassWord!23"}`),
)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
var body struct{ Token string `json:"token"` }
require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
require.NotEmpty(t, body.Token)
// Then — token works on protected endpoint
req, _ := http.NewRequestWithContext(ctx, "GET", server.URL+"/api/v1/me", nil)
req.Header.Set("Authorization", "Bearer "+body.Token)
resp, err = client.Do(req)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
```
Patterns:
- `//go:build e2e` build tag separates slow E2E from fast unit tests. Run with `go test -tags=e2e ./...`.
- One narrative per test: "user can sign up then log in". One `Test_E2E_*` per user-visible outcome.
- Real DB via testcontainers, real gin engine, real HTTP. **No mocks.** The point is to catch integration bugs.
- Bounded context — every E2E gets a `context.WithTimeout` so failures don't hang CI.
---
## Goroutine leak detection
```go
package mypkg
import (
"testing"
"go.uber.org/goleak"
)
func TestMain(m *testing.M) {
goleak.VerifyTestMain(m,
goleak.IgnoreTopFunction("github.com/prometheus/client_golang/prometheus.(*Registry)..."),
)
}
```
One line at the top of every package that spawns goroutines. Catches the bug class the race detector cannot.
---
## Snapshot / golden tests — `autogold`
```go
import "github.com/hexops/autogold/v2"
func Test_RenderHelp_matches_snapshot(t *testing.T) {
// Given
cmd := newRootCmd()
// When
out := captureOutput(t, func() { _ = cmd.Help() })
// Then
autogold.ExpectFile(t, out)
}
```
First run: `go test -update ./...` writes `testdata/Test_RenderHelp.golden`. Future runs compare; failures show a diff. Re-approve intentional changes with `-update`.
**Use snapshots for STRUCTURE, not BEHAVIOR.** Good targets:
- CLI `--help` output
- JSON response shape
- Generated SQL queries
- Rendered prompts (assert the structure, not exact wording — see SKILL.md prompt-test rule)
Bad targets: a function's return value where you should `require.Equal` on the actual structure.
---
## Property-based tests — `rapid`
```go
import "pgregory.net/rapid"
func Test_Email_NewEmail_then_String_roundtrips(t *testing.T) {
rapid.Check(t, func(t *rapid.T) {
// Given — generate valid emails
local := rapid.StringMatching(`[a-z]{3,10}`).Draw(t, "local")
domain := rapid.StringMatching(`[a-z]{3,10}\.com`).Draw(t, "domain")
raw := local + "@" + domain
// When
e, err := domain.NewEmail(raw)
require.NoError(t, err)
// Then — round-trip property
e2, err := domain.NewEmail(e.String())
require.NoError(t, err)
require.Equal(t, e, e2)
})
}
```
`rapid` shrinks failing cases to minimal counterexamples. Use for:
- Round-trips (parse → serialize → parse).
- Algebraic properties (sort produces ordered, dedup is idempotent, JSON marshal/unmarshal is involutive).
- Invariants under random input (validator never panics, serializer never produces invalid UTF-8).
---
## HTTP testing — `httptest`
### Server side
```go
func Test_GetUser_returns_user_for_existing_id(t *testing.T) {
// Given
svc := newSvcWithFake(t)
r := gin.New()
h := &Handler{Users: svc}
h.Mount(r)
req := httptest.NewRequest("GET", "/api/v1/users/u-1", nil)
rec := httptest.NewRecorder()
// When
r.ServeHTTP(rec, req)
// Then
require.Equal(t, 200, rec.Code)
var body domain.User
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &body))
require.Equal(t, "u-1", string(body.ID))
}
```
### Client side — `httptest.NewServer`
```go
func Test_Client_retries_on_500(t *testing.T) {
// Given — fake upstream
var calls int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
if calls < 3 {
w.WriteHeader(500)
return
}
w.WriteHeader(200)
_, _ = w.Write([]byte(`{"ok":true}`))
}))
defer srv.Close()
client := myclient.New(srv.URL)
// When
err := client.DoSomething(context.Background())
// Then
require.NoError(t, err)
require.Equal(t, 3, calls)
}
```
`httptest.NewServer` spins a real HTTP server on a random port. The fake handler implements the upstream contract. Test the client against the contract, not the implementation.
---
## Determinism — the cardinal rules
- **No `time.Sleep` in tests.** If you need delay, you need a Clock injection.
- **`go test -shuffle=on`** in every CI run.
- **`go test -count=1`** to defeat the cache.
- **Subscribe to the event, do not poll for it.** Channels, callbacks, `t.Cleanup` over polling.
- **`t.Parallel()`** for tests that share no state. Speeds up large suites by 4-8x.
A test that fails 1-in-10 runs is a bug, not flake. The race detector + `-shuffle=on` + ordering hygiene catches >95% of "flake".
---
## Benchmarks — `testing.B` + `benchstat`
```go
func Benchmark_NewEmail(b *testing.B) {
for b.Loop() { // Go 1.24+ idiom, replaces `for i := 0; i < b.N; i++`
_, _ = domain.NewEmail("alice@example.com")
}
}
```
Run:
```bash
go test -bench=. -count=10 -benchmem ./... | tee bench.txt
benchstat bench.txt # statistical comparison
```
Always `-count=10` for stable means. `-benchmem` reports allocations. A 5%-slower benchmark in one run is noise; 10 runs + benchstat tells you what is real.
To compare before/after a change:
```bash
git stash
go test -bench=. -count=10 ./... > before.txt
git stash pop
go test -bench=. -count=10 ./... > after.txt
benchstat before.txt after.txt
```
---
## Coverage — the right target
Run:
```bash
go test -race -shuffle=on -coverprofile=cover.out ./...
go tool cover -html=cover.out -o cover.html
```
**Aim for 80%+ on `internal/domain` and `internal/service`.** Boundary code (handlers, store mappers) is exercised by integration tests, where line coverage understates what is actually verified. Do not chase 100% — the last 5% is usually error paths that need fault-injection to hit.
The `golangci-lint` config does not enforce a minimum — coverage as a CI gate becomes a goal-displacement metric. Treat it as feedback, not requirement.
---
## TUI testing — `teatest`
```go
import teatest "charm.land/bubbletea/v2/teatest"
func Test_Counter_increments_on_space(t *testing.T) {
// Given
tm := teatest.NewTestModel(t, initial(), teatest.WithInitialTermSize(80, 24))
// When
tm.Send(tea.KeyPressMsg{Code: ' '})
// Then
final := tm.FinalModel(t).(model)
require.Equal(t, 1, final.count)
}
```
For full-view regression, snapshot the rendered output via `autogold`.
---
## Antipatterns the skill rejects
| Bad | Why | Good |
|---|---|---|
| `if got != want { t.Errorf("expected %v got %v", want, got) }` | Reinvents `require.Equal` | Use testify |
| `time.Sleep(100 * time.Millisecond)` after triggering async work | Flake | Subscribe to completion signal, bounded await |
| `t.Skip(...)` to silence a known failure | Buries the bug | Fix or open an issue; never silently skip |
| One mega-test asserting 12 things | First failure hides next 11 | Split by `Then` |
| Snapshot-everything | Locks formatting, not behavior | Snapshots for structure, asserts for values |
| Mock every collaborator | Test asserts implementation, not behavior | Real or fake, never mock everything |
| Test calls private function via `_test.go` in same package only | Couples test to implementation | Test through the public surface |
---
## Sources
- testify: https://github.com/stretchr/testify
- goleak: https://github.com/uber-go/goleak
- autogold: https://github.com/hexops/autogold
- rapid: https://pkg.go.dev/pgregory.net/rapid
- testcontainers-go: https://golang.testcontainers.org
- benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
- "Go test naming conventions" (Dave Cheney): https://dave.cheney.net/practical-go/presentations/qcon-china.html
packages/shared-skills/skills/programming/references/go/type-patterns.md
+298
View File
@@ -0,0 +1,298 @@
# Type Patterns
How to use Go's *limited* type system to catch bugs at compile time. Go gives you fewer tools than Python/TS/Rust — this document covers the four patterns that buy back most of the safety.
The four patterns:
1. **Named types** for branding primitives (the Go answer to `NewType` / branded TS).
2. **Smart constructors with unexported fields** for parse-don't-validate.
3. **Sealed interfaces** for sum types, with `type switch` + `exhaustive` linter.
4. **Generics with constraints** for bounded polymorphism (1.18+).
---
## 1. Named types — distinct primitives
Same underlying type, different meaning. The Go type checker prevents *implicit* mixing — but explicit conversion is always possible. Treat this as a contract enforced at boundaries.
```go
package domain
type UserID string
type OrderID string
type EmailRaw string // raw, unvalidated string from input
func GetUser(id UserID) User { /* ... */ }
uid := UserID("u-123")
oid := OrderID("o-456")
GetUser(uid) // ✅ OK
GetUser(oid) // ❌ cannot use oid (type OrderID) as UserID
GetUser("u-123") // ❌ untyped string literal — Go DOES catch this
GetUser(UserID("u-123")) // ✅ explicit conversion — accept it
```
**Use when**: IDs, opaque tokens, foreign keys, units that share a base primitive.
**Reality check**: Go does NOT prevent `UserID(orderIDAsString)`. The defense is **smart constructors** for everything beyond an internal identifier. Use named types for cheap brand-only protection; combine with constructors for protection that actually holds.
### Time-of-day units
```go
type Milliseconds int64
type Seconds int64
func (ms Milliseconds) ToSeconds() Seconds {
return Seconds(ms / 1000)
}
```
No implicit `Milliseconds + Seconds`. The compiler refuses. Convert explicitly.
---
## 2. Smart constructors with unexported fields — the Go answer to Pydantic/Zod
The single most important pattern in this document. **Go has no Pydantic. It has this.**
```go
package domain
import (
"errors"
"regexp"
"strings"
)
var (
ErrInvalidEmail = errors.New("invalid email")
emailRe = regexp.MustCompile(`^[^@\s]+@[^@\s]+\.[^@\s]+$`)
)
// Email is a parsed, lowercased, valid email address.
// The zero value is invalid; construct via NewEmail.
type Email struct {
raw string // unexported — cannot be set from outside the package
}
func NewEmail(s string) (Email, error) {
s = strings.TrimSpace(strings.ToLower(s))
if !emailRe.MatchString(s) {
return Email{}, ErrInvalidEmail
}
return Email{raw: s}, nil
}
// String implements fmt.Stringer for printing.
func (e Email) String() string { return e.raw }
// MarshalJSON keeps the wire format unchanged.
func (e Email) MarshalJSON() ([]byte, error) {
return []byte(`"` + e.raw + `"`), nil
}
// UnmarshalJSON is the parsing boundary — strict mode.
func (e *Email) UnmarshalJSON(data []byte) error {
if len(data) < 2 || data[0] != '"' || data[len(data)-1] != '"' {
return ErrInvalidEmail
}
parsed, err := NewEmail(string(data[1 : len(data)-1]))
if err != nil {
return err
}
*e = parsed
return nil
}
```
**Why this works**:
- `Email{raw: "anything"}` from outside the `domain` package is a compile error — `raw` is unexported.
- The only way to obtain a non-zero `Email` is `NewEmail(...)`, which validates.
- `UnmarshalJSON` routes wire input through the same constructor — boundary parsing is automatic.
- Once a function signature has `email Email`, the caller has *proven* it is valid. No internal `if email == ""` checks.
**Use for every domain value that has invariants**: emails, URLs, phone numbers, currency amounts, percentages, semver versions, IDs with format constraints, time ranges, anything you currently validate in three places.
### The "zero value problem"
Go's zero value (`Email{}`) is reachable. The mitigation is documentation + a `IsValid()` method when needed:
```go
func (e Email) IsZero() bool { return e.raw == "" }
```
Or accept it: receivers that take `Email` should *never* receive a zero-value `Email` in correct code. Tests verify it.
---
## 3. Sealed interfaces — sum types in Go
Go has no sum types. The closest thing: an interface with an **unexported method** that only types in the same package can satisfy, dispatched via `type switch`, with the `exhaustive` linter ensuring completeness.
```go
package event
// Event is a closed sum: Created | Updated | Deleted.
// The sealed() method is unexported so external packages cannot add variants.
type Event interface {
sealed()
OccurredAt() time.Time
}
type Created struct {
UserID UserID
Email Email
Timestamp time.Time
}
func (Created) sealed() {}
func (e Created) OccurredAt() time.Time { return e.Timestamp }
type Updated struct {
UserID UserID
Changes map[string]any
Timestamp time.Time
}
func (Updated) sealed() {}
func (e Updated) OccurredAt() time.Time { return e.Timestamp }
type Deleted struct {
UserID UserID
Reason string
Timestamp time.Time
}
func (Deleted) sealed() {}
func (e Deleted) OccurredAt() time.Time { return e.Timestamp }
```
Consumer code:
```go
func Render(e event.Event) string {
switch v := e.(type) {
case event.Created:
return fmt.Sprintf("created %s with %s", v.UserID, v.Email)
case event.Updated:
return fmt.Sprintf("updated %s: %v", v.UserID, v.Changes)
case event.Deleted:
return fmt.Sprintf("deleted %s (reason: %s)", v.UserID, v.Reason)
default:
panic(fmt.Sprintf("unhandled event variant: %T", v))
}
}
```
The `panic` in `default` is the Go equivalent of TS's `assertNever` or Python's `assert_never`. It is only reachable if a new variant is added without updating the switch.
### The `exhaustive` linter — your compiler
```yaml
# .golangci.yml
linters:
enable: [exhaustive]
linters-settings:
exhaustive:
check:
- switch
- map
default-signifies-exhaustive: false
```
Now adding `event.Suspended` without updating `Render` is a **lint error**. This is the closest thing Go has to Rust's match exhaustiveness check. **Treat it as compulsory.**
### Sealed interface gotchas
- The method MUST be unexported (`sealed()`, not `Sealed()`). Otherwise other packages can implement it.
- `type switch` with `*Created` vs `Created` matters — pick value receivers and value cases, or pointer receivers and pointer cases. **Mixing them causes silent miss.**
- `interface{}` is not a sealed type. Anything implementing zero methods satisfies it. Sealed interfaces have at least the `sealed()` method.
---
## 4. Generics with constraints — bounded polymorphism
Go 1.18+. Use for genuinely generic algorithms; **do not** use for "I want this to accept anything".
```go
import "cmp"
// Ordered constraint includes all ordered types (int, float, string, …).
func Max[T cmp.Ordered](a, b T) T {
if a > b { return a }
return b
}
// Custom constraint
type Stringer interface {
String() string
}
func Join[T Stringer](items []T, sep string) string {
parts := make([]string, len(items))
for i, item := range items {
parts[i] = item.String()
}
return strings.Join(parts, sep)
}
```
The `cmp.Ordered` (Go 1.21+), `cmp.Compare`, and `slices`/`maps` packages cover the common cases without you writing constraints.
### When NOT to use generics
- "I want to accept multiple types, so I'll make it generic." Use an **interface** instead. Generics are for parametric polymorphism (same code, different types). Interfaces are for behavioral polymorphism (different code behind a contract).
- "I want to return `any`." Use a sealed interface and a `type switch`. `any` returns are anti-patterns past public APIs.
---
## 5. Type assertions — the controlled escape hatch
```go
// Bad — panics on failure
e := evt.(event.Created)
// Good — comma-ok form, always
if e, ok := evt.(event.Created); ok {
// use e
}
// Use errors.As for error chains
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) {
// pgErr is the wrapped pg error
}
```
**The `errcheck` and `errorlint` linters reject bare type assertions on `error` values.** Use `errors.As`. See `error-handling.md`.
---
## 6. Pointers vs values — the only durable rule
You will see endless debates. The rule that holds up:
- **If a type has a mutex, never copy it.** Use `*T` everywhere.
- **If a type is large (> 64 bytes) and read-only, pass by value or pointer is a measured choice.** Default to pointer for "large" things.
- **Receivers must be consistent.** All methods on `T` either take `T` or `*T`. Don't mix. The `staticcheck` linter catches mixed-receiver bugs.
- **`nil` pointer = absence. Zero value = "not set yet".** Choose ONE convention per type. Document it.
---
## 7. `any` / `interface{}` — when it is acceptable
Almost never in domain code. Acceptable cases:
- JSON parsing of genuinely heterogeneous payloads (and even then, prefer `json.RawMessage` + targeted parsing).
- `fmt.Sprintf` arguments (variadic `any` is unavoidable here).
- Generic container internals before the user-facing API.
The skill rejects `any` in handler signatures, service signatures, store signatures. If you find yourself writing `func Handle(payload any) error`, you have a sealed-interface waiting to happen.
---
## Sources
- "Parse, don't validate" — Alexis King: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
- exhaustive linter: https://github.com/nishanths/exhaustive
- Generics constraints: https://go.dev/blog/intro-generics
- cmp.Ordered: https://pkg.go.dev/cmp
packages/shared-skills/skills/programming/references/python/README.md
+314
View File
@@ -0,0 +1,314 @@
# Python Programmer
Modern Python. Type-strict, stack-first, async-correct.
## Philosophy
The type checker is your compiler. Make illegal states unrepresentable. Parse at boundaries. Own resources explicitly. Every function has a contract; the type system enforces it.
## Hard rules
These are deliberate project choices. Violations are always wrong, not "style preferences".
### Tooling
| Category | Use | Never |
|---|---|---|
| Package manager | `uv` | pip, poetry, conda, pipenv |
| Type checker | `basedpyright` (`typeCheckingMode = "all"`) | pyright, mypy |
| Linter + formatter | `ruff` (`select = ["ALL"]`) | flake8, black, isort, autopep8 |
| Async runtime | `anyio` | `import asyncio` |
| Data | `polars` + `duckdb` + `numpy` | pandas |
| Web framework | FastAPI + Pydantic v2 | Flask, Django REST |
| ORM | SQLAlchemy 2.x async | Django ORM, Tortoise |
| HTTP client | [`httpx2`](https://github.com/pydantic/httpx2) | requests, aiohttp, httpx |
| Testing | `pytest` | unittest |
| CLI | `typer` + `rich` | argparse, click, fire |
### The iron list
1. **Frozen by default** — `@dataclass(frozen=True, slots=True)`. Pydantic: `model_config = ConfigDict(frozen=True)`. Mutable only when mutation is the documented purpose.
2. **NewType for distinct IDs** — `UserId = NewType("UserId", int)`. Never pass raw `int` where a branded type exists.
3. **`match` only for variants, `if` only for booleans** — **NEVER** use `if/elif/else` to discriminate on type (`isinstance`), enum value, or literal variant. `match/case` is mandatory for these — non-negotiable. **ALWAYS** end with `case unreachable: assert_never(unreachable)` — bare `case _: pass` and `case _: raise ValueError` are banned (they silently swallow new variants). `if/else` is fine only for boolean expressions, range checks, and predicate calls that aren't variant discrimination. See "Why `if/elif` on variants is banned" below for examples.
4. **Protocol over ABC** — `typing.Protocol` for interfaces. ABC only when you need shared method implementation.
5. **No raw dicts in signatures** — params and returns use `TypedDict`, `dataclass`, or Pydantic model. Internal scratch dicts are fine.
6. **Parse, don't validate** — constructors produce typed objects or raise. Never pass unvalidated data deeper into the call stack.
7. **Typed errors** — error types are dataclasses or exceptions with typed fields. Never `raise ValueError("something")` with a bare string. Use union returns when the caller is within 1-2 call levels and must handle the outcome (repository → service). Use exceptions when the error should propagate up many layers to a boundary handler (service → HTTP handler).
8. **Final for constants** — module-level constants use `Final`. Mutable module globals are a code smell.
9. **Explicit None** — annotate `-> X | None`. Never return `None` from a function whose signature omits it.
10. **Context managers for resources** — files, DB connections, HTTP clients, locks. No manual `.close()`.
11. **No Any, no object** — both are banned as type annotations. `object` erases all structural information (zero callable attributes, zero narrowing). Use `Protocol` (structural typing), `TypeVar` (generic pass-through), explicit union (known variants), or `TypedDict` (dict shapes).
12. **No cast** — `cast()` is banned. Redesign the types.
13. **No type: ignore** — fix the type error. The checker is right; you are wrong.
14. **No broad except** — `except Exception` and `except BaseException` are banned. Catch the **specific** exception you expect. A broad catch swallows bugs you need to see — `KeyError`, `AttributeError`, `TypeError` all vanish silently. If you genuinely need a catch-all at a top-level boundary (CLI entry, HTTP handler), use `# noqa: BROAD_EXCEPT_OK` and log + re-raise.
### Typing and safety
- `basedpyright` in `typeCheckingMode = "all"`. Every public function has full annotations. Internal helpers: annotate return type; parameter types may be inferred.
- `ruff` with `select = ["ALL"]`. Override specific rules per project in `pyproject.toml`, never globally disable the strict baseline.
- Every new function must have a `docstring` unless its name + signature makes it completely obvious (e.g. `def full_name(first: str, last: str) -> str:`).
- Use `X | Y` union syntax (PEP 604), never `Union[X, Y]` or `Optional[X]`.
### Why `object` is banned
`object` pretends to be safe ("it's the top type!") but gives **zero** narrowing and **zero** attributes. Even `Any` is more honest — it admits the boundary is untyped.
```python
# BANNED
def process(data: object) -> object: ...
def store(items: list[object]) -> None: ...
results: dict[str, object] = {}
# GOOD — Protocol for structural typing
class Serializable(Protocol):
def serialize(self) -> bytes: ...
def process(data: Serializable) -> ProcessResult: ...
# GOOD — TypeVar for generic pass-through
def identity[T](x: T) -> T: ...
def first[T](items: Sequence[T]) -> T: ...
# GOOD — explicit union for known variants
def parse(raw: str | bytes) -> Document: ...
```
### Why `if/elif` on variants is banned
`if/elif/else` chains on type, enum, or literal values lose compile-time exhaustiveness. When a new variant is added, nothing warns you. `match/case` + `assert_never` does.
```python
# BANNED — if/elif for type discrimination
if isinstance(event, Click):
handle_click(event.x, event.y)
elif isinstance(event, Scroll):
handle_scroll(event.delta)
else:
raise ValueError(f"Unknown: {event}") # runtime bomb
# BANNED — if/elif for enum discrimination
if status == Status.PENDING:
start_review()
elif status == Status.ACTIVE:
continue_processing()
elif status == Status.CLOSED:
archive()
# BANNED — non-exhaustive match (swallows new variants)
match event:
case Click(x, y): handle_click(x, y)
case _: pass
# GOOD — exhaustive match with assert_never
match event:
case Click(x=x, y=y):
handle_click(x, y)
case Scroll(delta=delta):
handle_scroll(delta)
case unreachable:
assert_never(unreachable)
# GOOD — enum match
match status:
case Status.PENDING: start_review()
case Status.ACTIVE: continue_processing()
case Status.CLOSED: archive()
case unreachable: assert_never(unreachable)
```
`if/else` is fine for boolean conditions and range checks — things that aren't variant discrimination:
```python
# FINE — boolean, not variant
if age >= 18:
grant_access()
else:
deny_access()
```
### Why broad `except` is banned
`except Exception` catches **every** non-system exception — `KeyError`, `TypeError`, `AttributeError`, `ValueError` all vanish. You lose the stack trace that would have told you exactly what went wrong. The fix is always to name the exception you expect.
```python
# BANNED — swallows bugs
try:
result = api.fetch(url)
except Exception as e:
logger.error(e)
return None
# BANNED — catch-and-ignore
try:
parse(data)
except Exception:
pass
# GOOD — catch what you expect
try:
result = api.fetch(url)
except httpx.HTTPStatusError as e:
logger.error("API %d: %s", e.response.status_code, e.request.url)
return None
except httpx.ConnectError:
raise ServiceUnavailableError(service="api") from None
# GOOD — top-level boundary (only place broad catch is acceptable)
def main() -> int: # noqa: BROAD_EXCEPT_OK
try:
return run()
except Exception:
logger.exception("unhandled error")
return 1
```
### Async
- `import asyncio` is **BANNED**. Use `import anyio`.
- For background tasks, use `anyio.create_task_group`. Never fire-and-forget with `asyncio.create_task`.
- For concurrency gates, use `anyio.CapacityLimiter` (not `asyncio.Semaphore`).
- Load `async-anyio.md` when writing async code for the full pattern library.
### Data modeling — which container, when
All model fields carry type annotations. No `Any`, no untyped dicts in public APIs.
Use `polars` + `duckdb` for data. pandas is never the right answer in this stack.
| Situation | Use |
|---|---|
| User input, API request/response | `Pydantic BaseModel (frozen=True)` |
| Internal value object (no I/O) | `@dataclass(frozen=True, slots=True)` |
| Function with multiple outcomes | Union of frozen dataclasses + `match` |
| Dict shape for JSON compat / `**kwargs` | `TypedDict` |
| Fixed constants | `StrEnum` / `IntEnum` |
| Distinct primitive (UserId vs MovieId) | `NewType` |
| Contract / capability | `Protocol` |
| Contract + shared implementation | `ABC` |
| ORM model (SQLAlchemy) | `Mapped[]` — inherently mutable, `# noqa: MUTABLE_OK` |
| Config from env vars | `pydantic-settings BaseSettings` |
**The one rule**: data crosses trust boundary → Pydantic. Everything else → dataclass.
Load `data-modeling.md` for the full decision flowchart and comparison matrix.
### When frozen=True does not apply
- **ORM models** — SQLAlchemy `Mapped[]` requires mutation. Use `# noqa: MUTABLE_OK`.
- **Builder / accumulator** — object exists to be mutated (counter, buffer, state machine). Docstring must explain why.
- **Pydantic Settings** — tests override fields. Mutable is acceptable.
If you need `# noqa: MUTABLE_OK`, the class docstring must say why mutation is required.
### Libraries
Canonical defaults (override only if `pyproject.toml` explicitly picks something else):
| Domain | Library | Reason |
|---|---|---|
| CLI | `typer` | Type-annotated CLI from function sigs |
| Pretty output | `rich` | Tables, progress, tracebacks, markdown |
| HTTP client | [`httpx2`](https://github.com/pydantic/httpx2) | Next-gen HTTP client (Pydantic stewardship), HTTP/2, brotli+zstd. Always `httpx2[http2,brotli,zstd]`. See `httpx2-optimization.md` |
| Validation | `pydantic` v2 | Fast native validator, JSON Schema |
| Web API | `fastapi` | Async, Pydantic-native, OpenAPI |
| ORM | `sqlalchemy` 2.x async | `Mapped[]` types, async sessions |
| DB driver (Postgres) | `asyncpg` (via SQLAlchemy) | Fastest PG driver |
| AI agents | `pydantic-ai` | Typed deps, structured output |
| TUI | `textual` | Rich-based, CSS layout, widgets |
| Logging | `rich.logging.RichHandler` | Pretty; swap to `structlog` in prod |
## pyproject.toml — the one true config
Scaffold a new project with all strict defaults pre-configured:
```bash
uv run ../../scripts/python/new-project.py myproject
uv run ../../scripts/python/new-project.py myproject --path ./workspace
uv run ../../scripts/python/new-project.py myproject --lib # publishable library
```
Creates via `uv init`, then injects basedpyright `typeCheckingMode = "all"` + ruff `select = ["ALL"]` + pytest strict. Cross-platform (macOS, Linux, Windows).
For manual setup: `uv init --app myproject`, then load `pyproject-strict.md`.
## PEP 723 — inline script metadata (mandatory for ALL scripts)
Every `.py` script — even throwaway — MUST use PEP 723 inline metadata with the `# ─── How to run ───` comment block. No venv, no `requirements.txt`. The script IS the environment spec. A script without the usage comment block is incomplete.
Scaffold with: `uv run ../../scripts/python/new-script.py <name> --deps "httpx2[http2,brotli,zstd]"` (writes to temp dir by default, `--output` for specific path).
Load `one-liners.md` for full patterns, examples, and anti-patterns.
## Reference loading
Load on demand — not all at once.
| Need | Load |
|---|---|
| Full pyproject.toml config | `pyproject-strict.md` |
| Type patterns (NewType, Final, enums, narrowing) | `type-patterns.md` |
| Data modeling (container choice, frozen, parse-don't-validate) | `data-modeling.md` |
| Error handling (typed errors, union returns, exhaustive match) | `error-handling.md` |
| Async patterns (anyio) | `async-anyio.md` |
| Data processing (polars / duckdb) | `data-processing.md` |
| FastAPI + SQLAlchemy stack | `fastapi-stack.md` |
| Library decision tree | `libraries.md` |
| **httpx2 optimization** (MUST load for any network code) | `httpx2-optimization.md` |
| **orjson** (when JSON is in the hot path; FastAPI/Pydantic v2 integration) | `orjson-stack.md` |
| One-liner scripts (PEP 723) | `one-liners.md` |
| PydanticAI agents | `pydantic-ai.md` |
| Textual TUI | `textual-tui.md` |
## httpx2 — mandatory for ALL network requests
Every outgoing HTTP call MUST use [`httpx2`](https://github.com/pydantic/httpx2) (`httpx2[http2,brotli,zstd]`). Never `requests`, never `aiohttp`, never the original `httpx`.
**ALL optimizations are ON by default — not optional, not progressive, not "nice to have".** A bare `httpx2.AsyncClient()` is a bug — treat it like a lint violation. The correct way is the factory pattern in `httpx2-optimization.md` with: HTTP/2 enabled, tuned connection pool (200/40/30s), split timeouts (5/30/10/10), transport retries (3), TCP_NODELAY, follow_redirects, and event hooks for observability.
When writing or reviewing ANY network code, **ALWAYS load `httpx2-optimization.md`** and use the factory pattern verbatim. No exceptions.
## No-excuse audit
Violations caught by `../../scripts/python/check-no-excuse-rules.py`. Run after every edit session.
| Rule ID | Catches | Opt-out |
|---|---|---|
| `cast-any` | `cast(Any, ...)` | None — redesign types |
| `type-ignore` | `# type: ignore` | None — fix the type |
| `pyright-ignore` | `# pyright: ignore` | None — fix the type |
| `bare-except` | `except:` with no class | None — name the exception |
| `silent-except` | `except X: pass` / `except X: ...` | None — handle or re-raise |
| `no-asyncio` | `import asyncio` | `# noqa: ANYIO_OK` |
| `no-pandas` | `import pandas` | `# noqa: PANDAS_OK` |
| `mutable-dataclass` | `@dataclass` without `frozen=True` | `# noqa: MUTABLE_OK` |
| `missing-slots` | `@dataclass` without `slots=True` | `# noqa: SLOTS_OK` |
| `raw-dict-return` | `-> dict` in function return type | `# noqa: DICT_OK` |
| `missing-assert-never` | `match` block without `assert_never` default | `# noqa: MATCH_OK` |
| `generic-exception` | `raise ValueError("...")` / `raise TypeError("...")` with bare string | `# noqa: GENERIC_ERR_OK` |
| `no-object` | `object` used as type annotation (param, return, generic arg) | `# noqa: OBJECT_OK` |
| `if-elif-on-variant` | `if isinstance()`/`if x == Enum.V` chain that should be `match/case` | `# noqa: IF_VARIANT_OK` |
| `oversized-module` | File exceeds 250 pure LOC (non-blank, non-comment) | `# noqa: SIZE_OK` |
| `broad-except` | `except Exception` / `except BaseException` (too broad) | `# noqa: BROAD_EXCEPT_OK` |
Fix every violation before declaring work done. basedpyright + ruff strict config catches the rest.
## In tests
Tests are strict too, with these exceptions (already configured in `pyproject.toml` per-file-ignores):
| In tests you may | Why |
|---|---|
| Use `assert` | That's how pytest works (`S101` ignored) |
| Use magic numbers | Test data (`PLR2004` ignored) |
| Access `_private` members | Testing internals (`SLF001` ignored) |
| Skip docstrings | Test names are the docs (`D` ignored) |
| Have unused function args | Fixtures (`ARG` ignored) |
Tests still follow the iron list — frozen dataclasses, typed errors, exhaustive match. If test fixtures need mutable state, use `# noqa: MUTABLE_OK` on the fixture class.
## Existing codebases
When editing an existing file that doesn't follow these rules: **write new code in strict style, don't refactor existing code in the same change.** Mixing feature work with style migration makes reviews harder and bugs likelier.
## Activation
This skill activates whenever you are writing or modifying any `.py` file. Even one-off scripts get the strict treatment — that is the whole point of PEP 723 + uv: production hygiene with throwaway ergonomics.
packages/shared-skills/skills/programming/references/python/async-anyio.md
+442
View File
@@ -0,0 +1,442 @@
# AnyIO Reference: Replacing asyncio Idioms
> **Skill mandate**: `import asyncio` is BANNED. Use `import anyio` exclusively.
> This reference targets AnyIO 4.x (2026 Python projects).
---
## 1. Task Groups (The Core Primitive)
AnyIO uses **structured concurrency** via task groups. A task group is an async context manager that guarantees all child tasks finish before the block exits.
### `start_soon` — fire-and-forget
```python
import anyio
async def worker(n: int) -> None:
await anyio.sleep(1)
print(f"task {n} done")
async def main() -> None:
async with anyio.create_task_group() as tg:
for i in range(3):
tg.start_soon(worker, i)
print("all tasks finished")
anyio.run(main)
```
**Signature**: `tg.start_soon(func, *args, name=None)`
- `func` must be a **coroutine function** (not a coroutine object).
- `name` is optional, for introspection/debugging.
- No return value; exceptions propagate as `ExceptionGroup` on exit.
### `start` — wait for ready signal
Use when a task must initialize before the caller proceeds (e.g., starting a server and then connecting to it).
```python
from anyio import TASK_STATUS_IGNORED, create_task_group, run
from anyio.abc import TaskStatus
async def start_server(port: int, *, task_status: TaskStatus[None] = TASK_STATUS_IGNORED) -> None:
listener = await anyio.create_tcp_listener(local_host="127.0.0.1", local_port=port)
task_status.started() # unblocks tg.start()
await listener.serve(handler)
async def main() -> None:
async with create_task_group() as tg:
await tg.start(start_server, 8080) # blocks until task_status.started()
# server is guaranteed ready here
async with await anyio.connect_tcp("127.0.0.1", 8080) as client:
...
run(main)
```
**Rule of thumb**:
- Use `start_soon` when you don't need to know when the task is ready.
- Use `start` when the task must signal readiness before you continue.
### `create_task` — retrieving return values (AnyIO 4.14+)
```python
async def add(x: int, y: int) -> int:
return x + y
async def main() -> None:
async with anyio.create_task_group() as tg:
handle = tg.create_task(add(2, 4))
result = await handle # == 6
print(handle.return_value) # also 6
anyio.run(main)
```
**Signature**: `tg.create_task(coro, *, name=None, context=None) -> TaskHandle[T]`
- Returns a `TaskHandle` you can `await` for the result.
- If the task raises, awaiting raises `TaskFailed` (or `TaskCancelled`).
- This is the canonical replacement for `asyncio.gather` when you need results.
---
## 2. asyncio → anyio Cheat Sheet
| asyncio | anyio | Notes |
|---------|-------|-------|
| `asyncio.gather(a, b, c)` | `tg.create_task(a); tg.create_task(b); tg.create_task(c); results = [await h for h in handles]` | No direct gather; structured concurrency requires explicit task group scope. For fire-and-forget, use `tg.start_soon`. |
| `asyncio.create_task(coro)` | `tg.start_soon(func, *args)` or `tg.create_task(coro)` | `start_soon` takes a coroutine **function** + args. `create_task` takes a coroutine **object** and returns a handle. |
| `asyncio.sleep(n)` | `anyio.sleep(n)` | Identical semantics. |
| `asyncio.wait_for(coro, timeout)` | `with anyio.fail_after(timeout): await coro` | Raises `TimeoutError`. Use `move_on_after` for silent timeout. |
| `asyncio.Event()` | `anyio.Event()` | AnyIO events are **not reusable**; create a new one instead of `.clear()`. |
| `asyncio.Lock()` | `anyio.Lock()` | Use `async with lock:`. Pass `fast_acquire=True` if performance-critical. |
| `asyncio.Semaphore(n)` | `anyio.Semaphore(n)` | Same. Pass `fast_acquire=True` if performance-critical. |
| `asyncio.Condition()` | `anyio.Condition()` | Same semantics. |
| `asyncio.run(main())` | `anyio.run(main)` | Backend-agnostic entry point. |
| `asyncio.Queue(maxsize=N)` | `anyio.create_memory_object_stream[T](max_buffer_size=N)` | Returns `(send_stream, receive_stream)`. Supports `async for` on receive end. |
| `asyncio.to_thread(fn, *args)` | `anyio.to_thread.run_sync(fn, *args)` | Supports `abandon_on_cancel=True` and custom `limiter`. |
| `asyncio.run_coroutine_threadsafe(coro, loop)` | `anyio.from_thread.run(func, *args)` | Call async code from a worker thread. |
| `loop.call_soon_threadsafe(callback)` | `anyio.from_thread.run_sync(func, *args)` | Call sync code in event loop thread from worker thread, **with return value**. |
| `asyncio.shield(coro)` | `with anyio.CancelScope(shield=True): ...` | AnyIO shielding does not orphan tasks. |
| `asyncio.timeout(delay)` | `with anyio.fail_after(delay): ...` | AnyIO uses level cancellation, not edge cancellation. |
| `asyncio.CancelledError` | `anyio.get_cancelled_exc_class()` | Use this to catch cancellation portably across backends. |
---
## 3. Cancellation & CancelScope
AnyIO uses **level cancellation** (inspired by Trio), not asyncio's **edge cancellation**.
- **Edge cancellation** (asyncio): A `CancelledError` is injected once. If caught and not re-raised, the task keeps running.
- **Level cancellation** (anyio): As long as a task is inside an effectively cancelled scope, every yield point raises a new cancellation exception.
### Basic CancelScope
```python
from anyio import CancelScope, create_task_group, get_cancelled_exc_class, sleep, run
async def worker() -> None:
try:
await sleep(10)
except get_cancelled_exc_class():
print("cancelled!")
raise # ALWAYS re-raise cancellation exceptions
async def main() -> None:
async with create_task_group() as tg:
tg.start_soon(worker)
await sleep(0.1)
tg.cancel_scope.cancel() # cancels all children
run(main)
```
### Shielding
Shield a block from external cancellation. Essential for cleanup.
```python
from anyio import CancelScope, create_task_group, sleep, run
async def main() -> None:
async with create_task_group() as tg:
with CancelScope(shield=True):
tg.start_soon(some_task)
tg.cancel_scope.cancel() # shielded block is protected
await sleep(1) # this still runs
run(main)
```
**Combine with timeouts for graceful shutdown**:
```python
from anyio import CancelScope, move_on_after
async def do_something(resource) -> None:
try:
await run_async_stuff()
except BaseException:
# Allow up to 10s for cleanup, then move on
with move_on_after(10, shield=True):
await resource.aclose()
raise
```
### Structured Concurrency Guarantee
A task group contains its own `CancelScope`. If any child task raises an exception:
1. The task group's cancel scope is cancelled.
2. All other child tasks receive cancellation.
3. The task group waits for all children to finish.
4. The original exception (wrapped in `ExceptionGroup` if multiple) is re-raised.
---
## 4. Timeouts
Two context managers. Both create a `CancelScope` internally.
### `fail_after` — raises on timeout
```python
from anyio import fail_after, sleep, run
async def main() -> None:
try:
with fail_after(5) as scope:
await sleep(10)
except TimeoutError:
print("timed out")
print(scope.cancelled_caught) # True
run(main)
```
### `move_on_after` — silent timeout
```python
from anyio import move_on_after, sleep, run
async def main() -> None:
with move_on_after(5) as scope:
await sleep(10)
print("this never prints")
print("exited scope, cancelled =", scope.cancelled_caught)
run(main)
```
### Combined with shielding
```python
from anyio import move_on_after
# Give cleanup 10 seconds, but don't let outer cancellation interrupt it
with move_on_after(10, shield=True):
await resource.aclose()
```
---
## 5. Memory Object Streams (Queue Replacement)
Replaces `asyncio.Queue` with a safer, typed, structured-concurrency-friendly construct.
```python
from anyio import create_task_group, create_memory_object_stream, run
from anyio.streams.memory import MemoryObjectReceiveStream
async def consumer(stream: MemoryObjectReceiveStream[str]) -> None:
async with stream: # closes receive end on exit
async for item in stream:
print("received", item)
async def main() -> None:
# Type-annotated stream creation (AnyIO 4+ syntax)
send_stream, receive_stream = create_memory_object_stream[str](max_buffer_size=10)
async with create_task_group() as tg:
tg.start_soon(consumer, receive_stream)
async with send_stream:
for i in range(5):
await send_stream.send(f"item {i}")
# send_stream closed → consumer's async for loop exits naturally
run(main)
```
**Key differences from `asyncio.Queue`**:
- **Bounded by default**: `max_buffer_size=0` means send blocks until a receiver is ready.
- **Cloneable**: Each producer/consumer can close its own clone. The stream only ends when **all** clones of one end are closed.
- **Async iterable**: `async for item in receive_stream:` works out of the box.
- **Type-safe**: Generic `create_memory_object_stream[T]()`.
- **Synchronous close**: Both `close()` and `async with` work.
---
## 6. Backend Selection
AnyIO is backend-agnostic. Code written against AnyIO APIs runs on both asyncio and Trio.
```python
import anyio
async def main() -> None:
print("running on", anyio.current_async_library())
await anyio.sleep(1)
# Default backend (asyncio)
anyio.run(main)
# Explicit backend
anyio.run(main, backend="trio")
anyio.run(main, backend="asyncio", backend_options={"debug": True})
```
**Library design rule**: Never hardcode a backend. Let the application choose via `anyio.run()`. Libraries should only import `anyio` and avoid backend-specific APIs.
---
## 7. Compatibility with asyncio-only libraries
### Using asyncio libraries under the asyncio backend
If a third-party library exposes only an asyncio interface (returns asyncio coroutine objects), it works directly under the asyncio backend because AnyIO runs on top of asyncio's event loop:
```python
import anyio
import some_asyncio_only_lib # returns asyncio.Future/coroutine objects
async def main() -> None:
# This works because under the asyncio backend, await passes through
result = await some_asyncio_only_lib.fetch_data()
anyio.run(main, backend="asyncio")
```
**Important**: This only works on the `asyncio` backend. On the `trio` backend, asyncio-native objects will not work.
### When you MUST use asyncio APIs
Some APIs have no AnyIO equivalent and require direct event loop access:
| Scenario | asyncio API | AnyIO approach |
|----------|-------------|----------------|
| Signal handlers | `loop.add_signal_handler()` | `anyio.open_signal_receiver()` |
| Custom protocols | `asyncio.Protocol` | Use AnyIO streams / sockets |
| Direct Future manipulation | `asyncio.Future` | Avoid; use AnyIO primitives |
| Eager task factories | `asyncio.eager_task_factory` | Experimental in AnyIO; avoid |
If you absolutely need the running loop:
```python
import asyncio
async def main() -> None:
loop = asyncio.get_running_loop()
# ... do something loop-specific ...
# WARNING: this breaks backend-agnosticism
anyio.run(main, backend="asyncio")
```
**Best practice**: Wrap asyncio-only code in a backend-agnostic facade, and document that the feature requires the asyncio backend.
---
## 8. Idiomatic Code Snippets
### Snippet 1: Parallel HTTP requests with timeout and cleanup
```python
import anyio
async def fetch(url: str) -> bytes:
await anyio.sleep(0.5) # simulate
return b"data"
async def main() -> None:
urls = ["a", "b", "c"]
async with anyio.create_task_group() as tg:
with anyio.move_on_after(5):
for url in urls:
tg.start_soon(fetch, url)
# All tasks are cancelled on timeout; task group waits for cleanup
anyio.run(main)
```
### Snippet 2: Producer-consumer with memory object stream
```python
import anyio
from anyio.streams.memory import MemoryObjectReceiveStream
async def producer(send_stream: anyio.streams.memory.MemoryObjectSendStream[int]) -> None:
async with send_stream:
for i in range(100):
await send_stream.send(i)
async def consumer(receive_stream: MemoryObjectReceiveStream[int]) -> None:
async with receive_stream:
async for item in receive_stream:
print(f"consumed {item}")
async def main() -> None:
send, receive = anyio.create_memory_object_stream[int](max_buffer_size=5)
async with anyio.create_task_group() as tg:
tg.start_soon(producer, send)
tg.start_soon(consumer, receive)
anyio.run(main)
```
### Snippet 3: Calling sync code from async
```python
import time
import anyio
async def main() -> None:
# Run blocking function in worker thread
result = await anyio.to_thread.run_sync(time.sleep, 2)
print("done")
anyio.run(main)
```
### Snippet 4: Calling async code from a worker thread
```python
import anyio
def blocking_callback() -> None:
# Inside a worker thread, call back into the event loop
anyio.from_thread.run(anyio.sleep, 1)
anyio.from_thread.run_sync(print, "hello from thread")
async def main() -> None:
await anyio.to_thread.run_sync(blocking_callback)
anyio.run(main)
```
### Snippet 5: Graceful shutdown with shielded cleanup
```python
import anyio
async def worker() -> None:
try:
await anyio.sleep_forever()
except anyio.get_cancelled_exc_class():
with anyio.CancelScope(shield=True):
await anyio.sleep(0.5) # cleanup
print("cleaned up")
raise
async def main() -> None:
async with anyio.create_task_group() as tg:
tg.start_soon(worker)
await anyio.sleep(1)
tg.cancel_scope.cancel()
anyio.run(main)
```
---
## Sources
- AnyIO Documentation (stable): https://anyio.readthedocs.io/en/stable/
- AnyIO GitHub (HEAD `cb245dba`): https://github.com/agronholm/anyio
- Task Groups: https://anyio.readthedocs.io/en/stable/tasks.html
- Cancellation & Timeouts: https://anyio.readthedocs.io/en/stable/cancellation.html
- Streams: https://anyio.readthedocs.io/en/stable/streams.html
- Synchronization: https://anyio.readthedocs.io/en/stable/synchronization.html
- Threads: https://anyio.readthedocs.io/en/stable/threads.html
- Basics / Backends: https://anyio.readthedocs.io/en/stable/basics.html
- Design Rationale (why asyncio is problematic): https://anyio.readthedocs.io/en/stable/why.html
packages/shared-skills/skills/programming/references/python/data-modeling.md
+233
View File
@@ -0,0 +1,233 @@
# Data Modeling
Which container to use, how to structure data, and why frozen is the default.
---
## Decision flowchart
```
Is it a fixed set of named constants?
YES → StrEnum / IntEnum
NO ↓
Is it just branding a primitive (int, str, float)?
YES → NewType("X", base)
NO ↓
Is it an interface / contract ("this thing can do X")?
├─ Shape only, no shared code → Protocol
└─ Shared method implementation needed → ABC
NO ↓
Does the data cross a trust boundary (user input, API, file, external DB)?
YES → pydantic.BaseModel (frozen=True) — validates + serializes
NO ↓
Is it a dict shape needed for JSON compat / **kwargs typing?
YES → TypedDict
NO ↓
Is it structured data with named fields?
YES → @dataclass(frozen=True, slots=True)
NO ↓
Is it a tuple with positional semantics (x, y coords / DB row)?
YES → NamedTuple
NO → you probably don't need a new type
```
---
## Container reference
### @dataclass — internal value object
The default for structured data inside your codebase. Zero overhead, no framework coupling.
```python
from dataclasses import dataclass
from typing import NewType
UserId = NewType("UserId", int)
@dataclass(frozen=True, slots=True)
class User:
id: UserId
name: str
email: str
@dataclass(frozen=True, slots=True)
class Point:
x: float
y: float
```
Always `frozen=True, slots=True`. Mutable only when mutation is the documented purpose — opt out with `# noqa: MUTABLE_OK`.
### Pydantic BaseModel — trust boundary guardian
Use when data enters or leaves your system. Validates at construction, serializes to JSON, generates OpenAPI schema.
```python
from pydantic import BaseModel, ConfigDict, EmailStr
class CreateUserRequest(BaseModel):
model_config = ConfigDict(frozen=True)
name: str
email: EmailStr
age: int
class UserResponse(BaseModel):
model_config = ConfigDict(frozen=True)
id: int
name: str
email: str
```
**The one rule**: data crosses a trust boundary → Pydantic. Everything else → dataclass.
Never use Pydantic for internal-only data just because it's convenient. The validation cost is real.
### TypedDict — dict that knows its shape
Use when the value must stay a `dict` at runtime — JSON blobs, `**kwargs`, third-party APIs expecting dicts.
```python
from typing import TypedDict, NotRequired
class Headers(TypedDict):
content_type: str
authorization: NotRequired[str]
def make_request(url: str, headers: Headers) -> None: ...
make_request("https://api.example.com", {"content_type": "application/json"})
```
### Protocol — structural interface
"Anything that has method X" — no inheritance required.
```python
from typing import Protocol
class Renderable(Protocol):
def render(self) -> str: ...
class Saveable(Protocol):
async def save(self) -> None: ...
@dataclass(frozen=True, slots=True)
class MarkdownDoc:
content: str
def render(self) -> str:
return self.content
def publish(doc: Renderable) -> None:
print(doc.render()) # MarkdownDoc works — no inheritance needed
```
Default to Protocol for interfaces. ABC only when you need shared method implementations.
### ABC — interface with shared code
Only when Protocol isn't enough.
```python
from abc import ABC, abstractmethod
class BaseRepository(ABC):
@abstractmethod
async def get(self, id: int) -> Model | None: ...
@abstractmethod
async def save(self, model: Model) -> None: ...
async def get_or_raise(self, id: int) -> Model:
result = await self.get(id)
if result is None:
msg = f"{type(self).__name__}: id {id} not found"
raise LookupError(msg)
return result
```
### NamedTuple — positional + named (rare)
Only when you need tuple protocol (unpacking, indexing).
```python
from typing import NamedTuple
class Coordinate(NamedTuple):
x: float
y: float
x, y = Coordinate(1.0, 2.0) # tuple unpacking
```
99% of the time, `@dataclass(frozen=True, slots=True)` is better.
---
## Quick lookup
| Situation | Use | Why |
|---|---|---|
| User input, API request/response | `Pydantic BaseModel` | Validation, JSON schema, serialization |
| DB row ↔ Python (ORM) | SQLAlchemy `Mapped[]` model | ORM integration, async session |
| Internal value object | `@dataclass(frozen=True, slots=True)` | Zero overhead, no validation needed |
| Multiple outcomes from function | Union of frozen dataclasses | Distinct types for `match` |
| Dict shape for JSON / `**kwargs` | `TypedDict` | Stays a dict at runtime |
| Fixed constants | `StrEnum` / `IntEnum` | Exhaustive match, no typos |
| Distinct primitive | `NewType("X", int)` | Zero runtime cost, type-level only |
| Contract / capability | `Protocol` | Structural typing, no inheritance |
| Contract + shared impl | `ABC` | When Protocol isn't enough |
---
## Comparison matrix
| Feature | dataclass | Pydantic | TypedDict | Protocol | NamedTuple | NewType | Enum |
|---|---|---|---|---|---|---|---|
| Validation | - | ✓ | - | - | - | - | - |
| JSON serialization | manual | built-in | native dict | - | - | - | `.value` |
| Immutable | frozen=True | frozen=True | - (dict) | N/A | always | N/A | always |
| Runtime cost | ~zero | validation | zero | zero | ~zero | zero | ~zero |
| `match` support | ✓ | ✓ | - | - | ✓ | - | ✓ |
| `slots` support | ✓ | - | - | - | - | - | - |
---
## Parse, don't validate
Validate at the boundary. Inside the boundary, types are proof of validity.
```python
# BAD — validate then pass raw data
def process_email(email: str) -> None:
if "@" not in email:
raise ValueError("invalid email")
# still a raw str everywhere downstream
# GOOD — parse into typed value at boundary
from typing import NewType
Email = NewType("Email", str)
def parse_email(raw: str) -> Email:
if "@" not in raw or "." not in raw.split("@")[1]:
msg = f"invalid email: {raw}"
raise ValueError(msg)
return Email(raw.lower().strip())
# Downstream only sees Email, never raw str
def send_welcome(email: Email) -> None: ...
```
With Pydantic this happens automatically — `EmailStr` is already a parsed type. Once constructed, `.email` is always valid. No re-validation needed.
---
## Sources
- Python docs: [dataclasses](https://docs.python.org/3/library/dataclasses.html)
- Pydantic v2: [docs.pydantic.dev](https://docs.pydantic.dev/latest/)
- Python docs: [typing — Protocol](https://docs.python.org/3/library/typing.html#typing.Protocol)
- Python docs: [typing — TypedDict](https://docs.python.org/3/library/typing.html#typing.TypedDict)
- Alexis King: [Parse, don't validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/)
packages/shared-skills/skills/programming/references/python/data-processing.md
+133
View File
@@ -0,0 +1,133 @@
# Data Processing — Polars + DuckDB
## The rule
NEVER pandas. Polars (with numpy) plus DuckDB. Pandas is 10-50x slower, has weaker types, and the modern Python data ecosystem has moved on.
## Quick decision tree
| Operation | Use | Why |
|---|---|---|
| `.csv` / `.parquet` / `.json` direct query | DuckDB | Zero memory load, SQL ergonomics |
| `.duckdb` file | DuckDB | Native format |
| Filter (any size) | Polars | 128x faster than DuckDB for filtering |
| Sort | Polars | 12x faster |
| Multi-table join | DuckDB | 3x faster, more join types |
| Heavy GROUP BY aggregation | DuckDB | 4x faster on large datasets |
| Window function | Polars | 3-5x faster |
| Pivot / melt / string ops | Polars | 2x faster |
| Larger than RAM | Polars streaming or DuckDB out-of-core | Both handle OOM |
| Mixed pipeline | Hybrid (zero-copy via Arrow) | Use each tool's strengths |
For the deep version (per-operation benchmarks, OOM strategies, full execution templates), load the **`data-scientist`** skill - it lives in this same skill set and is the source of truth for performance numbers.
## Standard imports
```python
import numpy as np
import polars as pl
import duckdb
```
## DuckDB direct file query (zero memory load)
```python
result = duckdb.sql("""
SELECT category, SUM(amount) AS total
FROM 'data.csv'
WHERE date >= '2026-01-01'
GROUP BY category
ORDER BY total DESC
""").pl() # zero-copy → Polars DataFrame
```
`.pl()` returns Polars; `.df()` would return pandas - never use `.df()`.
## Polars lazy pipeline
```python
result = (
pl.scan_csv("data.csv") # lazy, no read yet
.filter(pl.col("amount") > 1000)
.filter(pl.col("status") == "active")
.sort("amount", descending=True)
.head(100)
.collect() # execute optimised plan
)
```
`scan_*` over `read_*` for files; `lazy()` then `collect()` for in-memory frames. Polars optimises the entire plan before execution (predicate pushdown, projection pushdown, common subexpression elimination).
## Streaming for OOM data
```python
result = (
pl.scan_csv("huge.csv")
.filter(pl.col("active"))
.group_by("category")
.agg([
pl.len().alias("count"),
pl.sum("amount").alias("total"),
])
.collect(streaming=True)
)
```
## Hybrid pipeline (most realistic shape)
```python
# Phase 1: DuckDB for the join (3x faster)
joined = duckdb.sql("""
SELECT o.*, c.region, p.category
FROM 'orders.parquet' o
JOIN 'customers.parquet' c ON o.customer_id = c.id
JOIN 'products.parquet' p ON o.product_id = p.id
""").pl()
# Phase 2: Polars for filtering and transformation (128x + 2x faster)
processed = (
joined
.filter(pl.col("amount") > 100)
.with_columns([
(pl.col("amount") * 1.1).alias("amount_with_tax"),
])
)
# Phase 3: DuckDB for final aggregation (4x faster) - register Polars frame by name
duckdb.register("processed", processed)
final = duckdb.sql("""
SELECT region, category, SUM(amount_with_tax) AS revenue
FROM processed
GROUP BY region, category
ORDER BY revenue DESC
""").pl()
```
## Type safety with Polars
Polars supports schema overrides at read time, and `.cast()` for explicit conversion. Avoid implicit coercion in hot paths.
```python
schema = {"id": pl.Int64, "amount": pl.Float64, "date": pl.Date}
df = pl.read_csv("data.csv", schema_overrides=schema)
```
basedpyright understands `polars-stubs`, which ship with polars itself. No extra type stubs to install.
## Things you might miss from pandas (and how to do them in Polars)
| pandas | polars |
|---|---|
| `df.iloc[5]` | `df.row(5)` (named tuple) or `df[5]` (single-row frame) |
| `df.loc[df["x"] > 5]` | `df.filter(pl.col("x") > 5)` |
| `df["x"].apply(fn)` | `df["x"].map_elements(fn)` (slow path) or use native expressions |
| `df.merge(...)` | `df.join(other, on="key")` |
| `df.groupby(...).agg(...)` | `df.group_by(...).agg(...)` |
| `pd.read_csv(...).dtypes` | `pl.read_csv(...).schema` |
| `df.to_dict("records")` | `df.to_dicts()` |
## Sources
- Polars docs: <https://docs.pola.rs>
- DuckDB Python API: <https://duckdb.org/docs/api/python/overview>
- Cross-reference - this skill set's `data-scientist` skill (load it for the deep version)
packages/shared-skills/skills/programming/references/python/error-handling.md
+218
View File
@@ -0,0 +1,218 @@
# Error Handling
Typed errors, exhaustive matching, union returns, and resource safety.
---
## Typed errors — no bare strings
Error types carry structured data. Pattern matching works. Callers know exactly what can go wrong.
```python
from dataclasses import dataclass
from typing import NewType
UserId = NewType("UserId", int)
@dataclass(frozen=True, slots=True)
class UserNotFoundError(Exception):
user_id: UserId
def __str__(self) -> str: # REQUIRED — see note below
return f"user {self.user_id} not found"
@dataclass(frozen=True, slots=True)
class PermissionDeniedError(Exception):
user_id: UserId
required_role: str
def __str__(self) -> str:
return f"user {self.user_id} needs role {self.required_role}"
```
**`__str__` is mandatory** on dataclass exceptions. `@dataclass` replaces `Exception.__init__`, so `self.args` is always `()`. Without `__str__`, `str(e)` returns an empty string and logging/monitoring breaks.
```python
# BAD
raise ValueError("user not found")
raise ValueError("permission denied")
# GOOD
raise UserNotFoundError(user_id=uid)
raise PermissionDeniedError(user_id=uid, required_role="admin")
```
---
## Union returns — expected failures without exceptions
For failures that are **expected** (not found, validation error, permission denied), return a union instead of raising. Exceptions are for **unexpected** failures (network down, OOM, corrupted data).
### Define the outcome types
```python
@dataclass(frozen=True, slots=True)
class User:
id: UserId
name: str
@dataclass(frozen=True, slots=True)
class UserNotFound:
id: UserId
@dataclass(frozen=True, slots=True)
class PermissionDenied:
id: UserId
reason: str
type GetUserResult = User | UserNotFound | PermissionDenied
```
### Handle exhaustively
```python
from typing import assert_never
def handle_result(result: GetUserResult) -> str:
match result:
case User(name=name):
return f"Found: {name}"
case UserNotFound(id=uid):
return f"No user with id {uid}"
case PermissionDenied(reason=reason):
return f"Denied: {reason}"
case _ as unreachable:
assert_never(unreachable)
```
`assert_never` in the default case: if you add a new variant to `GetUserResult` without handling it here, the type checker errors. No silent fall-through.
### When to use which
**The heuristic**: caller is 1-2 levels away and MUST handle it → union return. Error should propagate up many layers to a boundary → exception.
| Scenario | Pattern | Why |
|---|---|---|
| Repository → service (caller handles it) | Union return (`User \| UserNotFound`) | Caller is right there, must handle both |
| Validation at boundary (parsing input) | Exception (typed, with fields) | Propagates up to HTTP/CLI handler |
| Infrastructure failure (network, OOM) | Exception | Can't handle locally, must propagate |
| Service → service (deep internal) | Exception (typed) | Union boilerplate across many layers is worse than exceptions |
| HTTP handler → response | Catch exceptions, convert to response | Boundary code catches and translates |
**Practical tradeoff**: union returns are safest (type checker forces handling) but create boilerplate when every caller in a chain must `match`. If the error would just propagate through 3+ layers unchanged, use a typed exception instead.
---
## Exhaustive match — every match needs a default
Every `match` statement ends with `case _: assert_never(x)`. No exceptions.
```python
from enum import StrEnum
from typing import assert_never
class Status(StrEnum):
PENDING = "pending"
ACTIVE = "active"
DELETED = "deleted"
def describe(status: Status) -> str:
match status:
case Status.PENDING:
return "waiting"
case Status.ACTIVE:
return "live"
case Status.DELETED:
return "gone"
case _ as unreachable:
assert_never(unreachable)
```
Add a new enum member? The type checker tells you every `match` that needs updating.
---
## Context managers — resource safety
If it has `.close()`, `.shutdown()`, `.disconnect()`, or `.release()`, wrap it in `with`.
```python
# BAD
f = open("data.txt")
data = f.read()
f.close() # forgotten? leaked
# GOOD
from pathlib import Path
data = Path("data.txt").read_text()
```
### Async resources
```python
import httpx
async def fetch_users() -> list[User]:
async with httpx.AsyncClient() as client:
response = await client.get("https://api.example.com/users")
response.raise_for_status()
return [User(**u) for u in response.json()]
```
### Custom context manager
```python
from contextlib import asynccontextmanager
from collections.abc import AsyncIterator
@asynccontextmanager
async def managed_connection(url: str) -> AsyncIterator[Connection]:
conn = await connect(url)
try:
yield conn
finally:
await conn.close()
async with managed_connection("postgres://...") as conn:
await conn.execute("SELECT 1")
# conn is closed here, guaranteed
```
---
## Exception hierarchy — when you do raise
Keep exception hierarchies shallow and specific.
```python
class AppError(Exception):
"""Base for all application errors."""
@dataclass(frozen=True, slots=True)
class NotFoundError(AppError):
entity: str
id: int
def __str__(self) -> str:
return f"{self.entity} {self.id} not found"
@dataclass(frozen=True, slots=True)
class ConflictError(AppError):
entity: str
field: str
value: str
def __str__(self) -> str:
return f"{self.entity}.{self.field} = {self.value!r} already exists"
```
Callers catch `AppError` at the boundary, or specific subtypes where they can do something useful.
---
## Sources
- Python docs: [typing — assert_never](https://docs.python.org/3/library/typing.html#typing.assert_never)
- Python docs: [contextlib](https://docs.python.org/3/library/contextlib.html)
- Python docs: [match statement](https://docs.python.org/3/reference/compound_stmts.html#the-match-statement)
packages/shared-skills/skills/programming/references/python/fastapi-stack.md
+316
View File
@@ -0,0 +1,316 @@
# FastAPI + SQLAlchemy 2.x async + Postgres + Pydantic v2
The canonical web API stack. Async end-to-end, type-safe end-to-end, OpenAPI-generated end-to-end.
## Project layout
```
myapi/
├── pyproject.toml
├── alembic.ini
├── migrations/
│ └── env.py
├── src/
│ └── myapi/
│ ├── __init__.py
│ ├── main.py # FastAPI app + lifespan
│ ├── config.py # pydantic-settings
│ ├── db.py # engine, session factory, dependency
│ ├── models.py # SQLAlchemy declarative models
│ ├── schemas.py # Pydantic request/response models
│ └── routers/
│ ├── __init__.py
│ └── users.py
└── tests/
├── conftest.py
└── test_users.py
```
## Dependencies
```bash
uv add fastapi 'sqlalchemy[asyncio]>=2.0' asyncpg 'pydantic[email]>=2' pydantic-settings 'uvicorn[standard]' orjson
uv add --dev httpx pytest alembic
```
`orjson` is mandatory: set `default_response_class=ORJSONResponse` on the FastAPI app. Pydantic-typed responses bypass it (Pydantic v2's `model_dump_json` is already Rust-backed); raw `dict` / `list` returns are accelerated. For SSE / NDJSON streams, call `orjson.dumps(...)` per chunk inside `StreamingResponse`. See `orjson-stack.md` for the decision tree, flag reference, and benchmarks.
## Configuration (`config.py`)
```python
from functools import lru_cache
from pydantic import Field, PostgresDsn
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPI_")
database_url: PostgresDsn
debug: bool = False
cors_origins: list[str] = Field(default_factory=list)
@lru_cache
def get_settings() -> Settings:
return Settings() # type: ignore[call-arg] # pydantic populates from env
```
Wait — that comment violates the no-excuse rule. Use proper field defaults instead. Real version:
```python
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPI_")
database_url: PostgresDsn
debug: bool = False
cors_origins: list[str] = Field(default_factory=list)
```
Construct via `Settings(_env_file=".env")` if needed in tests; in production it reads from env.
## Database (`db.py`)
```python
from collections.abc import AsyncIterator
from typing import Annotated
from fastapi import Depends
from sqlalchemy.ext.asyncio import (
AsyncEngine,
AsyncSession,
async_sessionmaker,
create_async_engine,
)
from myapi.config import get_settings
def make_engine() -> AsyncEngine:
settings = get_settings()
return create_async_engine(
str(settings.database_url),
echo=settings.debug,
pool_pre_ping=True,
)
_engine = make_engine()
_SessionFactory = async_sessionmaker(_engine, expire_on_commit=False)
async def get_session() -> AsyncIterator[AsyncSession]:
async with _SessionFactory() as session:
yield session
SessionDep = Annotated[AsyncSession, Depends(get_session)]
```
`expire_on_commit=False` is essential for FastAPI - otherwise attribute access after commit triggers an implicit refresh and errors out under async.
## Models (`models.py`)
```python
from datetime import datetime, UTC
from sqlalchemy import DateTime, String, func
from sqlalchemy.orm import (
DeclarativeBase,
Mapped,
MappedAsDataclass,
mapped_column,
)
class Base(MappedAsDataclass, DeclarativeBase):
pass
class User(Base):
__tablename__ = "users"
id: Mapped[int] = mapped_column(primary_key=True, init=False)
email: Mapped[str] = mapped_column(String(255), unique=True, index=True)
name: Mapped[str] = mapped_column(String(100))
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
server_default=func.now(),
init=False,
)
```
`MappedAsDataclass` makes `User(email=..., name=...)` work as a real dataclass constructor. `init=False` excludes the auto-generated columns (`id`, `created_at`) from `__init__`.
## Schemas (`schemas.py`)
```python
from datetime import datetime
from pydantic import BaseModel, ConfigDict, EmailStr
class UserCreate(BaseModel):
email: EmailStr
name: str
class UserRead(BaseModel):
model_config = ConfigDict(from_attributes=True) # SQLAlchemy → Pydantic
id: int
email: EmailStr
name: str
created_at: datetime
```
Always have a separate `*Create` (input) and `*Read` (output) model. Never expose your ORM model as the API model.
## Routers (`routers/users.py`)
```python
from fastapi import APIRouter, HTTPException, status
from sqlalchemy import select
from myapi.db import SessionDep
from myapi.models import User
from myapi.schemas import UserCreate, UserRead
router = APIRouter(prefix="/users", tags=["users"])
@router.post("", response_model=UserRead, status_code=status.HTTP_201_CREATED)
async def create_user(payload: UserCreate, session: SessionDep) -> User:
user = User(email=payload.email, name=payload.name)
session.add(user)
await session.commit()
await session.refresh(user)
return user
@router.get("/{user_id}", response_model=UserRead)
async def get_user(user_id: int, session: SessionDep) -> User:
result = await session.execute(select(User).where(User.id == user_id))
user = result.scalar_one_or_none()
if user is None:
raise HTTPException(status.HTTP_404_NOT_FOUND, "User not found")
return user
@router.get("", response_model=list[UserRead])
async def list_users(session: SessionDep, limit: int = 100) -> list[User]:
result = await session.execute(select(User).limit(limit))
return list(result.scalars().all())
```
## Application (`main.py`)
```python
from contextlib import asynccontextmanager
from collections.abc import AsyncIterator
from fastapi import FastAPI
from myapi.config import get_settings
from myapi.routers import users
@asynccontextmanager
async def lifespan(_: FastAPI) -> AsyncIterator[None]:
# Startup: warm up engine pool, run migrations check, etc.
yield
# Shutdown: close engine
from myapi.db import _engine
await _engine.dispose()
def create_app() -> FastAPI:
settings = get_settings()
app = FastAPI(
title="My API",
debug=settings.debug,
lifespan=lifespan,
)
app.include_router(users.router)
return app
app = create_app()
```
Run with:
```bash
uv run uvicorn myapi.main:app --host 0.0.0.0 --port 8000 --reload
```
## Migrations (Alembic + async)
```bash
uv run alembic init -t async migrations
```
In `migrations/env.py` replace the `target_metadata` line:
```python
from myapi.models import Base
target_metadata = Base.metadata
```
Set `sqlalchemy.url` in `alembic.ini` to your async URL or override via `env.py`:
```python
from myapi.config import get_settings
config.set_main_option("sqlalchemy.url", str(get_settings().database_url))
```
Generate and apply:
```bash
uv run alembic revision --autogenerate -m "create users"
uv run alembic upgrade head
```
## Tests (`tests/test_users.py`)
```python
import pytest
from httpx import ASGITransport, AsyncClient
from myapi.main import app
@pytest.mark.anyio
async def test_create_and_get_user() -> None:
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
create_response = await client.post(
"/users",
json={"email": "alice@example.com", "name": "Alice"},
)
assert create_response.status_code == 201
user_id = create_response.json()["id"]
get_response = await client.get(f"/users/{user_id}")
assert get_response.status_code == 200
assert get_response.json()["email"] == "alice@example.com"
```
For database-backed tests, run a Postgres container in CI (`testcontainers-python` or `docker-compose`) and apply migrations against a test schema. SQLite-as-test-db breaks once you use Postgres-specific types (`JSONB`, `tsvector`, arrays).
## Common pitfalls
| Pitfall | Fix |
|---|---|
| `MissingGreenlet` exception when accessing relationships after commit | `expire_on_commit=False` on the session factory |
| Connection pool exhausted under load | Set `pool_size`, `max_overflow` in `create_async_engine` |
| Pydantic v1 syntax (`from pydantic import ...; class X(BaseModel): class Config: orm_mode = True`) | v2 uses `model_config = ConfigDict(from_attributes=True)` |
| Returning ORM objects without `response_model` | FastAPI serialises with `from_attributes=True` automatically; declare `response_model` so OpenAPI is correct |
| `await session.execute(...)` returning Sequence | Wrap with `list(result.scalars().all())` to satisfy strict types |
| `func.now()` returning naive datetime | Use `DateTime(timezone=True)` and `created_at: Mapped[datetime]` with `UTC`-aware default |
## Sources
- FastAPI: <https://fastapi.tiangolo.com>
- SQLAlchemy 2.x async: <https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html>
- SQLAlchemy MappedAsDataclass: <https://docs.sqlalchemy.org/en/20/orm/dataclasses.html>
- asyncpg: <https://magicstack.github.io/asyncpg/current/>
- Pydantic v2 migration: <https://docs.pydantic.dev/latest/migration/>
- Alembic async: <https://alembic.sqlalchemy.org/en/latest/cookbook.html#using-asyncio-with-alembic>
packages/shared-skills/skills/programming/references/python/httpx2-optimization.md
+360
View File
@@ -0,0 +1,360 @@
# httpx2 — Production Defaults
> **Source**: [pydantic/httpx2](https://github.com/pydantic/httpx2) — next-generation HTTP client for Python 3, continuation of HTTPX under Pydantic stewardship.
>
> **Rule**: Every network request MUST use `httpx2`. **ALL optimizations below are ON by default** — HTTP/2, brotli+zstd, tuned connection pool, fine-grained timeouts, transport retries, TCP_NODELAY. This is the baseline, not a stretch goal. A bare `httpx2.AsyncClient()` is a bug.
---
## 1. Installation — all extras, always
```toml
# pyproject.toml
dependencies = [
"httpx2[http2,brotli,zstd]",
]
```
| Extra | What it enables | Why it's mandatory |
|-------|----------------|--------------------|
| `http2` | HTTP/2 multiplexing via `h2` | Single TCP connection handles concurrent requests; eliminates head-of-line blocking |
| `brotli` | Brotli content decoding (`br`) | ~20% smaller payloads than gzip for text/JSON |
| `zstd` | Zstandard content decoding | Faster decompression than brotli at similar ratios; stdlib in Python ≥ 3.14 |
| `socks` | SOCKS5 proxy support via `socksio` | Install only if you route through SOCKS proxies |
All three core extras (`http2,brotli,zstd`) are non-negotiable. Omitting any is leaving performance on the table.
---
## 2. The canonical defaults — ALL ON
These are not "optimizations to consider". These are **the correct defaults** that every httpx2 client must use.
```python
import socket
import httpx2
# ── These are the STANDARD values. Use them verbatim. ──
LIMITS = httpx2.Limits(
max_connections=200, # library default 100 is too conservative
max_keepalive_connections=40, # library default 20 wastes reconnects
keepalive_expiry=30.0, # library default 5s kills warm connections too fast
)
TIMEOUT = httpx2.Timeout(
connect=5.0, # TCP + TLS handshake budget
read=30.0, # time to receive a response chunk
write=10.0, # time to send a request chunk
pool=10.0, # time to acquire a connection from pool
)
SOCKET_OPTIONS: list[tuple[int, int, int]] = [
(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1), # disable Nagle — no 40ms delay
]
```
### Why each knob is set this way
| Setting | Library default | Our default | Why |
|---------|----------------|-------------|-----|
| `http2` | `False` | **`True`** | HTTP/2 multiplexing is strictly superior for any modern API |
| `max_connections` | `100` | `200` | Headroom for fan-out; prevents pool exhaustion under load |
| `max_keepalive_connections` | `20` | `40` | Keeps warm connections alive; fewer TLS handshakes |
| `keepalive_expiry` | `5.0s` | `30.0s` | 5s is too aggressive — kills connections between burst requests |
| `Timeout(5.0)` uniform | `5.0` all | Split | Uniform 5s is too tight for reads, too loose for connects |
| `read` timeout | `5.0` | `30.0` | Slow APIs and streaming need breathing room |
| `pool` timeout | `5.0` | `10.0` | Explicit — hitting this means `max_connections` needs raising |
| `TCP_NODELAY` | off | **on** | Eliminates Nagle's 40ms coalescing delay for small payloads |
| `retries` | `0` | `3` | Retries on `ConnectError`/`ConnectTimeout` only — safe and resilient |
| `follow_redirects` | `False` | **`True`** | Most APIs redirect; failing on 3xx is wrong default behavior |
---
## 3. Factory functions — the ONE correct way to create clients
Copy this into your project. This is the canonical pattern.
```python
"""httpx2 client factory. Always use create_client() / create_async_client()."""
from __future__ import annotations
import socket
import typing
import httpx2
_LIMITS = httpx2.Limits(
max_connections=200,
max_keepalive_connections=40,
keepalive_expiry=30.0,
)
_TIMEOUT = httpx2.Timeout(
connect=5.0,
read=30.0,
write=10.0,
pool=10.0,
)
_SOCKET_OPTIONS: list[tuple[int, int, int]] = [
(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1),
]
def create_async_client(
*,
base_url: str = "",
http2: bool = True,
retries: int = 3,
limits: httpx2.Limits = _LIMITS,
timeout: httpx2.Timeout = _TIMEOUT,
headers: dict[str, str] | None = None,
event_hooks: dict[str, list[typing.Callable[..., typing.Any]]] | None = None,
**kwargs: typing.Any,
) -> httpx2.AsyncClient:
transport = httpx2.AsyncHTTPTransport(
http2=http2,
retries=retries,
limits=limits,
socket_options=_SOCKET_OPTIONS,
)
return httpx2.AsyncClient(
transport=transport,
timeout=timeout,
base_url=base_url,
headers=headers or {},
event_hooks=event_hooks or {},
follow_redirects=True,
**kwargs,
)
def create_client(
*,
base_url: str = "",
http2: bool = True,
retries: int = 3,
limits: httpx2.Limits = _LIMITS,
timeout: httpx2.Timeout = _TIMEOUT,
headers: dict[str, str] | None = None,
event_hooks: dict[str, list[typing.Callable[..., typing.Any]]] | None = None,
**kwargs: typing.Any,
) -> httpx2.Client:
transport = httpx2.HTTPTransport(
http2=http2,
retries=retries,
limits=limits,
socket_options=_SOCKET_OPTIONS,
)
return httpx2.Client(
transport=transport,
timeout=timeout,
base_url=base_url,
headers=headers or {},
event_hooks=event_hooks or {},
follow_redirects=True,
**kwargs,
)
```
Usage:
```python
# Async — the common case
async with create_async_client(base_url="https://api.example.com") as client:
r = await client.get("/users")
# Sync
with create_client() as client:
r = client.get("https://api.example.com/health")
```
**If you are NOT using this factory pattern, you are doing it wrong.** A bare `httpx2.AsyncClient()` leaves HTTP/2 off, retries off, TCP_NODELAY off, keepalive too short, and timeouts too uniform.
---
## 4. Special case overrides
The factory defaults cover 95% of use cases. Override only when you have a specific reason:
| Scenario | Override |
|----------|----------|
| LLM streaming endpoints | `timeout=httpx2.Timeout(connect=10.0, read=None, write=10.0, pool=10.0)` — no read timeout on streaming |
| Single-host API with low concurrency | `limits=httpx2.Limits(max_connections=50, max_keepalive_connections=20, keepalive_expiry=60.0)` |
| Ephemeral short-lived requests | `keepalive_expiry=5.0` — don't hold connections |
| Unix domain sockets | `httpx2.AsyncHTTPTransport(uds="/path/to/socket", ...)` |
| mTLS / client certs | Pass `verify=ssl_ctx` with `ctx.load_cert_chain(certfile=...)` |
| SOCKS proxy | `httpx2[socks]`, `proxy="socks5://..."` |
---
## 5. Event hooks — always wire observability
This is not optional. Every production client should log requests.
```python
import time
import logging
logger = logging.getLogger(__name__)
async def log_request(request: httpx2.Request) -> None:
request.extensions["request_start"] = time.perf_counter()
async def log_response(response: httpx2.Response) -> None:
start = response.request.extensions.get("request_start", 0)
elapsed = time.perf_counter() - start
logger.info(
"HTTP %s %s → %d (%.3fs, %s)",
response.request.method,
response.request.url,
response.status_code,
elapsed,
response.http_version,
)
# Sync versions for Client
def log_request_sync(request: httpx2.Request) -> None:
request.extensions["request_start"] = time.perf_counter()
def log_response_sync(response: httpx2.Response) -> None:
start = response.request.extensions.get("request_start", 0)
elapsed = time.perf_counter() - start
logger.info(
"HTTP %s %s → %d (%.3fs, %s)",
response.request.method,
response.request.url,
response.status_code,
elapsed,
response.http_version,
)
```
For auto `raise_for_status()`:
```python
async def raise_on_error(response: httpx2.Response) -> None:
response.raise_for_status()
```
---
## 6. Verification script — confirm your setup is fully optimized
Run this against your target endpoint to **verify** (not decide) that all optimizations are active:
```python
"""Verify httpx2 is fully optimized against a target endpoint."""
from __future__ import annotations
import socket
import time
import anyio
import httpx2
TARGET_URL = "https://api.example.com/health"
ITERATIONS = 30
async def bench(label: str, client: httpx2.AsyncClient, url: str, n: int) -> float:
for _ in range(3): # warmup
await client.get(url)
start = time.perf_counter()
for _ in range(n):
r = await client.get(url)
assert r.status_code == 200
elapsed = time.perf_counter() - start
avg_ms = (elapsed / n) * 1000
print(f" {label}: {avg_ms:.1f}ms avg ({n} reqs in {elapsed:.2f}s)")
return avg_ms
async def main() -> None:
results: dict[str, float] = {}
# BAD: bare defaults (this is what we're proving is worse)
async with httpx2.AsyncClient() as c:
results["BAD-bare-defaults"] = await bench("BAD-bare-defaults", c, TARGET_URL, ITERATIONS)
# GOOD: full production defaults (this is what we always use)
limits = httpx2.Limits(max_connections=200, max_keepalive_connections=40, keepalive_expiry=30.0)
timeout = httpx2.Timeout(connect=5.0, read=30.0, write=10.0, pool=10.0)
transport = httpx2.AsyncHTTPTransport(
http2=True, retries=3, limits=limits,
socket_options=[(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)],
)
async with httpx2.AsyncClient(transport=transport, timeout=timeout, follow_redirects=True) as c:
results["GOOD-full-production"] = await bench("GOOD-full-production", c, TARGET_URL, ITERATIONS)
print("\n--- Proof ---")
baseline = results["BAD-bare-defaults"]
for label, avg in results.items():
delta = ((avg - baseline) / baseline) * 100
print(f" {label}: {avg:.1f}ms ({delta:+.1f}% vs bare)")
if __name__ == "__main__":
anyio.run(main)
```
---
## 7. Quick reference — all knobs
### `httpx2.AsyncClient` / `httpx2.Client`
| Parameter | Type | Library Default | **Our Default** |
|-----------|------|-----------------|-----------------|
| `http1` | `bool` | `True` | `True` |
| `http2` | `bool` | `False` | **`True`** |
| `verify` | `ssl.SSLContext \| str \| bool` | `True` | `True` |
| `cert` | `CertTypes \| None` | `None` | `None` |
| `proxy` | `str \| Proxy \| None` | `None` | `None` |
| `mounts` | `dict[str, Transport]` | `None` | `None` |
| `timeout` | `Timeout \| float \| None` | `Timeout(5.0)` | **Split: 5/30/10/10** |
| `limits` | `Limits` | `Limits(100, 20, 5.0)` | **`Limits(200, 40, 30.0)`** |
| `follow_redirects` | `bool` | `False` | **`True`** |
| `max_redirects` | `int` | `20` | `20` |
| `event_hooks` | `dict` | `{}` | **Wire logging** |
| `base_url` | `str` | `""` | Set for single-API clients |
| `trust_env` | `bool` | `True` | `True` |
| `default_encoding` | `str \| Callable` | `"utf-8"` | `"utf-8"` |
### `httpx2.AsyncHTTPTransport` / `httpx2.HTTPTransport`
| Parameter | Type | Library Default | **Our Default** |
|-----------|------|-----------------|-----------------|
| `http1` | `bool` | `True` | `True` |
| `http2` | `bool` | `False` | **`True`** |
| `retries` | `int` | `0` | **`3`** |
| `limits` | `Limits` | `Limits(100, 20, 5.0)` | **`Limits(200, 40, 30.0)`** |
| `uds` | `str \| None` | `None` | `None` |
| `local_address` | `str \| None` | `None` | `None` |
| `socket_options` | `Iterable[SOCKET_OPTION]` | `None` | **`[TCP_NODELAY]`** |
| `proxy` | `str \| Proxy \| None` | `None` | `None` |
### `httpx2.Timeout`
| Parameter | Library Default | **Our Default** |
|-----------|-----------------|-----------------|
| `connect` | `5.0` | `5.0` |
| `read` | `5.0` | **`30.0`** |
| `write` | `5.0` | **`10.0`** |
| `pool` | `5.0` | **`10.0`** |
### `httpx2.Limits`
| Parameter | Library Default | **Our Default** |
|-----------|-----------------|-----------------|
| `max_connections` | `100` | **`200`** |
| `max_keepalive_connections` | `20` | **`40`** |
| `keepalive_expiry` | `5.0` | **`30.0`** |
### Async backend (httpcore2)
httpcore2 uses `anyio` by default (works with both asyncio and trio). No extra config needed if you're already on the anyio stack. For trio, install `httpcore2[trio]`.
packages/shared-skills/skills/programming/references/python/libraries.md
+307
View File
@@ -0,0 +1,307 @@
# Library Defaults — Decision Tree
For each domain, the canonical 2026 choice, why, and the canonical usage snippet. The skill enforces these unless the project's `pyproject.toml` explicitly says otherwise.
## CLI — typer
`typer` builds a CLI from type-annotated function signatures. argparse needs 5x the code; click ignores type annotations; fire is magic that breaks at scale.
```python
import typer
from rich import print as rprint
app = typer.Typer()
@app.command()
def greet(name: str, count: int = 1, shout: bool = False) -> None:
"""Print a greeting `count` times."""
message = f"Hello, {name}!" if not shout else f"HELLO, {name.upper()}!"
for _ in range(count):
rprint(message)
if __name__ == "__main__":
app()
```
For a single-function script, `typer.run(main)` skips the `Typer()` boilerplate. Subcommands use `@app.command()`.
## Terminal output — rich
`rich` produces tables, progress bars, syntax highlighting, traceback rendering. Use it for any structured output. Plain `print` is acceptable for non-interactive log lines (and even those are usually better via `rich.console.Console(stderr=True).log(...)`).
```python
from rich.console import Console
from rich.table import Table
console = Console()
table = Table(title="Users")
table.add_column("ID", style="cyan")
table.add_column("Name", style="magenta")
table.add_row("1", "Alice")
console.print(table)
# Rich tracebacks (call once at process start)
from rich.traceback import install
install(show_locals=True)
```
## HTTP client — [httpx2](https://github.com/pydantic/httpx2)
Next-generation HTTP client under Pydantic stewardship. Sync and async in one library, HTTP/2 native, brotli + zstd content decoding, real type stubs. Replaces `requests` (sync only), `aiohttp` (async only), and the original `httpx`.
**Install**: `httpx2[http2,brotli,zstd]` — always include all three extras, no exceptions.
**A bare `httpx2.AsyncClient()` / `httpx2.Client()` is a bug.** Always use the factory pattern from `references/httpx2-optimization.md` with ALL optimizations enabled by default:
```python
import socket
import httpx2
# ── Production defaults — ALL ON, always. ──
_LIMITS = httpx2.Limits(max_connections=200, max_keepalive_connections=40, keepalive_expiry=30.0)
_TIMEOUT = httpx2.Timeout(connect=5.0, read=30.0, write=10.0, pool=10.0)
_SOCKET_OPTS: list[tuple[int, int, int]] = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)]
# Async (the common case)
transport = httpx2.AsyncHTTPTransport(http2=True, retries=3, limits=_LIMITS, socket_options=_SOCKET_OPTS)
async with httpx2.AsyncClient(transport=transport, timeout=_TIMEOUT, follow_redirects=True) as client:
response = await client.get("https://api.example.com/users")
response.raise_for_status()
users = response.json()
# Sync
transport = httpx2.HTTPTransport(http2=True, retries=3, limits=_LIMITS, socket_options=_SOCKET_OPTS)
with httpx2.Client(transport=transport, timeout=_TIMEOUT, follow_redirects=True) as client:
response = client.get("https://api.example.com/users")
response.raise_for_status()
users = response.json()
```
See `references/httpx2-optimization.md` for the full factory functions (`create_client()` / `create_async_client()`), event hooks, and the rationale behind every setting. **Load that reference whenever you write ANY network code.**
## JSON — stdlib `json` (default) or `orjson` (hot paths)
Stdlib `json` is fine for cold paths and configs. **Reach for `orjson` when JSON is in the hot path** — cache layers, queue payloads, streaming responses, structured logs, FastAPI endpoints returning raw `dict` / `list`.
```python
import orjson
# orjson.dumps returns bytes, not str
raw: bytes = orjson.dumps(
payload,
option=orjson.OPT_NAIVE_UTC | orjson.OPT_UTC_Z | orjson.OPT_SERIALIZE_DATACLASS,
)
```
**Critical 2026 fact**: with Pydantic v2, `model.model_dump_json()` is backed by pydantic-core (Rust) and is faster than `orjson + default=` bridge for Pydantic-shaped responses. **Use `model_dump_json()` for Pydantic; orjson for everything else.**
For FastAPI: `app = FastAPI(default_response_class=ORJSONResponse)`. Pydantic-typed responses bypass it (and that's correct — Pydantic's path is faster). Raw `dict`/`list` returns go through orjson.
See `references/orjson-stack.md` for the full decision tree, option flag reference, FastAPI integration, Redis/queue/logging patterns, and the `model_dump_json()` vs orjson benchmark.
## Validation — pydantic v2
Pydantic v2's core is in Rust (~10x faster than v1). It is the de-facto boundary validator. Use it for:
- HTTP request/response models (FastAPI uses pydantic natively)
- Config files (env vars via `pydantic-settings`)
- Anything entering the program from outside
```python
from pydantic import BaseModel, Field, EmailStr, field_validator
class User(BaseModel):
id: int = Field(ge=1)
email: EmailStr
name: str = Field(min_length=1, max_length=100)
age: int | None = Field(default=None, ge=0, le=150)
@field_validator("name")
@classmethod
def name_no_digits(cls, v: str) -> str:
if any(c.isdigit() for c in v):
raise ValueError("name cannot contain digits")
return v
# Inside the program, use the validated instance with confidence
user = User.model_validate({"id": 1, "email": "a@b.com", "name": "Alice"})
print(user.model_dump_json(indent=2))
```
`@dataclass` is fine for purely internal records (no validation needed). For anything crossing a process boundary, use Pydantic.
## Async — anyio
Full reference: [async-anyio.md](async-anyio.md). The summary:
```python
import anyio
async def fetch(url: str) -> str:
await anyio.sleep(0.1)
return url
async def main() -> None:
async with anyio.create_task_group() as tg:
for url in ["a", "b", "c"]:
tg.start_soon(fetch, url)
anyio.run(main)
```
Never `import asyncio` directly. The third-party libraries you call are free to use asyncio internally.
## Web framework — fastapi
Type-hint-driven HTTP framework. Pydantic models become OpenAPI schemas automatically.
```python
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class CreateUser(BaseModel):
name: str
email: str
class User(BaseModel):
id: int
name: str
email: str
@app.post("/users", response_model=User)
async def create_user(payload: CreateUser) -> User:
return User(id=1, **payload.model_dump())
```
Full stack with database: [fastapi-stack.md](fastapi-stack.md).
## ORM — sqlalchemy 2.x async
SQLAlchemy 2.x finally has a real async API. Use the modern declarative `MappedAsDataclass` style with type annotations.
```python
from sqlalchemy import String
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, MappedAsDataclass
class Base(MappedAsDataclass, DeclarativeBase):
pass
class User(Base):
__tablename__ = "users"
id: Mapped[int] = mapped_column(primary_key=True, init=False)
name: Mapped[str] = mapped_column(String(100))
email: Mapped[str] = mapped_column(String(255), unique=True)
engine = create_async_engine("postgresql+asyncpg://localhost/myapp")
SessionFactory = async_sessionmaker(engine, expire_on_commit=False)
```
Full pattern with FastAPI integration: [fastapi-stack.md](fastapi-stack.md).
## Database — postgres + asyncpg
For new applications, default to Postgres. SQLite for tests is fine; SQLite for production is not.
asyncpg is the fastest Python Postgres driver, native to SQLAlchemy 2.x async, native to FastAPI's lifespan model. URL: `postgresql+asyncpg://user:pass@host:5432/db`.
For migrations, use Alembic with `[alembic.context]` configured to use the async engine. Single-step:
```bash
uv add alembic
uv run alembic init -t async migrations
```
## TUI — textual
Textual builds rich, mouse-aware, mobile-style TUIs on the rich rendering engine. See [textual-tui.md](textual-tui.md).
## AI agents — pydantic-ai
The agent framework from the Pydantic team. Type-strict, structured outputs are first-class, model-agnostic. See [pydantic-ai.md](pydantic-ai.md).
## DataFrames — polars + numpy
Polars is 10-50x faster than pandas, has a real type system, and supports lazy evaluation. Numpy stays in the toolbox for arrays. See [data-processing.md](data-processing.md).
## OLAP / SQL — duckdb
DuckDB is the SQL engine for analytical workloads. Query CSV/Parquet/JSON files directly without loading into memory; perform joins and aggregations 3-4x faster than Polars; zero-copy interchange with Polars via Arrow. See [data-processing.md](data-processing.md).
## Tests — pytest
Plain `unittest` is fine for stdlib; everything else uses pytest. Conventions:
- File names `test_*.py`, function names `test_*`.
- Fixtures via `@pytest.fixture`. Async fixtures are anyio-aware (`@pytest.fixture` on an async function works under `pytest-anyio` which is bundled with anyio).
- Parametrise with `@pytest.mark.parametrize`.
- Mark async tests with `@pytest.mark.anyio` (provided by anyio's pytest plugin).
```python
import pytest
import anyio
@pytest.fixture
def sample_user() -> dict[str, str]:
return {"name": "Alice", "email": "a@b.com"}
@pytest.mark.parametrize("count,expected", [(1, "Hello"), (2, "Hello, Hello")])
def test_greet(count: int, expected: str) -> None:
result = ", ".join(["Hello"] * count)
assert result == expected
@pytest.mark.anyio
async def test_async_fetch() -> None:
await anyio.sleep(0)
assert True
```
`pyproject.toml`:
```toml
[tool.pytest.ini_options]
minversion = "8.0"
testpaths = ["tests"]
addopts = ["-ra", "--strict-config", "--strict-markers"]
```
## Settings / config — pydantic-settings
Loads env vars and `.env` files into a Pydantic model. Replaces ad-hoc `os.environ.get(...)` everywhere.
```python
from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPP_")
database_url: str
api_key: str = Field(min_length=1)
debug: bool = False
settings = Settings() # loads at import time; raises if any required var is missing
```
## Logging — stdlib logging + rich handler
Stdlib `logging` is fine; it gets a face-lift from `rich.logging.RichHandler`.
```python
import logging
from rich.logging import RichHandler
logging.basicConfig(
level=logging.INFO,
format="%(message)s",
datefmt="[%X]",
handlers=[RichHandler(rich_tracebacks=True, show_path=False)],
)
log = logging.getLogger(__name__)
log.info("ready")
```
For structured logging in production, swap to `structlog` (separate dep). Don't roll your own.
packages/shared-skills/skills/programming/references/python/one-liners.md
+268
View File
@@ -0,0 +1,268 @@
# One-liner Scripts (PEP 723 + uv)
Self-contained Python scripts with declared dependencies, run with no environment setup. The combination eliminates the historical reason to write small tools in Go or Bash.
**Rule: EVERY `.py` script — even throwaway — MUST use PEP 723 inline metadata with the usage comment block.** No venv, no requirements.txt, no setup.py. The script IS the environment spec.
## The two patterns
### Pattern 1: inline `uv run` invocation
```bash
uv run --with httpx2 --with rich python -c "
import httpx2
from rich import print
print(httpx2.get('https://api.github.com').json())
"
```
Use for terminal one-shots that you don't want to save. `--with PKG` may be repeated.
### Pattern 2: PEP 723 script with shebang (THE CANONICAL PATTERN)
A regular `.py` file with metadata in a comment block. uv reads the metadata, materialises a disposable venv (cached), and runs the script.
```python
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.13"
# dependencies = [
# "httpx2[http2,brotli,zstd]",
# "rich",
# ]
# ///
# ─── How to run ───
# 1. Install uv (if not installed):
# curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run directly (no venv, no pip install needed):
# uv run my_script.py
# 3. Or make executable and run:
# chmod +x my_script.py && ./my_script.py
# ──────────────────
from __future__ import annotations
import httpx2
from rich import print as rprint
def main() -> None:
with httpx2.Client(http2=True, follow_redirects=True) as client:
resp = client.get("https://api.github.com")
resp.raise_for_status()
rprint(resp.json())
if __name__ == "__main__":
main()
```
### Mandatory elements
Every PEP 723 script MUST include these, in order:
1. **Shebang**: `#!/usr/bin/env -S uv run --script`
2. **PEP 723 metadata block**: `# /// script` ... `# ///` with `requires-python` and `dependencies`
3. **Usage comment block**: How to install uv + how to run the script. Copy the template above verbatim.
4. **`from __future__ import annotations`**: Always first import.
5. **`if __name__ == "__main__": main()`**: Entry point guard.
### The usage comment block (NON-NEGOTIABLE)
```python
# ─── How to run ───
# 1. Install uv (if not installed):
# curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run directly (no venv, no pip install needed):
# uv run <SCRIPT_NAME>.py [ARGS]
# 3. Or make executable and run:
# chmod +x <SCRIPT_NAME>.py && ./<SCRIPT_NAME>.py
# ──────────────────
```
Replace `<SCRIPT_NAME>` with the actual filename. Add argument descriptions if the script takes CLI args. This block goes immediately after the `# ///` closing line, before any imports.
**Why mandatory**: Anyone who receives this script — colleague, CI, future you — must know how to run it without reading docs. The comment IS the docs.
## Template generator
Use `scripts/new-script.py` to scaffold a new PEP 723 script with all boilerplate pre-filled:
```bash
# Generate to temp directory (default)
uv run scripts/new-script.py my_tool
# Generate to specific path
uv run scripts/new-script.py my_tool --output ./scripts/my_tool.py
# With extra dependencies
uv run scripts/new-script.py my_tool --deps "polars" "duckdb" "rich"
```
## Common dependency sets
| Use case | Dependencies line |
|---|---|
| API client | `"httpx2[http2,brotli,zstd]"` |
| Data processing | `"polars"`, `"duckdb"` |
| CLI tool | `"typer"`, `"rich"` |
| Web scraping | `"httpx2[http2,brotli,zstd]"`, `"selectolax"` |
| File watcher | `"watchfiles"` |
| JSON pretty | `"rich"` |
| AI / LLM | `"pydantic-ai"`, `"httpx2[http2,brotli,zstd]"` |
## Real-world examples
### Fetch + print JSON
```python
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.13"
# dependencies = [
# "httpx2[http2,brotli,zstd]",
# "rich",
# ]
# ///
# ─── How to run ───
# 1. Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run: uv run fetch_json.py https://api.github.com/repos/pydantic/httpx2
# ──────────────────
from __future__ import annotations
import sys
import httpx2
from rich import print as rprint
def main() -> None:
url = sys.argv[1] if len(sys.argv) > 1 else "https://api.github.com"
with httpx2.Client(http2=True, follow_redirects=True) as client:
resp = client.get(url)
resp.raise_for_status()
rprint(resp.json())
if __name__ == "__main__":
main()
```
### CSV → Parquet conversion
```python
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.13"
# dependencies = [
# "polars",
# "typer",
# "rich",
# ]
# ///
# ─── How to run ───
# 1. Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run: uv run csv2parquet.py input.csv output.parquet
# ──────────────────
from __future__ import annotations
from pathlib import Path
import polars as pl
import typer
from rich import print as rprint
def main(input_path: Path, output_path: Path | None = None) -> None:
"""Convert CSV to Parquet."""
out = output_path or input_path.with_suffix(".parquet")
df = pl.read_csv(input_path)
df.write_parquet(out)
rprint(f"[green]✓[/green] {input_path} → {out} ({len(df)} rows)")
if __name__ == "__main__":
typer.run(main)
```
### Quick benchmark
```python
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.13"
# dependencies = [
# "httpx2[http2,brotli,zstd]",
# "rich",
# "anyio",
# ]
# ///
# ─── How to run ───
# 1. Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run: uv run bench.py https://api.example.com/health 50
# ──────────────────
from __future__ import annotations
import socket
import sys
import time
import anyio
import httpx2
from rich import print as rprint
async def main() -> None:
url = sys.argv[1] if len(sys.argv) > 1 else "https://api.github.com"
n = int(sys.argv[2]) if len(sys.argv) > 2 else 20
limits = httpx2.Limits(max_connections=200, max_keepalive_connections=40, keepalive_expiry=30.0)
timeout = httpx2.Timeout(connect=5.0, read=30.0, write=10.0, pool=10.0)
transport = httpx2.AsyncHTTPTransport(
http2=True, retries=3, limits=limits,
socket_options=[(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)],
)
async with httpx2.AsyncClient(transport=transport, timeout=timeout, follow_redirects=True) as client:
# warmup
for _ in range(3):
await client.get(url)
start = time.perf_counter()
for _ in range(n):
r = await client.get(url)
assert r.status_code == 200
elapsed = time.perf_counter() - start
avg_ms = (elapsed / n) * 1000
rprint(f"[bold]{url}[/bold]: {avg_ms:.1f}ms avg over {n} requests ({elapsed:.2f}s total, {r.http_version})")
if __name__ == "__main__":
anyio.run(main)
```
## Anti-patterns
| ❌ Don't | ✅ Do |
|---|---|
| `pip install httpx2 && python script.py` | `uv run script.py` |
| `requirements.txt` alongside script | PEP 723 inline metadata |
| `python -m venv .venv && ...` | `uv run --script` handles it |
| Script without usage comment | Always include the "How to run" block |
| `import asyncio; asyncio.run(main())` | `import anyio; anyio.run(main)` |
| Bare `httpx2.AsyncClient()` | Full production defaults (see `references/httpx2-optimization.md`) |
## Sources
- PEP 723 - Inline script metadata: <https://peps.python.org/pep-0723/>
- uv `run --script` docs: <https://docs.astral.sh/uv/guides/scripts/>
- Original article: <https://www.cottongeeks.com/articles/2025-06-24-fun-with-uv-and-pep-723>
- Simon Willison on one-shot Python tools: <https://simonwillison.net/2024/Dec/19/one-shot-python-tools/>
packages/shared-skills/skills/programming/references/python/orjson-stack.md
+378
View File
@@ -0,0 +1,378 @@
# orjson — When to Use, How to Integrate
`orjson` is the fastest JSON library on PyPI — written in Rust, 6–11× faster than stdlib `json` on serialization, 1.5–4× faster on deserialization. It also supports types the stdlib refuses to serialize: `datetime`, `date`, `UUID`, `numpy` arrays, `dataclass`, Pydantic models (via a small bridge).
This document covers the production patterns. **Not every project needs orjson.** The decision tree is in §1.
---
## 1. Decision tree — should you adopt orjson?
```
Are you serializing/deserializing JSON in a hot path?
├─ NO → stdlib `json` is fine. Stop here.
└─ YES ↓
Is the project FastAPI?
├─ YES ↓
│
│ Is your response body fully described by a Pydantic v2 model?
│ ├─ YES → Use FastAPI's default JSON response (uses Pydantic's
│ │ Rust-backed serializer; orjson saves nothing in this path).
│ │ Adopt orjson only for *non-Pydantic* responses below.
│ └─ NO → Use `ORJSONResponse` for endpoints that return dicts,
│ lists, or arbitrary structures.
│
└─ NOT FastAPI ↓
Are you serializing Pydantic v2 models repeatedly?
├─ YES → Use `model.model_dump_json()` directly — backed by pydantic-core
│ (Rust), within ~10% of orjson on the same payload, and respects
│ every Pydantic feature (computed fields, aliases, validators).
└─ NO ↓
Are you serializing dicts / lists / dataclasses / datetime / UUID?
├─ YES → orjson is the right answer.
└─ NO → stdlib `json`.
```
**The crucial 2026 fact**: with Pydantic v2's `model_dump_json()`, **Pydantic-shaped responses no longer need orjson**. Adopt orjson where you are still going through `dict` / `list` / `dataclass`.
---
## 2. Install
```toml
# pyproject.toml
dependencies = [
"orjson>=3.10",
]
```
orjson wheels are published for every major CPython version and platform (macOS, Linux glibc/musl, Windows, ARM64). No compilation step on install.
---
## 3. Basic usage
```python
import orjson
# Serialization — returns bytes, not str
raw: bytes = orjson.dumps({"hello": "world", "ts": datetime.now(UTC)})
# Deserialization
data = orjson.loads(raw)
```
Two things to internalize:
1. **`orjson.dumps` returns `bytes`**, not `str`. Stdlib `json.dumps` returns `str`. This is by design — most JSON destinations (sockets, files in binary mode, HTTP bodies) want bytes anyway, and skipping the encode/decode round trip is part of the speedup.
2. **No `indent` arg.** orjson supports `OPT_INDENT_2` (and only 2-space indent) via flags. If you need other indentation, use stdlib `json`.
---
## 4. The option flags you actually use
```python
import orjson
orjson.dumps(
payload,
option=(
orjson.OPT_NAIVE_UTC # treat naive datetimes as UTC (recommended)
| orjson.OPT_UTC_Z # render UTC as "...Z" instead of "+00:00"
| orjson.OPT_SERIALIZE_NUMPY # serialize numpy arrays natively
| orjson.OPT_SERIALIZE_DATACLASS # serialize @dataclass instances
| orjson.OPT_NON_STR_KEYS # allow int / UUID / datetime dict keys
# | orjson.OPT_SORT_KEYS # only when you need deterministic output
# | orjson.OPT_INDENT_2 # only for human-readable output (slower)
),
)
```
Each flag is opt-in for a reason — orjson defaults to spec-strict JSON.
The flag combination above is a sensible "production default" for application code. The `OPT_NAIVE_UTC | OPT_UTC_Z` pair is especially important: it produces RFC 3339 timestamps that every parser on earth accepts.
---
## 5. orjson + FastAPI
### 5.1 The legacy pattern: `ORJSONResponse`
```python
from fastapi import FastAPI
from fastapi.responses import ORJSONResponse
app = FastAPI(default_response_class=ORJSONResponse)
@app.get("/items")
async def get_items() -> dict[str, list[dict[str, int]]]:
return {"items": [{"id": i, "qty": i * 2} for i in range(1000)]}
```
`default_response_class=ORJSONResponse` swaps the global JSON encoder for orjson. **This affects only the response body serialization**, not request parsing — for request parsing, FastAPI still uses Pydantic.
### 5.2 The 2026 reality — Pydantic v2 vs orjson
With FastAPI 0.100+ on Pydantic v2:
- If your response is annotated with a Pydantic model, FastAPI calls `model_dump_json()` directly. **orjson is bypassed** even with `default_response_class=ORJSONResponse`, because the Pydantic serializer is already Rust-backed.
- If your response is a raw `dict` / `list` / Python object, `ORJSONResponse` does kick in and saves real time.
The benchmark in `tiangolo/fastapi#11728` (Apr 2024) showed `model_dump_json()` is ~10–15% faster than `ORJSONResponse + model_dump()` for Pydantic-shaped responses. The shape of the data matters; on mixed-shape APIs, keep `ORJSONResponse` as the default and trust Pydantic's path for typed responses.
### 5.3 Recommended setup
```python
from fastapi import FastAPI
from fastapi.responses import ORJSONResponse
app = FastAPI(
default_response_class=ORJSONResponse, # benefits dict/list returns
# Pydantic-typed returns automatically use pydantic-core serialization
)
```
**Do NOT** wrap Pydantic models manually:
```python
# BAD — defeats Pydantic's optimized path
@app.get("/users/{id}", response_class=ORJSONResponse)
async def get_user(id: int) -> ORJSONResponse:
user = await fetch_user(id)
return ORJSONResponse(content=user.model_dump()) # extra dict trip
# GOOD — let FastAPI serialize the model
@app.get("/users/{id}")
async def get_user(id: int) -> User:
return await fetch_user(id)
```
### 5.4 Streaming responses
`ORJSONResponse` does not stream — it buffers the whole response. For SSE, NDJSON, or chunked JSON, use `StreamingResponse` and call `orjson.dumps` per chunk:
```python
from fastapi.responses import StreamingResponse
import orjson
async def ndjson_stream():
async for row in fetch_rows():
yield orjson.dumps(row) + b"\n"
@app.get("/export")
async def export():
return StreamingResponse(ndjson_stream(), media_type="application/x-ndjson")
```
This is where orjson shines — per-chunk serialization in a tight loop, zero buffering.
---
## 6. orjson + Pydantic v2 (no FastAPI)
When you have a Pydantic model and want orjson's output for non-FastAPI contexts:
```python
from pydantic import BaseModel
import orjson
class User(BaseModel):
id: int
email: str
created: datetime
user = User(id=1, email="a@b.com", created=datetime.now(UTC))
# Option A — Pydantic's built-in Rust serializer (USE THIS by default)
raw: bytes = user.model_dump_json().encode()
# 2026: ~1.2× faster than orjson on the same payload, supports
# every Pydantic feature (aliases, computed fields, json_schema_extra, etc.)
# Option B — orjson bridge for cases Pydantic does not cover
raw: bytes = orjson.dumps(
user,
default=lambda obj: obj.model_dump() if isinstance(obj, BaseModel) else None,
)
# Useful when serializing nested non-Pydantic structures that contain
# BaseModels — e.g. a list of dicts that each may contain a BaseModel.
```
For routine "serialize one Pydantic model to JSON", `model_dump_json()` wins on speed AND feature parity. Reach for orjson only at the *container* level (a dict of mixed types).
### Custom `default=` callback — the universal extension point
```python
import orjson
from decimal import Decimal
from pydantic import BaseModel
def _default(obj):
if isinstance(obj, BaseModel):
return obj.model_dump()
if isinstance(obj, Decimal):
return str(obj)
if isinstance(obj, set):
return list(obj)
raise TypeError(f"orjson: cannot serialize {type(obj).__name__}")
orjson.dumps(payload, default=_default, option=orjson.OPT_NAIVE_UTC | orjson.OPT_UTC_Z)
```
The `default=` callback runs once per unrecognized type, then orjson caches the path. Performance impact on subsequent calls is negligible.
---
## 7. Caching, queues, logging — the prime orjson use cases
These are where orjson pays off most clearly because there is no Pydantic in the loop:
### Redis cache
```python
import orjson
import redis.asyncio as redis
r = redis.from_url("redis://localhost")
async def set_cache(key: str, value: dict) -> None:
await r.set(key, orjson.dumps(value), ex=3600)
async def get_cache(key: str) -> dict | None:
raw = await r.get(key)
return orjson.loads(raw) if raw else None
```
`orjson` over stdlib `json` here saves ~5–10× on the serialize step for typical cache payloads. Multiply by request rate.
### Task queue payloads (Celery, RQ, dramatiq)
```python
# Celery custom serializer
from kombu.serialization import register
import orjson
def _orjson_dumps(obj):
return orjson.dumps(obj, option=orjson.OPT_NAIVE_UTC | orjson.OPT_UTC_Z).decode()
def _orjson_loads(s):
return orjson.loads(s)
register("orjson", _orjson_dumps, _orjson_loads,
content_type="application/x-orjson",
content_encoding="utf-8")
```
Same speedup, applied to every task payload encode/decode.
### Structured logging (structlog, custom slog)
```python
import structlog
import orjson
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.add_log_level,
structlog.processors.JSONRenderer(serializer=orjson.dumps),
],
)
```
structlog's `JSONRenderer` accepts any callable; orjson is the obvious default. Logging hot paths benefit dramatically — every log line at info level becomes ~5× cheaper to render.
---
## 8. Gotchas
### `orjson.dumps` returns bytes, not str
```python
# BAD — concatenating bytes and str
log.info("payload: " + orjson.dumps(data)) # TypeError
# GOOD
log.info("payload: %s", orjson.dumps(data).decode())
# or
log.info("payload: %s", orjson.dumps(data)) # let the formatter handle it
```
### No `cls=` argument for custom encoders
orjson uses `default=` only. If you have a custom `JSONEncoder` subclass from stdlib `json`, port its `default()` method to a `default=` callable.
### Subclasses of `dict` / `list` are NOT serialized as their parent
```python
class StrictDict(dict): ...
d = StrictDict({"k": "v"})
import json
json.dumps(d) # OK — stdlib walks subclasses
orjson.dumps(d) # TypeError — orjson is strict by design
orjson.dumps(d, option=orjson.OPT_PASSTHROUGH_SUBCLASS) # then route via default=
```
Set `OPT_PASSTHROUGH_SUBCLASS` and handle the subclass in `default=`. The design discourages accidental subclass usage that breaks elsewhere.
### `int` overflow
orjson refuses to encode integers larger than 2⁵³ - 1 by default (the IEEE-754 double-precision safe-integer limit — what JavaScript can round-trip). For larger ints, opt in:
```python
orjson.dumps(huge_int, option=orjson.OPT_STRICT_INTEGER) # error
orjson.dumps(huge_int) # default — int is encoded as JSON number
# JavaScript clients lose precision past 2^53; consider sending as string
```
This is more spec-strict than stdlib `json`, which silently emits ints of any size.
### Timezone-naive datetimes
By default, orjson treats naive `datetime` as the system local timezone — almost never what you want. **Always set `OPT_NAIVE_UTC`** to treat naive datetimes as UTC, or use timezone-aware datetimes (which is the better long-term habit).
---
## 9. Benchmark — should I actually adopt this?
The numbers below are 2024–2026 averages from `tiangolo/fastapi#11728` and orjson's own benchmark suite, on Python 3.13, modern x86_64:
| Payload | stdlib `json` | `orjson` | `model_dump_json()` (Pydantic v2) |
|---|---|---|---|
| Small dict (100 fields) | 1.0× | **8×** | n/a |
| List of 10k dicts | 1.0× | **11×** | n/a |
| Pydantic model with 20 fields | 1.0× (after `model_dump()`) | 5× (with `default=` bridge) | **6×** |
| Datetime-heavy payload | 1.0× (after manual ISO conv) | **9×** | 6× |
| numpy array (1M floats) | impossible without manual conv | **20×** vs json+tolist | n/a |
The takeaways:
- For raw dict/list/datetime, **orjson is dramatically faster**.
- For Pydantic models, **`model_dump_json()` is already faster than orjson+bridge**.
- For numpy, orjson is the only sane choice.
In production, the actual measured win on a FastAPI app with mixed payloads is typically 5–15% reduction in p99 latency. Worth the one-line `default_response_class=ORJSONResponse` switch.
---
## 10. When NOT to adopt orjson
- The codebase is small, JSON is not a bottleneck, and you have no measured perf concern.
- You depend on stdlib `json`'s `cls=` arg or its lax tolerance for non-spec input (NaN, Infinity, comments).
- You need pretty-printed JSON with custom indent — orjson only supports 2-space indent via the flag.
- You need pure-Python portability (e.g., MicroPython, no-wheel platforms) — orjson is a compiled Rust extension.
If the choice is "add a dependency that does 5–10× the speed on serialization for free", the answer is almost always yes. The "almost" is in the bullets above.
---
## Sources
- orjson: https://github.com/ijl/orjson
- Pydantic v2 `model_dump_json`: https://docs.pydantic.dev/latest/concepts/serialization/#modelmodel_dump_json
- FastAPI `ORJSONResponse`: https://fastapi.tiangolo.com/advanced/custom-response/#use-orjsonresponse
- "FastAPI + orjson vs Pydantic v2" benchmark: https://github.com/fastapi/fastapi/discussions/11728
- structlog JSON rendering: https://www.structlog.org/en/stable/api.html#structlog.processors.JSONRenderer
packages/shared-skills/skills/programming/references/python/pydantic-ai.md
+285
View File
@@ -0,0 +1,285 @@
# PydanticAI Reference (v1.x, 2026)
> Canonical patterns for wiring PydanticAI agents. Target: production usage, late-2025 / 2026.
> Source: [ai.pydantic.dev](https://ai.pydantic.dev) and [pydantic/pydantic-ai@`cad9569`](https://github.com/pydantic/pydantic-ai/blob/cad956910079737ea0886b50cef15777208f92e6).
---
## 1. Agent Constructor
```python
from pydantic_ai import Agent
agent = Agent(
'openai:gpt-5.2', # model (str | Model | None)
output_type=MyOutputModel, # structured output type; default=str
instructions='You are a...', # static or callable instructions
system_prompt='Be concise.', # static system prompt(s)
deps_type=MyDeps, # dependency type for type-checking only
name='my-agent', # optional, inferred from var name if omitted
retries=1, # default retries for tools + output validation
output_retries=None, # override retries for output validation only
tools=[my_tool], # list of Tool objects or plain functions
defer_model_check=False, # set True to skip env-var check at init time
end_strategy='early', # 'early' | 'graceful' | 'exhaustive'
)
```
**Breaking change (v1.88.0)**: `result_type` was renamed to `output_type`. Use `output_type`.
---
## 2. Model Strings
Format: `provider:model-name`. The framework infers the provider from the prefix.
| Provider prefix | Example |
|---|---|
| `openai:` | `'openai:gpt-5.2'`, `'openai:gpt-4o'` |
| `anthropic:` | `'anthropic:claude-sonnet-4-6'`, `'anthropic:claude-opus-4-1'` |
| `google-gla:` | `'google-gla:gemini-3-flash-preview'` |
| `google-vertex:` | `'google-vertex:gemini-3-pro-preview'` |
| `bedrock:` | `'bedrock:anthropic.claude-sonnet-4-6'` |
| `xai:` / `grok:` | `'xai:grok-3'`, `'grok:grok-3-fast'` |
| `deepseek:` | `'deepseek:deepseek-chat'` |
| `cohere:` | `'cohere:command-r-08-2024'` |
| `gateway/...` | `'gateway/openai:gpt-5.2'` (PydanticAI Gateway) |
Model can also be omitted at construction and passed per-run: `agent.run(prompt, model='openai:gpt-5.2')`.
---
## 3. Tools
### Decorator syntax
```python
from pydantic_ai import Agent, RunContext
agent = Agent('openai:gpt-5.2', deps_type=str)
@agent.tool # default: receives RunContext as first arg
async def greet(ctx: RunContext[str], name: str) -> str:
return f"Hello {ctx.deps}, {name}!"
@agent.tool_plain # no context needed
async def roll_dice(sides: int) -> int:
import random
return random.randint(1, sides)
```
### `RunContext[Deps]`
First parameter of `@agent.tool` functions. Carries:
- `ctx.deps` — the dependency instance
- `ctx.model` — the model being used
- `ctx.usage` — token usage so far
- `ctx.messages` — conversation history
- `ctx.retry` / `ctx.max_retries` — current retry count
- `ctx.agent` — the running agent instance
Use `@agent.tool_plain` when the tool does **not** need any of the above.
---
## 4. Structured Output
Pass a Pydantic `BaseModel` (or `bool`, `int`, `list[str]`, etc.) as `output_type`. The result is accessed via `.output`.
```python
from pydantic import BaseModel
from pydantic_ai import Agent
class City(BaseModel):
name: str
country: str
population_millions: float
agent = Agent('openai:gpt-5.2', output_type=City)
result = agent.run_sync('Tell me about Tokyo')
print(result.output) # City(name='Tokyo', country='Japan', ...)
print(result.output.name) # 'Tokyo'
```
**Note**: `result.data` was renamed; the canonical accessor is `result.output`.
---
## 5. Async vs Sync
| Method | Mode | Returns |
|---|---|---|
| `await agent.run(prompt, ...)` | async | `AgentRunResult[OutputDataT]` |
| `agent.run_sync(prompt, ...)` | sync | `AgentRunResult[OutputDataT]` |
| `async with agent.run_stream(prompt, ...) as response:` | async streaming | `StreamedRunResult` |
```python
# Sync
result = agent.run_sync('What is the capital of Italy?')
print(result.output)
# Async
result = await agent.run('What is the capital of France?')
print(result.output)
# Streaming
async with agent.run_stream('What is the capital of the UK?') as response:
async for text in response.stream_text():
print(text, end='')
# After streaming finishes:
print(response.output)
```
`run_sync()` is a convenience wrapper over `loop.run_until_complete(self.run(...))`. Do not use it inside an active async context.
---
## 6. Dependencies
Use a `@dataclass` container, pass the **type** to `deps_type`, and pass an **instance** to `deps` at run time.
```python
from dataclasses import dataclass
import httpx
from pydantic_ai import Agent, RunContext
@dataclass
class Deps:
api_key: str
http_client: httpx.AsyncClient
agent = Agent(
'openai:gpt-5.2',
deps_type=Deps,
)
@agent.tool
async def fetch_data(ctx: RunContext[Deps], endpoint: str) -> str:
r = await ctx.deps.http_client.get(
endpoint,
headers={'Authorization': f'Bearer {ctx.deps.api_key}'},
)
r.raise_for_status()
return r.text
async def main():
async with httpx.AsyncClient() as client:
deps = Deps(api_key='sk-...', http_client=client)
result = await agent.run('Get /users', deps=deps)
print(result.output)
```
---
## 7. Error Types & Retrying from a Tool
```python
from pydantic_ai import Agent, ModelRetry, UnexpectedModelBehavior, capture_run_messages
agent = Agent('openai:gpt-5.2', retries=3)
@agent.tool_plain
def calc_volume(size: int) -> int:
if size == 42:
return size ** 3
raise ModelRetry('Please try again with size 42.')
with capture_run_messages() as messages:
try:
result = agent.run_sync('Get the volume of a box with size 6.')
except UnexpectedModelBehavior as e:
print('Error:', e) # "Tool 'calc_volume' exceeded max retries count of 3"
print('Cause:', e.__cause__) # ModelRetry('Please try again...')
print('Messages:', messages)
```
- **`ModelRetry`** — raise from a tool, output validator, or capability hook to ask the model to retry.
- **`UnexpectedModelBehavior`** — raised when the retry limit is exceeded or the model API returns an unrecoverable error.
- **`capture_run_messages()`** — context manager that records all messages exchanged during a run for debugging.
---
## 8. Logfire Integration
One-line setup if the `logfire` extra is installed (included in the default `pydantic-ai` package):
```python
import logfire
logfire.configure() # reads token from .logfire directory
logfire.instrument_pydantic_ai() # auto-traces all agent runs
```
Alternatively, set `instrument=True` on the agent:
```python
agent = Agent('openai:gpt-5.2', instrument=True)
```
---
## 9. Minimal Complete Snippets
### (a) Basic agent with structured output
```python
from pydantic import BaseModel
from pydantic_ai import Agent
class City(BaseModel):
name: str
country: str
agent = Agent('openai:gpt-5.2', output_type=City)
result = agent.run_sync('Tell me about Paris')
print(result.output) # City(name='Paris', country='France')
```
### (b) Agent with tools and dependencies
```python
from dataclasses import dataclass
from pydantic_ai import Agent, RunContext
@dataclass
class Deps:
api_key: str
agent = Agent('openai:gpt-5.2', deps_type=Deps)
@agent.tool
async def get_secret(ctx: RunContext[Deps], code: str) -> str:
if code == '1234':
return f'secret-for-{ctx.deps.api_key}'
return 'wrong code'
result = agent.run_sync('My code is 1234', deps=Deps(api_key='sk-abc'))
print(result.output)
```
### (c) Async streaming
```python
import anyio
from pydantic_ai import Agent
agent = Agent('openai:gpt-5.2')
async def main() -> None:
async with agent.run_stream('Write a haiku about Python') as response:
async for text in response.stream_text():
print(text, end='')
print('\n---')
print('Final:', response.output)
anyio.run(main)
```
---
## Version Notes
- **V1** reached API stability in September 2025. Breaking changes are reserved for V2 (earliest April 2026).
- **v1.88.0** renamed `result_type` → `output_type` and `result_tool_name` / `result_tool_description` were removed. Use `output_type`.
- The canonical accessor for run results is `result.output` (not `result.data`).
packages/shared-skills/skills/programming/references/python/pyproject-strict.md
+232
View File
@@ -0,0 +1,232 @@
# Strict pyproject.toml (basedpyright + ruff + uv)
The canonical "super strict but sane" config for modern Python projects. Copy-paste, then add your own dependencies.
## Bootstrap
```bash
# Application
uv init --app myproject
cd myproject
# Library (publishable to PyPI)
uv init --lib mylibrary
cd mylibrary
# Add dev tools
uv add --dev basedpyright ruff pytest
```
`uv init` creates `pyproject.toml`, `.python-version`, and `src/` layout. Replace its `pyproject.toml` `[tool.*]` sections with the block below.
## The full pyproject.toml
```toml
[project]
name = "myproject"
version = "0.1.0"
description = "..."
readme = "README.md"
requires-python = ">=3.13"
dependencies = []
[dependency-groups]
dev = [
"basedpyright>=1.21",
"ruff>=0.8",
"pytest>=8",
"pytest-cov>=5",
]
# ─────────────────────────────────────────────────────────────────
# basedpyright - typeCheckingMode = "all" sets every report flag to error
# Source: https://docs.basedpyright.com/latest/configuration/config-files/
# ─────────────────────────────────────────────────────────────────
[tool.basedpyright]
typeCheckingMode = "all"
pythonVersion = "3.13"
pythonPlatform = "All" # default in basedpyright; explicit for clarity
include = ["src", "tests"]
exclude = ["**/__pycache__", "**/.venv", "**/build", "**/dist"]
# Strict enforcement extras (most are already "error" under "all" mode,
# but listing them explicitly documents the intent)
reportUnusedCallResult = "warning" # flag ignored return values
reportUnnecessaryTypeIgnoreComment = "error" # stale type: ignore comments must die
reportUnusedVariable = "error" # unused variables are errors
reportMissingParameterType = "error" # every parameter must have a type
reportMissingReturnType = "error" # every function must declare its return type
reportPrivateUsage = "error" # respect _private convention
# Optional: gradual adoption baseline
# baselineFile = "./.basedpyright/baseline.json"
# ─────────────────────────────────────────────────────────────────
# ruff - select = ["ALL"] enables every rule, then we ignore the
# small set that conflicts with the formatter or is not useful.
# Source: https://docs.astral.sh/ruff/linter/#rule-selection
# ─────────────────────────────────────────────────────────────────
[tool.ruff]
target-version = "py313"
line-length = 88 # ruff/black default; 100 or 120 also fine
src = ["src", "tests"]
[tool.ruff.lint]
select = ["ALL"]
ignore = [
# Formatter conflicts (ruff itself tells you to ignore these)
"COM812", # missing trailing comma
"ISC001", # implicit string concat
# Docstyle conflicts (pick D211 over D203, D212 over D213)
"D203",
"D213",
# Project-specific noise
"CPY001", # missing copyright notice
"FBT001", # boolean positional arg in def
"FBT002", # boolean positional default in def
"TD002", # missing TODO author
"TD003", # missing TODO link
"FIX002", # line contains TODO (TODOs are allowed)
]
fixable = ["ALL"]
unfixable = []
[tool.ruff.lint.per-file-ignores]
"tests/**/*.py" = [
"S101", # `assert` is the entire point of pytest
"ARG", # unused args (fixtures appear unused)
"PLR2004", # magic numbers in test data
"SLF001", # tests need access to private members
"D", # docstrings not required in tests
]
"scripts/**/*.py" = [
"T201", # `print` allowed in scripts
"INP001", # implicit namespace package
]
[tool.ruff.lint.pydocstyle]
convention = "google" # or "numpy" / "pep257"
[tool.ruff.lint.flake8-bugbear]
# typer / fastapi rely on call-as-default for parameter metadata.
# Without this, ruff B008 ("function call in default") fires on every typer/fastapi route.
extend-immutable-calls = [
"typer.Argument",
"typer.Option",
"fastapi.Depends",
"fastapi.Query",
"fastapi.Path",
"fastapi.Body",
"fastapi.Header",
"fastapi.Cookie",
"fastapi.File",
"fastapi.Form",
]
[tool.ruff.format]
quote-style = "double"
indent-style = "space"
docstring-code-format = true
docstring-code-line-length = "dynamic"
# ─────────────────────────────────────────────────────────────────
# pytest
# ─────────────────────────────────────────────────────────────────
[tool.pytest.ini_options]
minversion = "8.0"
testpaths = ["tests"]
addopts = [
"-ra",
"--strict-config",
"--strict-markers",
]
filterwarnings = ["error"]
# ─────────────────────────────────────────────────────────────────
# coverage
# ─────────────────────────────────────────────────────────────────
[tool.coverage.run]
source = ["src"]
branch = true
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"if TYPE_CHECKING:",
"if typing.TYPE_CHECKING:",
"raise NotImplementedError",
"@(abc\\.)?abstractmethod",
]
```
## Why these settings
### basedpyright `typeCheckingMode = "all"`
basedpyright's modes, strictest first:
| Mode | Behavior |
|---|---|
| `"all"` | Every diagnostic at `error` |
| `"recommended"` | Same rules; less severe ones at `warning`; `failOnWarnings = true` makes CI still fail |
| `"strict"` | pyright's strict mode |
| `"standard"` | Default |
| `"basic"` / `"off"` | Loose / disabled |
`"all"` enables basedpyright-exclusive rules pyright lacks: `reportImplicitOverride`, `reportImplicitStringConcatenation`, `reportIncompatibleUnannotatedOverride`, `reportUnannotatedClassAttribute`. No need to opt-in to additional flags.
`pythonPlatform = "All"` is basedpyright's default (better than pyright's host-OS default) - it errors on platform-specific imports that fail on other OSes.
### ruff `select = ["ALL"]`
The official docs say *"Use ALL with discretion. Enabling ALL will implicitly enable new rules whenever you upgrade."* For a strict skill that is the intended behavior - every new ruff rule should be considered an error until you justify ignoring it.
The minimal ignore set:
| Rule | Reason |
|---|---|
| `COM812`, `ISC001` | Conflict with `ruff format` (ruff itself documents this) |
| `D203` vs `D211`, `D213` vs `D212` | Mutually-exclusive docstring conventions; pick the modern one |
| `CPY001` | Most projects don't need a copyright header on every file |
| `FBT001`, `FBT002` | Boolean flags are ergonomic for CLI/typer; ban makes typer awkward |
| `TD002`, `TD003`, `FIX002` | TODOs without a JIRA link are fine in solo / internal code |
`ANN101` and `ANN102` were **removed in ruff 0.8.0** (Nov 2024). Do NOT include them in `ignore` - ruff errors on unknown rule codes.
`per-file-ignores` for `tests/**` is the standard pattern from real-world repos like `community-of-python/auto-typing-final` and `Preston-Landers/concurrent-log-handler`.
## CI gate
```bash
# In CI, fail on any violation:
uv run basedpyright
uv run ruff check
uv run ruff format --check
uv run pytest
```
A single `make ci` target combining the four works fine.
## Enforcement summary
The config above, combined with `scripts/check-no-excuse-rules.py`, enforces:
| What | How |
|---|---|
| Exhaustive match | basedpyright `all` mode + `assert_never` |
| No `Any` | basedpyright `all` mode + script `cast-any` rule |
| Ignored return values | `reportUnusedCallResult = "warning"` |
| Immutable default | Script `mutable-dataclass` + `missing-slots` rules |
| No null surprise | basedpyright strict `None` analysis |
| Constants are const | basedpyright catches `Final` reassignment |
| Unused variables | `reportUnusedVariable = "error"` |
## Sources
- basedpyright modes: <https://docs.basedpyright.com/latest/configuration/config-files/#type-check-diagnostics-settings>
- basedpyright `"all"` vs `"recommended"`: <https://docs.basedpyright.com/latest/configuration/config-files/#recommended-and-all>
- basedpyright better defaults: <https://docs.basedpyright.com/latest/benefits-over-pyright/better-defaults/>
- ruff rule selection: <https://docs.astral.sh/ruff/linter/#rule-selection>
- ruff ANN101/ANN102 removed: <https://github.com/astral-sh/ruff/pull/14384>
- Real-world ALL config: <https://github.com/community-of-python/auto-typing-final/blob/main/pyproject.toml>
- PEP 735 dependency-groups: <https://peps.python.org/pep-0735/>
packages/shared-skills/skills/programming/references/python/textual-tui.md
+201
View File
@@ -0,0 +1,201 @@
# Textual TUI
Textual builds rich, mouse-aware, scrollable, mobile-style TUIs on top of `rich`. Replaces curses, urwid, blessed.
## Install
```bash
uv add textual
uv add --dev textual-dev # textual console + run --dev for hot reload
```
## Minimal app
```python
from textual.app import App, ComposeResult
from textual.widgets import Header, Footer, Button, Label
from textual.containers import Vertical
class CounterApp(App[None]):
"""A trivial counter app."""
BINDINGS = [("q", "quit", "Quit")]
CSS = """
#count {
height: 3;
content-align: center middle;
background: $boost;
}
"""
count: int = 0
def compose(self) -> ComposeResult:
yield Header()
with Vertical():
yield Label("0", id="count")
yield Button("Increment", id="inc", variant="primary")
yield Button("Reset", id="reset", variant="warning")
yield Footer()
def on_button_pressed(self, event: Button.Pressed) -> None:
if event.button.id == "inc":
self.count += 1
elif event.button.id == "reset":
self.count = 0
self.query_one("#count", Label).update(str(self.count))
if __name__ == "__main__":
CounterApp().run()
```
Run:
```bash
uv run python counter.py
```
For hot reload during development:
```bash
uv run textual run --dev counter.py
```
## Reactive attributes
Textual's `reactive()` descriptor turns a class attribute into something that watches assignments and re-renders automatically. Replaces the manual `query_one` + `update` dance.
```python
from textual.app import App, ComposeResult
from textual.reactive import reactive
from textual.widgets import Label
class CountWidget(Label):
count: reactive[int] = reactive(0)
def render(self) -> str:
return f"Count: {self.count}"
class CounterApp(App[None]):
def compose(self) -> ComposeResult:
yield CountWidget()
def on_key(self, event) -> None:
if event.key == "space":
self.query_one(CountWidget).count += 1
```
`reactive()` triggers `render()` (or `watch_<attr>` and `validate_<attr>` callbacks if defined). Use `recompose=True` if you need to call `compose()` again on change.
## Async work — workers
NEVER block the event loop. For network/disk/CPU work, use `@work` (creates a worker) or `run_worker`.
```python
import httpx
from textual.app import App, ComposeResult
from textual.widgets import Input, Static
from textual.work import work
class FetchApp(App[None]):
def compose(self) -> ComposeResult:
yield Input(placeholder="URL", id="url")
yield Static(id="result")
@work(exclusive=True)
async def fetch(self, url: str) -> None:
async with httpx.AsyncClient(timeout=10.0) as client:
response = await client.get(url)
self.query_one("#result", Static).update(f"{response.status_code} - {len(response.text)} bytes")
def on_input_submitted(self, event: Input.Submitted) -> None:
self.fetch(event.value)
```
`exclusive=True` cancels the previous worker if the user submits a new URL before the first finishes. Workers integrate with Textual's lifecycle - they're cancelled when the app exits.
`@work` is asyncio-flavoured under the hood. That is fine - it does not violate the no-asyncio rule because you are calling Textual's API, not importing asyncio yourself. Inside the worker body, use `httpx.AsyncClient` and other anyio-friendly libraries.
## Action handlers
Bind keys to method calls via `BINDINGS` and `action_*` methods.
```python
class App(App):
BINDINGS = [
("ctrl+s", "save", "Save"),
("ctrl+r", "reload", "Reload"),
]
def action_save(self) -> None:
# Called on ctrl+s
...
def action_reload(self) -> None:
...
```
Bindings can also include the `priority=True` flag to fire before children get a chance.
## CSS
Textual's CSS supports selectors, variables (`$primary`, `$boost`), animations. Inline via `CSS = "..."` or external via `CSS_PATH = "app.tcss"`.
```css
Screen {
background: $surface;
color: $text;
layout: vertical;
}
#sidebar {
width: 30;
background: $boost;
}
Button.danger {
background: $error;
}
```
Reload with `r` in dev mode (`textual run --dev`).
## Testing
```python
import pytest
from myapp import CounterApp
@pytest.mark.anyio
async def test_counter_increments() -> None:
app = CounterApp()
async with app.run_test() as pilot:
await pilot.click("#inc")
await pilot.click("#inc")
assert app.count == 2
```
`pilot.click(selector)`, `pilot.press("q")`, `pilot.pause()` for waiting on the next frame.
## When NOT to use Textual
| Need | Use |
|---|---|
| One-off CLI with structured output | typer + rich |
| Progress bar in a script | rich.progress |
| Tabular display of query results | rich.table |
| Full-screen app with state, input, mouse | Textual |
A pretty CLI is not a TUI. Reach for Textual when the user expects to navigate a UI, not when you want colours.
## Sources
- Textual docs: <https://textual.textualize.io>
- Textual tutorial: <https://textual.textualize.io/tutorial/>
- API reference: <https://textual.textualize.io/api/>
packages/shared-skills/skills/programming/references/python/type-patterns.md
+176
View File
@@ -0,0 +1,176 @@
# Type Patterns
How to use Python's type system to catch bugs at check time, not runtime.
---
## NewType — distinct primitives
Same runtime type, different meaning. The type checker prevents mixing.
```python
from typing import NewType
UserId = NewType("UserId", int)
MovieId = NewType("MovieId", int)
Email = NewType("Email", str)
Seconds = NewType("Seconds", float)
Milliseconds = NewType("Milliseconds", float)
def get_user(user_id: UserId) -> User: ...
def get_movie(movie_id: MovieId) -> Movie: ...
def sleep(duration: Seconds) -> None: ...
uid = UserId(42)
mid = MovieId(42)
get_user(uid) # OK
get_user(mid) # type error: MovieId is not UserId
get_user(42) # type error: int is not UserId
sleep(Milliseconds(100.0)) # type error
```
**Use when**: IDs, indices, keys, units of measurement — any pair where swapping is a bug.
**Skip when**: ephemeral local math where branding adds noise with zero safety gain.
---
## Final — constants are const
Module-level constants declare their intent. Reassignment is a type error.
```python
from typing import Final
MAX_RETRIES: Final = 3
API_BASE_URL: Final = "https://api.example.com"
DEFAULT_TIMEOUT: Final = 30.0
MAX_RETRIES = 5 # type error: cannot assign to Final
```
If it changes at runtime, it's not a constant — make it a function parameter or config field.
---
## TypeAlias — name complex types
If a union or generic appears more than once, give it a name.
```python
# Python 3.12+
type JsonValue = str | int | float | bool | None | list["JsonValue"] | dict[str, "JsonValue"]
type Headers = dict[str, str]
type Middleware = Callable[[Request], Awaitable[Response]]
# Pre-3.12
from typing import TypeAlias
JsonValue: TypeAlias = str | int | float | bool | None | list["JsonValue"] | dict[str, "JsonValue"]
```
---
## StrEnum / IntEnum — closed sets
Any fixed set of known values. No string literals scattered through code.
```python
from enum import StrEnum, IntEnum, unique
@unique
class Role(StrEnum):
ADMIN = "admin"
USER = "user"
GUEST = "guest"
@unique
class HttpStatus(IntEnum):
OK = 200
NOT_FOUND = 404
INTERNAL_ERROR = 500
# BAD
def check_role(role: str) -> bool: ...
# GOOD
def check_role(role: Role) -> bool: ...
```
`StrEnum` when values serialize as strings (API, DB). `IntEnum` for numeric codes. Plain `Enum` for pure labels.
---
## Type narrowing — let the checker follow your logic
`isinstance`, `is None`, and `match` narrow types automatically. Use them instead of `cast`.
```python
def process(value: str | int | None) -> str:
if value is None:
return "nothing"
# checker knows: str | int
if isinstance(value, str):
return value.upper()
# checker knows: int
return str(value * 2)
```
### TypeGuard for custom narrowing
```python
from typing import TypeGuard
def is_valid_email(value: str) -> TypeGuard[Email]:
return "@" in value and "." in value.split("@")[1]
def send(addr: str) -> None:
if not is_valid_email(addr):
raise ValueError(addr)
# checker knows: addr is Email
deliver(addr)
```
### TypeIs (Python 3.13+) — the strict version
`TypeIs` is stricter than `TypeGuard` — it narrows in both `if` and `else` branches.
```python
from typing import TypeIs
def is_str(value: str | int) -> TypeIs[str]:
return isinstance(value, str)
def handle(v: str | int) -> None:
if is_str(v):
print(v.upper()) # checker knows: str
else:
print(v + 1) # checker knows: int
```
---
## Union syntax
Always `X | Y`. Never `Union[X, Y]` or `Optional[X]`.
```python
# BAD
from typing import Union, Optional
def f(x: Optional[int]) -> Union[str, int]: ...
# GOOD
def f(x: int | None) -> str | int: ...
```
---
## Sources
- Python docs: [typing — NewType](https://docs.python.org/3/library/typing.html#newtype)
- Python docs: [typing — Final](https://docs.python.org/3/library/typing.html#typing.Final)
- Python docs: [typing — TypeGuard](https://docs.python.org/3/library/typing.html#typing.TypeGuard)
- PEP 604: [Union syntax X | Y](https://peps.python.org/pep-0604/)
- PEP 742: [TypeIs](https://peps.python.org/pep-0742/)
packages/shared-skills/skills/programming/references/rust-ub/README.md
+289
View File
@@ -0,0 +1,289 @@
# Rust Undefined Behavior Exorcist
You are a UB hunter. Your job is to find, classify, prove, and eliminate every instance of undefined behavior in Rust code. **Miri is your primary weapon** — everything else supplements where Miri cannot reach.
## Core Philosophy
1. **Miri first, always.** Before reading a single line of `unsafe`, run Miri. Before proposing a fix, run Miri. After applying a fix, run Miri. Miri is the oracle.
2. **Classify before fixing.** Every UB finding gets classified against the 14-category taxonomy (see [ub-taxonomy.md](ub-taxonomy.md)). This prevents misdiagnosis and ensures the fix targets the root cause, not a symptom.
3. **Prove the fix.** A fix is not done until Miri passes with full paranoia flags. If Miri cannot run the test (FFI, I/O), the fix is not done until the appropriate sanitizer passes.
4. **Bead handoff.** Each resolved UB instance is a "bead" — a discrete, documented, proven fix. Hand it off with: the UB category, the root cause, the fix, and the Miri proof.
## The UB Taxonomy
14 categories. The full reference is in [ub-taxonomy.md](ub-taxonomy.md). Memorize the categories; classify every finding:
| # | Category | Miri? |
|---|----------|-------|
| 1 | Aliasing violations (Stacked/Tree Borrows) | YES |
| 2 | Data races | YES |
| 3 | Use-after-free / dangling pointers | YES |
| 4 | Uninitialized memory | YES |
| 5 | Invalid values (type invariant violations) | YES |
| 6 | Misaligned pointer access | YES |
| 7 | Pin invariant violations | PARTIAL |
| 8 | FFI boundary UB | LIMITED |
| 9 | Incorrect Send/Sync implementations | YES (via race) |
| 10 | Out-of-bounds memory access | YES |
| 11 | Provenance violations | YES (strict mode) |
| 12 | Double free / invalid free | YES |
| 13 | Library / unsafe contract violations | PARTIAL |
| 14 | Unwinding across extern "C" | PARTIAL |
## The Hunt Workflow
### Phase 1: Reconnaissance
1. **Find all `unsafe` blocks and `unsafe impl`s:**
```bash
rg 'unsafe\s*(fn|impl|{|\{)' --type rust -n
```
2. **Find all `unsafe` trait implementations:**
```bash
rg 'unsafe\s+impl\s+(Send|Sync)' --type rust -n
```
3. **Find transmute / pointer casts / raw pointer derefs:**
```bash
rg '(transmute|transmute_copy|from_raw|into_raw|as_ptr|as_mut_ptr|offset|add|sub|read|write|copy|ptr::null)' --type rust -n
```
4. **Find FFI boundaries:**
```bash
rg 'extern\s+"C"' --type rust -n
```
5. **Count and catalog.** Create a hit list: file, line, `unsafe` category, initial risk assessment (high/medium/low based on the UB taxonomy).
### Phase 2: Miri Sweep (THE CRITICAL PHASE)
Run Miri with escalating strictness. **Do not skip any level.**
**Level 1 — Default (Stacked Borrows):**
```bash
cargo +nightly miri test 2>&1
```
**Level 2 — Strict Provenance + Symbolic Alignment:**
```bash
MIRIFLAGS="-Zmiri-strict-provenance -Zmiri-symbolic-alignment-check -Zmiri-backtrace=full" \
cargo +nightly miri test 2>&1
```
**Level 3 — Full Paranoia (the audit standard):**
```bash
MIRIFLAGS="\
-Zmiri-strict-provenance \
-Zmiri-symbolic-alignment-check \
-Zmiri-preemption-rate=0.1 \
-Zmiri-backtrace=full \
-Zmiri-disable-isolation" \
cargo +nightly miri test 2>&1
```
**Level 4 — Tree Borrows (second model confirmation):**
```bash
MIRIFLAGS="\
-Zmiri-tree-borrows \
-Zmiri-strict-provenance \
-Zmiri-symbolic-alignment-check \
-Zmiri-preemption-rate=0.1 \
-Zmiri-backtrace=full \
-Zmiri-disable-isolation" \
cargo +nightly miri test 2>&1
```
**Interpret results:**
- Fails at Level 1 → Definite UB. Fix immediately.
- Passes Level 1, fails Level 2 → Provenance or alignment UB. Fix.
- Passes Levels 1-3, fails Level 4 → Tree Borrows found something Stacked Borrows missed (unusual). Investigate — may be a Tree Borrows false positive, but usually indicates fragile aliasing.
- Passes all 4 → Miri-clean. Proceed to supplementary tools.
### Phase 3: Supplementary Scans
For code Miri cannot fully cover:
**Concurrent code with custom atomics:**
```bash
RUSTFLAGS="--cfg loom" cargo test --lib --release -- loom_tests 2>&1
```
**FFI-heavy code:**
```bash
RUSTFLAGS="-Zsanitizer=address" cargo +nightly test -Zbuild-std --target $(rustc -vV | rg host | awk '{print $2}') 2>&1
```
**Untrusted input parsing:**
```bash
cargo +nightly fuzz run <target> -- -max_total_time=300 2>&1
```
### Phase 4: Fix-and-Prove Loop
For each UB finding:
1. **Classify** against the 14-category taxonomy.
2. **Write the SAFETY comment** explaining what is wrong and what the fix must achieve.
3. **Apply the minimal fix.** Do not refactor — fix the UB and nothing else.
4. **Run Miri (Level 3 minimum) on the specific test that triggered the UB.**
5. **Run Miri (Level 3) on the full test suite** to check for regressions.
6. **Document the bead:**
```
BEAD: [Category #] [Short description]
FILE: [path:line]
ROOT CAUSE: [one sentence]
FIX: [one sentence]
PROOF: Miri Level [N] pass — [command used]
```
### Phase 5: Hardening (Post-Fix)
After all beads are resolved:
1. **Add Miri to CI** if not already present (see [miri-sanitizers-loom.md](miri-sanitizers-loom.md) for the GitHub Actions config).
2. **Add `#[cfg(miri)]` regression tests** for each bead — these are the tests that originally caught the UB, locked in so it never returns.
3. **Review SAFETY comments** on every remaining `unsafe` block. Each must name the specific invariant from the taxonomy.
4. **Run the full paranoia sweep one final time** to confirm clean.
## Miri-First Decision Protocol
When the agent encounters `unsafe` code during ANY Rust task (not just audits):
```
Is there unsafe code in the changeset?
YES → Run Miri Level 1 before proceeding.
│ Miri fails?
│ YES → Stop. Classify. Fix. Prove. Then continue.
│ NO → Run Miri Level 2 (strict provenance).
│ Miri fails?
│ YES → Stop. Classify. Fix. Prove. Then continue.
│ NO → Proceed with the original task.
NO → Proceed normally.
```
This is not optional. **Every `unsafe` block gets Miri'd before it ships.**
## SAFETY Comment Standard
Every `unsafe` block requires a SAFETY comment within 5 lines above it. The comment must:
1. **Name the UB category** it could trigger (from the taxonomy).
2. **State the invariant** that makes this safe.
3. **Name who/what guarantees** the invariant (caller contract, type system, runtime check).
```rust
// SAFETY: [Category 4 — Uninitialized Memory]
// All N elements have been written to via `ptr::write` in the loop above.
// The loop runs exactly `len` times, and `len` was validated against the
// allocation size at line 42. MaybeUninit::assume_init is therefore sound.
unsafe { buf.assume_init() }
```
Bad SAFETY comments that must be rejected:
- `// SAFETY: we know this is safe` — Says nothing.
- `// SAFETY: this is fine because we tested it` — Testing does not prove absence of UB.
- `// SAFETY: the caller ensures correctness` — Which invariant? What is the contract?
- No SAFETY comment at all — Immediate failure.
## Audit Report Format
When completing a UB audit, produce a summary:
```markdown
## UB Audit Report
**Scope:** [crate/module/file]
**Miri version:** [output of `cargo +nightly miri --version`]
**Date:** [date]
### Findings
| # | Category | File:Line | Severity | Status |
|---|----------|-----------|----------|--------|
| 1 | Aliasing | src/buf.rs:42 | High | Fixed (Bead #1) |
| 2 | Uninit | src/ffi.rs:98 | High | Fixed (Bead #2) |
### Beads
#### Bead #1: Aliasing violation in buffer resize
- **Root cause:** `&mut` created while `&` to same slice existed
- **Fix:** Restructured to drop shared ref before taking mutable
- **Proof:** `cargo +nightly miri test -- test_buffer_resize` passes Level 3
### Miri CI Status
- [ ] Miri added to CI (Level 2 minimum)
- [ ] All SAFETY comments reviewed
- [ ] Regression tests added for each bead
```
## Common Fix Patterns
### Aliasing → Use `UnsafeCell` or restructure borrows
```rust
// BEFORE (UB: &mut while & exists)
let ptr = slice.as_ptr();
let mut_ref = &mut slice[0]; // UB: ptr still usable
// AFTER
let mut_ref = &mut slice[0];
// ptr is never created / used across the mutable borrow
```
### Uninitialized → Use `MaybeUninit::write` + `assume_init`
```rust
// BEFORE (UB: mem::uninitialized)
let x: T = unsafe { std::mem::uninitialized() };
// AFTER
let x: T = unsafe {
let mut uninit = MaybeUninit::<T>::uninit();
uninit.write(initial_value);
uninit.assume_init()
};
```
### Provenance → Use `expose_provenance` / `with_exposed_provenance`
```rust
// BEFORE (UB: provenance lost)
let addr = ptr as usize;
let recovered = addr as *const T;
// AFTER
let addr = ptr.expose_provenance();
let recovered = std::ptr::with_exposed_provenance::<T>(addr);
```
### Send/Sync → Remove manual impl, use PhantomData
```rust
// BEFORE (unsound)
unsafe impl Send for MyType {}
// AFTER — if MyType truly needs Send, prove it:
// SAFETY: [Category 9 — Send/Sync]
// MyType's only non-Send field is `*mut Buffer`. Access to the buffer
// is guarded by `self.lock: Mutex<()>`, which provides the
// happens-before guarantee required by Send.
unsafe impl Send for MyType {}
```
### FFI → Validate at boundary
```rust
// BEFORE (UB: null pointer from C becomes &T)
let result = unsafe { ffi_call() };
// AFTER
let raw = unsafe { ffi_call() };
let result = NonNull::new(raw).ok_or(Error::NullFromFfi)?;
```
## Activation
This skill activates when:
- The user requests a "UB audit", "miri sweep", "unsafe audit", "soundness check", "rustonomicon audit", "race hunt"
- The agent encounters `unsafe` code during a Rust task and needs to verify it
- Miri reports a failure and the agent needs to classify and fix it
- The user asks "is this sound?" about Rust code
**Miri is not optional. Miri is the proof. Ship nothing `unsafe` without Miri's blessing.**
packages/shared-skills/skills/programming/references/rust-ub/miri-sanitizers-loom.md
+411
View File
@@ -0,0 +1,411 @@
# Miri, Sanitizers, Loom, and Fuzzing — The UB Detection Arsenal
Miri is the **primary weapon**. Everything else is supplementary for the gaps Miri cannot reach.
---
## Miri — The First and Last Line of Defense
### What Miri Is
Miri is an interpreter for Rust's MIR (Mid-level IR). It executes your test suite inside a virtual machine that tracks every byte of memory for validity, provenance, alignment, initialization, and aliasing. It is **deterministic** — same inputs, same result — and it can find UB that no amount of testing on real hardware will ever trigger.
### Why Miri Is Non-Negotiable
- Detects 12 of 14 UB categories (see `ub-taxonomy.md`).
- Catches aliasing violations that compile and run correctly on every platform today but are UB that future compiler optimizations will exploit.
- Catches data races under a configurable scheduling model.
- Catches provenance violations that are impossible to observe on real hardware.
- **Zero false positives** — if Miri says it is UB, it is UB. Period.
### Installation
```bash
rustup install nightly
rustup component add miri rust-src --toolchain nightly
```
Verify:
```bash
cargo +nightly miri --version
```
### Running Miri
**Default run (Stacked Borrows, standard checks):**
```bash
cargo +nightly miri test
```
**With nextest (recommended for projects already using nextest):**
```bash
cargo +nightly miri nextest run
```
**Specific test:**
```bash
cargo +nightly miri test -- test_name
```
**Run a binary:**
```bash
cargo +nightly miri run
```
### MIRIFLAGS — The Dial-Up Knobs
These flags are set via the `MIRIFLAGS` environment variable. The agent should use ALL of the strictness flags during a UB audit.
#### Aliasing Model
```bash
# Default: Stacked Borrows (strict)
cargo +nightly miri test
# Tree Borrows (newer, more permissive — use as a second pass)
MIRIFLAGS="-Zmiri-tree-borrows" cargo +nightly miri test
```
**Protocol:** Run Stacked Borrows first. If it fails, fix it. Then run Tree Borrows to confirm. Code that passes Stacked Borrows is sound under both models.
#### Strict Provenance
```bash
MIRIFLAGS="-Zmiri-strict-provenance" cargo +nightly miri test
```
Catches `ptr as usize as *const T` roundtrips where provenance is lost. **Should be ON for every audit.**
#### Symbolic Alignment Checks
```bash
MIRIFLAGS="-Zmiri-symbolic-alignment-check" cargo +nightly miri test
```
Catches alignment UB that happens to be aligned on your machine but is not guaranteed by the type system.
#### Data Race Detection Tuning
```bash
# Increase preemption rate to stress-test race conditions
MIRIFLAGS="-Zmiri-preemption-rate=0.5" cargo +nightly miri test
# Disable preemption (sequential scheduling — fewer races found but deterministic)
MIRIFLAGS="-Zmiri-preemption-rate=0" cargo +nightly miri test
```
#### The Full Paranoia Sweep (Use This for Audits)
```bash
MIRIFLAGS="\
-Zmiri-strict-provenance \
-Zmiri-symbolic-alignment-check \
-Zmiri-preemption-rate=0.1 \
-Zmiri-backtrace=full \
-Zmiri-disable-isolation" \
cargo +nightly miri test
```
Then a second pass with Tree Borrows:
```bash
MIRIFLAGS="\
-Zmiri-tree-borrows \
-Zmiri-strict-provenance \
-Zmiri-symbolic-alignment-check \
-Zmiri-preemption-rate=0.1 \
-Zmiri-backtrace=full \
-Zmiri-disable-isolation" \
cargo +nightly miri test
```
#### Isolation and I/O
Miri runs in isolation by default — no file I/O, no network, no system calls. If your tests need the filesystem:
```bash
MIRIFLAGS="-Zmiri-disable-isolation" cargo +nightly miri test
```
Use sparingly — isolation is a feature, not a limitation. Tests that need I/O should have a separate `#[cfg(not(miri))]` path.
### Miri Limitations
| Cannot do | Workaround |
|-----------|-----------|
| Execute FFI / C code | ASAN, MSAN, Valgrind |
| Run I/O-heavy tests (default) | `-Zmiri-disable-isolation` or `#[cfg(not(miri))]` |
| Exhaustive interleaving exploration | loom |
| Find performance bugs | criterion, flamegraph |
| Run inline assembly | skip with `#[cfg(not(miri))]` |
| Test OS-specific behavior | real hardware + sanitizers |
### Miri in CI
```yaml
# GitHub Actions example
miri:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: miri, rust-src
- name: Miri test (Stacked Borrows + strict provenance)
run: |
MIRIFLAGS="-Zmiri-strict-provenance -Zmiri-symbolic-alignment-check -Zmiri-backtrace=full" \
cargo +nightly miri test
- name: Miri test (Tree Borrows)
run: |
MIRIFLAGS="-Zmiri-tree-borrows -Zmiri-strict-provenance -Zmiri-symbolic-alignment-check -Zmiri-backtrace=full" \
cargo +nightly miri test
```
### Miri-Incompatible Test Gating
```rust
#[test]
#[cfg_attr(miri, ignore)] // Miri cannot run this (FFI, I/O, inline asm)
fn test_requires_real_hardware() {
// ...
}
// Or conditionally compile the test body:
#[test]
fn test_with_miri_fallback() {
#[cfg(miri)]
{
// Simplified version that avoids FFI
}
#[cfg(not(miri))]
{
// Full version with FFI
}
}
```
---
## Sanitizers — Where Miri Cannot Reach
Sanitizers are compiler instrumentation passes. They run your actual binary on real hardware with extra checks injected. Use them for FFI, I/O-heavy code, and integration tests.
### AddressSanitizer (ASAN)
Detects: use-after-free, buffer overflow, stack-use-after-return, double-free, memory leaks.
```bash
RUSTFLAGS="-Zsanitizer=address" cargo +nightly test -Zbuild-std --target x86_64-unknown-linux-gnu
```
On macOS:
```bash
RUSTFLAGS="-Zsanitizer=address" cargo +nightly test -Zbuild-std --target aarch64-apple-darwin
```
### ThreadSanitizer (TSAN)
Detects: data races on non-atomic accesses across threads.
```bash
RUSTFLAGS="-Zsanitizer=thread" cargo +nightly test -Zbuild-std --target x86_64-unknown-linux-gnu
```
**When to use over Miri:** Integration tests involving real threads + real I/O + FFI. Miri's data-race detector is superior for pure-Rust code.
### MemorySanitizer (MSAN)
Detects: reads of uninitialized memory.
```bash
RUSTFLAGS="-Zsanitizer=memory -Zsanitizer-memory-track-origins" cargo +nightly test -Zbuild-std --target x86_64-unknown-linux-gnu
```
**When to use over Miri:** FFI code where C/C++ may return uninitialized memory into Rust.
### UndefinedBehaviorSanitizer (UBSAN)
Detects: integer overflow, misaligned access, null dereference, and other C/C++-style UB at the LLVM level.
```bash
RUSTFLAGS="-Zsanitizer=undefined" cargo +nightly test -Zbuild-std --target x86_64-unknown-linux-gnu
```
### Sanitizer Limitations
- Require nightly + `-Zbuild-std` (rebuilds the standard library with instrumentation).
- MSAN requires ALL dependencies (including C libs) to be instrumented — practically hard.
- Cannot catch aliasing violations (that is Miri's domain).
- Significant runtime overhead (2-15x slower).
- Linux has the best support; macOS works for ASAN; Windows support is minimal.
---
## Loom — Exhaustive Concurrency Testing
Loom explores all possible thread interleavings of a bounded concurrent program. It is mandatory for lock-free and wait-free primitives.
### When to Use Loom
- Any `unsafe` code involving atomics with ordering weaker than `SeqCst`.
- Custom lock implementations.
- Lock-free queues, stacks, or other concurrent data structures.
- Any code where you chose `Relaxed`, `Acquire`, or `Release` ordering.
### When NOT to Use Loom
- Code using only `Mutex`/`RwLock` from std or `parking_lot` — the locks are sound, your usage is the question, and Miri + TSAN cover that.
- Async code (loom does not model async runtimes — use `tokio::test` + Miri instead).
### Setup
```toml
[dev-dependencies]
loom = "0.7"
```
### Loom Test Pattern
```rust
#[cfg(loom)]
mod loom_tests {
use loom::sync::atomic::{AtomicUsize, Ordering};
use loom::sync::Arc;
use loom::thread;
#[test]
fn concurrent_increment_is_sound() {
loom::model(|| {
let counter = Arc::new(AtomicUsize::new(0));
let threads: Vec<_> = (0..2).map(|_| {
let c = counter.clone();
thread::spawn(move || {
c.fetch_add(1, Ordering::SeqCst);
})
}).collect();
for t in threads {
t.join().unwrap();
}
assert_eq!(counter.load(Ordering::SeqCst), 2);
});
}
}
```
### Conditional Compilation for Loom
```rust
#[cfg(loom)]
use loom::sync::atomic::{AtomicUsize, Ordering};
#[cfg(not(loom))]
use std::sync::atomic::{AtomicUsize, Ordering};
```
### Running Loom Tests
```bash
# Loom tests only (use cfg flag)
RUSTFLAGS="--cfg loom" cargo test --lib -- loom_tests
# With release optimizations (loom is slow)
RUSTFLAGS="--cfg loom" cargo test --lib --release -- loom_tests
```
### Loom + Miri Interaction
Loom and Miri solve different problems:
- **Miri** checks a single execution for UB (aliasing, validity, provenance).
- **Loom** checks all interleavings for correctness (ordering, atomicity).
Run BOTH on lock-free code:
```bash
# Step 1: loom for interleaving correctness
RUSTFLAGS="--cfg loom" cargo test --lib --release -- loom_tests
# Step 2: Miri for UB in each path
cargo +nightly miri test -- concurrent_tests
```
---
## Cargo-Fuzz — Property-Based UB Hunting
Fuzzing generates random inputs to maximize code coverage and find crashes, panics, and UB.
### Setup
```bash
cargo install cargo-fuzz
cargo fuzz init
```
### Fuzz Target
```rust
// fuzz/fuzz_targets/parse_input.rs
#![no_main]
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
// Your parsing/deserialization/processing code here.
// If it panics or triggers UB, the fuzzer catches it.
let _ = my_crate::parse(data);
});
```
### Running
```bash
# Run until interrupted
cargo +nightly fuzz run parse_input
# Run with ASAN (catches memory bugs in unsafe code)
cargo +nightly fuzz run parse_input -- -rss_limit_mb=4096
# Minimize a crashing input
cargo +nightly fuzz tmin parse_input artifacts/parse_input/crash-xxxxx
```
### Fuzz + Miri Pipeline
When the fuzzer finds a crashing input:
1. Minimize it with `cargo fuzz tmin`.
2. Add it as a regression test.
3. Run the regression test under Miri to classify whether it is a panic (safe) or UB (must fix).
```bash
# After adding the input as a test case:
cargo +nightly miri test -- test_fuzz_regression_001
```
---
## Tool Selection Decision Tree
```
Start
│
├── Is it pure Rust (no FFI, no I/O)?
│ YES → Miri (full paranoia flags)
│ │ └── Also: loom (if atomics/lock-free)
│ │ └── Also: proptest (if parsing/serialization)
│ │ └── Also: cargo-fuzz (if untrusted input)
│ │
│ NO → Does it involve FFI?
│ YES → ASAN + MSAN on integration tests
│ │ └── Miri on the Rust-side handling
│ │ └── cbindgen in CI for layout verification
│ │
│ NO → Is it I/O-heavy?
│ YES → TSAN for thread safety
│ │ └── Miri with -Zmiri-disable-isolation where possible
│ │
│ NO → Miri (full paranoia flags)
│
└── Always: Miri is the default. Other tools supplement.
```
## The One Rule
> **When in doubt, run Miri.** If Miri cannot run it, write a version it can run, and test that under Miri. Then test the real version under sanitizers. Never ship `unsafe` code that has not passed Miri.
packages/shared-skills/skills/programming/references/rust-ub/ub-taxonomy.md
+269
View File
@@ -0,0 +1,269 @@
# Rust Undefined Behavior Taxonomy
Every category of UB the Rust compiler, Miri, and the language specification recognize. The agent must know the full surface to hunt systematically. Each entry names the UB class, its root cause, canonical trigger, Miri detection status, and the canonical fix.
## 1. Aliasing Violations (Stacked Borrows / Tree Borrows)
**Root cause:** Two pointers access the same memory in ways that violate Rust's borrowing model — even through raw pointers inside `unsafe`.
**Canonical triggers:**
- Creating a `&mut T` while another `&T` or `&mut T` to the same location exists.
- Dereferencing a raw pointer derived from a reference after that reference was invalidated (e.g., `&mut` was retaken).
- Calling `slice::from_raw_parts_mut` on overlapping regions.
- Interior mutability through `UnsafeCell` without going through the `UnsafeCell` API.
- Casting `&T` to `*mut T` and writing through it (even via FFI).
**Miri detection:** YES — Stacked Borrows is the default model. Tree Borrows (`-Zmiri-tree-borrows`) is the newer, more permissive model. Run both:
```bash
cargo +nightly miri test # Stacked Borrows (stricter)
MIRIFLAGS="-Zmiri-tree-borrows" cargo +nightly miri test # Tree Borrows (relaxed)
```
If code passes Tree Borrows but fails Stacked Borrows, it is *likely* sound but *possibly* relying on unspecified behavior. Fix it anyway — Stacked Borrows is the conservative bet.
**Fix pattern:** Use `UnsafeCell` for all interior mutability. Never cast `&T` to `*mut T`. Derive mutable pointers from `*mut T` obtained via `UnsafeCell::get()` or `addr_of_mut!()`.
---
## 2. Data Races
**Root cause:** Two threads access the same non-atomic memory location, at least one is a write, and there is no happens-before ordering between them.
**Canonical triggers:**
- `unsafe impl Send for T` on a type containing `*mut U` without synchronization.
- `unsafe impl Sync for T` on a type containing `Cell<T>` or `UnsafeCell<T>` without a lock.
- Using `std::ptr::write` from multiple threads to the same allocation.
- Shared `&T` where `T` has interior mutability but no atomic/lock guard.
**Miri detection:** YES — Miri's data-race detector is on by default. It detects races on non-atomic accesses. For **preemptive scheduling** stress, use:
```bash
MIRIFLAGS="-Zmiri-preemption-rate=0.1" cargo +nightly miri test
```
**Complementary tools:** `loom` for exhaustive interleaving exploration on lock-free algorithms. ThreadSanitizer (TSAN) for integration tests Miri cannot run (I/O, FFI).
**Fix pattern:** Wrap in `Mutex`/`RwLock`/`AtomicXxx`. Never `unsafe impl Sync` unless you can name the synchronization primitive guarding every mutable field.
---
## 3. Use After Free / Dangling Pointers
**Root cause:** A pointer or reference outlives the allocation it points to.
**Canonical triggers:**
- Returning a reference to a local variable (compiler catches most, but raw pointers escape).
- `Box::into_raw` → manual `Box::from_raw` with wrong lifetime.
- `Vec` reallocation invalidating raw pointers obtained from `as_ptr()` / `as_mut_ptr()`.
- `Pin<Box<T>>` unpinned and moved after self-referential pointers were set up.
**Miri detection:** YES — allocation tracking catches use-after-free on the exact operation.
**Fix pattern:** Borrow checker for references. For raw pointers: tie pointer validity to an explicit lifetime via a `PhantomData<&'a T>` in the wrapper, or use arena allocation (`bumpalo`) so all pointers share one lifetime.
---
## 4. Uninitialized Memory
**Root cause:** Reading a value from memory that was never written to.
**Canonical triggers:**
- `MaybeUninit::assume_init()` before all bytes are written.
- `mem::uninitialized()` (deprecated, still compiles).
- `alloc::alloc(layout)` returns uninitialized memory — reading it before writing is UB.
- Padding bytes in structs read via `transmute` or raw pointer casts.
- `read_unaligned` on uninitialized memory.
**Miri detection:** YES — tracks initialization state per byte. Catches partial-init structs, padding reads, and premature `assume_init`.
**Fix pattern:** Use `MaybeUninit::zeroed()` when zero-init is acceptable. Write every field before calling `assume_init()`. Use `MaybeUninit::write()` instead of raw pointer writes. Never `transmute` structs with padding unless you zeroed the padding.
---
## 5. Invalid Values (Type Invariant Violations)
**Root cause:** Producing a value that violates the type's validity invariant.
**Canonical triggers:**
- `bool` not 0 or 1.
- `char` outside Unicode scalar range.
- Enum discriminant not matching any variant.
- `NonZeroU32` containing 0.
- `&T` or `&mut T` that is null or dangling.
- `str` containing non-UTF-8 bytes.
- `fn` pointer that is null.
**Miri detection:** YES — validity checks are on by default. Extra strictness:
```bash
MIRIFLAGS="-Zmiri-strict-provenance" cargo +nightly miri test
```
**Fix pattern:** Validate before transmuting. Use `TryFrom` at boundaries. Never `transmute` to enum types — use a checked conversion function.
---
## 6. Misaligned Pointer Access
**Root cause:** Dereferencing a pointer that is not aligned to the type's required alignment.
**Canonical triggers:**
- Casting `*const u8` to `*const u64` and dereferencing (alignment goes from 1 to 8).
- `#[repr(packed)]` struct field references (the compiler warns, but raw pointers bypass the warning).
- Network buffer parsing where offsets are arbitrary.
**Miri detection:** YES — immediate trap on misaligned read/write.
**Fix pattern:** Use `read_unaligned` / `write_unaligned` for packed data. Use `bytemuck` or `zerocopy` for safe reinterpretation with alignment checks.
---
## 7. Violating `Pin` Invariants
**Root cause:** Moving a value that was pinned and relied on its address stability (self-referential types, intrusive linked lists).
**Canonical triggers:**
- `mem::swap` on a `Pin<&mut T>` after `unsafe` deref.
- Implementing `Unpin` for a type that contains self-referential pointers.
- Manually calling `Pin::new_unchecked` on a movable allocation.
**Miri detection:** PARTIAL — Miri detects the resulting aliasing/use-after-free if the self-referential pointer is actually used. It does not detect "Pin contract violated but pointer was never dereferenced."
**Fix pattern:** Never `impl Unpin` for self-referential types. Use `pin_project` or `pin_project_lite` for safe pin projections. Review every `Pin::new_unchecked` call.
---
## 8. FFI Boundary UB
**Root cause:** Mismatch between Rust's ABI expectations and the foreign code's actual behavior.
**Canonical triggers:**
- C function returning uninitialized memory into a Rust `&T`.
- Wrong `#[repr(C)]` layout (padding differs between platforms).
- Passing a Rust `enum` to C without `#[repr(C)]` or `#[repr(i32)]`.
- Null pointer passed where C expects non-null (and Rust wraps it in `&T`).
- C code writing to Rust-owned memory through a pointer Rust considers immutable.
- Forgetting to mark FFI functions as `unsafe extern "C"`.
- longjmp/setjmp across Rust frames (unwinding UB).
**Miri detection:** LIMITED — Miri cannot execute foreign code. It detects UB in the Rust-side handling of FFI return values.
**Complementary tools:** AddressSanitizer (ASAN), MemorySanitizer (MSAN) for detecting actual FFI-side corruption. Valgrind as a last resort.
**Fix pattern:** Validate every FFI return at the boundary. Use `Option<NonNull<T>>` for nullable pointers. Use `CStr`/`CString` for strings. Add `cbindgen` to CI to verify layout agreement. Wrap every FFI call in a safe Rust function that checks preconditions.
---
## 9. Incorrect `Send` / `Sync` Implementations
**Root cause:** Manually implementing `Send` or `Sync` for a type that does not actually uphold the required invariant.
**Canonical triggers:**
- `unsafe impl Send for Wrapper(*mut T)` when `T` is not `Send`.
- `unsafe impl Sync for Wrapper(UnsafeCell<T>)` without a lock, atomic, or other synchronization.
- Types containing `Rc<T>` with a manual `Send` impl (Rc is explicitly !Send).
**Miri detection:** YES for the *resulting* data race if exercised. Miri's data-race detector will fire when two threads access the same location unsynchronized.
**Fix pattern:** Never manually implement `Send`/`Sync` unless you can write a SAFETY proof naming the synchronization mechanism. Use `PhantomData<*const ()>` to opt-out of auto-`Send`/`Sync` when in doubt.
---
## 10. Out-of-Bounds Memory Access
**Root cause:** Pointer arithmetic or indexing that escapes the allocation.
**Canonical triggers:**
- `ptr.offset(n)` where `n` exceeds the allocation size.
- `slice::from_raw_parts(ptr, len)` where `len` is too large.
- Off-by-one in manual buffer management.
- Integer overflow in size calculations leading to undersized allocation.
**Miri detection:** YES — allocation-precise bounds checking.
**Fix pattern:** Use checked arithmetic (`checked_add`, `checked_mul`) for size calculations. Use `slice::from_raw_parts` only with validated lengths. Prefer safe indexing (`get()`, iterators) over raw pointer arithmetic.
---
## 11. Provenance Violations
**Root cause:** Using a pointer whose provenance does not grant access to the target memory, even if the address is numerically correct.
**Canonical triggers:**
- Casting an integer to a pointer and dereferencing it (`addr as *const T`).
- Roundtripping a pointer through `usize` and back (`ptr as usize as *const T`) — the provenance is lost.
- Using `ptr::from_exposed_addr` without a corresponding `ptr.expose_provenance()`.
**Miri detection:** YES with strict provenance:
```bash
MIRIFLAGS="-Zmiri-strict-provenance" cargo +nightly miri test
```
**Fix pattern:** Use `ptr::with_exposed_provenance` / `ptr.expose_provenance()` for legitimate int-to-ptr roundtrips. Avoid `as usize as *const T` entirely. Use `sptr` crate for provenance-safe pointer manipulation on stable.
---
## 12. Double Free / Invalid Free
**Root cause:** Freeing the same allocation twice, or freeing memory not obtained from the allocator.
**Canonical triggers:**
- `Box::from_raw` called twice on the same pointer.
- Manual `dealloc` on a pointer already freed.
- `ManuallyDrop` dropped explicitly then the outer type also drops it.
**Miri detection:** YES — immediate trap.
**Fix pattern:** Enforce single ownership via RAII. Use `ManuallyDrop` with extreme care — document who is responsible for the drop. Never clone a raw pointer and `Box::from_raw` both copies.
---
## 13. Library / Unsafe Contract Violations
**Root cause:** Violating the documented safety invariant of a safe or unsafe API, where the library author relied on the invariant for soundness.
**Canonical triggers:**
- `Vec::set_len(n)` where the first `n` elements are not initialized.
- `String::from_utf8_unchecked` on non-UTF-8 bytes.
- `HashMap` key mutated after insertion (violates hash invariant — not UB per se, but unsound and Miri may detect downstream effects).
- `BTreeMap` key with broken `Ord` impl (the standard library assumes a total order).
**Miri detection:** DEPENDS — Miri catches the downstream UB (e.g., reading uninitialized bytes from a `Vec` with inflated len). It does not catch "you violated the documented contract" if no memory-level UB results.
**Fix pattern:** Read the `# Safety` section of every `unsafe fn` you call. Document the invariant in your SAFETY comment. When in doubt, use the safe API and pay the cost.
---
## 14. Unwinding Across `extern "C"` Boundaries
**Root cause:** A Rust panic unwinding through a frame that uses the C calling convention.
**Canonical triggers:**
- `panic!()` inside a `#[no_mangle] extern "C" fn` callback passed to C code.
- `unwrap()` inside FFI callbacks.
**Miri detection:** PARTIAL — Miri does not model foreign unwinding, but it can detect the immediate UB if the panic reaches the FFI boundary.
**Fix pattern:** Use `std::panic::catch_unwind` at every FFI entry point. Mark FFI callbacks as `extern "C-unwind"` when panic propagation is intentional (nightly). Prefer returning `Result`-like error codes from FFI callbacks.
---
## Summary Table
| # | Category | Miri Detects? | Complementary Tool |
|---|----------|--------------|-------------------|
| 1 | Aliasing (Stacked/Tree Borrows) | YES | — |
| 2 | Data races | YES | loom, TSAN |
| 3 | Use-after-free / dangling | YES | ASAN |
| 4 | Uninitialized memory | YES | MSAN |
| 5 | Invalid values | YES | — |
| 6 | Misaligned access | YES | UBSAN |
| 7 | Pin invariant violation | PARTIAL | manual review |
| 8 | FFI boundary UB | LIMITED | ASAN, MSAN, Valgrind |
| 9 | Incorrect Send/Sync | YES (via race) | loom |
| 10 | Out-of-bounds access | YES | ASAN |
| 11 | Provenance violations | YES (strict mode) | — |
| 12 | Double free | YES | ASAN |
| 13 | Library contract violations | PARTIAL | proptest, fuzzing |
| 14 | Unwinding across FFI | PARTIAL | — |
## Miri Coverage Assessment
Miri catches categories 1-6, 9-12 with high confidence. Categories 7, 8, 13, 14 require supplementary tools or manual audit. **Miri is the single highest-leverage tool** — it should run on every PR that touches `unsafe`, and ideally on the full test suite regularly.
packages/shared-skills/skills/programming/references/rust/README.md
+317
View File
@@ -0,0 +1,317 @@
# Rust Programmer
Production Rust in 2026. **Explicit allocation, compile-time proof, zero hidden cost.** Type-state-first, unsafe-banished-by-default, agent-proof.
## Identity — What Kind of Rust You Write
You write Rust that looks like a Zig programmer designed it and a Rust compiler enforces it. Every allocation is visible. Every cost is explicit. Every invariant is encoded in the type system. Every cleanup is deterministic. The borrow checker, lifetime analysis, trait bounds, and `miri` then guarantee what Zig leaves to discipline.
**Five pillars, every file, no exceptions:**
| Pillar | Default Behavior | Reference |
|---|---|---|
| **Explicit allocation** | Arena for hot paths, `&[T]`/`Cow` over `Vec`/`String` in signatures, `try_*` when allocation can fail | [zero-cost-safety.md §1](zero-cost-safety.md) |
| **Compile-time proof** | `const fn` everything const-eligible, `const { assert!(...) }` for compile-time guards, const generics for sized buffers | [zero-cost-safety.md §2](zero-cost-safety.md) |
| **Zero hidden cost** | Slice-based APIs where caller owns memory, no hidden `.clone()`/`.to_string()`, `Cow` to defer allocation | [zero-cost-safety.md §3](zero-cost-safety.md) |
| **Type-encoded invariants** | Newtype wrappers for every semantic unit, type-state for state machines, branded IDs | [type-state.md](type-state.md) |
| **Deterministic cleanup** | `scopeguard::guard` for errdefer, `Drop` for RAII, defuse-on-success for rollback | [zero-cost-safety.md §5](zero-cost-safety.md) |
The two highest-leverage tools Rust gives a coding agent:
1. **Bounded polymorphism** (traits). Real, machine-checked, composable constraints.
2. **Newtype-as-coordinate-space.** `Point<Screen>` and `Point<World>` are distinct types — the agent literally cannot pass one where the other is expected. This is the `euclid` crate pattern; generalize ruthlessly to money, durations, IDs, byte offsets, char offsets, paths rooted at different bases. Full patterns → [type-state.md](type-state.md).
---
## Hard Rules (Every `.rs` File)
### 1. No `unwrap()`, No `expect()` Outside Tests
```rust
// WRONG
let val = map.get("key").unwrap();
// RIGHT — propagate or provide context
let val = map.get("key").context("missing 'key' in config")?;
```
Typed errors for libraries ([thiserror](https://docs.rs/thiserror)), ad-hoc errors for binaries ([anyhow](https://docs.rs/anyhow) / [color-eyre](https://docs.rs/color-eyre)). Full stack → [libraries.md](libraries.md).
### 2. No `unsafe` Without Miri Proof
If `unsafe` is unavoidable, you have miri. Run it. Always. **Load [`../rust-ub/README.md`](../rust-ub/README.md) plus every file under [`../rust-ub/`](../rust-ub/)** for the full UB taxonomy, Miri escalation protocol (4 strictness levels), and the fix-and-prove workflow. Every `unsafe` block needs the three components from [unsafe-discipline.md](unsafe-discipline.md): safe wrapper, `// SAFETY:` comment, miri test.
```bash
cargo +nightly miri nextest run
```
### 3. Explicit Allocation — Arena by Default in Hot Paths
**Do not scatter `Box::new()` / `Vec::new()` across hot loops.** Use arena allocation to make allocation scope visible and bulk-freeable. Full recipes → [zero-cost-safety.md §1](zero-cost-safety.md).
```rust
use bumpalo::Bump;
fn parse_frame<'a>(arena: &'a Bump, raw: &[u8]) -> Frame<'a> {
let header = arena.alloc(parse_header(raw));
let payload = arena.alloc_slice_copy(&raw[HEADER_LEN..]);
Frame { header, payload }
}
// Caller owns arena. Caller decides when memory dies. Zero individual frees.
```
When arena is overkill (simple CLI, one-shot allocation), `Vec`/`String` are fine — but **function signatures still prefer borrows**:
```rust
// WRONG — forces caller to allocate
fn process(input: String) -> String { ... }
// RIGHT — caller chooses allocation strategy
fn process(input: &str) -> Cow<'_, str> { ... }
// BEST for hot paths — zero allocation, caller provides buffer
fn process(input: &[u8], output: &mut [u8]) -> usize { ... }
```
### 4. Compile-Time First — const fn Everything Const-Eligible
If a function CAN be `const fn`, it MUST be `const fn`. Full recipes → [zero-cost-safety.md §2](zero-cost-safety.md).
```rust
// Lookup tables computed at compile time — zero runtime cost
const CRC_TABLE: [u32; 256] = {
let mut table = [0u32; 256];
let mut i = 0;
while i < 256 {
let mut crc = i as u32;
let mut j = 0;
while j < 8 {
crc = if crc & 1 != 0 { (crc >> 1) ^ 0xEDB88320 } else { crc >> 1 };
j += 1;
}
table[i] = crc;
i += 1;
}
table
};
// Compile-time assertions — catch violations at build time, not runtime
const { assert!(std::mem::size_of::<Header>() == 12, "Header must be 12 bytes") };
```
Use `const generics` for stack-allocated buffers with compile-time size:
```rust
struct RingBuffer<T, const N: usize> {
data: [MaybeUninit<T>; N],
head: usize,
len: usize,
}
```
### 5. Scope Guards — Deterministic Cleanup on Every Path
Zig's `errdefer` in Rust. Full recipes → [zero-cost-safety.md §5](zero-cost-safety.md).
```rust
use scopeguard::guard;
fn deploy(artifact: &Path) -> Result<(), DeployError> {
let backup = snapshot_current()?;
// errdefer: restore on failure
let rollback = guard(backup, |b| { let _ = restore(&b); });
upload(artifact)?;
health_check()?;
// Success: defuse the guard
scopeguard::ScopeGuard::into_inner(rollback);
Ok(())
}
```
### 6. Bit-Level Layout — zerocopy for Wire Formats
Never hand-write `transmute` or pointer casts for parsing binary data. Full recipes → [zero-cost-safety.md §4](zero-cost-safety.md).
```rust
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable};
#[derive(FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct PacketHeader {
magic: [u8; 4],
version: u8,
flags: u8,
length: [u8; 2], // use byte array for packed fields, decode via from_le_bytes
}
```
### 7. Exhaustive Match — No Wildcard on Enums You Control
```rust
// WRONG — silently ignores new variants
match status {
Status::Ok => handle_ok(),
_ => handle_error(),
}
// RIGHT — compiler forces update when variants change
match status {
Status::Ok => handle_ok(),
Status::NotFound => handle_not_found(),
Status::Timeout => handle_timeout(),
}
```
For `#[non_exhaustive]` enums from external crates, the wildcard `_` is required — but add a `tracing::warn!` in the catch-all so you notice when new variants appear.
### 8. Type-State Over Runtime Checks
Never `if self.state == State::Validated`. Encode states as distinct types so the compiler refuses invalid transitions. Full patterns → [type-state.md](type-state.md).
```rust
struct Order<S: OrderState> { data: OrderData, _state: PhantomData<S> }
struct Draft;
struct Validated;
struct Paid;
impl Order<Draft> {
fn validate(self) -> Result<Order<Validated>, ValidationError> { ... }
}
impl Order<Validated> {
fn pay(self, payment: Payment) -> Result<Order<Paid>, PaymentError> { ... }
}
// Order<Draft> has no .pay() method. Compiler enforces the workflow.
```
---
## Standard Library Defaults
Full decision tree with rationale and code snippets → [libraries.md](libraries.md).
| Category | Crate | Why |
|---|---|---|
| Async runtime | `tokio` | Ecosystem standard. Patterns → [async-tokio.md](async-tokio.md) |
| HTTP server | `axum` + `tower` | Type-safe extractors, tower middleware. Stack → [axum-stack.md](axum-stack.md) |
| CLI | `clap` derive + `color-eyre` | Typed args, beautiful errors. Stack → [clap-stack.md](clap-stack.md) |
| Serialization | `serde` + `serde_json` | Non-negotiable for any boundary type |
| Error (library) | `thiserror` | Derive `Error` with zero boilerplate |
| Error (binary) | `anyhow` / `color-eyre` | Context-rich ad-hoc errors |
| Database | `sqlx` (compile-time checked) | No runtime SQL surprises |
| Arena alloc | `bumpalo` / `typed-arena` | Explicit allocation scope. Patterns → [zero-cost-safety.md §1](zero-cost-safety.md) |
| Zero-copy parse | `zerocopy` | Safe binary parsing, no transmute. Patterns → [zero-cost-safety.md §4](zero-cost-safety.md) |
| Scope guard | `scopeguard` | errdefer/defer. Patterns → [zero-cost-safety.md §5](zero-cost-safety.md) |
| Stack collections | `smallvec` / `arrayvec` / `tinyvec` | Stack-first, heap-spillover. Patterns → [zero-cost-safety.md §3](zero-cost-safety.md) |
| Bitfield | `bitfield` / `modular-bitfield` | Bit-packed flags. Patterns → [zero-cost-safety.md §4](zero-cost-safety.md) |
| Testing | `proptest` + `insta` | Property + snapshot tests. Patterns → [proptest-insta.md](proptest-insta.md) |
| Concurrency | `tokio::sync` / `parking_lot` | Channel-first, lock-second. Patterns → [concurrency.md](concurrency.md) |
---
## Cargo Strict Configuration
Every new project gets the strict lint config from [cargo-strict.md](cargo-strict.md). The non-negotiable CI gate:
```bash
cargo fmt --all -- --check && \
cargo clippy --all-targets --all-features -- -D warnings && \
cargo nextest run && \
cargo +nightly miri nextest run # when unsafe is involved
```
---
## Code Review Checklist (Post-Write, Every PR)
Run through this list after writing any Rust code. Every item links to its recipe.
| # | Check | Fix Reference |
|---|---|---|
| 1 | Every function signature prefers `&[T]`/`&str`/`Cow` over owned types | [zero-cost-safety.md §3](zero-cost-safety.md) |
| 2 | Hot-path allocations use arena (`bumpalo`) not scattered `Box`/`Vec` | [zero-cost-safety.md §1](zero-cost-safety.md) |
| 3 | Const-eligible functions are `const fn` | [zero-cost-safety.md §2](zero-cost-safety.md) |
| 4 | Lookup tables / config constants computed at compile time | [zero-cost-safety.md §2](zero-cost-safety.md) |
| 5 | Binary format parsing uses `zerocopy`, not `transmute` | [zero-cost-safety.md §4](zero-cost-safety.md) |
| 6 | Cleanup logic uses `scopeguard` or `Drop`, never manual `if err` cleanup | [zero-cost-safety.md §5](zero-cost-safety.md) |
| 7 | Distinct semantic units are newtypes, not primitive aliases | [type-state.md](type-state.md) |
| 8 | State machines use type-state, not runtime `if state ==` | [type-state.md](type-state.md) |
| 9 | No `unwrap()`/`expect()` outside `#[cfg(test)]` | [libraries.md](libraries.md) |
| 10 | Every `unsafe` has SAFETY comment + miri test | [unsafe-discipline.md](unsafe-discipline.md), [../rust-ub/](../rust-ub/) |
| 11 | Match on owned enums is exhaustive (no `_ =>`) | This file §7 |
| 12 | Clippy pedantic passes with zero warnings | [cargo-strict.md](cargo-strict.md) |
| 13 | Property tests exist for any function with a nontrivial domain | [proptest-insta.md](proptest-insta.md) |
| 14 | Concurrency uses channels first, locks second, atomics last | [concurrency.md](concurrency.md) |
| 15 | Async code uses `JoinSet` for structured concurrency | [async-tokio.md](async-tokio.md) |
---
## Default Cargo.toml Dependencies — Zero-Cost Safety Stack
Every new project starts with these alongside the standard deps from [cargo-strict.md](cargo-strict.md):
```toml
# Zero-cost safety stack
bumpalo = { version = "3", features = ["collections"] }
scopeguard = "1"
smallvec = { version = "1", features = ["union", "const_generics"] }
zerocopy = { version = "0.8", features = ["derive"] }
# Add when needed:
# typed-arena = "2" # homogeneous arena
# arrayvec = "0.7" # fixed-capacity stack vec
# tinyvec = { version = "1", features = ["alloc"] }
# bitfield = "0.17" # bit-packed flags
# modular-bitfield = "0.11" # richer bitfield API
# bytemuck = { version = "1", features = ["derive"] }
```
---
## Reference Index
| File | When to Load |
|---|---|
| [zero-cost-safety.md](zero-cost-safety.md) | Arena, allocator, const fn, comptime, zero-alloc, bitfield, repr, scopeguard, errdefer, Zig-like patterns |
| [type-state.md](type-state.md) | Newtype wrappers, type-state machines, branded IDs, phantom types |
| [unsafe-discipline.md](unsafe-discipline.md) | Any `unsafe` block — SAFETY comments, safe wrappers, miri proof |
| [libraries.md](libraries.md) | Library selection, crate decision tree, dependency audit |
| [cargo-strict.md](cargo-strict.md) | Project bootstrap, lint config, CI gate commands |
| [async-tokio.md](async-tokio.md) | Async runtime, spawning, cancellation, `JoinSet`, `select!` |
| [axum-stack.md](axum-stack.md) | HTTP services — axum + sqlx + tower + tracing |
| [clap-stack.md](clap-stack.md) | CLI tools — clap derive + color-eyre + indicatif |
| [concurrency.md](concurrency.md) | Locks, atomics, channels, loom model checker |
| [proptest-insta.md](proptest-insta.md) | Property tests, snapshot tests, round-trip invariants |
| [one-liners.md](one-liners.md) | `rust-script` one-liners, disposable scripts, inline deps |
| [../rust-ub/README.md](../rust-ub/README.md) | UB hunting — miri escalation, sanitizers, fuzzing |
| [../rust-ub/ub-taxonomy.md](../rust-ub/ub-taxonomy.md) | 14-category UB taxonomy with detection status |
| [../rust-ub/miri-sanitizers-loom.md](../rust-ub/miri-sanitizers-loom.md) | Miri flags, ASAN/TSAN/MSAN, loom, cargo-fuzz |
---
## The Shape of Every Function
```rust
/// One-line doc explaining WHAT, not HOW.
///
/// # Errors
/// Returns `FooError::Bar` when the input is invalid.
const fn frobnicate<'a>(
arena: &'a Bump, // explicit allocator when arena is in play
input: &[u8], // borrow, not owned
output: &mut [u8], // caller-provided buffer
) -> Result<&'a Frob, FrobError> {
// ...
}
```
**Why this shape:** the caller sees every cost. Allocation scope is the arena's lifetime. Input is borrowed. Output buffer is caller-owned. Error is typed. The compiler enforces all of it.
---
## Activation
This skill activates whenever you are writing or modifying any `.rs` file or `Cargo.toml`. One-off scripts get the strict treatment too — `rust-script` + the same lints, the same gates. Details → [one-liners.md](one-liners.md).
**The promise:** production hygiene with throwaway ergonomics. Explicit allocation, compile-time proof, zero hidden cost, and **agent-proof safety at any volume**.
packages/shared-skills/skills/programming/references/rust/async-tokio.md
+299
View File
@@ -0,0 +1,299 @@
# Async with Tokio
Structured concurrency, cancellation, blocking-work isolation, channel selection. The patterns the agent should reach for by default.
## Runtime selection
```rust
// Default for services and CLIs that do real work
#[tokio::main(flavor = "multi_thread", worker_threads = 8)]
async fn main() -> anyhow::Result<()> { ... }
// For tiny CLIs or wasm where you measured single-thread is enough
#[tokio::main(flavor = "current_thread")]
async fn main() -> anyhow::Result<()> { ... }
```
Pick worker count explicitly. The default (`num_cpus`) is fine for servers; for desktop tools you usually want 2-4.
## Spawning
`tokio::spawn` returns a `JoinHandle<T>`. The future runs to completion even if the handle is dropped (detached). To enforce structured concurrency, use `JoinSet`:
```rust
use tokio::task::JoinSet;
let mut set = JoinSet::new();
for url in urls {
let client = client.clone();
set.spawn(async move { fetch(&client, &url).await });
}
let mut results = Vec::new();
while let Some(joined) = set.join_next().await {
match joined {
Ok(Ok(body)) => results.push(body),
Ok(Err(error)) => tracing::warn!(%error, "fetch failed"),
Err(panicked) if panicked.is_panic() => {
tracing::error!(?panicked, "worker panicked");
// Choose: re-raise, or continue with degraded result set.
}
Err(other) => tracing::error!(?other, "worker join error"),
}
}
```
`JoinSet`:
- Knows when all spawned tasks finish.
- Dropping the set aborts every still-running task.
- Lets you handle failures one by one rather than all-or-nothing.
For wait-for-all semantics with one type, `join!`:
```rust
let (a, b, c) = tokio::join!(load_a(), load_b(), load_c());
let a = a?; let b = b?; let c = c?;
```
For first-of-many, `select!`:
```rust
tokio::select! {
biased; // bias to top-to-bottom checking when ordering matters
_ = shutdown.recv() => {
tracing::info!("shutdown signal");
return Ok(());
}
request = listener.accept() => {
handle_request(request?).await?;
}
}
```
Without `biased`, branches are polled in random order each iteration (good for fairness). Use `biased` only when you need deterministic priority (shutdown signal first, etc).
## Cancellation
A future is cancelled when it is dropped (e.g., the `select!` arm wins another branch). **Always think: if this future is dropped mid-await, what state is left behind?**
Cancel-safe futures (you can drop without lasting effect):
- `recv()` on channels
- `accept()` on listeners
- `wait_for` on `watch::Receiver`
- `read_buf`/`write_all` on streams **only when buffers are owned by the future**, otherwise no
Cancel-unsafe futures (dropping mid-way leaves partial state):
- Manual `read_exact` into an external buffer
- Custom futures that perform partial side effects before suspending
If a function is cancel-unsafe, document it in a rustdoc `# Cancel Safety` section.
To explicitly opt out of cancellation, use `tokio_util::sync::CancellationToken`:
```rust
use tokio_util::sync::CancellationToken;
let token = CancellationToken::new();
let child = token.child_token();
tokio::spawn(async move {
tokio::select! {
_ = child.cancelled() => { /* clean up */ }
result = work() => { /* normal */ }
}
});
// later
token.cancel();
```
Pass child tokens down the call tree so the whole tree can be cancelled together.
## Timeouts
```rust
use tokio::time::{timeout, Duration};
match timeout(Duration::from_secs(5), fetch(url)).await {
Ok(Ok(body)) => Ok(body),
Ok(Err(error)) => Err(error.into()),
Err(_elapsed) => Err(anyhow::anyhow!("timed out fetching {url}")),
}
```
Set timeouts on every external I/O boundary. Defaults of "wait forever" are bugs.
## Blocking work
NEVER block inside an async task. Symptoms: deadlock, every future stalled, latency cliffs.
Heavy CPU or sync I/O → `spawn_blocking`:
```rust
let result = tokio::task::spawn_blocking(|| {
// CPU-bound: parsing, hashing, image processing
// Or sync I/O: rusqlite, OS APIs without async wrappers
expensive_pure_computation()
}).await?;
```
Long-running blocking jobs (more than ~1 second of CPU) → use a dedicated thread pool (`rayon`), not tokio's blocking pool which is sized for short bursts.
## Channels
| Need | Use |
|---|---|
| 1-many producers → 1 consumer, async | `tokio::sync::mpsc::channel(cap)` |
| Same as above, both sync + async | `flume::bounded(cap)` |
| 1 → many fan-out, latest-value semantics | `tokio::sync::watch::channel(initial)` |
| 1 → many fan-out, queued | `tokio::sync::broadcast::channel(cap)` |
| One-shot reply | `tokio::sync::oneshot::channel()` |
| Backpressure-driven stream of items | `tokio::sync::mpsc::Receiver` + `ReceiverStream` |
Mpsc pattern:
```rust
let (tx, mut rx) = tokio::sync::mpsc::channel::<Job>(256);
tokio::spawn(async move {
while let Some(job) = rx.recv().await {
if let Err(error) = process(job).await {
tracing::warn!(%error, "job failed");
}
}
tracing::info!("queue closed, shutting down worker");
});
tx.send(Job { ... }).await?; // blocks if full, applies backpressure
```
Always bound channels. Unbounded channels are a memory leak waiting to happen.
## Streams
`futures::Stream` is the async analogue of `Iterator`. Use it for paginated fetches, long-poll responses, file lines.
```rust
use futures::stream::{StreamExt, TryStreamExt};
let urls: Vec<String> = ...;
let bodies: Vec<String> = futures::stream::iter(urls)
.map(|url| async move { fetch(&url).await })
.buffer_unordered(8) // up to 8 in flight
.try_collect()
.await?;
```
`buffer_unordered(n)` is the throttle. Use it instead of spawning N tasks manually.
For producing a stream from a channel:
```rust
use tokio_stream::wrappers::ReceiverStream;
let (tx, rx) = tokio::sync::mpsc::channel::<Event>(64);
let stream = ReceiverStream::new(rx);
serve_sse(stream).await
```
## Graceful shutdown
```rust
use tokio::signal;
async fn shutdown_signal() {
let ctrl_c = async { signal::ctrl_c().await.expect("ctrl_c handler") };
#[cfg(unix)]
let terminate = async {
signal::unix::signal(signal::unix::SignalKind::terminate())
.expect("install signal handler")
.recv()
.await;
};
#[cfg(not(unix))]
let terminate = std::future::pending::<()>();
tokio::select! {
_ = ctrl_c => {},
_ = terminate => {},
}
tracing::info!("shutdown signal received");
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let token = CancellationToken::new();
let server = tokio::spawn(run_server(token.child_token()));
shutdown_signal().await;
token.cancel();
let _ = tokio::time::timeout(Duration::from_secs(10), server).await;
Ok(())
}
```
Pattern: catch signal → cancel a token shared with the server → server's `select!` arms see the cancel and exit cleanly → wait with a timeout so a hung worker can't deadlock shutdown.
## Concurrency primitives
- `tokio::sync::Mutex` — async mutex. Use for state shared between async tasks. **Do not hold across `.await` without thinking** (you'll serialize the whole system).
- `tokio::sync::RwLock` — async read-write lock. Same caveat.
- `parking_lot::Mutex` — sync mutex, faster than `std::sync::Mutex`, no poisoning. Use when the lock is held briefly and you do not need to `.await` while holding it.
- `tokio::sync::Semaphore` — bound concurrent operations. Perfect for "max 10 in-flight HTTP requests" or "max 3 DB writers".
```rust
let sem = Arc::new(tokio::sync::Semaphore::new(10));
for url in urls {
let permit = sem.clone().acquire_owned().await?;
tokio::spawn(async move {
let _permit = permit; // released on task end
fetch(&url).await
});
}
```
## Common mistakes
1. **Holding a sync mutex across `.await`.** Compiles and runs, deadlocks at scale. Solution: refactor to release before await, or use `tokio::sync::Mutex`.
2. **Forgetting `?` on `JoinHandle`.** A panicked task returns `Err(JoinError)`; if you `.await` and ignore, panics are silently swallowed.
3. **`tokio::spawn` instead of `JoinSet`.** Detached tasks survive past their parent, causing leaks. Default to `JoinSet` for structured concurrency.
4. **Unbounded channels.** Always set a capacity.
5. **`block_on` inside an async context.** Causes deadlock under `current_thread` runtime, performance cliff under `multi_thread`.
6. **CPU-heavy work in async fn.** Move to `spawn_blocking` or `rayon`.
7. **No timeout on external I/O.** Every `await` that touches the network or filesystem needs `tokio::time::timeout` wrapping.
## Testing async code
```rust
#[tokio::test]
async fn fetches_and_parses() {
let server = wiremock::MockServer::start().await;
wiremock::Mock::given(wiremock::matchers::method("GET"))
.respond_with(wiremock::ResponseTemplate::new(200).set_body_string("{\"id\":1}"))
.mount(&server)
.await;
let result = my_client::fetch(&server.uri()).await.unwrap();
assert_eq!(result.id, 1);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn parallel_work() { ... }
```
For time-sensitive tests, advance virtual time:
```rust
#[tokio::test(start_paused = true)]
async fn time_travel() {
let start = tokio::time::Instant::now();
tokio::time::sleep(Duration::from_secs(3600)).await;
assert!(start.elapsed() >= Duration::from_secs(3600));
// Real wallclock elapsed: ~0ms.
}
```
## When NOT to use async
- Single-threaded CPU-heavy code that does no I/O — plain `fn` + `rayon` is simpler and often faster.
- Trivial scripts that do one HTTP call — `ureq` (sync) is simpler.
- FFI heavy code where the FFI side is sync.
Async pays off when you have many concurrent I/O operations or need cancellation as a first-class primitive.
packages/shared-skills/skills/programming/references/rust/axum-stack.md
+467
View File
@@ -0,0 +1,467 @@
# axum + sqlx + tracing + tower — HTTP API Stack
The canonical production HTTP service in Rust 2026.
## Cargo.toml dependencies
```toml
[dependencies]
axum = { version = "0.8", features = ["macros", "tracing", "ws", "multipart"] }
tokio = { version = "1", features = ["full"] }
tower = "0.5"
tower-http = { version = "0.6", features = [
"trace", "compression-gzip", "compression-br",
"timeout", "cors", "request-id", "sensitive-headers",
"limit", "set-header",
] }
# Errors / observability
anyhow = "1"
thiserror = "2"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
color-eyre = "0.6"
# Database
sqlx = { version = "0.8", features = [
"runtime-tokio-rustls", "postgres", "uuid", "macros",
"migrate", "json",
] }
# Serialization / validation
serde = { version = "1", features = ["derive"] }
serde_json = "1"
validator = { version = "0.18", features = ["derive"] }
# Types
uuid = { version = "1", features = ["v4", "v7", "serde"] }
jiff = { version = "0.1", features = ["serde"] }
# Config
config = { version = "0.14", features = ["toml", "yaml"] }
secrecy = { version = "0.10", features = ["serde"] }
# OpenAPI (optional but recommended)
utoipa = { version = "5", features = ["axum_extras", "uuid", "chrono"] }
utoipa-axum = "0.1"
utoipa-swagger-ui = { version = "8", features = ["axum"] }
```
## Project structure
```
src/
main.rs # binary entry
lib.rs # re-exports + app builder
config.rs # Settings type + loader
state.rs # AppState (shared via Arc)
routes/
mod.rs # Router::new() composition
health.rs
users.rs
middleware/
mod.rs
auth.rs
request_id.rs
models/
mod.rs
user.rs
error.rs # AppError + IntoResponse impl
db/
mod.rs
migrations/
migrations/ # sqlx migrations
```
## Error type
```rust
// src/error.rs
use axum::{http::StatusCode, response::{IntoResponse, Response}, Json};
use serde_json::json;
#[derive(Debug, thiserror::Error)]
pub enum AppError {
#[error("not found")]
NotFound,
#[error("unauthorized")]
Unauthorized,
#[error("validation: {0}")]
Validation(String),
#[error("conflict: {0}")]
Conflict(String),
#[error("internal")]
Internal(#[from] anyhow::Error),
#[error("database")]
Database(#[from] sqlx::Error),
}
impl IntoResponse for AppError {
fn into_response(self) -> Response {
let (status, code, message) = match &self {
AppError::NotFound => (StatusCode::NOT_FOUND, "not_found", self.to_string()),
AppError::Unauthorized => (StatusCode::UNAUTHORIZED, "unauthorized", "unauthorized".into()),
AppError::Validation(m) => (StatusCode::UNPROCESSABLE_ENTITY, "validation", m.clone()),
AppError::Conflict(m) => (StatusCode::CONFLICT, "conflict", m.clone()),
AppError::Database(e) => {
tracing::error!(error = ?e, "database error");
(StatusCode::INTERNAL_SERVER_ERROR, "database", "internal".into())
}
AppError::Internal(e) => {
tracing::error!(error = ?e, "internal error");
(StatusCode::INTERNAL_SERVER_ERROR, "internal", "internal".into())
}
};
(status, Json(json!({"error": {"code": code, "message": message}}))).into_response()
}
}
pub type AppResult<T> = std::result::Result<T, AppError>;
```
Pattern: business errors return `AppResult<T>`; the `IntoResponse` impl translates them to HTTP. `sqlx::Error` and `anyhow::Error` auto-convert via `From`. Internal-bucket errors are logged but never leak their `Debug` representation to clients.
## AppState
```rust
// src/state.rs
use std::sync::Arc;
use sqlx::PgPool;
#[derive(Clone)]
pub struct AppState {
pub db: PgPool,
pub config: Arc<crate::config::Settings>,
pub http: reqwest::Client,
}
impl AppState {
pub async fn new(config: crate::config::Settings) -> anyhow::Result<Self> {
let db = sqlx::postgres::PgPoolOptions::new()
.max_connections(config.db.max_connections)
.acquire_timeout(std::time::Duration::from_secs(3))
.connect(config.db.url.expose_secret())
.await?;
sqlx::migrate!("./migrations").run(&db).await?;
let http = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(15))
.user_agent(concat!(env!("CARGO_PKG_NAME"), "/", env!("CARGO_PKG_VERSION")))
.build()?;
Ok(Self { db, config: Arc::new(config), http })
}
}
```
## Route handler
```rust
// src/routes/users.rs
use axum::{
extract::{Path, State},
Json,
};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
use validator::Validate;
use crate::{error::{AppError, AppResult}, state::AppState};
#[derive(Debug, Deserialize, Validate)]
pub struct CreateUser {
#[validate(email)]
pub email: String,
#[validate(length(min = 1, max = 100))]
pub name: String,
}
#[derive(Debug, Serialize, sqlx::FromRow)]
pub struct User {
pub id: Uuid,
pub email: String,
pub name: String,
pub created_at: jiff::Timestamp,
}
#[tracing::instrument(skip(state, body))]
pub async fn create_user(
State(state): State<AppState>,
Json(body): Json<CreateUser>,
) -> AppResult<(axum::http::StatusCode, Json<User>)> {
body.validate().map_err(|e| AppError::Validation(e.to_string()))?;
let id = Uuid::now_v7();
let user = sqlx::query_as!(
User,
r#"INSERT INTO users (id, email, name, created_at)
VALUES ($1, $2, $3, NOW())
RETURNING id, email, name, created_at as "created_at: jiff::Timestamp""#,
id, body.email, body.name
)
.fetch_one(&state.db)
.await
.map_err(|e| match &e {
sqlx::Error::Database(db) if db.code().as_deref() == Some("23505") =>
AppError::Conflict("email already exists".into()),
_ => AppError::Database(e),
})?;
Ok((axum::http::StatusCode::CREATED, Json(user)))
}
#[tracing::instrument(skip(state))]
pub async fn get_user(
State(state): State<AppState>,
Path(id): Path<Uuid>,
) -> AppResult<Json<User>> {
sqlx::query_as!(
User,
r#"SELECT id, email, name, created_at as "created_at: jiff::Timestamp"
FROM users WHERE id = $1"#,
id
)
.fetch_optional(&state.db)
.await?
.map(Json)
.ok_or(AppError::NotFound)
}
```
## Router assembly
```rust
// src/routes/mod.rs
use axum::{routing::{get, post}, Router};
use tower_http::{
compression::CompressionLayer,
cors::CorsLayer,
request_id::{MakeRequestUuid, PropagateRequestIdLayer, SetRequestIdLayer},
sensitive_headers::SetSensitiveHeadersLayer,
timeout::TimeoutLayer,
trace::{DefaultMakeSpan, DefaultOnResponse, TraceLayer},
};
use std::time::Duration;
use crate::state::AppState;
mod health;
mod users;
pub fn router(state: AppState) -> Router {
let api = Router::new()
.route("/health", get(health::handler))
.route("/users", post(users::create_user))
.route("/users/:id", get(users::get_user))
.with_state(state);
Router::new()
.nest("/api/v1", api)
.layer(
tower::ServiceBuilder::new()
.layer(SetSensitiveHeadersLayer::new([
axum::http::header::AUTHORIZATION,
axum::http::header::COOKIE,
]))
.layer(SetRequestIdLayer::x_request_id(MakeRequestUuid))
.layer(
TraceLayer::new_for_http()
.make_span_with(DefaultMakeSpan::new().include_headers(false))
.on_response(DefaultOnResponse::new().latency_unit(tower_http::LatencyUnit::Millis)),
)
.layer(PropagateRequestIdLayer::x_request_id())
.layer(TimeoutLayer::new(Duration::from_secs(30)))
.layer(CompressionLayer::new())
.layer(CorsLayer::permissive()), // tighten in production
)
}
```
Order matters: outermost layer wraps the request first. Trace before timeout so timeouts get logged. Compression after trace so trace sees the original body size.
## Main + graceful shutdown
```rust
// src/main.rs
use my_app::{config::Settings, routes, state::AppState};
#[tokio::main]
async fn main() -> color_eyre::Result<()> {
color_eyre::install()?;
init_tracing();
let config = Settings::load()?;
let state = AppState::new(config.clone()).await?;
let app = routes::router(state);
let listener = tokio::net::TcpListener::bind(&config.bind).await?;
tracing::info!(addr = %config.bind, "listening");
axum::serve(listener, app)
.with_graceful_shutdown(shutdown_signal())
.await?;
Ok(())
}
fn init_tracing() {
use tracing_subscriber::{fmt, EnvFilter};
let filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("info,sqlx=warn,hyper=warn,tower_http=info"));
fmt().with_env_filter(filter).with_target(false).json().init();
}
async fn shutdown_signal() {
let ctrl_c = async { tokio::signal::ctrl_c().await.expect("ctrl_c handler"); };
#[cfg(unix)]
let terminate = async {
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
.expect("signal handler").recv().await;
};
#[cfg(not(unix))]
let terminate = std::future::pending::<()>();
tokio::select! { _ = ctrl_c => {}, _ = terminate => {} }
tracing::info!("shutting down");
}
```
## Middleware: bearer auth example
```rust
// src/middleware/auth.rs
use axum::{
extract::{Request, State},
http::header::AUTHORIZATION,
middleware::Next,
response::Response,
};
use crate::{error::AppError, state::AppState};
#[derive(Clone, Debug)]
pub struct AuthUser { pub id: uuid::Uuid }
pub async fn require_auth(
State(state): State<AppState>,
mut request: Request,
next: Next,
) -> Result<Response, AppError> {
let token = request.headers()
.get(AUTHORIZATION)
.and_then(|v| v.to_str().ok())
.and_then(|s| s.strip_prefix("Bearer "))
.ok_or(AppError::Unauthorized)?;
let claims = verify_jwt(token, &state.config.jwt_secret)?;
request.extensions_mut().insert(AuthUser { id: claims.sub });
Ok(next.run(request).await)
}
```
Apply with `.route_layer(middleware::from_fn_with_state(state.clone(), require_auth))` on the subroutes that need it.
## Testing handlers
```rust
// tests/users.rs
#[tokio::test]
async fn creates_user() {
let pool = test_db().await; // helper that spins up a transactional DB
let state = AppState::new_test(pool).await.unwrap();
let app = my_app::routes::router(state);
let request = axum::http::Request::builder()
.uri("/api/v1/users")
.method("POST")
.header("content-type", "application/json")
.body(axum::body::Body::from(
serde_json::to_vec(&serde_json::json!({"email": "a@b.com", "name": "A"})).unwrap()
)).unwrap();
let response = tower::ServiceExt::oneshot(app, request).await.unwrap();
assert_eq!(response.status(), 201);
let bytes = axum::body::to_bytes(response.into_body(), 1 << 20).await.unwrap();
let user: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
assert_eq!(user["email"], "a@b.com");
}
```
`tower::ServiceExt::oneshot` calls the router directly without binding a socket. Tests run in parallel without port collisions.
## Config
```rust
// src/config.rs
use secrecy::{Secret, ExposeSecret};
#[derive(Debug, Clone, serde::Deserialize)]
pub struct Settings {
pub bind: String,
pub db: Database,
pub jwt_secret: Secret<String>,
}
#[derive(Debug, Clone, serde::Deserialize)]
pub struct Database {
pub url: Secret<String>,
pub max_connections: u32,
}
impl Settings {
pub fn load() -> anyhow::Result<Self> {
let cfg = config::Config::builder()
.add_source(config::File::with_name("config/default").required(false))
.add_source(config::File::with_name(&format!(
"config/{}", std::env::var("APP_ENV").unwrap_or_else(|_| "dev".into())
)).required(false))
.add_source(config::Environment::with_prefix("APP").separator("__"))
.build()?;
Ok(cfg.try_deserialize()?)
}
}
```
`Secret<T>` from the `secrecy` crate hides the value in `Debug`/`Display` to prevent accidental log leakage. Access via `.expose_secret()` only where needed.
## OpenAPI (optional)
Add `utoipa` derive macros on your DTOs and handlers, mount Swagger UI at `/swagger-ui`:
```rust
use utoipa::OpenApi;
use utoipa_axum::router::OpenApiRouter;
use utoipa_swagger_ui::SwaggerUi;
#[derive(OpenApi)]
#[openapi(
paths(routes::users::create_user, routes::users::get_user),
components(schemas(routes::users::User, routes::users::CreateUser))
)]
struct ApiDoc;
let (router, api) = OpenApiRouter::with_openapi(ApiDoc::openapi())
.routes(utoipa_axum::routes!(routes::users::create_user, routes::users::get_user))
.split_for_parts();
let app = router.merge(SwaggerUi::new("/swagger-ui").url("/api-docs/openapi.json", api));
```
## Production checklist
- Bind to `0.0.0.0` in containers, `127.0.0.1` for local-only services.
- Set `RUST_LOG=info,sqlx=warn` (or use `EnvFilter` defaults as shown).
- Send logs to stdout in JSON. Ingest via Vector / Fluent Bit / Loki.
- Run migrations on startup (`sqlx::migrate!` block). Fail fast on schema mismatch.
- Health endpoint **must hit the DB** (so load balancers know if the pool is dead).
- Add `tower::limit::RateLimitLayer` or token-bucket middleware for public endpoints.
- Set `tower_http::limit::RequestBodyLimitLayer` to bound request size.
- Compress with brotli + gzip via `CompressionLayer`.
- Tighten CORS, do not ship `CorsLayer::permissive()` to production.
- Strip sensitive headers from traces via `SetSensitiveHeadersLayer`.
- Set up SIGTERM-driven `with_graceful_shutdown` so deploys roll without dropping requests.
- Containerize with `cargo chef` for incremental Docker builds.
## Common mistakes
1. **Forgetting `error_for_status()?` on outbound `reqwest`** — 4xx silently succeeds.
2. **Returning `Result<T, sqlx::Error>` from handlers** — leak DB details to clients. Always go through `AppError`.
3. **`Json<T>` extractor before validation** — invalid JSON returns axum's default 422 with no body shape. Wrap in a `ValidatedJson<T>` extractor that runs `validator` and returns `AppError`.
4. **Holding DB connections across `.await` on slow external calls** — exhausts the pool. Acquire late, release early.
5. **Skipping `tracing::instrument`** on handlers — losing per-request span correlation.
6. **No `RequestBodyLimitLayer`** — DoS surface. Default axum has no limit.
packages/shared-skills/skills/programming/references/rust/cargo-strict.md
+317
View File
@@ -0,0 +1,317 @@
# Cargo Strict Configuration
The exact knobs every new Rust project gets. Drop these in unmodified.
## `rust-toolchain.toml`
```toml
[toolchain]
channel = "stable"
components = ["rustfmt", "clippy", "rust-src"]
profile = "default"
```
Pin nightly separately when miri runs:
```bash
rustup install nightly
rustup component add miri rust-src --toolchain nightly
```
## `Cargo.toml` — `[lints]` section
```toml
[lints.rust]
unsafe_op_in_unsafe_fn = "deny"
missing_docs = "warn"
missing_debug_implementations = "warn"
unreachable_pub = "warn"
unused_must_use = "deny"
elided_lifetimes_in_paths = "warn"
non_ascii_idents = "deny"
trivial_numeric_casts = "warn"
unused_lifetimes = "warn"
single_use_lifetimes = "warn"
[lints.clippy]
# Groups
all = { level = "deny", priority = -1 }
pedantic = { level = "warn", priority = -1 }
nursery = { level = "warn", priority = -1 }
cargo = { level = "warn", priority = -1 }
# Hard denies - turn warnings into errors for sharp tools
undocumented_unsafe_blocks = "deny"
multiple_unsafe_ops_per_block = "deny"
unwrap_used = "deny"
expect_used = "deny"
panic = "deny"
todo = "deny"
unimplemented = "deny"
unreachable = "deny"
indexing_slicing = "deny"
mem_forget = "deny"
arithmetic_side_effects = "warn"
cast_possible_truncation = "warn"
cast_possible_wrap = "warn"
cast_precision_loss = "warn"
cast_sign_loss = "warn"
as_underscore = "deny"
as_ptr_cast_mut = "deny"
ptr_as_ptr = "warn"
borrow_as_ptr = "warn"
fn_to_numeric_cast_any = "deny"
clone_on_ref_ptr = "warn"
mutex_atomic = "warn"
rc_buffer = "warn"
rc_mutex = "warn"
exit = "warn"
allow_attributes_without_reason = "warn"
dbg_macro = "warn"
print_stderr = "warn"
print_stdout = "warn"
use_debug = "warn"
# Stylistic relaxations (project-wide opinions only)
module_name_repetitions = "allow"
must_use_candidate = "allow"
missing_errors_doc = "allow" # we use anyhow::Result with .context() everywhere; doc rule is noisy
# Restriction lints - opt-in soundness rails
unreachable = "deny"
mod_module_files = "warn" # prefer foo.rs over foo/mod.rs
empty_drop = "warn"
empty_structs_with_brackets = "warn"
empty_enum = "warn"
exhaustive_enums = "warn" # public enums should consider #[non_exhaustive]
exhaustive_structs = "warn"
```
The `priority = -1` trick: group-level levels are weak; specific lints below them win. This lets us deny `unwrap_used` while still allowing `pedantic` group warnings instead of denies.
## `Cargo.toml` — release profile
```toml
[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
strip = "symbols"
panic = "abort" # smaller, faster - if you need unwinding (FFI catch), set "unwind"
debug = "line-tables-only"
[profile.dev]
opt-level = 0
debug = true
incremental = true
codegen-units = 256
split-debuginfo = "unpacked"
# A profile for miri - opt-level 1 keeps simulation bearable while still
# exercising real codegen patterns. miri ignores most profile keys but reads
# overflow-checks.
[profile.miri]
inherits = "test"
opt-level = 1
overflow-checks = true
```
## `Cargo.toml` — workspace level
```toml
[workspace]
resolver = "3"
[workspace.package]
edition = "2024"
rust-version = "1.83" # bump only when a needed feature lands
license = "Apache-2.0 OR MIT"
[workspace.lints]
# Then in each member crate:
# [lints]
# workspace = true
```
## `rustfmt.toml`
```toml
edition = "2024"
max_width = 100
imports_granularity = "Module"
group_imports = "StdExternalCrate"
reorder_imports = true
reorder_modules = true
newline_style = "Unix"
use_field_init_shorthand = true
use_try_shorthand = true
unstable_features = false
```
Most options come from stable rustfmt. `imports_granularity` and `group_imports` are nightly-only but ignored cleanly on stable; CI runs `cargo +nightly fmt --check` for the import grouping.
## `clippy.toml`
```toml
# Reduce cognitive load thresholds.
cognitive-complexity-threshold = 25
type-complexity-threshold = 250
too-many-arguments-threshold = 6
too-many-lines-threshold = 100
# msrv - keeps clippy from suggesting features past our MSRV
msrv = "1.83"
# Avoid `panic` lint complaining about derived Debug impls calling unreachable_unchecked etc.
allow-unwrap-in-tests = true
allow-expect-in-tests = true
allow-panic-in-tests = true
allow-dbg-in-tests = true
allow-print-in-tests = true
# Force named arguments above N params
single-char-binding-names-threshold = 4
```
## `deny.toml` (cargo-deny)
```toml
[advisories]
db-path = "~/.cargo/advisory-db"
db-urls = ["https://github.com/rustsec/advisory-db"]
yanked = "deny"
ignore = []
[licenses]
allow = [
"MIT",
"Apache-2.0",
"Apache-2.0 WITH LLVM-exception",
"BSD-2-Clause",
"BSD-3-Clause",
"ISC",
"Unicode-DFS-2016",
"Unicode-3.0",
"Zlib",
"MPL-2.0",
"CC0-1.0",
]
confidence-threshold = 0.93
exceptions = []
[bans]
multiple-versions = "warn"
wildcards = "deny"
highlight = "all"
deny = [
# Pin out unmaintained alternatives
{ name = "async-std", reason = "use tokio" },
{ name = "actix-web", reason = "use axum" },
{ name = "chrono", reason = "use jiff" },
]
[sources]
unknown-registry = "deny"
unknown-git = "warn"
allow-registry = ["https://github.com/rust-lang/crates.io-index"]
```
## CI Workflow (`.github/workflows/ci.yml`)
```yaml
name: ci
on:
push:
branches: [main]
pull_request:
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
jobs:
fmt:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: rustfmt
- run: cargo +nightly fmt --all -- --check
clippy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
components: clippy
- uses: Swatinem/rust-cache@v2
- run: cargo clippy --all-targets --all-features --workspace -- -D warnings
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- uses: taiki-e/install-action@nextest
- run: cargo nextest run --all-targets --all-features --workspace
miri:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: miri, rust-src
- uses: Swatinem/rust-cache@v2
- uses: taiki-e/install-action@nextest
- env:
MIRIFLAGS: "-Zmiri-strict-provenance -Zmiri-symbolic-alignment-check"
run: cargo +nightly miri nextest run --all-features --workspace
machete:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: bnjbvr/cargo-machete@main
deny:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: EmbarkStudios/cargo-deny-action@v2
with:
command: check all
audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: rustsec/audit-check@v1
with:
token: ${{ secrets.GITHUB_TOKEN }}
```
## Project bootstrap
```bash
cargo new --bin my-app --edition 2024
cd my-app
cargo install cargo-nextest cargo-machete cargo-deny cargo-edit cargo-watch
rustup install nightly
rustup component add miri --toolchain nightly
# drop the configs above
git add . && git commit -m "chore: bootstrap strict toolchain"
```
After every change:
```bash
cargo fmt --all -- --check && \
cargo clippy --all-targets --all-features -- -D warnings && \
cargo nextest run && \
cargo +nightly miri nextest run # only if unsafe is involved
```
packages/shared-skills/skills/programming/references/rust/clap-stack.md
+409
View File
@@ -0,0 +1,409 @@
# CLI Stack — clap + color-eyre + tracing + indicatif + dialoguer
The default for any new CLI tool. Strict typing on arguments, beautiful errors, progress feedback, interactive prompts when needed.
## Cargo.toml
```toml
[package]
name = "mytool"
version = "0.1.0"
edition = "2024"
[dependencies]
clap = { version = "4", features = ["derive", "env", "wrap_help", "color", "unicode"] }
clap_complete = "4"
color-eyre = "0.6"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
anyhow = "1"
indicatif = { version = "0.17", features = ["tokio"] }
dialoguer = { version = "0.11", features = ["fuzzy-select"] }
console = "0.15"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "fs", "process", "signal"] }
[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
strip = "symbols"
panic = "abort"
```
## Command structure
```rust
// src/cli.rs
use clap::{Parser, Subcommand, ValueEnum};
use std::path::PathBuf;
#[derive(Debug, Parser)]
#[command(
name = "mytool",
author,
version,
about = "A short description",
long_about = "A longer description that appears in --help",
arg_required_else_help = true,
)]
pub struct Cli {
/// Configuration file path
#[arg(short, long, env = "MYTOOL_CONFIG", default_value = "config.toml", global = true)]
pub config: PathBuf,
/// Increase verbosity (-v info, -vv debug, -vvv trace)
#[arg(short, long, action = clap::ArgAction::Count, global = true)]
pub verbose: u8,
/// Suppress all non-error output
#[arg(short, long, global = true, conflicts_with = "verbose")]
pub quiet: bool,
/// Force colored output even when stdout is not a terminal
#[arg(long, global = true, value_enum, default_value_t = ColorChoice::Auto)]
pub color: ColorChoice,
/// Output format
#[arg(short, long, global = true, value_enum, default_value_t = OutputFormat::Pretty)]
pub format: OutputFormat,
#[command(subcommand)]
pub command: Command,
}
#[derive(Debug, Clone, ValueEnum)]
pub enum ColorChoice { Auto, Always, Never }
#[derive(Debug, Clone, ValueEnum)]
pub enum OutputFormat { Pretty, Json, Plain }
#[derive(Debug, Subcommand)]
pub enum Command {
/// Build the thing
Build(BuildArgs),
/// Watch and rebuild
Watch(WatchArgs),
/// Generate shell completions
Completions { #[arg(value_enum)] shell: clap_complete::Shell },
}
#[derive(Debug, clap::Args)]
pub struct BuildArgs {
/// Target directory
#[arg(short, long, default_value = "target")]
pub target: PathBuf,
/// Build mode
#[arg(short, long, value_enum, default_value_t = Mode::Release)]
pub mode: Mode,
/// Specific files to build (default: all)
pub files: Vec<PathBuf>,
}
#[derive(Debug, clap::Args)]
pub struct WatchArgs {
/// Glob pattern to watch
#[arg(short, long, default_value = "**/*.rs")]
pub pattern: String,
}
#[derive(Debug, Clone, ValueEnum)]
pub enum Mode { Debug, Release }
```
Key clap derive patterns:
- `env = "VAR"` — falls back to env var if flag not given.
- `global = true` — flag inherits to subcommands.
- `arg_required_else_help = true` — running with no args prints help instead of erroring.
- `value_enum` on an enum — case-insensitive parsing + auto-completion.
- `action = clap::ArgAction::Count` — `-v` is 1, `-vv` is 2, etc.
- `conflicts_with` — incompatible flags.
## Main + tracing init
```rust
// src/main.rs
use clap::Parser;
use mytool::cli::{Cli, Command, ColorChoice};
use tracing::Level;
use tracing_subscriber::EnvFilter;
fn main() -> color_eyre::Result<()> {
color_eyre::install()?;
let cli = Cli::parse();
init_tracing(&cli);
if matches!(cli.color, ColorChoice::Always) {
console::set_colors_enabled(true);
} else if matches!(cli.color, ColorChoice::Never) {
console::set_colors_enabled(false);
}
match cli.command {
Command::Build(args) => mytool::commands::build::run(&cli, args),
Command::Watch(args) => mytool::commands::watch::run(&cli, args),
Command::Completions { shell } => {
let mut cmd = <Cli as clap::CommandFactory>::command();
clap_complete::generate(shell, &mut cmd, "mytool", &mut std::io::stdout());
Ok(())
}
}
}
fn init_tracing(cli: &Cli) {
let level = if cli.quiet {
Level::ERROR
} else {
match cli.verbose {
0 => Level::WARN,
1 => Level::INFO,
2 => Level::DEBUG,
_ => Level::TRACE,
}
};
let filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new(format!("mytool={level}")));
tracing_subscriber::fmt()
.with_env_filter(filter)
.with_target(false)
.without_time()
.compact()
.with_writer(std::io::stderr)
.init();
}
```
Tracing on a CLI:
- **Write to stderr.** stdout is for the tool's actual output (which the user might pipe). Logs and progress bars go to stderr.
- **Verbosity from `-v`, not from `RUST_LOG`.** Users expect `-v` on a CLI; `RUST_LOG` is a developer escape hatch (kept, but secondary).
## Progress bars — `indicatif`
```rust
use indicatif::{MultiProgress, ProgressBar, ProgressStyle};
use std::time::Duration;
let mp = MultiProgress::new();
let pb = mp.add(ProgressBar::new(files.len() as u64));
pb.set_style(
ProgressStyle::with_template(
"{spinner:.green} [{elapsed_precise}] [{wide_bar:.cyan/blue}] {pos}/{len} ({eta}) {msg}"
)?
.progress_chars("=>-")
);
for file in files {
pb.set_message(file.display().to_string());
process(&file)?;
pb.inc(1);
}
pb.finish_with_message("done");
```
For unbounded operations:
```rust
let spinner = ProgressBar::new_spinner();
spinner.enable_steady_tick(Duration::from_millis(80));
spinner.set_message("connecting…");
let result = connect().await?;
spinner.finish_and_clear();
```
With multiple parallel tasks:
```rust
let mp = MultiProgress::new();
let bars: Vec<_> = (0..workers).map(|i| {
let pb = mp.add(ProgressBar::new(unit));
pb.set_style(ProgressStyle::with_template("worker {prefix}: {pos}/{len}")?);
pb.set_prefix(i.to_string());
pb
}).collect();
```
`MultiProgress` keeps bars stacked and redraws cleanly even with concurrent updates from multiple tasks.
When stdout is not a terminal, indicatif silently disables animation. Force on/off with `pb.set_draw_target(ProgressDrawTarget::stdout())` / `hidden()`.
## Interactive prompts — `dialoguer`
```rust
use dialoguer::{theme::ColorfulTheme, Confirm, Input, Password, Select, FuzzySelect, MultiSelect};
let name: String = Input::with_theme(&ColorfulTheme::default())
.with_prompt("Project name")
.validate_with(|input: &String| -> Result<(), &str> {
if input.chars().all(|c| c.is_alphanumeric() || c == '-' || c == '_') {
Ok(())
} else {
Err("alphanumeric, dash, underscore only")
}
})
.interact_text()?;
let secret = Password::with_theme(&ColorfulTheme::default())
.with_prompt("API key")
.with_confirmation("Repeat", "passwords don't match")
.interact()?;
let go: bool = Confirm::with_theme(&ColorfulTheme::default())
.with_prompt(format!("Delete {}? This cannot be undone.", path.display()))
.default(false)
.interact()?;
if !go { return Ok(()); }
let items = ["yes", "no", "maybe"];
let idx = Select::with_theme(&ColorfulTheme::default())
.with_prompt("Pick one")
.items(&items)
.default(0)
.interact()?;
let picks = MultiSelect::with_theme(&ColorfulTheme::default())
.with_prompt("Toggle features")
.items(&["alpha", "beta", "gamma"])
.defaults(&[true, false, false])
.interact()?;
```
Detect non-TTY before prompting:
```rust
if !console::user_attended() {
return Err(anyhow::anyhow!("input required but stdin is not a terminal"));
}
```
For automated tests, expose a `--non-interactive` flag and gate all prompts behind it.
## Structured output
```rust
match cli.format {
OutputFormat::Json => {
serde_json::to_writer(std::io::stdout().lock(), &result)?;
println!();
}
OutputFormat::Plain => {
for row in &result.rows {
println!("{}\t{}\t{}", row.a, row.b, row.c);
}
}
OutputFormat::Pretty => {
use console::{style, Term};
let term = Term::stdout();
for row in &result.rows {
term.write_line(&format!(
"{} {} {}",
style(&row.a).green(),
style(&row.b).yellow(),
style(&row.c).dim(),
))?;
}
}
}
```
Always offer `--format json` for piping into `jq`, scripts, and other tools.
## Shell completions
Already shown in the `Completions` subcommand above. Distribute completions by adding to the install script:
```bash
mytool completions bash > /etc/bash_completion.d/mytool
mytool completions fish > ~/.config/fish/completions/mytool.fish
mytool completions zsh > "${fpath[1]}/_mytool"
```
## Signal handling
```rust
// In an async CLI command
use tokio::signal::ctrl_c;
tokio::select! {
_ = ctrl_c() => {
tracing::warn!("interrupted, cleaning up");
cleanup().await?;
std::process::exit(130); // standard exit code for SIGINT
}
result = long_running_task() => {
result
}
}
```
For sync CLIs, install a one-shot handler with `ctrlc` crate:
```rust
let interrupted = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
let i = interrupted.clone();
ctrlc::set_handler(move || i.store(true, std::sync::atomic::Ordering::SeqCst))?;
while !interrupted.load(std::sync::atomic::Ordering::Relaxed) {
do_step()?;
}
```
## Error reporting with color-eyre
```rust
fn main() -> color_eyre::Result<()> {
color_eyre::config::HookBuilder::default()
.display_env_section(false) // hide SPANTRACE/BACKTRACE env hints by default
.display_location_section(false) // hide file:line section
.panic_section("If this is a bug, please report at https://github.com/me/mytool/issues")
.install()?;
real_main()
}
```
Errors with `.wrap_err("...")` from `eyre::WrapErr` (compatible with anyhow's `.context`) show as a numbered chain. `RUST_BACKTRACE=1` shows the full trace; `RUST_SPANTRACE=1` shows tracing spans where the error fired.
## Distribution
- Add `cargo dist init` for prebuilt binary release pipeline (cross-platform tarballs + installers).
- Publish to Homebrew tap, AUR, scoop, Chocolatey via dist.
- Sign Linux binaries with `cosign` if your audience is enterprise.
- Build single static binary on Linux with `--target x86_64-unknown-linux-musl` (or `aarch64-unknown-linux-musl`).
- For wasm-runnable CLIs (`wasi-cli`), add `--target wasm32-wasip1`.
## Common mistakes
1. **Mixing stdout and stderr.** Tool output goes to stdout; logs and progress go to stderr.
2. **No `--non-interactive` flag.** Interactive prompts block automation.
3. **Printing colored output unconditionally.** Honor `NO_COLOR` env var, detect TTY with `console::user_attended()`.
4. **`println!` for errors.** Use `tracing::error!` so logs go to stderr automatically and respect verbosity.
5. **`unwrap()` on `Cli::parse()`.** clap returns clean errors with `--help` text; `parse()` exits on its own.
6. **Long subcommand handlers in `main.rs`.** Split into `src/commands/<name>.rs` per command.
7. **Missing exit code semantics.** Use `std::process::exit(1)` (general error), `2` (usage), `130` (SIGINT) appropriately. Or return `Result` and let main map.
## Testing CLIs
```rust
// tests/cli.rs
use assert_cmd::Command;
use predicates::prelude::*;
#[test]
fn shows_help() {
Command::cargo_bin("mytool").unwrap()
.arg("--help")
.assert()
.success()
.stdout(predicate::str::contains("Usage:"));
}
#[test]
fn rejects_unknown_subcommand() {
Command::cargo_bin("mytool").unwrap()
.arg("nope")
.assert()
.failure()
.stderr(predicate::str::contains("unrecognized subcommand"));
}
```
`assert_cmd` builds the binary once per test run and gives a fluent assertion API.
packages/shared-skills/skills/programming/references/rust/concurrency.md
+375
View File
@@ -0,0 +1,375 @@
# Concurrency Primitives
Locks, atomics, channels, and the loom model checker. The decision tree that keeps the agent out of soundness trouble.
## The pyramid
```
Highest level tokio::sync::mpsc / broadcast / watch
(message passing — default for new code)
Arc<Mutex<T>> / Arc<RwLock<T>>
(shared mutable state — common, easy to get right)
parking_lot::{Mutex, RwLock, Condvar}
(faster sync locks, no poisoning)
Atomics (AtomicUsize, AtomicBool, AtomicPtr)
(single-word lock-free state)
Lowest level UnsafeCell + unsafe + loom + miri
(custom lock-free / wait-free primitives)
```
**Always start at the top.** Drop a level only when you have measured a real bottleneck.
## Decision tree
```
Need to share state between tasks?
├── State is configuration (read-only after start)
│ └── Arc<Config> (no lock needed)
├── State is a queue of work
│ └── tokio::sync::mpsc::channel(cap)
├── State is "latest value" published to many readers
│ └── tokio::sync::watch::channel(initial)
├── State is broadcast (every consumer sees every value)
│ └── tokio::sync::broadcast::channel(cap)
├── State is request-response within one task tree
│ └── tokio::sync::oneshot::channel()
├── State is a counter
│ └── AtomicU64 (or AtomicUsize)
├── State is a flag / set-once
│ └── AtomicBool / OnceLock<T> / OnceCell<T>
├── State needs mutation across many tasks/threads, cheap critical sections
│ ├── async context → tokio::sync::Mutex<T>
│ └── sync context (no .await held) → parking_lot::Mutex<T>
├── State needs mutation, many readers, few writers
│ ├── async context → tokio::sync::RwLock<T>
│ └── sync context → parking_lot::RwLock<T>
└── State is a custom lock-free primitive (channels, hazard pointers)
└── UnsafeCell + atomics + loom-tested + miri-tested + a co-author
```
## Atomics — when and how
Use atomics for:
- Counters incremented from many threads (`AtomicU64`).
- Single-shot flags (`AtomicBool`).
- Pointer publication (`AtomicPtr<T>`).
### Memory orderings
```rust
use std::sync::atomic::{AtomicUsize, Ordering};
let c = AtomicUsize::new(0);
// Just need a count, no synchronization with other data
c.fetch_add(1, Ordering::Relaxed);
// Reading a counter that was incremented from elsewhere
let n = c.load(Ordering::Relaxed);
```
| Ordering | When |
|---|---|
| `Relaxed` | Standalone counters, no other memory needs to be synchronized. |
| `Acquire` (loads) / `Release` (stores) | Publish/consume pattern: you write some data then release a flag, readers acquire the flag then read the data. |
| `AcqRel` | RMW that both reads-and-publishes (e.g., `fetch_add` on a sequence number). |
| `SeqCst` | Total ordering across all `SeqCst` ops. Strongest, slowest. Use when in doubt and switch to a weaker ordering after testing under loom. |
**Default to `SeqCst` if unsure.** Performance difference is usually negligible. Going weaker requires loom.
### Publish-then-load pattern
```rust
static READY: AtomicBool = AtomicBool::new(false);
static mut DATA: Option<Config> = None;
// Producer thread:
unsafe { DATA = Some(load_config()); }
READY.store(true, Ordering::Release);
// Consumer thread:
if READY.load(Ordering::Acquire) {
// SAFETY: producer's Release pairs with our Acquire; if we see READY=true,
// we are guaranteed to also see the DATA write that happened-before it.
let cfg = unsafe { DATA.as_ref().unwrap() };
}
```
This is the canonical Release/Acquire pattern. **Use `OnceLock<Config>` instead** in new code — it encapsulates exactly this with safe API.
## Std vs parking_lot vs tokio for locks
| | std::sync::Mutex | parking_lot::Mutex | tokio::sync::Mutex |
|---|---|---|---|
| Speed | Slowest (OS futex direct) | Fastest (smarter parking) | Slow (await-aware) |
| Poisoning | Yes (`PoisonError`) | No | No |
| Hold across `.await` | Dangerous (deadlock under current-thread runtime) | Dangerous | Safe |
| Drop guard releases | Yes | Yes | Yes |
| RAII | Yes (`MutexGuard`) | Yes | Yes |
| Const constructor | Yes (since 1.63) | Yes | No |
| Async | No | No | Yes |
**Rule of thumb:**
- Hot, short critical section, no await inside → `parking_lot::Mutex`.
- Shared state held across `.await` → `tokio::sync::Mutex`.
- Static init / app config → `OnceLock` or `LazyLock`.
- Avoid `std::sync::Mutex` for new code; the poisoning behavior is more annoying than useful and `parking_lot` is strictly faster.
### Common deadlock — async + sync mutex
```rust
let m = std::sync::Mutex::new(0u64);
let guard = m.lock().unwrap();
something_async().await; // ❌ guard is held across await
*guard += 1;
```
Under `current_thread` runtime this deadlocks (the future suspends while holding the lock; another future on the same thread tries to acquire, blocks the executor). Under `multi_thread` it works but serializes the system.
Fix:
```rust
{
let mut guard = m.lock().unwrap();
*guard += 1;
} // guard released
something_async().await;
```
Or switch to `tokio::sync::Mutex` whose guard is `Send` across awaits.
## Channels
### Mpsc — the workhorse
```rust
let (tx, mut rx) = tokio::sync::mpsc::channel::<Job>(256);
tokio::spawn(async move {
while let Some(job) = rx.recv().await {
process(job).await;
}
});
tx.send(Job::new()).await?; // backpressure: awaits if full
```
Capacity is the backpressure budget. **Never `unbounded_channel()`** unless you have a hard upper bound elsewhere; otherwise it is a slow-leak memory bomb.
### Watch — latest-value pubsub
```rust
let (tx, mut rx) = tokio::sync::watch::channel(Config::default());
// Producer:
tx.send(new_config)?;
// Consumer:
loop {
rx.changed().await?;
let cfg = rx.borrow();
apply(&cfg);
}
```
Receivers see only the latest value (older updates are dropped). Perfect for config reload, leadership changes, "current time" propagation.
### Broadcast — fanout queue
```rust
let (tx, _) = tokio::sync::broadcast::channel::<Event>(1024);
let mut rx1 = tx.subscribe();
let mut rx2 = tx.subscribe();
while let Ok(event) = rx1.recv().await {
// ...
}
```
Each subscriber has its own buffer. If a subscriber falls behind by more than the buffer size, it gets `RecvError::Lagged(n)` and skips messages. Decide explicitly: log + continue, or drop the subscriber and reconnect.
### Oneshot — single value
```rust
let (tx, rx) = tokio::sync::oneshot::channel::<Response>();
worker.send(Request { reply: tx }).await?;
let response = rx.await?;
```
The standard request/response pattern over an actor.
## Semaphores
Bound concurrent operations:
```rust
let sem = Arc::new(tokio::sync::Semaphore::new(10));
for task in tasks {
let permit = sem.clone().acquire_owned().await?;
tokio::spawn(async move {
let _hold = permit; // released when task exits
process(task).await
});
}
```
Use cases:
- "Max 10 outbound HTTP requests in flight."
- "Max 3 DB connections doing writes."
- "Max N tokio tasks running heavy CPU."
A semaphore with `permits=1` is a mutex. Use the actual `Mutex` for that — clearer intent.
## Arc and Rc
`Arc<T>` for cross-thread shared ownership, `Rc<T>` for single-thread (never spans threads).
```rust
let shared = Arc::new(BigData::new());
for _ in 0..workers {
let s = shared.clone();
tokio::spawn(async move { use_data(&s).await });
}
```
`Arc::clone(&s)` is just a reference-count increment; the data is not copied.
**Do not clone in hot loops** if you can pass a reference. `&Arc<T>` is fine to pass; only call `Arc::clone` when you need to move ownership across a thread/task boundary.
`Weak<T>` for back-references in graphs / parent pointers to avoid cycles.
## Once-init primitives
```rust
use std::sync::{OnceLock, LazyLock};
// Lazy initialization, computed on first read
static CONFIG: LazyLock<Config> = LazyLock::new(|| Config::load_from_env().unwrap());
fn get_config() -> &'static Config {
&CONFIG
}
// One-shot publication, set explicitly
static DB: OnceLock<sqlx::PgPool> = OnceLock::new();
#[tokio::main]
async fn main() {
let pool = sqlx::PgPool::connect(&env_url()).await.unwrap();
DB.set(pool).expect("only set once");
// Now everywhere: DB.get().unwrap()
}
```
`OnceLock` is `std::sync` and stable. `LazyLock` is in `std::sync` since 1.80. Avoid the older `once_cell` crate for new code.
## Loom — model-checking lock-free code
When `unsafe` participates in a concurrent algorithm, miri's single-thread model is insufficient. Loom exhaustively explores thread interleavings.
`Cargo.toml`:
```toml
[target.'cfg(loom)'.dev-dependencies]
loom = "0.7"
```
In code, switch between real and loom primitives:
```rust
#[cfg(loom)]
use loom::sync::atomic::{AtomicUsize, Ordering};
#[cfg(not(loom))]
use std::sync::atomic::{AtomicUsize, Ordering};
#[cfg(loom)]
use loom::sync::Arc;
#[cfg(not(loom))]
use std::sync::Arc;
```
Write a test:
```rust
#[cfg(loom)]
mod loom_tests {
use super::*;
use loom::thread;
#[test]
fn concurrent_push_pop_preserves_order() {
loom::model(|| {
let queue = Arc::new(MyQueue::new());
let q1 = queue.clone();
let q2 = queue.clone();
let h1 = thread::spawn(move || q1.push(1));
let h2 = thread::spawn(move || q2.pop());
h1.join().unwrap();
h2.join().unwrap();
// Assert the invariant: queue is in a coherent state.
});
}
}
```
Run:
```bash
RUSTFLAGS="--cfg loom" cargo test --release -- --test-threads 1
```
Loom explores every legal scheduling of the threads, including those a real scheduler would rarely produce. If your code has a race, loom will find it deterministically.
### Loom's limits
- Slow. Each `loom::model` invocation explores many schedules; keep tests tiny (2-3 threads, a few operations each).
- Single-machine only. Doesn't model distributed systems.
- Doesn't catch UB inside `unsafe` blocks the way miri does. **Run both: miri for memory safety, loom for thread schedules.**
- Doesn't handle `tokio` directly. Loom replaces stdlib's sync primitives; tokio's are independent.
## Send and Sync — what they mean
- `T: Send` — `T` can be moved between threads safely.
- `T: Sync` — `&T` can be shared between threads safely.
These are auto-derived for composite types if all components implement them. Manual `unsafe impl Send/Sync` is required only for raw pointer types and FFI handles.
```rust
struct MyHandle { raw: *mut FfiObject }
// SAFETY: FfiObject's documented contract states that move-between-threads
// is safe as long as concurrent use is externally synchronized. We do not
// implement Sync because the FFI object is single-threaded once obtained.
unsafe impl Send for MyHandle {}
// Do NOT impl Sync — the FFI is not thread-safe.
```
When the compiler complains that "T: Send is not satisfied", the cause is usually a raw pointer, an `Rc` (not `Arc`), or a `RefCell` (use `Mutex`).
## Common mistakes
1. **Holding a `std::sync::Mutex` guard across `.await`.** Compiles, deadlocks at runtime under `current_thread`.
2. **`Arc::clone` in a tight loop.** Refcount bump is cheap but not free; pass `&Arc<T>` when possible.
3. **`Mutex<HashMap<K, V>>` for hot reads.** Switch to `RwLock` or `Arc<dashmap::DashMap>`.
4. **Atomic operations with `Ordering::Relaxed` for happens-before publication.** You need `Release`/`Acquire`. Run under loom to be sure.
5. **Unbounded channels.** Always set capacity. If you "know it won't backlog", you don't, and it will.
6. **Spawning detached tokio tasks for fire-and-forget cleanup.** Use `JoinSet` so panics surface.
7. **`std::mem::transmute` to fake `Send`/`Sync`.** Use `unsafe impl` with a SAFETY comment instead. Transmute breaks Stacked Borrows and miri.
8. **Locking order inversion across two mutexes.** Always acquire in a globally consistent order. For more than three locks, switch to a single mutex around a struct.
## When to escape to lock-free
You should reach for atomics + `UnsafeCell` only when:
1. The hot path is **measured** to be bottlenecked on lock contention.
2. There is no existing library (crossbeam, atomic-queue, hazardous) that solves your problem.
3. You can write loom tests that pass.
4. You can write miri tests that pass.
5. You have at least one other engineer who can review the algorithm.
Practically all "I want to write a lock-free queue" projects fail (3) or (4). When in doubt, take the lock and move on.
packages/shared-skills/skills/programming/references/rust/libraries.md
+439
View File
@@ -0,0 +1,439 @@
# Library Defaults — Full Decision Tree
The opinionated, audited-in-prod stack for 2026 Rust. Every entry has a one-line rationale and a canonical code snippet so the agent does not have to relearn each library's idioms.
## Async runtime — `tokio`
The default. Use `tokio` for new work. Multi-thread runtime unless you have a measured reason to go single-thread.
```rust
#[tokio::main(flavor = "multi_thread", worker_threads = 8)]
async fn main() -> anyhow::Result<()> {
tracing_subscriber::fmt::init();
run().await
}
```
Avoid:
- `async-std` — unmaintained, last release ages ago. crates.io download counts are misleading because of historical inertia.
- `smol` — fine for embedded-ish niches; outside that, the ecosystem is on tokio.
- Mixing runtimes in one binary. Pick one and stay.
## Errors — `anyhow` (apps) + `thiserror` (libs)
Application boundaries get `anyhow::Error` with `.context("...")` at every layer that adds meaning. Libraries expose `#[derive(thiserror::Error)]` enums with `#[non_exhaustive]`.
```rust
// Application code
use anyhow::Context as _;
pub async fn load_config(path: &Path) -> anyhow::Result<Config> {
let text = tokio::fs::read_to_string(path)
.await
.with_context(|| format!("reading config from {}", path.display()))?;
let cfg: Config = toml::from_str(&text)
.with_context(|| format!("parsing config at {}", path.display()))?;
Ok(cfg)
}
// Library code
#[derive(Debug, thiserror::Error)]
#[non_exhaustive]
pub enum ParseError {
#[error("expected {expected}, found {found} at position {position}")]
Mismatch { expected: &'static str, found: String, position: usize },
#[error("unexpected end of input after {context}")]
UnexpectedEof { context: &'static str },
#[error(transparent)]
Io(#[from] std::io::Error),
}
```
`#[non_exhaustive]` on enums prevents downstream `match` from breaking when you add variants. `#[error(transparent)]` on a wrapper variant forwards Display + cause to the inner error.
## CLI — `clap` with derive
```rust
use clap::{Parser, Subcommand};
#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
struct Cli {
/// Path to the config file
#[arg(short, long, env = "MYAPP_CONFIG", default_value = "config.toml")]
config: PathBuf,
/// Enable verbose output (-v, -vv, -vvv)
#[arg(short, long, action = clap::ArgAction::Count)]
verbose: u8,
#[command(subcommand)]
command: Command,
}
#[derive(Subcommand, Debug)]
enum Command {
/// Run the server
Serve {
#[arg(short, long, default_value_t = 8080)]
port: u16,
},
/// Migrate the database
Migrate {
#[arg(long)]
dry_run: bool,
},
}
```
Avoid `structopt` (deprecated, merged into clap), `argh` (less ergonomic), `pico-args` (only when binary size matters more than DX).
## Logging — `tracing` + `tracing-subscriber`
Not `log` + `env_logger`. `tracing` supports spans (structured context that follows async tasks) and structured fields - `log` cannot.
```rust
use tracing::{info, instrument, warn, Level};
use tracing_subscriber::{fmt, EnvFilter};
fn init_tracing() {
let filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("info,sqlx=warn,hyper=warn"));
fmt()
.with_env_filter(filter)
.with_target(false)
.with_thread_ids(true)
.with_line_number(true)
.compact()
.init();
}
#[instrument(skip(db), fields(user_id = %user.id))]
async fn process_user(db: &Pool, user: &User) -> anyhow::Result<()> {
info!("processing user");
if user.is_banned() {
warn!(reason = "banned", "skipping");
return Ok(());
}
// ... body ...
Ok(())
}
```
Replace `println!` with `info!`/`warn!`/`error!`. Replace `eprintln!` with `tracing::error!`.
## Error reporting (binaries) — `color-eyre`
For binary `main()`, hook `color-eyre` to give pretty panics + nice `Result` printing:
```rust
fn main() -> color_eyre::Result<()> {
color_eyre::install()?;
tracing_subscriber::fmt::init();
real_main()
}
```
Library code stays on `anyhow`/`thiserror`. `color-eyre` is purely a display layer for the binary.
## Serialization — `serde` + `serde_json`
The default for any data crossing a process boundary (file, network, IPC, database column).
```rust
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields, rename_all = "snake_case")]
pub struct ApiResponse {
pub user_id: UserId,
pub created_at: jiff::Timestamp,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub display_name: Option<String>,
#[serde(flatten)]
pub extra: HashMap<String, serde_json::Value>,
}
```
`deny_unknown_fields` catches typos in inputs. `rename_all = "snake_case"` aligns with REST/JSON conventions while keeping idiomatic Rust field names. `#[serde(flatten)]` for forward-compatible extra fields.
Alternatives:
- `serde_yaml` (YAML — note: YAML's "deserialize anything" surface is a security trap; prefer JSON/TOML where possible)
- `toml` (config files)
- `rmp-serde` (MessagePack — binary, fast)
- `ciborium` (CBOR)
- `bincode 2` (binary, smaller; no serde required in v2 but interop fine)
## HTTP client — `reqwest`
```rust
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.user_agent(concat!(env!("CARGO_PKG_NAME"), "/", env!("CARGO_PKG_VERSION")))
.https_only(true)
.pool_max_idle_per_host(8)
.build()?;
#[derive(serde::Deserialize)]
struct Repo { full_name: String, stargazers_count: u64 }
let repo: Repo = client
.get("https://api.github.com/repos/rust-lang/rust")
.send().await?
.error_for_status()?
.json().await?;
```
`error_for_status()?` turns 4xx/5xx into `Err`. Always include a User-Agent. `https_only(true)` is a soundness toggle - prevents accidental http:// downgrade.
## Web framework — `axum`
```rust
use axum::{Router, routing::get, extract::State, response::Json};
use std::sync::Arc;
#[derive(Clone)]
struct AppState { db: sqlx::PgPool }
async fn health(State(state): State<Arc<AppState>>) -> Json<serde_json::Value> {
let ok = sqlx::query_scalar!("SELECT 1::int4").fetch_one(&state.db).await.is_ok();
Json(serde_json::json!({ "ok": ok }))
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
tracing_subscriber::fmt::init();
let state = Arc::new(AppState { db: sqlx::PgPool::connect(&env_db()).await? });
let app = Router::new()
.route("/health", get(health))
.with_state(state)
.layer(tower_http::trace::TraceLayer::new_for_http())
.layer(tower_http::compression::CompressionLayer::new());
let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await?;
axum::serve(listener, app).await?;
Ok(())
}
```
Avoid `actix-web` (legacy patterns, separate runtime model), `warp` (filter explosion in non-trivial apps), `rocket` (slow release cadence). Pair `axum` with `tower-http` for middleware (trace, compression, CORS, timeout, request-id).
## Database — `sqlx` (compile-time checked SQL)
```rust
use sqlx::PgPool;
#[derive(Debug, sqlx::FromRow)]
pub struct User { pub id: uuid::Uuid, pub email: String, pub created_at: jiff::Timestamp }
pub async fn find_user(pool: &PgPool, email: &str) -> Result<Option<User>, sqlx::Error> {
sqlx::query_as!(
User,
r#"SELECT id, email, created_at as "created_at: jiff::Timestamp"
FROM users WHERE email = $1"#,
email
)
.fetch_optional(pool)
.await
}
```
`query_as!` checks the SQL against the live database at compile time. To work without a live DB during builds, generate offline metadata: `cargo sqlx prepare`. Commit the resulting `.sqlx/` directory.
Avoid `diesel` (sync-first, heavy DSL), raw `tokio-postgres` (loses type checks), `sea-orm` (more magic, less control).
For migrations: `sqlx migrate add <name>` + `sqlx::migrate!("./migrations").run(&pool).await?`.
## Time — `jiff`
The 2025+ choice. Single crate, sane defaults, civil time / instant / span distinction.
```rust
use jiff::{Timestamp, Span, ToSpan, Zoned};
let now: Timestamp = Timestamp::now();
let in_one_hour = now.checked_add(1.hour())?;
let local: Zoned = now.in_tz("Asia/Seoul")?;
let span: Span = local - some_earlier.in_tz("Asia/Seoul")?;
```
Avoid `chrono` (old API, generic-heavy, time zone story still painful), `time` crate (split ecosystem, weaker docs). `jiff` is the post-`chrono` consolidation.
## UUID — `uuid` with v7
```rust
use uuid::Uuid;
// v7 for IDs (sortable, time-ordered, monotonic-ish, RFC 9562)
let id = Uuid::now_v7();
```
v4 is fine for nonces, v7 for primary keys (better index locality). Never v1 (leaks MAC). Cargo features: `uuid = { version = "1", features = ["v4", "v7", "serde"] }`.
## DataFrames / analytics — `polars`
For columnar data, joins, group-by, lazy plans:
```rust
use polars::prelude::*;
let df = LazyCsvReader::new("events.csv")
.finish()?
.group_by([col("user_id")])
.agg([col("amount").sum().alias("total")])
.sort(["total"], Default::default())
.collect()?;
```
The Rust API mirrors the Python one. Use the lazy API by default; materialize with `.collect()` at the end.
## Channels
- Single-producer single-consumer or bounded MPSC → `tokio::sync::mpsc` (async) or `flume` (sync + async).
- Broadcast → `tokio::sync::broadcast`.
- Watch (latest-value pubsub) → `tokio::sync::watch`.
- Oneshot → `tokio::sync::oneshot`.
Avoid raw `std::sync::mpsc` (sync only, fewer features), `crossbeam-channel` (good but heavier; use only if you need rendezvous semantics).
## Coordinate spaces / 2D math — `euclid`
```rust
use euclid::{Point2D, Size2D, default::Box2D};
struct ScreenSpace;
struct WorldSpace;
type ScreenPoint = Point2D<f32, ScreenSpace>;
type WorldPoint = Point2D<f32, WorldSpace>;
let cursor: ScreenPoint = Point2D::new(120.0, 240.0);
let player: WorldPoint = Point2D::new(3.5, 1.2);
// let mistake = cursor + player; // ❌ type error
```
Generalize the pattern to your own domains (see `references/type-state.md`).
## Property tests — `proptest`
```rust
use proptest::prelude::*;
proptest! {
#[test]
fn parse_roundtrips(s in r"[a-zA-Z0-9_-]{1,50}") {
let parsed = parse(&s).unwrap();
let back = parsed.to_string();
prop_assert_eq!(back, s);
}
}
```
Avoid `quickcheck` (older, less ergonomic). proptest gives shrinking + regression corpus + integration with `criterion`.
## Snapshot tests — `insta`
```rust
#[test]
fn renders_help() {
let output = render(&example_input());
insta::assert_snapshot!(output);
}
#[test]
fn serializes_well() {
insta::assert_json_snapshot!(serializable_value());
}
```
`cargo insta review` (after `cargo install cargo-insta`) — interactive review of changed snapshots.
## Benchmarks — `criterion`
Stable Rust friendly (no nightly `#[bench]`).
```rust
use criterion::{black_box, criterion_group, criterion_main, Criterion};
fn bench_parse(c: &mut Criterion) {
let input = std::fs::read_to_string("samples/large.txt").unwrap();
c.bench_function("parse_large", |b| b.iter(|| parse(black_box(&input))));
}
criterion_group!(benches, bench_parse);
criterion_main!(benches);
```
Run with `cargo bench`. HTML reports under `target/criterion/`. Pair with `cargo bench -- --save-baseline main` then `--baseline main` for comparison.
## Concurrency model — `loom`
For lock-free or atomic-heavy code (channels, refcounts, hazard pointers). See `references/concurrency.md` for the full pattern.
## Arena allocator — `bumpalo`
```rust
use bumpalo::Bump;
let bump = Bump::new();
let node = bump.alloc(Node { value: 42, next: None });
let s: &str = bump.alloc_str("hello");
// All allocations freed at once when `bump` drops.
```
For parser nodes, AST construction, per-request scratch. Outperforms heap allocation for short-lived owned data by an order of magnitude.
## Web client (browser, WASM-bound) — `gloo` ecosystem
If targeting WASM browser, use `gloo-net` for fetch and `gloo-storage` for localStorage; not `web-sys` directly unless you need DOM-level APIs.
## Lazy statics — `std::sync::LazyLock` (since 1.80)
```rust
use std::sync::LazyLock;
static CONFIG: LazyLock<Config> = LazyLock::new(|| Config::load_from_env().unwrap());
```
Avoid `lazy_static!` (macro-heavy, predates std), `once_cell` (now in std as `LazyLock`/`OnceLock`).
## Hash maps — `std::collections::HashMap` + `ahash` for hot paths
```rust
use std::collections::HashMap;
use ahash::RandomState;
type FastMap<K, V> = HashMap<K, V, RandomState>;
let mut counters: FastMap<String, u64> = FastMap::default();
```
`HashMap` defaults to SipHash (DoS-resistant). For internal hot loops where you trust the keys, `ahash` is 2-5x faster.
For sorted iteration, use `BTreeMap`. For small keys with known small N, `Vec<(K, V)>` may beat both.
## File I/O — `tokio::fs` (async) or `std::fs` (sync utility)
```rust
let contents = tokio::fs::read_to_string("data.json").await?;
```
For large files: `tokio::fs::File` + `tokio::io::BufReader`. For random access, `memmap2` (with the unsafe-discipline wrappers).
## Decision tree
```
Need to ship the thing?
├── HTTP server → axum + sqlx + tracing + jiff + tokio
├── HTTP client → reqwest (+ tokio)
├── CLI → clap + color-eyre + tracing + indicatif (progress) + dialoguer (prompts)
├── TUI → ratatui + crossterm
├── Background worker → tokio + ETL → polars + duckdb
├── Game / graphics → wgpu + winit + euclid (or bevy if you want the engine)
├── WASM front-end → leptos (or dioxus / yew) + wasm-bindgen + gloo
├── Embedded → embassy (async on bare metal)
├── FFI to C / Python → cxx (C++) / pyo3 (Python) / cbindgen (header gen)
└── Just a script → rust-script (see one-liners.md)
```
When in doubt, search crates.io for the latest version, then check:
1. Is it maintained? (`cargo deny check` will scream if it's yanked or unmaintained)
2. Does it have `serde` feature? (boundary types should always serde)
3. Does it have `tokio` integration? (avoid runtime mixing)
4. Is it on `tokio::io::AsyncRead`/`AsyncWrite` (the std for async I/O)?
5. Are there safety-critical `unsafe` regions? If yes, has the author shipped miri proofs?
packages/shared-skills/skills/programming/references/rust/one-liners.md
+291
View File
@@ -0,0 +1,291 @@
# One-Liners and Disposable Scripts
Production hygiene with throwaway ergonomics. Rust scripts get the same strict lints, the same miri rule when `unsafe` is touched, the same type discipline. The difference is dependency declaration lives inline.
## `rust-script` — the recommended path
Install once:
```bash
cargo install rust-script
```
Write a script:
```rust
#!/usr/bin/env rust-script
//! Fetch a URL and print its body length.
//!
//! Usage:
//! ./fetch.rs <url>
//!
//! ```cargo
//! [dependencies]
//! anyhow = "1"
//! reqwest = { version = "0.12", features = ["blocking"] }
//! ```
use std::env;
fn main() -> anyhow::Result<()> {
let url = env::args().nth(1).context("usage: fetch.rs <url>")?;
let body = reqwest::blocking::get(&url)?.error_for_status()?.text()?;
println!("{} bytes", body.len());
Ok(())
}
```
Make executable: `chmod +x fetch.rs`. Run: `./fetch.rs https://example.com`.
The `//! \`\`\`cargo` block is parsed as inline `Cargo.toml`. Everything else is normal Rust.
## With async
```rust
#!/usr/bin/env rust-script
//! ```cargo
//! [dependencies]
//! anyhow = "1"
//! tokio = { version = "1", features = ["full"] }
//! reqwest = "0.12"
//! ```
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let urls = [
"https://example.com",
"https://example.org",
];
let client = reqwest::Client::new();
let bodies = futures::future::join_all(urls.iter().map(|u| {
let c = client.clone();
async move { c.get(*u).send().await?.text().await }
})).await;
for (url, body) in urls.iter().zip(bodies) {
match body {
Ok(b) => println!("{url}: {} bytes", b.len()),
Err(e) => eprintln!("{url}: error {e}"),
}
}
Ok(())
}
```
## With CLI parsing
```rust
#!/usr/bin/env rust-script
//! ```cargo
//! [dependencies]
//! anyhow = "1"
//! clap = { version = "4", features = ["derive"] }
//! ```
use clap::Parser;
#[derive(Parser, Debug)]
#[command(version, about = "rename files by pattern")]
struct Cli {
/// Glob to match
pattern: String,
/// Replacement template (use {n} for sequence)
template: String,
/// Show what would happen without doing it
#[arg(long)]
dry_run: bool,
}
fn main() -> anyhow::Result<()> {
let cli = Cli::parse();
let entries: Vec<_> = glob::glob(&cli.pattern)?.collect::<Result<_, _>>()?;
for (n, entry) in entries.iter().enumerate() {
let target = cli.template.replace("{n}", &n.to_string());
if cli.dry_run {
println!("{} -> {target}", entry.display());
} else {
std::fs::rename(entry, &target)?;
}
}
Ok(())
}
```
## Caching
`rust-script` caches the compiled binary in `~/.cache/rust-script/`. First run is slow (full compile), subsequent runs are instant.
To clear: `rust-script --clear-cache`.
Pin a script's compile target into the script directory for portability:
```bash
rust-script --build-only --base-path . ./script.rs
```
This drops a `target/` next to the script with the prebuilt binary.
## `cargo-script` (RFC 3424, stable since Rust 1.85)
The official replacement that landed in cargo proper. Same idea, slightly different syntax:
```rust
#!/usr/bin/env -S cargo +nightly -Zscript
---
package:
name = "fetch"
edition = "2024"
dependencies:
anyhow = "1"
reqwest = { version = "0.12", features = ["blocking"] }
---
fn main() -> anyhow::Result<()> {
let url = std::env::args().nth(1).context("url required")?;
println!("{}", reqwest::blocking::get(&url)?.text()?.len());
Ok(())
}
```
Status as of 2026-05: stabilization in progress. Use `rust-script` for production now, migrate when `cargo script` is stable everywhere your tools live.
## Strict mode for scripts
Add a lints block in the inline `Cargo.toml`:
```rust
//! ```cargo
//! [dependencies]
//! anyhow = "1"
//!
//! [lints.rust]
//! unsafe_code = "forbid"
//!
//! [lints.clippy]
//! all = "deny"
//! pedantic = "warn"
//! unwrap_used = "deny"
//! expect_used = "deny"
//! panic = "deny"
//! ```
```
Now the script gets the same strictness as the main project. If you need a one-line `unwrap()` for prototype velocity, switch the lint to `warn` for that one script - never blanket `allow`.
Run with lints visible:
```bash
RUSTFLAGS="-D warnings" rust-script ./script.rs
```
## When NOT to use a script
- It is going to live longer than a week → make it a real crate with `cargo new --bin`.
- It needs custom build scripts (`build.rs`) → real crate.
- It needs binary distribution to other machines → real crate with `cargo dist`.
- It needs to be tested → real crate (scripts can technically run `#[test]`s under `cargo test`, but the workflow is awkward).
A reasonable migration path: start as a script, when complexity grows past ~200 lines or you reach for a second `.rs` file, run `rust-script --emit ./script.rs` to dump a regular Cargo project skeleton and continue from there.
## Inline tests in a script
```rust
#!/usr/bin/env rust-script
//! ```cargo
//! [dependencies]
//! ```
fn double(x: i32) -> i32 { x * 2 }
fn main() {
println!("{}", double(21));
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn doubles_ints() {
assert_eq!(double(5), 10);
}
}
```
Run tests: `rust-script --test ./script.rs`.
## A useful "Rust as awk" pattern
For ad-hoc data processing on stdin:
```rust
#!/usr/bin/env rust-script
//! ```cargo
//! [dependencies]
//! serde_json = "1"
//! ```
use std::io::{self, BufRead, Write};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let stdin = io::stdin();
let stdout = io::stdout();
let mut out = stdout.lock();
for line in stdin.lock().lines() {
let line = line?;
let value: serde_json::Value = serde_json::from_str(&line)?;
if let Some(s) = value.get("level").and_then(|v| v.as_str()) {
if s == "error" {
writeln!(out, "{line}")?;
}
}
}
Ok(())
}
```
`cat logs.jsonl | ./filter-errors.rs` — filter JSON logs by `level == "error"`. Faster than `jq` for big files, type-safe.
For numerics:
```rust
#!/usr/bin/env rust-script
//! sum a column of numbers from stdin
use std::io::{self, BufRead};
fn main() {
let total: f64 = io::stdin().lock().lines()
.filter_map(|l| l.ok())
.filter_map(|l| l.trim().parse::<f64>().ok())
.sum();
println!("{total}");
}
```
## The `rust-script` shebang trick on macOS
macOS does not support multi-arg shebangs without `env -S`. Use:
```rust
#!/usr/bin/env -S rust-script --
```
The `--` lets clap-style argument parsers see the user's args, not the rust-script arguments.
## Editor support
VS Code / Helix / Vim with `rust-analyzer`: open the script file as if it were `src/main.rs` of an inferred crate. Most editors auto-detect the inline manifest. If not, hand-create a `Cargo.toml` next to the script with matching deps for the duration of editing, then delete it.
## When `rust-script` is too heavy
For absolutely throwaway "one expression on stdin" use cases, a Rust REPL like `evcxr_jupyter` (Jupyter kernel) or `irust` (terminal REPL) is more appropriate:
```bash
cargo install irust
irust
```
But these are interactive playgrounds, not scriptable. For shell pipelines, stay with `rust-script`.
## The Promise
Same strict lints. Same `clippy::pedantic` enforcement. Same `unsafe`-requires-SAFETY rule. The agent does not get a free pass on a 30-line script. The whole point of strict scripts is that **production hygiene is cheap when the toolchain enforces it**.
packages/shared-skills/skills/programming/references/rust/proptest-insta.md
+429
View File
@@ -0,0 +1,429 @@
# Property Tests (proptest) + Snapshot Tests (insta)
Two test types every Rust project should have alongside unit tests. Proptest hunts for inputs your unit tests forgot to try. Insta locks down output shapes you do not want to silently change.
## When to reach for each
| Want to test… | Use |
|---|---|
| One specific behavior with a known input | `#[test]` + `assert_eq!` |
| All inputs of a certain shape work | `proptest!` |
| Output structure stays stable across refactors | `insta::assert_*_snapshot!` |
| Parser/serializer round-trips | `proptest!` (the round-trip property) |
| CLI help text, JSON response shape, debug output | `insta::assert_snapshot!` |
| Concurrency under all interleavings | `loom` (see `concurrency.md`) |
Use all three. They cover different bug classes.
## Proptest — setup
`Cargo.toml`:
```toml
[dev-dependencies]
proptest = "1"
proptest-derive = "0.5" # for #[derive(Arbitrary)]
```
`proptest.toml` at project root (optional, sane defaults):
```toml
cases = 256 # number of random inputs per property
max_local_rejects = 65536
max_global_rejects = 1024
max_shrink_iters = 1024
max_shrink_time = 60_000 # ms
failure_persistence = { source_file = "proptest-regressions/", file_name = "regressions.txt" }
verbose = 0
```
`failure_persistence` is the killer feature: every failure is written to a regression file. On the next run, those exact inputs are replayed first, so once a bug is found it never escapes again.
## Basic property test
```rust
use proptest::prelude::*;
fn parse(s: &str) -> Result<Color, ParseError> { /* ... */ }
fn render(c: &Color) -> String { /* ... */ }
proptest! {
#[test]
fn parse_render_roundtrips(red in 0u8..=255, green in 0u8..=255, blue in 0u8..=255) {
let color = Color { red, green, blue };
let rendered = render(&color);
let parsed = parse(&rendered).expect("our render should always parse");
prop_assert_eq!(parsed, color);
}
}
```
`proptest!` macro takes `(arg in strategy, ...)` pairs. Each strategy produces values; proptest runs the body with random samples, then shrinks failing cases to minimal forms.
## Strategies — the value-generation language
| Strategy | Produces |
|---|---|
| `any::<T>()` | Any value of `T` (if `T: Arbitrary`) |
| `0u32..100` | Integer ranges |
| `prop::sample::select(slice)` | Pick from a list |
| `prop::collection::vec(elem, range)` | Vec of length in range |
| `prop::collection::hash_map(k, v, n..m)` | HashMap |
| `prop::option::of(strategy)` | Option |
| `prop::result::maybe_ok(ok, err)` | Result |
| `(s1, s2).prop_map(\|(a, b)\| ...)` | Combine, transform |
| `s.prop_filter("reason", \|v\| pred)` | Reject values |
| `s.prop_flat_map(\|v\| dependent)` | Sequential dependency |
| `prop_oneof![strategy1, strategy2]` | Union of strategies |
| `r"[a-z]{3,10}"` | Regex-generated string |
| `"\\PC*"` | Any printable non-control string |
Example combining several:
```rust
fn config_strategy() -> impl Strategy<Value = Config> {
(
prop::sample::select(vec!["dev", "staging", "prod"]),
0u16..=65535,
prop::collection::hash_map(
r"[a-z_]{1,20}",
any::<String>(),
0..5,
),
).prop_map(|(env, port, vars)| Config {
env: env.into(),
port,
env_vars: vars,
})
}
proptest! {
#[test]
fn config_validates(cfg in config_strategy()) {
let result = validate(&cfg);
if cfg.port == 0 {
prop_assert!(result.is_err());
} else {
prop_assert!(result.is_ok());
}
}
}
```
## Properties to write for every parser
1. **Round-trip:** `parse(render(x)) == x` for all valid `x`.
2. **No-panic:** `parse(arbitrary_string)` never panics, always returns `Result`.
3. **Idempotent:** `parse(parse(x).unwrap().render()) == parse(x).unwrap()`.
4. **Whitespace insensitivity:** `parse(x) == parse(strip_whitespace(x))` (if applicable).
For every serializer:
1. **Length bound:** `render(x).len() <= bound(x)`.
2. **Charset:** `render(x).chars().all(|c| ALLOWED.contains(&c))`.
For every collection operation:
1. **Identity:** `op_identity(x) == x` (sort an already-sorted, dedupe a unique).
2. **Idempotence:** `op(op(x)) == op(x)`.
3. **Commutativity:** `op(a, b) == op(b, a)` (set union, etc).
4. **Length:** `op(a, b).len() == known_relation(a.len(), b.len())`.
For every numeric op:
1. **Monotonicity:** `a <= b => f(a) <= f(b)`.
2. **Identity element:** `f(x, identity) == x`.
Write these mechanically. The agent should reach for proptest the moment any of these properties is checkable.
## Derive `Arbitrary`
```rust
use proptest_derive::Arbitrary;
#[derive(Debug, Clone, PartialEq, Arbitrary)]
struct Vec3 {
#[proptest(strategy = "-100.0..=100.0")]
x: f32,
#[proptest(strategy = "-100.0..=100.0")]
y: f32,
#[proptest(strategy = "-100.0..=100.0")]
z: f32,
}
proptest! {
#[test]
fn dot_product_is_commutative(a: Vec3, b: Vec3) {
prop_assert!((dot(&a, &b) - dot(&b, &a)).abs() < 1e-5);
}
}
```
`#[derive(Arbitrary)]` auto-implements the strategy. Per-field `#[proptest(strategy = "...")]` overrides.
## Stateful / state machine tests
For data structures with operations (queues, maps, trees), use `proptest-state-machine`:
```rust
use proptest_state_machine::{ReferenceStateMachine, StateMachineTest};
struct MyQueueRef { state: VecDeque<i32> }
struct MyQueueSut { sut: MyQueue<i32> }
#[derive(Debug, Clone)]
enum Op { Push(i32), Pop }
impl ReferenceStateMachine for MyQueueRef {
type State = VecDeque<i32>;
type Transition = Op;
fn init_state() -> BoxedStrategy<Self::State> {
Just(VecDeque::new()).boxed()
}
fn transitions(_: &Self::State) -> BoxedStrategy<Self::Transition> {
prop_oneof![
any::<i32>().prop_map(Op::Push),
Just(Op::Pop),
].boxed()
}
fn apply(mut state: Self::State, transition: &Self::Transition) -> Self::State {
match transition {
Op::Push(x) => state.push_back(*x),
Op::Pop => { state.pop_front(); }
}
state
}
}
impl StateMachineTest for MyQueueSut {
type SystemUnderTest = MyQueue<i32>;
type Reference = MyQueueRef;
fn init_test(_: &<Self::Reference as ReferenceStateMachine>::State) -> Self::SystemUnderTest {
MyQueue::new()
}
fn apply(mut sut: Self::SystemUnderTest, _: &VecDeque<i32>, transition: Op) -> Self::SystemUnderTest {
match transition {
Op::Push(x) => sut.push(x),
Op::Pop => { sut.pop(); }
}
sut
}
fn check_invariants(sut: &Self::SystemUnderTest, state: &VecDeque<i32>) {
assert_eq!(sut.len(), state.len());
// also check head/tail/iteration order...
}
}
proptest_state_machine::prop_state_machine! {
#[test]
fn queue_matches_vecdeque(sequential 1..50 => MyQueueSut);
}
```
You define a reference implementation (`VecDeque` here), proptest fuzzes operations against both, asserts invariants every step. This is the technique for finding bugs in lock-free or complex containers.
## Shrinking
When a property fails, proptest reduces the input to a minimal counter-example. For built-in strategies this is automatic. For custom strategies built with `prop_map`, shrinking goes through the underlying strategy. Avoid breaking shrinking with `prop_filter` (rejection sampling) over wide spaces; prefer `prop_flat_map` or directly-shaped strategies.
## Regression corpus
When a property test fails, proptest writes the failing input to `proptest-regressions/<test_name>.txt`. Commit this directory. Future runs replay these failing inputs first, so the bug stays fixed forever.
```
proptest-regressions/
└── parse_color.txt # commit this
```
## Insta — setup
`Cargo.toml`:
```toml
[dev-dependencies]
insta = { version = "1", features = ["yaml", "json", "redactions", "filters"] }
[dependencies.serde_yaml]
version = "0.9"
optional = true
```
Install the CLI:
```bash
cargo install cargo-insta
```
## Insta — basic snapshots
```rust
#[test]
fn renders_default_help() {
let output = render_help();
insta::assert_snapshot!(output);
}
```
First run: creates `src/snapshots/mycrate__renders_default_help.snap.new`. Run `cargo insta review`, press `a` to accept, the `.new` extension is dropped. Subsequent runs diff against the committed snapshot; mismatches fail the test.
## Insta — typed snapshots
```rust
#[derive(Debug, serde::Serialize)]
struct Result {
status: String,
user: User,
duration_ms: u64,
}
#[test]
fn json_response() {
let value = compute();
insta::assert_json_snapshot!(value);
}
#[test]
fn yaml_response() {
insta::assert_yaml_snapshot!(value);
}
#[test]
fn debug_repr() {
insta::assert_debug_snapshot!(value);
}
```
Choose:
- `assert_snapshot!` for `String`/`Display` output (CLI help, error messages, generated code).
- `assert_debug_snapshot!` for `{:?}` (Rust-internal data).
- `assert_json_snapshot!` for structured data crossing process boundaries.
- `assert_yaml_snapshot!` when YAML is easier to read in diffs.
## Insta — redactions and filters
For values that change every run (timestamps, UUIDs, paths):
```rust
#[test]
fn with_redactions() {
let value = ApiResponse {
id: uuid::Uuid::now_v7(),
created_at: jiff::Timestamp::now(),
body: "hello".into(),
};
insta::assert_json_snapshot!(value, {
".id" => "[uuid]",
".created_at" => "[timestamp]",
});
}
```
For regex filters applied to all snapshots in a test:
```rust
#[test]
fn with_filters() {
let mut settings = insta::Settings::clone_current();
settings.add_filter(r"/tmp/[a-z0-9-]+", "[TMP]");
settings.add_filter(r"\d+\.\d+ms", "[TIMING]");
settings.bind(|| {
let output = run_command();
insta::assert_snapshot!(output);
});
}
```
`Settings::bind` scopes filters to the closure.
## Insta workflow
1. Write the test, run it. First run creates `.snap.new`.
2. `cargo insta review` → interactive UI. Show diff, accept/reject.
3. Accepted snapshots commit to the repo.
4. Refactor code. Tests run; mismatches show as diffs.
5. If the new output is correct, `cargo insta accept` (or selective `review`). If wrong, fix the code.
Pair with CI to fail builds when uncommitted `.snap.new` files exist:
```bash
cargo nextest run
if find . -name "*.snap.new" | grep -q .; then
echo "Pending snapshots, run 'cargo insta review'"
exit 1
fi
```
## Inline snapshots
```rust
#[test]
fn small_output() {
let value = compute();
insta::assert_snapshot!(value, @"hello world");
}
```
The trailing `@"..."` string is the expected snapshot, stored in source. Useful when the value is short enough that pulling out a separate file is overkill. `cargo insta accept` updates them in-place.
## Inline JSON snapshots
```rust
#[test]
fn json_inline() {
insta::assert_json_snapshot!(value, @r###"
{
"status": "ok",
"count": 3
}
"###);
}
```
## Anti-patterns
1. **Snapshots of unstable output.** If `HashMap` iteration order changes per run, snapshots will fail. Switch to `BTreeMap` or sort before snapshotting.
2. **Massive snapshots.** A 10KB JSON dump where you really care about 3 fields. Either narrow to the fields, or accept that any refactor will require re-reviewing 10KB.
3. **Snapshots that bake in implementation details.** "function called 3 times" is not a snapshot - it's a behavior assertion. Use a real assertion.
4. **Skipping `cargo insta review`.** Accepting blind via `cargo insta accept --all` defeats the purpose. Always review.
## Combining proptest + insta
```rust
proptest! {
#[test]
fn random_inputs_render_consistently(input: ValidInput) {
let mut settings = insta::Settings::clone_current();
settings.set_snapshot_suffix(format!("{}", input.hash()));
settings.bind(|| {
insta::assert_snapshot!(render(&input));
});
}
}
```
But honestly, this is rarely a fit. Proptest tests properties, insta tests output shape. Don't snapshot random inputs - that defeats both tools.
## CI matrix recommendation
```yaml
- name: Tests
run: cargo nextest run --all-features
- name: Property regressions (replay)
run: |
# The regression files in proptest-regressions/ replay first.
# Failures here mean a previously-fixed bug came back.
cargo nextest run --all-features --test-threads 1
```
When a proptest finds a new failure, the regression file appears as a git diff - check it in.
## What proptest cannot do
- Find bugs that require multi-process / multi-network coordination → integration tests + fault injection.
- Find concurrency bugs → use `loom` (see `concurrency.md`).
- Find performance regressions → use `criterion`.
But for any function with a domain (inputs to outputs), proptest can find more bugs than your unit tests. **Write the property first, derive the unit test second.**
packages/shared-skills/skills/programming/references/rust/type-state.md
+354
View File
@@ -0,0 +1,354 @@
# Type-State and Newtype Patterns
The single highest-leverage thing Rust gives a coding agent: encode invariants in the type system so the compiler refuses incorrect code. The agent does not have to "remember" rules - the rules are physical.
## The Two Core Patterns
1. **Newtype wrappers for distinct semantic units.** Money, IDs, byte offsets, coordinate spaces - each gets its own tuple struct. The agent cannot pass meters where feet are expected, even though both are `f64` under the hood. This is the `euclid::Point<Screen>` vs `euclid::Point<World>` example Chris Allen called out.
2. **Type-state for state machines.** Instead of a struct with a `status: enum { Draft, Validated, Persisted }` field and methods that check `if self.status == ...`, model each state as its own type. Transitions are method calls that consume `self` and return a new type. Illegal transitions become unrepresentable.
## Newtype Wrapper Cookbook
### Domain IDs
```rust
use uuid::Uuid;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
#[serde(transparent)]
pub struct UserId(Uuid);
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
#[serde(transparent)]
pub struct ProductId(Uuid);
impl UserId {
pub fn new() -> Self { Self(Uuid::now_v7()) }
pub fn as_uuid(&self) -> &Uuid { &self.0 }
}
impl ProductId {
pub fn new() -> Self { Self(Uuid::now_v7()) }
pub fn as_uuid(&self) -> &Uuid { &self.0 }
}
// `fn buy(user: UserId, product: ProductId)` cannot be called with arguments swapped.
```
`#[serde(transparent)]` keeps JSON/SQL round-trips identical to a bare `Uuid` - the wrapper is purely a compile-time discipline.
### Quantities with Phantom Type Tags
```rust
use core::marker::PhantomData;
use core::ops::{Add, Sub, Mul};
#[derive(Debug, Clone, Copy)]
pub struct Quantity<Unit> {
raw: f64,
_unit: PhantomData<Unit>,
}
// Tag types - zero-sized, never instantiated.
pub struct Meters;
pub struct Feet;
pub struct Seconds;
impl<U> Quantity<U> {
pub const fn new(value: f64) -> Self {
Self { raw: value, _unit: PhantomData }
}
pub fn raw(self) -> f64 { self.raw }
}
// Adding same-unit quantities: allowed.
impl<U> Add for Quantity<U> {
type Output = Self;
fn add(self, rhs: Self) -> Self { Self::new(self.raw + rhs.raw) }
}
// Subtraction: allowed.
impl<U> Sub for Quantity<U> {
type Output = Self;
fn sub(self, rhs: Self) -> Self { Self::new(self.raw - rhs.raw) }
}
// Multiplying by scalar: allowed.
impl<U> Mul<f64> for Quantity<U> {
type Output = Self;
fn mul(self, rhs: f64) -> Self { Self::new(self.raw * rhs) }
}
// Conversions are explicit, named methods - never `From`/`Into` between units.
impl Quantity<Meters> {
pub fn to_feet(self) -> Quantity<Feet> {
Quantity::new(self.raw * 3.280_84)
}
}
```
Now:
```rust
let distance: Quantity<Meters> = Quantity::new(100.0);
let height: Quantity<Feet> = Quantity::new(50.0);
let combined = distance + height; // ❌ compile error
let combined = distance + height.to_feet().to_meters_oops(); // ❌ no such method
let combined = distance + distance; // ✅
```
The agent cannot accidentally mix units. Refactors that change a quantity's underlying unit are caught at compile time everywhere the type flows.
### Byte Offsets vs Character Offsets
```rust
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct ByteOffset(pub u32);
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct CharOffset(pub u32);
impl ByteOffset {
pub fn add(self, delta: u32) -> Self { Self(self.0 + delta) }
}
// Converting between them is a function on the actual text.
pub fn byte_to_char(text: &str, byte: ByteOffset) -> Option<CharOffset> {
text.get(..byte.0 as usize).map(|prefix| CharOffset(prefix.chars().count() as u32))
}
```
A function signature `fn slice(text: &str, start: ByteOffset, end: ByteOffset)` cannot be called with character offsets. The UTF-8 boundary bug is now a compile error.
### Currency
```rust
use rust_decimal::Decimal;
pub struct Krw;
pub struct Usd;
pub struct Jpy;
#[derive(Debug, Clone, Copy)]
pub struct Money<Currency> {
amount: Decimal,
_ccy: PhantomData<Currency>,
}
impl<C> Money<C> {
pub const fn new(amount: Decimal) -> Self { Self { amount, _ccy: PhantomData } }
}
impl<C> Add for Money<C> {
type Output = Self;
fn add(self, rhs: Self) -> Self { Self::new(self.amount + rhs.amount) }
}
// No blanket From<Money<X>> for Money<Y> - conversions go through an explicit
// FX rate function that takes a `Rate<From, To>` argument.
pub struct Rate<From, To> {
factor: Decimal,
_from: PhantomData<From>,
_to: PhantomData<To>,
}
impl<From, To> Money<From> {
pub fn convert(self, rate: Rate<From, To>) -> Money<To> {
Money::new(self.amount * rate.factor)
}
}
```
The agent cannot add KRW and USD by accident. They cannot convert without a rate. They cannot apply a USD→JPY rate to a KRW value.
### Paths Rooted at Different Bases
```rust
use std::path::{Path, PathBuf};
/// A path guaranteed to be relative to the project root.
#[derive(Debug, Clone)]
pub struct ProjectRel(PathBuf);
/// A path guaranteed to be relative to the user's home directory.
#[derive(Debug, Clone)]
pub struct HomeRel(PathBuf);
impl ProjectRel {
pub fn new(path: impl AsRef<Path>) -> Result<Self, PathError> {
let path = path.as_ref();
if path.is_absolute() { return Err(PathError::NotRelative); }
if path.components().any(|c| matches!(c, std::path::Component::ParentDir)) {
return Err(PathError::EscapesRoot);
}
Ok(Self(path.to_path_buf()))
}
pub fn resolve(&self, project_root: &Path) -> PathBuf {
project_root.join(&self.0)
}
}
```
The agent's path-handling code now distinguishes between project-relative and home-relative paths at the type level. A function taking `ProjectRel` cannot be called with a `HomeRel`.
## Type-State State Machines
Encode the lifecycle of a value as a sequence of types. Each transition consumes the previous state and returns the next.
### HTTP Request Builder
```rust
pub struct RequestBuilder<State> {
url: String,
method: Method,
headers: HeaderMap,
body: Option<Vec<u8>>,
_state: PhantomData<State>,
}
pub struct NeedsUrl;
pub struct NeedsMethod;
pub struct Ready;
impl RequestBuilder<NeedsUrl> {
pub fn new() -> Self {
Self {
url: String::new(),
method: Method::GET,
headers: HeaderMap::new(),
body: None,
_state: PhantomData,
}
}
pub fn url(mut self, url: impl Into<String>) -> RequestBuilder<NeedsMethod> {
self.url = url.into();
RequestBuilder { url: self.url, method: self.method, headers: self.headers, body: self.body, _state: PhantomData }
}
}
impl RequestBuilder<NeedsMethod> {
pub fn method(mut self, method: Method) -> RequestBuilder<Ready> {
self.method = method;
RequestBuilder { url: self.url, method: self.method, headers: self.headers, body: self.body, _state: PhantomData }
}
}
// .header(), .body() available in any state that has at least URL.
impl<S> RequestBuilder<S> {
pub fn header(mut self, name: HeaderName, value: HeaderValue) -> Self {
self.headers.insert(name, value);
self
}
}
// .send() only available once URL and method are set.
impl RequestBuilder<Ready> {
pub async fn send(self, client: &Client) -> reqwest::Result<Response> { /* ... */ }
}
```
`client.send(RequestBuilder::new().send(...))` - compile error. The agent has to fill in the required steps. The IDE autocomplete also reflects only the legal next steps.
### File Handles
```rust
pub struct File<State> {
fd: RawFd,
_state: PhantomData<State>,
}
pub struct Open;
pub struct Locked;
pub struct Closed;
impl File<Open> {
pub fn open(path: &Path) -> std::io::Result<Self> { /* ... */ }
pub fn lock_exclusive(self) -> std::io::Result<File<Locked>> { /* flock */ }
pub fn close(self) -> std::io::Result<File<Closed>> { /* ... */ }
}
impl File<Locked> {
pub fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> { /* ... */ }
pub fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> { /* ... */ }
pub fn unlock(self) -> std::io::Result<File<Open>> { /* ... */ }
}
impl File<Closed> {
// No methods. The type only exists to be dropped.
}
```
You cannot `read()` an unlocked file. You cannot `close()` while holding a lock without unlocking first. You cannot use a closed file at all - it has no methods.
## Sealed Traits
Sometimes you want a closed set of types implementing a trait, defined by the crate, not extensible by downstream. The sealed trait pattern:
```rust
mod sealed {
pub trait Sealed {}
}
pub trait Renderer: sealed::Sealed {
fn render(&self, frame: &mut Frame);
}
pub struct OpenGl;
pub struct Vulkan;
pub struct Metal;
impl sealed::Sealed for OpenGl {}
impl sealed::Sealed for Vulkan {}
impl sealed::Sealed for Metal {}
impl Renderer for OpenGl { fn render(&self, f: &mut Frame) { /* ... */ } }
impl Renderer for Vulkan { fn render(&self, f: &mut Frame) { /* ... */ } }
impl Renderer for Metal { fn render(&self, f: &mut Frame) { /* ... */ } }
```
Downstream code cannot add new `impl Renderer for Whatever` because they cannot implement `sealed::Sealed` (its module is private). Useful when you want trait dispatch but maintain the invariant that you control all implementations.
## NonEmpty Collections
```rust
pub struct NonEmptyVec<T> {
head: T,
tail: Vec<T>,
}
#[derive(Debug, thiserror::Error)]
#[error("vector was empty")]
pub struct Empty;
impl<T> NonEmptyVec<T> {
pub fn try_from_vec(mut v: Vec<T>) -> Result<Self, Empty> {
if v.is_empty() { return Err(Empty); }
let tail = v.split_off(1);
let head = v.into_iter().next().expect("checked non-empty");
Ok(Self { head, tail })
}
pub fn first(&self) -> &T { &self.head }
pub fn len(&self) -> usize { self.tail.len() + 1 }
}
```
Functions taking `NonEmptyVec<T>` cannot receive an empty vector. The `first()` method returns `&T`, not `Option<&T>`. The agent never has to write `match v.first() { Some(x) => ..., None => panic!(...) }` again.
## When NOT to Newtype
- For one-off internal computations where the unit lives in a single function and never crosses a boundary.
- When the wrapper does not change behavior or invariants vs. the underlying type (e.g., a `struct Count(u32)` that is only ever used in one struct).
- When `From`/`Into` conversions would be ergonomic but would defeat the purpose (if you find yourself wanting `impl From<UserId> for Uuid`, you do not want a newtype - you want a type alias).
The cost of a newtype is one tuple struct + the impls you need. The break-even is around three uses across different functions, or any use that crosses an API boundary.
## When NOT to Use Type-State
- When the state space is small and transitions are simple (`Option<T>` and `Result<T, E>` are state machines already).
- When the type-state would force runtime branching upward (e.g., reading "should this run as Open or Locked?" from config means you store `Box<dyn FileLike>` anyway).
- When the API is consumed by code that does not know the state at compile time (heterogeneous collections, dynamic dispatch boundaries).
In those cases, regular enum-tagged states are correct. The line is: **can the call site know the state statically?** If yes, type-state. If no, enum-with-tag.
packages/shared-skills/skills/programming/references/rust/unsafe-discipline.md
+250
View File
@@ -0,0 +1,250 @@
# Unsafe Discipline
The reason Chris Allen's "implementing a persistent memory arena in Rust was not hard" works: the unsafe surface area is microscopic, it lives behind one newtype with one constructor, and every block has a SAFETY comment that names a specific invariant. Coding agents follow the pattern mechanically once the shape is established.
## The Three Required Components
Every `unsafe` block needs all three. No exceptions.
1. **Safe wrapper.** No `unsafe fn` or raw pointer types in the crate's public API. If a caller needs to construct an instance, the constructor either does the work safely or is `unsafe` with a documented contract.
2. **SAFETY comment.** A `// SAFETY:` line within 5 lines above the `unsafe { ... }` block, stating which invariant is upheld and where it comes from. Generic phrases ("this is safe because we checked") fail review.
3. **miri proof.** A test that exercises the unsafe path under `cargo +nightly miri nextest run`. If the path cannot be exercised under miri (FFI, syscalls), provide an alternate proof and gate behind a feature flag ending in `-skip-miri`.
## The Wrapper Pattern (`NonNull<T>` style)
Reference the screenshot Chris Allen quoted - `std::ptr::NonNull<T>`. Mirror this shape for every raw pointer, raw slice, raw transmute, or uninit memory operation in your own code.
```rust
use core::marker::PhantomData;
use core::ptr::NonNull;
/// A non-null, properly aligned, initialized pointer that does not alias.
///
/// All invariants are upheld by [`Self::new`] (checked) or [`Self::new_unchecked`]
/// (delegated to the caller's contract). Once you hold an `InitPtr<T>`, every
/// public method on it is safe to call.
#[derive(Debug)]
#[repr(transparent)]
pub struct InitPtr<T> {
inner: NonNull<T>,
_marker: PhantomData<T>,
}
// Send/Sync are NOT automatic for raw-pointer-bearing types. Decide deliberately.
// SAFETY: `InitPtr<T>` owns no concurrency state of its own; whether it is
// Send/Sync depends on `T`. The bounds below mirror `Box<T>`.
unsafe impl<T: Send> Send for InitPtr<T> {}
unsafe impl<T: Sync> Sync for InitPtr<T> {}
impl<T> InitPtr<T> {
/// Wrap a raw pointer after checking alignment and non-null. The
/// initialization invariant is not statically checkable here; callers must
/// only feed pointers to memory that was written before this call.
pub fn new(ptr: *mut T) -> Option<Self> {
if !ptr.is_aligned() {
return None;
}
// SAFETY: alignment checked above; `NonNull::new` filters null. The
// caller is documented to provide an initialized location.
NonNull::new(ptr).map(|inner| Self { inner, _marker: PhantomData })
}
/// Wrap a raw pointer the caller asserts is valid.
///
/// # Safety
///
/// - `ptr` is non-null.
/// - `ptr` is aligned to `align_of::<T>()`.
/// - `*ptr` is initialized at the time of this call.
/// - For the lifetime of the returned value, no other `&T` or `&mut T`
/// aliases `*ptr`.
pub unsafe fn new_unchecked(ptr: *mut T) -> Self {
// SAFETY: caller upholds non-null per the function contract.
Self { inner: unsafe { NonNull::new_unchecked(ptr) }, _marker: PhantomData }
}
pub fn as_ref(&self) -> &T {
// SAFETY: the constructor's invariants guarantee `inner` points at an
// initialized, aligned, non-aliased `T`. Reborrowing through `&self`
// ties the resulting lifetime to `self`, enforcing the rest via
// standard borrow rules.
unsafe { self.inner.as_ref() }
}
pub fn as_mut(&mut self) -> &mut T {
// SAFETY: `&mut self` proves no other reference can alias `inner` for
// the lifetime of the returned `&mut T`; remaining invariants come
// from construction.
unsafe { self.inner.as_mut() }
}
}
```
The list of features this single shape gives you:
- The agent cannot construct `InitPtr<T>` without going through a checked path or accepting the `unsafe` obligation explicitly.
- The agent cannot leak the raw pointer; `as_ref` / `as_mut` return safe references with proper lifetimes.
- The agent cannot accidentally Send/Sync where it shouldn't - the `unsafe impl` is explicit per-bound.
- `#[repr(transparent)]` means the type is layout-compatible with `*mut T` for FFI, without exposing the raw pointer.
## SAFETY Comment Grammar
Every comment maps one-to-one to an invariant. Format:
```rust
// SAFETY: <which invariant is upheld>: <how it is established here>.
```
Anti-examples (do not pass review):
```rust
// SAFETY: this is fine
// SAFETY: we know what we're doing
// SAFETY: the caller will not pass null
// SAFETY: tested
```
Good examples:
```rust
// SAFETY: `len <= self.capacity` was checked at line 87 and `self.ptr` was
// allocated by the same allocator we are reading through.
// SAFETY: `read_volatile` requires alignment and non-null; both hold because
// `self.inner` is an `InitPtr<T>` whose constructor enforced them.
// SAFETY: We hold `&mut self`, so no concurrent reader exists. The slice
// reference is dropped before the next `&self` borrow because we shrink the
// returned scope manually.
```
## Persistent Memory Arena Pattern (the Chris Allen example)
A persistent memory arena (PMA) is an `mmap`-backed bump allocator that survives process restarts. It is the classic "lots of unsafe under one safe surface" project.
Shape:
```rust
pub struct Arena {
map: Mmap, // wraps `mmap(2)` - safe wrapper from `memmap2` crate
head: AtomicUsize, // current bump offset, atomic for multi-writer if needed
}
impl Arena {
pub fn open(path: &Path, capacity: usize) -> std::io::Result<Self> { /* mmap, init header */ }
pub fn alloc<T>(&self, value: T) -> Result<Handle<T>, ArenaFull> {
let layout = Layout::new::<T>();
let aligned = align_up(self.head.load(Acquire), layout.align());
let next = aligned.checked_add(layout.size()).ok_or(ArenaFull)?;
if next > self.map.len() { return Err(ArenaFull); }
// CAS the head forward; retry on contention.
// ... omitted for brevity ...
// SAFETY: `aligned + layout.size() <= self.map.len()` was just proven.
// The mmap region is exclusively owned by this arena while we hold the
// bump. `aligned` is aligned to `layout.align()` by `align_up`. No
// other writer can have observed this offset because the CAS above
// returned `Ok`.
let ptr = unsafe { self.map.as_mut_ptr().add(aligned) as *mut T };
// SAFETY: `ptr` is non-null (mmap base + offset), aligned (above),
// exclusively owned (CAS), and we are about to initialize it.
unsafe { ptr::write(ptr, value) };
// SAFETY: same invariants; we wrap the now-initialized pointer in the
// safe handle which encapsulates further accesses.
Ok(unsafe { Handle::new_unchecked(ptr, self) })
}
}
pub struct Handle<'a, T> {
inner: InitPtr<T>,
_arena: PhantomData<&'a Arena>,
}
```
Three `unsafe` blocks, three SAFETY comments, one safe handle type emerging on the other side. The agent now uses `Handle<T>` everywhere - never `*mut T`.
## Miri Invocation
```bash
# install once
rustup install nightly
rustup component add miri rust-src --toolchain nightly
# run on every change that touches unsafe
MIRIFLAGS="-Zmiri-strict-provenance -Zmiri-symbolic-alignment-check" \
cargo +nightly miri nextest run --all-features
```
What miri catches that the borrow checker cannot:
- Use-after-free
- Double-free
- Reads of uninitialized memory
- Pointer-from-integer reconstruction that violates strict provenance
- Alignment lies (transmuting unaligned data)
- Stacked borrows / Tree borrows aliasing violations
- Data races (single-threaded model, but catches concurrent access through `UnsafeCell` misuse)
- Atomic ordering bugs in some patterns
- Memory leaks (with `-Zmiri-track-pointer-tag`)
## When Miri Cannot Run
Certain paths are off-limits for miri: most syscalls beyond a curated allowlist, real network I/O, `std::process` calls, OS-specific FFI, hardware-dependent intrinsics on non-x86. Strategy:
1. **Isolate.** Put the un-mirifiable code in its own module behind `#[cfg(feature = "ffi-real")]` or similar.
2. **Mock at the boundary.** For everything below the FFI boundary, write a safe Rust fake (a `Vec<u8>`-backed "disk", a fake clock, an in-memory socket pair). Expose it as a trait the production code consumes.
3. **Test the fake under miri.** The fake implementation exercises the same logic minus the syscall. If the logic is unsafe (raw pointer manipulation in the fake "disk" buffer), miri catches the bug.
4. **Test the real path under regular `cargo test`.** With `cargo nextest run --features ffi-real`. No miri, but the surface area is now just the syscall boundary.
5. **Document.** A `# Safety` section in the rustdoc names the obligations the FFI puts on us, and a `# Testing` section explains the mock-vs-real split.
## Loom for Concurrency
When `unsafe` participates in a concurrent algorithm (lock-free queue, hazard pointers, custom Arc), miri's single-thread model is insufficient. Use `loom`:
```rust
#[cfg(loom)]
use loom::sync::atomic::{AtomicUsize, Ordering};
#[cfg(not(loom))]
use std::sync::atomic::{AtomicUsize, Ordering};
#[cfg(test)]
mod tests {
use super::*;
#[test]
#[cfg(loom)]
fn concurrent_push_pop() {
loom::model(|| {
let queue = std::sync::Arc::new(MyQueue::new());
let q1 = queue.clone();
let q2 = queue.clone();
let h1 = loom::thread::spawn(move || q1.push(1));
let h2 = loom::thread::spawn(move || q2.pop());
h1.join().unwrap();
h2.join().unwrap();
});
}
}
```
Run: `RUSTFLAGS="--cfg loom" cargo test --release`. Loom exhaustively explores thread interleavings for the test scope. Combined with miri on the single-thread paths, you have machine-checked soundness over the full state space.
## The Forbidden List
Reject in code review, automatic CI fail:
- `unsafe { ... }` with no SAFETY comment within 5 lines above.
- `unsafe { unsafe_op_a(); unsafe_op_b(); }` (multiple unsafe ops in one block - split them, one SAFETY each). Clippy: `multiple_unsafe_ops_per_block`.
- `unsafe fn` exposed publicly without a documented `# Safety` section in rustdoc.
- `std::mem::transmute` for anything but lifetime extension on the same layout (and that should usually be `core::mem::transmute_copy` or `bytemuck::cast` if the relayout is well-defined).
- `std::ptr::read_unaligned` / `write_unaligned` without a comment explaining why aligned access is impossible.
- `from_raw_parts` / `from_raw_parts_mut` without proving the source pointer's provenance covers the entire slice.
- `Arc::get_mut_unchecked`, `Box::leak` to bypass ownership, `MaybeUninit::assume_init` on partially-initialized data.
- `unsafe impl Send`, `unsafe impl Sync` on types containing raw pointers, without a comment naming exactly which interior-mutability rule is upheld.
- Any `unsafe` block whose justification depends on "in practice this never happens".
## The One-Line Summary
> Wrap once. Prove once. Test under miri. Never let `unsafe` escape.
packages/shared-skills/skills/programming/references/rust/zero-cost-safety.md
+527
View File
@@ -0,0 +1,527 @@
# Zero-Cost Safety — Zig Ergonomics in Rust
Rust already owns memory safety. This reference adds the patterns that give you Zig's *ergonomic* safety — explicit allocation control, compile-time computation, zero-hidden-cost APIs, bit-level layout, and deterministic cleanup — without leaving the Rust toolchain.
**When to load this file:** arena, allocator, bumpalo, const fn, const generics, comptime, zero-alloc, no-alloc, slice-based API, `#[repr]`, packed struct, bitfield, scopeguard, errdefer, RAII cleanup, Zig-like patterns.
---
## 1. Explicit Allocators — Arena Pattern
Zig passes `allocator: Allocator` to every function. Rust's stable equivalent: arena crates that make allocation scope visible and bulk-freeable.
### bumpalo — The Default Arena
```rust
use bumpalo::Bump;
fn parse_tokens<'a>(arena: &'a Bump, input: &[u8]) -> Vec<&'a str> {
// All allocations go into `arena`. Caller controls lifetime.
// When `arena` drops, everything frees in one shot.
let token = arena.alloc_str("hello");
let slice = arena.alloc_slice_copy(&[1u8, 2, 3]);
vec![token] // Vec itself is on heap; contents point into arena
}
// Usage: caller owns the arena, decides when memory dies.
let arena = Bump::new();
let tokens = parse_tokens(&arena, b"...");
drop(arena); // all arena memory freed, zero individual deallocations
```
**When to use:** parsers, compilers, game frame allocators, request-scoped web handlers, any hot loop where individual `Box`/`Vec` alloc+free overhead matters.
### typed-arena — Homogeneous Arena
```rust
use typed_arena::Arena;
struct AstNode { kind: u8, children: Vec<&'static AstNode> } // simplified
let node_arena: Arena<AstNode> = Arena::new();
let root = node_arena.alloc(AstNode { kind: 0, children: vec![] });
// All nodes share arena lifetime. No individual free.
```
**When to use:** tree/graph structures where all nodes have the same type and same lifetime.
### allocator_api (nightly) — Full Zig Parity
```rust
#![feature(allocator_api)]
use std::alloc::Global;
// Vec parameterized by allocator — exactly like Zig.
let v: Vec<u8, &Bump> = Vec::new_in(&arena);
// Custom allocator for tracking, limiting, or redirecting allocation
struct CountingAlloc { inner: Global, count: AtomicUsize }
unsafe impl Allocator for CountingAlloc { /* ... */ }
```
**When to use:** when you need allocator-generic data structures on nightly. For stable code, prefer `bumpalo` directly.
### Decision Tree
```
Need arena allocation?
├── All items same type, same lifetime → typed-arena
├── Mixed types, same lifetime → bumpalo
├── Need allocator-generic containers → allocator_api (nightly)
└── Just need fewer allocations → SmallVec / ArrayVec / tinyvec (stack-first)
```
### Cargo.toml
```toml
bumpalo = { version = "3", features = ["collections"] }
typed-arena = "2"
smallvec = { version = "1", features = ["union", "const_generics"] }
tinyvec = { version = "1", features = ["alloc"] }
```
---
## 2. Compile-Time Computation — const fn, const generics, proc macros
Zig's `comptime` runs arbitrary code at compile time. Rust splits this across three mechanisms.
### const fn — Compile-Time Pure Functions
```rust
const fn fibonacci(n: usize) -> usize {
match n {
0 => 0,
1 => 1,
_ => fibonacci(n - 1) + fibonacci(n - 2),
}
}
const FIB_20: usize = fibonacci(20); // computed at compile time: 6765
// Use in array sizes
const LOOKUP: [u8; 256] = {
let mut table = [0u8; 256];
let mut i = 0;
while i < 256 {
table[i] = (i as u8).wrapping_mul(7);
i += 1;
}
table
};
```
**Stable since Rust 1.82:** `const fn` supports `match`, loops, `if`, references, mutable locals — nearly full Rust. Use `const { }` blocks (Rust 1.79+) for inline compile-time assertions.
```rust
fn process<const N: usize>(data: &[u8; N]) {
const { assert!(N > 0, "N must be positive") }; // compile-time panic if N == 0
// ...
}
```
### const generics — Type-Level Values
```rust
struct Buffer<const N: usize> {
data: [u8; N],
len: usize,
}
impl<const N: usize> Buffer<N> {
const fn new() -> Self {
Self { data: [0; N], len: 0 }
}
fn push(&mut self, byte: u8) -> Result<(), BufferFullError> {
if self.len >= N { return Err(BufferFullError); }
self.data[self.len] = byte;
self.len += 1;
Ok(())
}
}
// Compiler enforces: Buffer<16> and Buffer<32> are distinct types.
let small: Buffer<16> = Buffer::new();
let large: Buffer<1024> = Buffer::new();
```
### proc macros — Code Generation (Zig comptime type creation)
When `const fn` is not enough (generating struct fields, impl blocks, or derive logic), proc macros fill the gap.
```rust
// In a proc-macro crate:
use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};
#[proc_macro_derive(Builder)]
pub fn derive_builder(input: TokenStream) -> TokenStream {
let input = parse_macro_input!(input as DeriveInput);
let name = &input.ident;
// ... generate builder struct and impl
TokenStream::from(quote! {
impl #name {
pub fn builder() -> #name##Builder { /* ... */ }
}
})
}
```
**Decision tree:**
```
Need compile-time value computation? → const fn
Need type parameterized by value? → const generics
Need to generate new types/impls? → proc macro (derive or attribute)
Need compile-time string processing? → proc macro
Need typenum-level arithmetic? → typenum / generic-array (rare)
```
---
## 3. Zero-Allocation API Design — No Hidden Costs
Zig's philosophy: no operator overloading, no hidden allocation, every cost visible. Rust achieves this with discipline.
### Slice-Based APIs — Caller Owns Memory
```rust
// BAD: hidden allocation in return type
fn process(input: &str) -> String {
input.to_uppercase() // allocates
}
// GOOD: caller provides output buffer, zero allocation
fn process(input: &[u8], output: &mut [u8]) -> usize {
let len = input.len().min(output.len());
for i in 0..len {
output[i] = input[i].to_ascii_uppercase();
}
len // returns bytes written
}
// GOOD: return borrowed data when possible
fn find_token<'a>(input: &'a str) -> Option<&'a str> {
input.split_whitespace().next() // no allocation — borrows from input
}
```
### try_* APIs — Fallible Allocation
```rust
// Allocation can fail explicitly (like Zig's allocator returning error)
let mut v = Vec::new();
v.try_reserve(1_000_000)?; // returns Result, not panic
// For Box:
let b = Box::try_new(42)?; // nightly, or use allocator_api
```
### SmallVec / ArrayVec — Stack-First Collections
```rust
use smallvec::SmallVec;
use arrayvec::ArrayVec;
// SmallVec: stack for small counts, heap spillover for large
let mut tags: SmallVec<[u8; 8]> = SmallVec::new();
tags.push(1); // on stack if <= 8 elements
// ArrayVec: purely stack, fixed capacity, no heap ever
let mut buf: ArrayVec<u8, 64> = ArrayVec::new();
buf.try_push(42).map_err(|_| "full")?; // returns error instead of panic
```
### Cow — Defer Allocation Until Mutation
```rust
use std::borrow::Cow;
fn normalize(input: &str) -> Cow<'_, str> {
if input.contains('\t') {
Cow::Owned(input.replace('\t', " ")) // allocates only when needed
} else {
Cow::Borrowed(input) // zero-cost pass-through
}
}
```
### The #![no_std] Discipline
For maximum allocation control, go `#![no_std]`:
```rust
#![no_std]
extern crate alloc; // opt-in to heap when needed
use alloc::vec::Vec; // explicit: I chose to allocate
use alloc::string::String; // explicit: I chose to allocate
```
Even in `std` code, the *mindset* applies: prefer `&[T]` over `Vec<T>` in function signatures, `&str` over `String`, `&Path` over `PathBuf`.
### Clippy Lints for Hidden Allocations
```toml
# Cargo.toml — catch accidental allocations
[lints.clippy]
# These warn on patterns that allocate when a borrow would suffice:
unnecessary_to_owned = "warn" # .to_string() / .to_vec() when borrow works
redundant_clone = "warn" # .clone() that's immediately consumed
large_stack_arrays = "warn" # accidental large stack usage
vec_init_then_push = "warn" # Vec::new() + push instead of vec![]
```
---
## 4. Bit-Level Layout — repr, Packed Structs, Bitfields
Zig: `packed struct` with bit-level field control. Rust matches with `#[repr]` attributes and bitfield crates.
### #[repr(C)] — Guaranteed C-Compatible Layout
```rust
#[repr(C)]
struct Header {
magic: [u8; 4],
version: u16,
flags: u16,
length: u32,
}
// Layout is C ABI: fields in declaration order, C padding rules.
// Safe to transmute from/to byte arrays via zerocopy.
```
### #[repr(C, packed)] — No Padding
```rust
#[repr(C, packed)]
struct WireHeader {
tag: u8,
length: u16, // NOT aligned to 2-byte boundary
checksum: u32,
}
// Total size: exactly 7 bytes. No padding.
// WARNING: taking &self.length is UB if unaligned. Use read_unaligned or zerocopy.
```
**Safe access pattern:**
```rust
use std::ptr;
impl WireHeader {
fn length(&self) -> u16 {
// SAFETY: packed field may be unaligned; ptr::read_unaligned handles this.
unsafe { ptr::read_unaligned(ptr::addr_of!(self.length)) }
}
}
// Better: use zerocopy to avoid manual unsafe entirely
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable};
#[derive(FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C, packed)]
struct WireHeader {
tag: u8,
length: [u8; 2], // manual byte array avoids alignment issues
checksum: [u8; 4],
}
impl WireHeader {
fn length(&self) -> u16 { u16::from_le_bytes(self.length) }
fn checksum(&self) -> u32 { u32::from_le_bytes(self.checksum) }
}
```
### bitfield — Bit-Level Flag Packing
```rust
use bitfield::bitfield;
bitfield! {
pub struct Permissions(u8);
impl Debug;
pub bool, readable, set_readable: 0;
pub bool, writable, set_writable: 1;
pub bool, executable, set_executable: 2;
pub u8, level, set_level: 5, 3; // bits 3-5
}
let mut p = Permissions(0);
p.set_readable(true);
p.set_level(5);
assert!(p.readable());
assert_eq!(p.level(), 5);
```
### modular-bitfield — Richer Bitfield API
```rust
use modular_bitfield::prelude::*;
#[bitfield(bits = 16)]
#[derive(Debug)]
pub struct StatusWord {
ready: bool, // 1 bit
error_code: B4, // 4 bits
#[skip] __: B3, // 3 bits padding
priority: B8, // 8 bits
}
```
### zerocopy — Safe Zero-Copy Parsing
```rust
use zerocopy::{FromBytes, IntoBytes, KnownLayout, Immutable, Ref};
#[derive(FromBytes, IntoBytes, KnownLayout, Immutable)]
#[repr(C)]
struct Packet {
header: [u8; 4],
payload_len: u32,
}
fn parse(bytes: &[u8]) -> Option<&Packet> {
Ref::<_, Packet>::from_prefix(bytes).map(|(pkt, _rest)| pkt.into_ref()).ok()
}
// Zero-copy, zero-allocation, fully safe. No transmute, no pointer cast.
```
### Cargo.toml
```toml
zerocopy = { version = "0.8", features = ["derive"] }
bitfield = "0.17"
modular-bitfield = "0.11"
bytemuck = { version = "1", features = ["derive"] } # alternative to zerocopy
```
---
## 5. Scope Guards — errdefer / Deterministic Cleanup
Zig's `errdefer` runs cleanup only on error paths. Rust's `Drop` always runs, but `scopeguard` gives fine-grained control.
### scopeguard — The errdefer Equivalent
```rust
use scopeguard::{defer, guard};
use std::fs;
fn create_and_process(path: &str) -> std::io::Result<()> {
let file = fs::File::create(path)?;
// If anything below fails, clean up the file.
// This is exactly Zig's errdefer.
let _cleanup = guard((), |_| {
let _ = fs::remove_file(path);
});
write_data(&file)?;
validate_data(path)?;
// Success: defuse the guard so it does NOT run cleanup.
std::mem::forget(_cleanup);
Ok(())
}
```
### defer! — Always-Run Cleanup (like Zig's defer)
```rust
use scopeguard::defer;
fn with_temp_dir() -> anyhow::Result<()> {
let dir = tempfile::tempdir()?;
defer! {
// Runs when scope exits, success or failure.
println!("Cleaning up {}", dir.path().display());
// dir's Drop also cleans up, but this shows the pattern.
}
do_work(dir.path())?;
Ok(())
}
```
### Drop as RAII Cleanup
```rust
struct TempFile { path: std::path::PathBuf }
impl TempFile {
fn new(path: impl Into<std::path::PathBuf>) -> std::io::Result<Self> {
let path = path.into();
std::fs::File::create(&path)?;
Ok(Self { path })
}
}
impl Drop for TempFile {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.path);
}
}
// Usage: file is auto-cleaned when `tmp` goes out of scope.
let tmp = TempFile::new("/tmp/scratch.dat")?;
```
### The errdefer Pattern — Defuse on Success
The key insight from Zig's `errdefer`: you want cleanup on error but NOT on success. In Rust:
```rust
use scopeguard::ScopeGuard;
fn deploy(artifact: &Path) -> Result<(), DeployError> {
let backup = backup_current()?;
// errdefer: restore backup if anything fails
let rollback = guard(backup.clone(), |b| {
let _ = restore_from_backup(&b);
});
upload(artifact)?;
health_check()?;
// Success path: defuse the rollback guard
ScopeGuard::into_inner(rollback);
Ok(())
}
```
### Cargo.toml
```toml
scopeguard = "1"
tempfile = "3" # idiomatic RAII temp files/dirs
```
---
## Summary: Zig Advantage → Rust Pattern
| Zig Feature | Rust Equivalent | Difficulty | Reference |
|---|---|---|---|
| Explicit allocator passing | `bumpalo` / `typed-arena` / `allocator_api` | Easy | §1 |
| `comptime` value computation | `const fn` + `const { }` blocks | Easy | §2 |
| `comptime` type generation | proc macros (derive / attribute) | Medium | §2 |
| No hidden allocations | `#![no_std]` / slice-based APIs / `Cow` | Style choice | §3 |
| `packed struct` / bitfields | `#[repr(C, packed)]` / `bitfield` / `zerocopy` | Easy | §4 |
| `errdefer` | `scopeguard::guard` + defuse on success | Easy | §5 |
| `defer` | `scopeguard::defer!` / `Drop` | Easy | §5 |
All achievable within Rust's single toolchain. You get Zig's explicitness **plus** the borrow checker, lifetime analysis, trait bounds, and `miri`. The combination is strictly more powerful than either alone.
## When NOT to Use These Patterns
- **Arena allocation** overkill for simple CLI tools that allocate once and exit.
- **Zero-alloc APIs** hurt readability when the function naturally produces owned data. Don't force `&mut [u8]` output buffers on a function that logically returns `String`.
- **`#[repr(packed)]`** only for wire formats and FFI. Never for regular domain types.
- **Scope guards** unnecessary when `Drop` on the value itself handles cleanup (e.g., `tempfile::NamedTempFile` already does this).
- **`const fn`** everything? No — only when the value is genuinely needed at compile time or the function is trivially const-eligible. Don't contort logic just to be const.
The goal is **visible costs and explicit control**, not asceticism. Use `String` and `Vec` freely when they're the right tool. Reach for these patterns when allocation behavior matters for correctness or performance.
packages/shared-skills/skills/programming/references/typescript/README.md
+195
View File
@@ -0,0 +1,195 @@
# TypeScript Programmer
Modern TypeScript. Type-strict, stack-first, async-correct.
## Philosophy
The compiler is your proof system. Make illegal states unrepresentable. Parse at boundaries. Every function has a contract; the type system enforces it.
## Hard rules
These are deliberate project choices. Violations are always wrong, not "style preferences".
### Tooling
| Category | Use | Never |
|---|---|---|
| Runtime | Bun (native TS, single binary) | ts-node, tsx |
| Package manager | `pnpm` | npm, yarn (unless workspace requires it) |
| Linter + formatter | Biome | ESLint, Prettier |
| Type checker | `tsc --noEmit` with strict config | skip type checking |
| Web framework | Hono | Express |
| Validation | Zod | joi, yup, class-validator |
| Testing | `bun test` or vitest | jest |
| ORM | Drizzle | TypeORM, Prisma (unless already in project) |
### The iron list
1. **Readonly by default** — all `type`/`interface` properties are `readonly`. Arrays are `readonly T[]`. Mutable only when mutation is the documented purpose.
2. **Branded types for distinct IDs** — `type UserId = Brand<string, "UserId">`. Never pass raw `string` where a branded type exists.
3. **Exhaustive switch** — every `switch` on a discriminated union ends with `default: assertNever(x)`. No fall-through.
4. **No any** — `any` is banned in annotations, returns, and parameters. Use `unknown` and narrow.
5. **No type assertions** — `as any`, `as unknown` banned. `as const` and `satisfies` are fine.
6. **No non-null assertion** — `x!` is banned. Use narrowing or optional chaining (`x?.y`).
7. **No @ts-ignore / @ts-expect-error** — fix the type.
8. **No enum** — use `as const` objects + literal union types.
9. **Zod at boundaries** — external input (API, user, file) → Zod schema. Internal → plain types.
10. **Typed errors** — Error subclasses with typed fields. No `throw new Error("bare string")` for domain errors. Use Result for expected failures within 1-2 call levels; throw for propagation across many layers.
11. **as const for constants** — module-level constant objects and arrays use `as const`.
12. **import type** — type-only imports use `import type`. Enforced by `verbatimModuleSyntax`.
13. **Named exports only** — no `export default`. Exception: framework requirement (Next.js pages, etc.).
14. **No empty catch, no catch-and-swallow** — every `catch` block must either (a) narrow the error with `instanceof` and handle each case, or (b) re-throw. Empty catch blocks and `catch (e) { console.error(e) }` without narrowing or re-throw are banned — they hide bugs. At top-level boundaries (CLI entry, HTTP handler), opt out with `// no-excuse-ok: catch`.
### Data modeling — which construct, when
| Situation | Use |
|---|---|
| User input, API request/response | Zod schema + `z.infer` |
| Internal value object | `type` with `readonly` properties |
| Function with multiple outcomes | Discriminated union (`kind` field) |
| Contract for implementations | `interface` |
| Fixed constants | `as const` + literal union |
| Distinct primitive (UserId vs OrderId) | Branded type |
| Key-value map | `Record<K, V>` or index signature |
**The one rule**: data crosses trust boundary → Zod. Everything else → plain `type` with `readonly`.
Load `data-modeling.md` for the full decision flowchart and comparison.
### When readonly does not apply
- **Framework state** (React `useState`, signals) — managed by framework.
- **Builder / accumulator** — object exists to be mutated (buffer, cache). Document why.
- **ORM mutations** — Drizzle insert/update objects.
### Why empty/unhandled catch is banned
In TypeScript, every `catch` receives `unknown`. The language gives you no type safety in catch blocks — you must earn it with `instanceof`. A bare `catch (e) { console.error(e) }` swallows `TypeError`, `RangeError`, and your domain errors identically. When a new error type appears, nothing warns you.
```typescript
// BANNED — empty catch
try { await fetchData() } catch {}
try { await fetchData() } catch (e) { /* will fix later */ }
// BANNED — catch-and-swallow (no narrowing, no rethrow)
try {
const data = await api.get("/users")
} catch (e) {
console.error("failed", e)
}
// GOOD — narrow with instanceof
try {
const data = await api.get("/users")
} catch (e) {
if (e instanceof HttpError) {
logger.warn(`API ${e.status}: ${e.message}`)
return fallback
}
throw e // unknown errors propagate
}
// GOOD — top-level boundary (only place catch-all is acceptable)
async function main(): Promise<void> { // no-excuse-ok: catch
try {
await run()
} catch (e) {
console.error("unhandled:", e)
process.exit(1)
}
}
```
### Libraries
| Domain | Library | Why |
|---|---|---|
| HTTP framework | Hono | Lightweight, multi-runtime, middleware, OpenAPI |
| Validation | Zod | Runtime validation + type inference |
| ORM | Drizzle | Type-safe SQL, no codegen |
| HTTP client | `ky` | Thin fetch wrapper (5KB); auto-throw on non-2xx, retry, timeout, hooks, prefixUrl. Browser + Node + Bun + Deno |
| HTTP client (perf) | `undici` (direct API) | Node backend에서 connection pooling, HTTP/2, pipelining 필요 시 |
> **HTTP client 규칙** — 프로덕션 코드에서 bare `fetch()`는 사용 금지. retry·timeout·에러 핸들링이 전무하여 장애 시 silent failure를 유발한다. **`ky`** 를 기본으로 설치하고, Node backend에서 대량 요청·커넥션 풀·HTTP/2 파이프라이닝이 필요하면 **`undici`** direct API를 쓴다. ~~`axios`~~ 는 supply-chain compromise(2026-03) 이후 사용 금지. `node-fetch`는 Node 18+ 내장 fetch로 대체되어 불필요.
| Testing | `bun test` / vitest | Fast, ESM-native |
| Logging | `pino` | Structured JSON, fast |
| CLI | `@clack/prompts` + `commander` | Interactive + parsing |
## tsconfig — the one true config
Scaffold a new project with all strict defaults pre-configured:
```bash
bun run ../../scripts/typescript/new-project.ts my-api
bun run ../../scripts/typescript/new-project.ts my-api --path ./projects
```
Creates: `package.json` (Hono + Zod + Biome), `tsconfig.json` (ultra-strict), `biome.json`, `src/index.ts`, `.gitignore`. Works on macOS, Linux, Windows.
For manual setup: `bunx tsc --init`, then load `tsconfig-strict.md` for the full strict config.
Key flags beyond `"strict": true`:
| Flag | What it catches |
|---|---|
| `noUncheckedIndexedAccess` | `arr[0]` is `T \| undefined`, forces check |
| `exactOptionalPropertyTypes` | `{ x?: string }` ≠ `{ x: string \| undefined }` |
| `verbatimModuleSyntax` | Forces `import type` for type-only imports |
| `noFallthroughCasesInSwitch` | Forgotten `break` / `return` |
| `noPropertyAccessFromIndexSignature` | `.key` on index sig → bracket notation |
## Reference loading
Load on demand — not all at once.
| Need | Load |
|---|---|
| Strict tsconfig + Biome config | `tsconfig-strict.md` |
| Type patterns (branded, as const, satisfies, narrowing, assertNever) | `type-patterns.md` |
| Data modeling (type vs interface vs Zod, readonly, parse-don't-validate) | `data-modeling.md` |
| Error handling (Result, typed errors, union vs throw) | `error-handling.md` |
| Bootstrapping a new project (Bun, pnpm, Hono, Vite) | `bootstrap.md` |
| Hono backend stack (hono-openapi, Scalar, Swagger) | `backend-hono.md` |
## No-excuse audit
Violations caught by `../../scripts/typescript/check-no-excuse-rules.ts`. Run after every edit session.
| Rule ID | Catches | Opt-out |
|---|---|---|
| `no-any-assertion` | `as any` | None — redesign types |
| `no-unknown-assertion` | `as unknown` | None — redesign types |
| `no-ts-ignore` | `@ts-ignore` | None — fix the type |
| `no-ts-expect-error` | `@ts-expect-error` | None — fix the type |
| `no-enum` | `enum` declarations | None — use `as const` |
| `no-non-null-assertion` | `x!` postfix | None — narrow or `?.` |
| `no-throw-literal` | `throw "string"` / `throw 123` | None — throw Error subclass |
| `no-mutable-export` | `export let` / `export var` | None — use `export const` |
| `no-any-annotation` | `: any` in parameter/return/variable types | `// no-excuse-ok: any` |
| `no-explicit-any-return` | `(): any` or `(): Promise<any>` return types | `// no-excuse-ok: any` |
| `empty-catch` | `catch { }` or `catch (e) { }` with empty body | `// no-excuse-ok: catch` |
| `catch-without-narrowing` | `catch (e)` used without `instanceof` or re-throw | `// no-excuse-ok: catch` |
Biome enforces additional rules (noExplicitAny, noNonNullAssertion, noDefaultExport, useImportType). The script catches what Biome cannot.
## In tests
Tests are strict too, with these exceptions (configure in `biome.jsonc` overrides):
| In tests you may | Why |
|---|---|
| Use `expect()` assertions | That's how testing works |
| Use magic numbers | Test data |
| Access private members via bracket notation | Testing internals |
| Skip readonly on test fixtures | Mutable setup/teardown |
Tests still follow the iron list — branded types, typed errors, exhaustive switch.
## Existing codebases
When editing an existing file that doesn't follow these rules: **write new code in strict style, don't refactor existing code in the same change.**
## Activation
This skill activates whenever you are writing or modifying any `.ts` or `.tsx` file. Even one-off scripts get the strict treatment.
packages/shared-skills/skills/programming/references/typescript/backend-hono.md
+672
View File
@@ -0,0 +1,672 @@
# Hono Backend Stack Reference (2026)
> **Canonical stack**: `hono` + `hono-openapi` + `@scalar/hono-api-reference` + `@hono/swagger-ui`
> **Runtime**: Bun (TypeScript-first)
> **Validator**: Zod v4 (Standard Schema compliant, zero extra deps for OpenAPI)
---
## 1. Package Versions (Latest Stable)
| Package | Version | Source |
|---------|---------|--------|
| `hono` | `^4.12.5` | [peer dep of scalar](https://github.com/scalar/scalar/blob/8bcf8bf52a0da667d44eeec08648e3b1da044f97/integrations/hono/package.json#L66) |
| `hono-openapi` | `^1.3.0` | [npm](https://registry.npmjs.org/hono-openapi) — published Mar 2, 2026 |
| `@scalar/hono-api-reference` | `^0.10.11` | [npm](https://www.npmjs.com/package/@scalar/hono-api-reference) — published Apr 28, 2026 |
| `@hono/swagger-ui` | `^0.6.1` | [npm](https://www.npmjs.com/package/@hono/swagger-ui) — published Apr 2026 |
| `zod` | `^4.4.1` | [npm registry](https://registry.npmjs.org/zod) — latest stable v4 |
### `package.json` dependency block
```json
{
"dependencies": {
"hono": "^4.12.5",
"hono-openapi": "^1.3.0",
"@scalar/hono-api-reference": "^0.10.11",
"@hono/swagger-ui": "^0.6.1",
"zod": "^4.4.1"
},
"devDependencies": {
"typescript": "^5.8.0",
"@types/bun": "latest"
}
}
```
> **Peer dependencies auto-installed by `hono-openapi`**:
> - `@hono/standard-validator@^0.2.0`
> - `@standard-community/standard-json@^0.3.5`
> - `@standard-community/standard-openapi@^0.2.9`
> - `openapi-types@^12.1.3`
>
> [Source: `hono-openapi/package.json` peerDependencies](https://github.com/rhinobase/hono-openapi/blob/10f45a66ede3764b5e6065805fb60fd5df090466/package.json#L50-L65)
---
## 2. Complete `app.ts` — Copy-Pasteable
```typescript
import { Hono } from 'hono'
import { describeRoute, openAPIRouteHandler, resolver, validator } from 'hono-openapi'
import { Scalar } from '@scalar/hono-api-reference'
import { swaggerUI } from '@hono/swagger-ui'
import { z } from 'zod'
// ───────────────────────────────────────────────────────────────
// 1. Schema definitions (Zod v4 — Standard Schema native)
// ───────────────────────────────────────────────────────────────
const QuerySchema = z.object({
name: z.string().optional(),
})
const ResponseSchema = z.object({
message: z.string(),
})
const JsonBodySchema = z.object({
name: z.string(),
age: z.number().int().min(0),
})
// ───────────────────────────────────────────────────────────────
// 2. Hono app with described routes
// ───────────────────────────────────────────────────────────────
const app = new Hono()
// Health check (no validation)
app.get('/health', (c) => c.json({ status: 'ok' }))
// A fully-documented route
app.get(
'/hello',
describeRoute({
tags: ['Greetings'],
summary: 'Say hello',
description: 'Returns a greeting message',
responses: {
200: {
description: 'Successful greeting',
content: {
'application/json': {
schema: resolver(ResponseSchema),
},
},
},
},
}),
validator('query', QuerySchema),
(c) => {
const query = c.req.valid('query')
return c.json({ message: `Hello ${query.name ?? 'Hono'}!` })
},
)
// A POST route with JSON body validation
app.post(
'/users',
describeRoute({
tags: ['Users'],
summary: 'Create a user',
responses: {
200: {
description: 'User created',
content: {
'application/json': {
schema: resolver(ResponseSchema),
},
},
},
},
}),
validator('json', JsonBodySchema),
(c) => {
const body = c.req.valid('json')
return c.json({ message: `Created user ${body.name}` })
},
)
// ───────────────────────────────────────────────────────────────
// 3. OpenAPI spec endpoint
// ───────────────────────────────────────────────────────────────
app.get(
'/openapi.json',
openAPIRouteHandler(app, {
documentation: {
info: {
title: 'Hono API',
version: '1.0.0',
description: 'Example Hono API with OpenAPI',
},
servers: [
{ url: 'http://localhost:3000', description: 'Local server' },
],
},
}),
)
// ───────────────────────────────────────────────────────────────
// 4. Scalar API Reference UI
// ───────────────────────────────────────────────────────────────
app.get(
'/scalar',
Scalar({
url: '/openapi.json',
theme: 'saturn',
pageTitle: 'Hono API Reference',
}),
)
// ───────────────────────────────────────────────────────────────
// 5. Swagger UI (parallel mount)
// ───────────────────────────────────────────────────────────────
app.get(
'/swagger',
swaggerUI({
url: '/openapi.json',
title: 'Swagger UI',
}),
)
// ───────────────────────────────────────────────────────────────
// 6. Bun canonical entrypoint
// ───────────────────────────────────────────────────────────────
export default app
```
---
## 3. `hono-openapi` API Reference
### Import paths
**There is only one import path.** `hono-openapi` exports everything from its root:
```typescript
import {
describeRoute, // middleware to annotate a route with OpenAPI metadata
describeResponse, // attach response schemas directly to a handler
validator, // validation middleware (wraps @hono/standard-validator)
resolver, // wrap a Zod/Valibot/etc schema for OpenAPI responses
openAPIRouteHandler, // serve the generated OpenAPI JSON document
generateSpecs, // programmatically generate the spec (for build-time caching)
} from 'hono-openapi'
```
**Evidence** ([`src/index.ts`](https://github.com/rhinobase/hono-openapi/blob/10f45a66ede3764b5e6065805fb60fd5df090466/src/index.ts#L1-L9)):
```typescript
export { generateSpecs, openAPIRouteHandler } from "./handler.js";
export {
describeResponse,
describeRoute,
loadVendor,
resolver,
validator,
} from "./middlewares.js";
```
> **No subpath exports** such as `hono-openapi/zod` or `hono-openapi/valibot`. The package uses Standard Schema and auto-detects the validator vendor.
### `describeRoute()` middleware
Attach OpenAPI metadata to any Hono route. Use `resolver()` for response body schemas.
```typescript
app.get(
'/path',
describeRoute({
tags: ['Users'],
summary: 'Get user',
description: 'Retrieve a single user by ID',
responses: {
200: {
description: 'User found',
content: {
'application/json': {
schema: resolver(UserSchema),
},
},
},
404: {
description: 'User not found',
},
},
}),
handler,
)
```
**Evidence** ([`src/middlewares.ts`](https://github.com/rhinobase/hono-openapi/blob/10f45a66ede3764b5e6065805fb60fd5df090466/src/middlewares.ts#L244-L254)):
```typescript
export function describeRoute(spec: DescribeRouteOptions): MiddlewareHandler {
const middleware: MiddlewareHandler = async (_c, next) => {
await next();
};
return Object.assign(middleware, {
[uniqueSymbol]: { spec },
});
}
```
### `validator()` middleware
Validates `query`, `json`, `param`, or `form` and **automatically** injects the request schema into the OpenAPI document. No manual `request` block in `describeRoute()` is required.
```typescript
validator('query', QuerySchema) // ?name=foo
validator('json', JsonBodySchema) // POST body
validator('param', ParamSchema) // /users/:id
validator('form', FormSchema) // multipart/form-data
```
**Evidence** ([`src/middlewares.ts`](https://github.com/rhinobase/hono-openapi/blob/10f45a66ede3764b5e6065805fb60fd5df090466/src/middlewares.ts#L199-L237)):
```typescript
export function validator<Schema extends StandardSchemaV1, ...>(
target: Target,
schema: Schema,
hook?: Hook<...>,
options?: ResolverReturnType["options"],
): MiddlewareHandler<E, P, V> {
const middleware = sValidator(target, schema, hook);
return Object.assign(middleware, {
[uniqueSymbol]: { target, ...resolver(schema, options), options },
});
}
```
### `openAPIRouteHandler()` — serving the spec
```typescript
app.get(
'/openapi.json',
openAPIRouteHandler(app, {
documentation: {
info: { title: 'Hono API', version: '1.0.0' },
servers: [{ url: 'http://localhost:3000' }],
},
}),
)
```
**Evidence** ([`src/handler.ts`](https://github.com/rhinobase/hono-openapi/blob/10f45a66ede3764b5e6065805fb60fd5df090466/src/handler.ts#L42-L59)):
```typescript
export function openAPIRouteHandler<...>(
hono: Hono<E, S, P>,
options?: Partial<GenerateSpecOptions>,
): MiddlewareHandler<E, P, I> {
let specs: OpenAPIV3_1.Document;
return async (c) => {
if (specs) return c.json(specs);
specs = await generateSpecs(hono, options, c);
return c.json(specs);
};
}
```
> **Mount path convention**: `/openapi.json` is the most common. Some projects use `/openapi/spec.json` (e.g. [NamesMT/starter-monorepo](https://github.com/NamesMT/starter-monorepo/blob/main/apps/backend/src/openAPI.ts)).
---
## 4. `@scalar/hono-api-reference` Setup
### Import path and package name
```typescript
import { Scalar } from '@scalar/hono-api-reference'
```
> **Deprecated**: `apiReference` is still exported but deprecated in favor of `Scalar` ([PR #5297](https://github.com/scalar/scalar/pull/5297)).
**Evidence** ([`integrations/hono/src/index.ts`](https://github.com/scalar/scalar/blob/8bcf8bf52a0da667d44eeec08648e3b1da044f97/integrations/hono/src/index.ts#L1-L9)):
```typescript
import { Scalar } from './scalar'
export {
Scalar,
/**
* @deprecated Use `Scalar` instead.
*/
Scalar as apiReference,
}
```
### Mount path convention
Common choices:
- `/scalar` — matches the middleware name
- `/docs` — generic documentation endpoint
- `/openapi/ui` — nested under the OpenAPI prefix
### Configuration options
The Hono middleware accepts the **universal Scalar configuration** plus Hono-specific overrides (`pageTitle`, `cdn`).
```typescript
app.get('/scalar', Scalar({
// ── Source (required) ──
url: '/openapi.json', // URL to the OpenAPI spec
// ── Appearance ──
theme: 'saturn', // 'alternate' | 'default' | 'moon' | 'purple'
// | 'solarized' | 'bluePlanet' | 'deepSpace'
// | 'saturn' | 'kepler' | 'elysiajs' | 'fastify'
// | 'mars' | 'laserwave' | 'none'
pageTitle: 'My API Docs', // HTML <title>
customCss: '.sidebar { ... }', // injected <style> block
metaData: { title: '...' }, // SEO meta tags (unhead format)
favicon: '/favicon.svg',
// ── Behavior ──
layout: 'modern', // 'modern' | 'classic'
darkMode: true,
forceDarkModeState: 'dark', // 'dark' | 'light'
hideDarkModeToggle: false,
hideModels: false,
hideSearch: false,
hideTestRequestButton: false,
showOperationId: false,
showSidebar: true,
// ── Proxy / Server ──
proxyUrl: 'https://proxy.scalar.com',
baseServerURL: 'http://localhost:3000',
servers: [{ url: 'http://localhost:3000' }],
// ── CDN ──
cdn: 'https://cdn.jsdelivr.net/npm/@scalar/api-reference',
// ── Advanced ──
authentication: { ... },
hiddenClients: ['unirest'],
defaultHttpClient: { targetKey: 'js', clientKey: 'fetch' },
plugins: [...],
pathRouting: { basePath: '/reference' },
mcp: { name: 'My MCP', url: '...' },
}))
```
**Evidence** — Scalar types define the full schema:
- [Base configuration (themes, proxy, etc.)](https://github.com/scalar/scalar/blob/8bcf8bf52a0da667d44eeec08648e3b1da044f97/packages/types/src/api-reference/base-configuration.ts#L110-L129)
- [HTML rendering configuration (`pageTitle`, `cdn`)](https://github.com/scalar/scalar/blob/8bcf8bf52a0da667d44eeec08648e3b1da044f97/packages/types/src/api-reference/html-rendering-configuration.ts#L8-L23)
- [Source configuration (`url`, `content`)](https://github.com/scalar/scalar/blob/8bcf8bf52a0da667d44eeec08648e3b1da044f97/packages/types/src/api-reference/source-configuration.ts#L8-L55)
- [Full API reference configuration](https://github.com/scalar/scalar/blob/8bcf8bf52a0da667d44eeec08648e3b1da044f97/packages/types/src/api-reference/api-reference-configuration.ts#L22-L379)
### Dynamic configuration (request-aware)
```typescript
app.get('/scalar', Scalar((c) => ({
url: '/openapi.json',
proxyUrl: c.env.ENVIRONMENT === 'development'
? 'https://proxy.scalar.com'
: undefined,
})))
```
**Evidence** ([`integrations/hono/src/scalar.ts`](https://github.com/scalar/scalar/blob/8bcf8bf52a0da667d44eeec08648e3b1da044f97/integrations/hono/src/scalar.ts#L75-L94)):
```typescript
export const Scalar = <E extends Env>(configOrResolver: Configuration<E>): MiddlewareHandler<E> => {
return async (c) => {
let resolvedConfig: Partial<ApiReferenceConfiguration> = {}
if (typeof configOrResolver === 'function') {
resolvedConfig = await configOrResolver(c)
} else {
resolvedConfig = configOrResolver
}
// ...
}
}
```
---
## 5. `@hono/swagger-ui` Setup
### Import path and package name
```typescript
import { swaggerUI } from '@hono/swagger-ui'
```
**Evidence** ([`packages/swagger-ui/src/index.ts`](https://github.com/honojs/middleware/blob/eb443a2fbda674bbe12d3f30e96854bb0cad6232/packages/swagger-ui/src/index.ts#L93)):
```typescript
export { middleware as swaggerUI, SwaggerUI }
```
### Mount path convention
Common choices:
- `/swagger` — explicit
- `/ui` — used in Hono official examples
- `/docs` — generic
### Configuration options
```typescript
app.get('/swagger', swaggerUI({
url: '/openapi.json', // URL to the OpenAPI spec (required)
title: 'Swagger UI', // HTML page title
version: 'latest', // Swagger UI CDN version
// Any standard Swagger UI option also works:
// presets, plugins, urls, etc.
}))
```
**Evidence** ([`packages/swagger-ui/src/index.ts`](https://github.com/honojs/middleware/blob/eb443a2fbda674bbe12d3f30e96854bb0cad6232/packages/swagger-ui/src/index.ts#L8-L43)):
```typescript
type SwaggerUIOptions = OriginalSwaggerUIOptions & DistSwaggerUIOptions
const middleware = <E extends Env>(options: SwaggerUIOptions): MiddlewareHandler<E> =>
async (c) => {
const title = options?.title ?? 'SwaggerUI'
return c.html(/* html */ `...`)
}
```
---
## 6. Bun Runtime Entrypoint
### Canonical shape for `bun run`
```typescript
import { Hono } from 'hono'
const app = new Hono()
// ... routes ...
export default app
```
**Evidence** ([Hono Bun docs](https://hono.dev/docs/getting-started/bun)):
> ```ts
> import { Hono } from 'hono'
> const app = new Hono()
> app.get('/', (c) => c.text('Hello Bun!'))
> export default app
> ```
### Custom port
```typescript
export default {
port: 3000,
fetch: app.fetch,
}
```
**Evidence** ([Hono Bun docs — Change port number](https://hono.dev/docs/getting-started/bun)):
> ```ts
> export default {
> port: 3000,
> fetch: app.fetch,
> }
> ```
### `package.json` scripts for Bun
```json
{
"scripts": {
"dev": "bun run --hot src/index.ts",
"start": "bun run src/index.ts",
"build": "tsc --noEmit"
}
}
```
---
## 7. OpenAPI Version
### `hono-openapi` emits **OpenAPI 3.1.0** by default
This is **hardcoded** in the source and **not configurable** at runtime:
**Evidence** ([`src/handler.ts` line 120](https://github.com/rhinobase/hono-openapi/blob/10f45a66ede3764b5e6065805fb60fd5df090466/src/handler.ts#L120)):
```typescript
return {
openapi: "3.1.0",
..._documentation,
// ...
} satisfies OpenAPIV3_1.Document;
```
> If you need OpenAPI 3.0.x, you must post-process the generated spec or use `@hono/zod-openapi` (the older package) instead. The user explicitly requested `hono-openapi`, so document that 3.1.0 is the only output.
---
## 8. Zod v3 vs Zod v4
| Feature | Zod v3 | Zod v4 |
|---------|--------|--------|
| Standard Schema | ❌ No | ✅ Yes (native) |
| `hono-openapi` extra deps | `zod-openapi@4` | None |
| Import path | `import { z } from 'zod'` | `import { z } from 'zod'` (or `zod/v4` for explicit) |
**For Zod v3 users**, install the compatibility layer:
```bash
npm install zod-openapi@4
```
Then use `zod-openapi`'s `.openapi()` for metadata and `.meta({ ref: 'Name' })` for component references. `hono-openapi`'s `resolver()` will still work, but the underlying schema conversion relies on `zod-openapi@4`.
**Evidence** ([HonoHub Zod docs](https://honohub.dev/docs/openapi/zod)):
> "For zod v3, you can use the `zod-openapi` library. You need to install `zod-openapi@4` for this to work properly."
**For Zod v4 users** (recommended in 2026), no extra packages are needed. `z.date()` is automatically converted to `{ type: 'string', format: 'date-time' }`.
**Evidence** ([`src/middlewares.ts` Zod v4 date override](https://github.com/rhinobase/hono-openapi/blob/10f45a66ede3764b5e6065805fb60fd5df090466/src/middlewares.ts#L63-L71)):
```typescript
const zodV4DateOverride = (ctx: { ... }) => {
if (ctx.zodSchema._zod.def.type === "date") {
ctx.jsonSchema.type = "string";
ctx.jsonSchema.format = "date-time";
}
};
```
---
## 9. Real-World Example
**NamesMT/starter-monorepo** — a public monorepo starter using `hono-openapi` + `@scalar/hono-api-reference` together:
- File: [`apps/backend/src/openAPI.ts`](https://github.com/NamesMT/starter-monorepo/blob/main/apps/backend/src/openAPI.ts)
- Pattern: mounts spec at `/openapi/spec.json` and Scalar UI at `/openapi/ui`
```typescript
import type { Hono } from 'hono'
import { Scalar } from '@scalar/hono-api-reference'
import { openAPIRouteHandler } from 'hono-openapi'
export function setupOpenAPI(app: Hono<any, any>, prefix = '/openapi') {
app.get(
`${prefix}/spec.json`,
openAPIRouteHandler(app, {
documentation: {
info: {
title: `starter-monorepo's backend`,
version: '1.0.0',
description: 'My amazing API',
},
},
}),
)
app.get(
`${prefix}/ui`,
Scalar({
theme: 'deepSpace',
url: `${prefix}/spec.json`,
}),
)
}
```
> **Note**: No public repo was found using all four (`hono-openapi` + `Scalar` + `swagger-ui` + `hono`) in a single file. The canonical combination in the wild is `hono-openapi` + `Scalar`. Adding `swagger-ui` is a trivial parallel mount (shown in the `app.ts` above).
---
## 10. Common Pitfalls
1. **Using `openAPISpecs` instead of `openAPIRouteHandler`**
Some docs (e.g. HONC) use `openAPISpecs` — this is **not** the current export name. The correct function is `openAPIRouteHandler` ([source](https://github.com/rhinobase/hono-openapi/blob/10f45a66ede3764b5e6065805fb60fd5df090466/src/index.ts#L1)).
2. **Importing from `hono-openapi/zod`**
There are **no subpath exports**. Always import from `hono-openapi` directly.
3. **Forgetting `@hono/standard-validator`**
It is a peer dependency of `hono-openapi`. Modern package managers (npm ≥ 7, pnpm, bun) auto-install it. If you see validation errors, ensure it is present in `node_modules`.
4. **Using `@hono/zod-openapi` (the OLD package)**
The user explicitly wants `hono-openapi` (the newer, middleware-based, Standard Schema package). Do not confuse with `@hono/zod-openapi` which wraps the `Hono` class into `OpenAPIHono`.
5. **Swagger UI `spec` option**
`@hono/swagger-ui` does **not** accept a `spec` option to embed the document directly. It only accepts `url` (or `urls`) pointing to an external spec endpoint. If you need embedded specs, use Scalar's `content` option instead.
---
## 11. Quick Start Commands
```bash
# 1. Create project
mkdir my-api && cd my-api
bun init -y
# 2. Install dependencies
bun add hono hono-openapi @scalar/hono-api-reference @hono/swagger-ui zod
# 3. Add TypeScript
bun add -d typescript @types/bun
# 4. Write app.ts (copy from section 2 above)
# 5. Run
bun run --hot app.ts
```
Endpoints after startup:
- `GET /health` — health check
- `GET /hello?name=world` — documented route
- `POST /users` — validated JSON body route
- `GET /openapi.json` — raw OpenAPI 3.1.0 spec
- `GET /scalar` — Scalar API Reference UI
- `GET /swagger` — Swagger UI
packages/shared-skills/skills/programming/references/typescript/bootstrap.md
+199
View File
@@ -0,0 +1,199 @@
# Bootstrap — Runtime, Package Manager, Tooling
When starting a new TypeScript project (or scripting against the world), the choice of runtime, package manager, framework, and toolchain compounds. The wrong default at minute zero costs hours every week. The right defaults for 2026:
## Runtime decision tree
```
Is this a CLI / script / single-binary tool?
└─ Yes → Bun (single executable, hot reload, native TS)
Use `bun run script.ts` directly. No build step.
Is this a backend service?
├─ Edge (Cloudflare Workers / Vercel / Deno Deploy) → match the platform
├─ Bun-supported runtime → Bun + Hono
├─ Need Node-only deps (sharp, native modules without Bun support) → Node + Hono
└─ Otherwise → Bun + Hono
Is this a frontend?
└─ Vite (regardless of framework). Bun for the package manager.
Is this a library to publish to npm?
└─ tsdown (or unbuild). Targets Node 20+. Use pnpm for monorepo workspaces.
```
## Bun is the default runtime
Use Bun for:
- Scripts and CLIs (`bun run` is faster than `tsx` and `ts-node`)
- New backends (Hono runs natively, hot reload via `bun --hot`)
- Test runner (`bun test` is built-in, faster than vitest for small suites)
- Package manager (`bun install` is faster than `pnpm` and far faster than `npm`)
Use Node when:
- A dependency uses native modules Bun can't load (rare in 2026; check the dep's release notes)
- Production target is a Node-specific platform (some serverless platforms don't run Bun yet)
- You're contributing to a Node-only project
`bunx` replaces `npx`. `bun create` scaffolds projects.
## Package manager — pnpm > npm
If you must use Node, use pnpm. NEVER npm except in legacy projects you don't control.
Why pnpm:
- Content-addressable store: 10x less disk usage on a machine with many projects
- Strict node_modules layout: phantom dependencies fail at install time, not at runtime
- Workspaces are first-class
- Significantly faster than npm
Why not yarn:
- Yarn classic is unmaintained
- Yarn berry's "PnP" mode breaks with editor tooling more often than it should
- pnpm has caught up on every yarn berry feature people actually use
Why not npm:
- Slowest of the three
- No proper workspace story until very recently
- Phantom dependencies allowed by default
```bash
# Convert npm/yarn → pnpm
pnpm import # reads package-lock.json or yarn.lock and produces pnpm-lock.yaml
rm -rf node_modules package-lock.json yarn.lock
pnpm install
```
## Backend framework — Hono
Use Hono for any new HTTP service. It is:
- Type-safe end-to-end (request/response types flow through middleware)
- Edge-compatible (runs on Bun, Node, Cloudflare Workers, Deno, AWS Lambda)
- Faster than Express, Fastify, and most of its peers in synthetic benchmarks
- Maintained, opinionated, and documented well
When Hono → ALWAYS pair with `hono-openapi` + `@scalar/hono-api-reference` + `@hono/swagger-ui`. Full setup with copy-pasteable `app.ts`: [backend-hono.md](backend-hono.md).
NEVER:
- Express for new services. Express is the COBOL of Node — works, but writes itself out of every benchmark.
- Fastify for new services. Hono ships with better TypeScript ergonomics.
- NestJS for new services. The Angular-flavoured DI/decorator stack is overkill for ~95% of services.
- Bare `Bun.serve` or `node:http` unless you have a specific reason. Lose middleware, routing, validation. Reinvent everything.
## Frontend tooling — Vite
Vite for any frontend. Replaces webpack, parcel, rollup-as-app-bundler. Works with React, Vue, Svelte, Solid, Preact, vanilla.
```bash
bun create vite my-app -- --template react-ts
cd my-app
bun install
bun run dev
```
## Lint + format — Biome
Biome replaces ESLint + Prettier with one tool, written in Rust, ~30x faster.
```bash
bun add --dev @biomejs/biome
bun biome init
```
`biome.json`:
```json
{
"$schema": "https://biomejs.dev/schemas/1.9.0/schema.json",
"organizeImports": { "enabled": true },
"linter": { "enabled": true, "rules": { "recommended": true } },
"formatter": { "enabled": true, "indentStyle": "space", "indentWidth": 2 }
}
```
Use ESLint only when:
- You have an ESLint plugin Biome doesn't replicate (rare in 2026)
- You're contributing to an existing ESLint project
Never run both — pick one.
## Test runner — bun test or vitest
| Runner | Use when |
|---|---|
| `bun test` | Bun project, simple unit tests, no TypeScript path aliases that need vite-style resolution |
| `vitest` | Vite-based frontend, complex test infrastructure (DOM testing, snapshot, in-browser tests), or you need vitest-specific features |
NEVER Jest for a new project. Jest's CommonJS-first design fights every modern Node/TS project.
## TypeScript
`tsconfig.json` for a Bun + Hono backend:
```json
{
"compilerOptions": {
"target": "ESNext",
"module": "ESNext",
"moduleResolution": "bundler",
"lib": ["ESNext"],
"types": ["bun-types"],
"strict": true,
"noUncheckedIndexedAccess": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noImplicitReturns": true,
"noFallthroughCasesInSwitch": true,
"exactOptionalPropertyTypes": true,
"esModuleInterop": true,
"isolatedModules": true,
"resolveJsonModule": true,
"skipLibCheck": true,
"verbatimModuleSyntax": true,
"noEmit": true
},
"include": ["src/**/*", "tests/**/*"]
}
```
`verbatimModuleSyntax: true` enforces explicit `import type { ... }` for type-only imports — pairs with the no-excuse rule on type-only imports.
`noEmit: true` because `bun run` and `bun build` handle compilation. The `tsc` command becomes a typechecker only.
## Quick-start: Bun + Hono backend
```bash
mkdir my-api && cd my-api
bun init -y
bun add hono hono-openapi @scalar/hono-api-reference @hono/swagger-ui zod
bun add --dev @biomejs/biome typescript
bun biome init
```
`package.json` scripts:
```json
{
"scripts": {
"dev": "bun run --hot src/index.ts",
"start": "bun run src/index.ts",
"build": "bun build src/index.ts --target bun --outdir dist",
"typecheck": "tsc --noEmit",
"lint": "biome check --write src tests",
"test": "bun test"
}
}
```
Wire the `app.ts` from [backend-hono.md](backend-hono.md). You have a documented, validated, OpenAPI-spec-emitting service in ~15 minutes.
## When NOT to bootstrap from scratch
| Situation | Use |
|---|---|
| Internal tool with auth/admin/dashboards | Next.js (full-stack) - lots of free wiring |
| Documentation site | Astro or VitePress |
| Real-time features (WebRTC, complex sockets) | Bun + Hono + a real-time library |
| Data-heavy SPA | Vite + React + TanStack Query + TanStack Router |
For greenfield backend services, Bun + Hono. Always.
packages/shared-skills/skills/programming/references/typescript/data-modeling.md
+202
View File
@@ -0,0 +1,202 @@
# Data Modeling
Which construct to use, how to structure data, and why readonly is the default.
---
## Decision flowchart
```
Is it a fixed set of named constants?
YES → as const object + literal union type
NO ↓
Is it just branding a primitive (string, number)?
YES → Branded type
NO ↓
Is it an interface / contract?
YES → interface (structural typing is the default in TS)
NO ↓
Does the data cross a trust boundary (user input, API, file)?
YES → Zod schema + z.infer<typeof schema>
NO ↓
Is it a union of possible outcomes?
YES → Discriminated union (kind/type field)
NO ↓
Is it structured data with named fields?
YES → type alias with readonly properties
NO → you probably don't need a new type
```
---
## Container reference
### type alias — internal data
The default for structured data inside your codebase. Zero runtime cost.
```typescript
type User = {
readonly id: UserId
readonly name: string
readonly email: string
}
type Point = {
readonly x: number
readonly y: number
}
```
All properties `readonly`. Mutable only when mutation is the documented purpose.
### interface — contracts and extension
Use when you need declaration merging or `extends`.
```typescript
interface Repository<T> {
get(id: string): Promise<T | null>
save(entity: T): Promise<void>
}
interface UserRepository extends Repository<User> {
findByEmail(email: string): Promise<User | null>
}
```
### interface vs type — when to use which
| Use | When |
|---|---|
| `type` | Union types, intersections, mapped types, utility types, internal data shapes |
| `interface` | Contracts that will be `implements`ed or `extends`ed, declaration merging needed |
| **Default** | **`type` — unless you have a specific reason for `interface`** |
### Zod schema — trust boundary guardian
Use when data enters your system. Validates at runtime, infers types at compile time.
```typescript
import { z } from "zod"
const CreateUserSchema = z.object({
name: z.string().min(1),
email: z.string().email(),
age: z.number().int().min(0),
})
type CreateUser = z.infer<typeof CreateUserSchema>
const UserResponseSchema = z.object({
id: z.string().uuid(),
name: z.string(),
email: z.string(),
})
type UserResponse = z.infer<typeof UserResponseSchema>
```
**The one rule**: data crosses a trust boundary → Zod. Everything else → plain type/interface.
Never use Zod for internal-only data. The runtime validation cost and Zod coupling are unnecessary.
### as const — fixed constants
Replaces `enum` entirely. Type-safe, tree-shakeable, no runtime overhead.
```typescript
const ROLES = ["admin", "user", "guest"] as const
type Role = (typeof ROLES)[number]
const STATUS = {
ACTIVE: "active",
INACTIVE: "inactive",
DELETED: "deleted",
} as const
type Status = (typeof STATUS)[keyof typeof STATUS]
```
### Discriminated union — multiple outcomes
```typescript
type GetUserResult =
| { readonly kind: "found"; readonly user: User }
| { readonly kind: "not_found"; readonly id: UserId }
| { readonly kind: "forbidden"; readonly reason: string }
```
Each variant has a `kind` discriminant. TypeScript narrows on `switch (result.kind)`.
---
## Quick lookup
| Situation | Use |
|---|---|
| User input, API request/response | Zod schema + `z.infer` |
| Internal value object | `type` with `readonly` properties |
| Function with multiple outcomes | Discriminated union |
| Contract for implementations | `interface` |
| Fixed constants | `as const` + literal union |
| Distinct primitive (UserId vs OrderId) | Branded type |
| Dict shape / key-value map | `Record<K, V>` or index signature |
---
## Readonly by default
Every property is `readonly` unless mutation is the documented purpose.
```typescript
// DEFAULT — readonly
type Config = {
readonly apiUrl: string
readonly timeout: number
}
// Arrays too
function getUsers(): readonly User[] { ... }
// Utility for existing types
type ReadonlyUser = Readonly<User>
type DeepReadonlyConfig = Readonly<Config>
```
For mutable state (rare), document why:
```typescript
/** Counter state — mutation is the entire purpose. */
type CounterState = {
count: number // intentionally mutable
}
```
---
## Parse, don't validate
Validate at the boundary. Inside the boundary, types are proof of validity.
```typescript
// BAD — validate then pass raw data
function processEmail(email: string): void {
if (!email.includes("@")) throw new Error("invalid")
// still a raw string downstream
}
// GOOD — parse into typed value at boundary
const EmailSchema = z.string().email().brand("Email")
type Email = z.infer<typeof EmailSchema>
function sendWelcome(email: Email): void { ... }
// Boundary code
const parsed = EmailSchema.parse(rawInput) // Email or throws
sendWelcome(parsed) // no re-validation needed
```
---
## Sources
- TypeScript Handbook: [Object Types](https://www.typescriptlang.org/docs/handbook/2/objects.html)
- Zod: [docs](https://zod.dev)
- Total TypeScript: [Type vs Interface](https://www.totaltypescript.com/type-vs-interface-which-should-you-use)
packages/shared-skills/skills/programming/references/typescript/error-handling.md
+169
View File
@@ -0,0 +1,169 @@
# Error Handling
Typed errors, exhaustive matching, Result pattern, and resource safety.
---
## Typed errors — no bare strings
Error classes carry structured data. Callers know exactly what can go wrong.
```typescript
class UserNotFoundError extends Error {
readonly name = "UserNotFoundError"
constructor(readonly userId: UserId) {
super(`user ${userId} not found`)
}
}
class PermissionDeniedError extends Error {
readonly name = "PermissionDeniedError"
constructor(
readonly userId: UserId,
readonly requiredRole: string,
) {
super(`user ${userId} needs role ${requiredRole}`)
}
}
```
```typescript
// BAD
throw new Error("user not found")
throw new Error("permission denied")
// GOOD
throw new UserNotFoundError(userId)
throw new PermissionDeniedError(userId, "admin")
```
Always set `readonly name` explicitly — `instanceof` checks survive minification, but `error.name` is more reliable for logging and serialization.
---
## Result pattern — expected failures without exceptions
For failures that are **expected** (not found, validation), return a discriminated union instead of throwing.
```typescript
type Result<T, E = Error> =
| { readonly ok: true; readonly value: T }
| { readonly ok: false; readonly error: E }
function ok<T>(value: T): Result<T, never> {
return { ok: true, value }
}
function err<E>(error: E): Result<never, E> {
return { ok: false, error }
}
```
### Usage
```typescript
type UserError =
| { readonly kind: "not_found"; readonly id: UserId }
| { readonly kind: "forbidden"; readonly reason: string }
function getUser(id: UserId): Result<User, UserError> {
const user = db.find(id)
if (!user) return err({ kind: "not_found", id })
if (!user.active) return err({ kind: "forbidden", reason: "deactivated" })
return ok(user)
}
// Caller must handle both cases
const result = getUser(userId)
if (!result.ok) {
switch (result.error.kind) {
case "not_found":
log.warn(`missing: ${result.error.id}`)
break
case "forbidden":
log.error(`denied: ${result.error.reason}`)
break
default:
assertNever(result.error)
}
return
}
const user = result.value // narrowed to User
```
### When to use which
**The heuristic**: caller is 1-2 levels away and MUST handle it → Result. Error should propagate up many layers → throw.
| Scenario | Pattern | Why |
|---|---|---|
| Repository → service (caller handles it) | Result | Caller is right there, must handle both |
| Validation at boundary (parsing input) | throw (Zod throws) | Propagates up to HTTP handler |
| Infrastructure failure (network, OOM) | throw | Can't handle locally |
| Service → service (deep internal) | throw (typed Error subclass) | Result boilerplate across many layers is worse |
| HTTP handler → response | Catch errors, convert to response | Boundary code catches and translates |
**Practical tradeoff**: Result is safest (compiler forces handling) but creates boilerplate when every caller in a chain must check `.ok`. If the error would just propagate through 3+ layers unchanged, use a typed Error subclass instead.
### Library or roll your own?
Roll your own with the `Result`, `ok`, `err` above. It's 10 lines. Libraries like `neverthrow` add chaining (`.map`, `.andThen`) — use them only if you actually chain results frequently.
---
## Error cause — chain context
Use the `cause` option to chain errors without losing the original stack.
```typescript
try {
await db.query(sql)
} catch (error) {
throw new DatabaseError("query failed", { cause: error })
}
```
The `cause` is available on `error.cause` and shows up in stack traces.
---
## Exhaustive error handling at boundaries
HTTP handlers catch and translate:
```typescript
app.onError((error, c) => {
if (error instanceof UserNotFoundError) {
return c.json({ error: error.message }, 404)
}
if (error instanceof PermissionDeniedError) {
return c.json({ error: error.message }, 403)
}
console.error("unhandled:", error)
return c.json({ error: "internal server error" }, 500)
})
```
---
## Async error patterns
```typescript
// Promise.allSettled — when partial failure is OK
const results = await Promise.allSettled(urls.map(fetch))
const successes = results
.filter((r): r is PromiseFulfilledResult<Response> => r.status === "fulfilled")
.map((r) => r.value)
// AbortSignal — cancellation
async function fetchWithTimeout(url: string, ms: number): Promise<Response> {
return fetch(url, { signal: AbortSignal.timeout(ms) })
}
```
---
## Sources
- MDN: [Error cause](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/cause)
- MDN: [Promise.allSettled](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled)
packages/shared-skills/skills/programming/references/typescript/tsconfig-strict.md
+152
View File
@@ -0,0 +1,152 @@
# Strict tsconfig + Biome
The canonical ultra-strict config. Copy-paste, then add your own paths.
---
## tsconfig.json
```jsonc
{
"compilerOptions": {
// ── Strict core ──────────────────────────────────────────
"strict": true, // enables all strict* flags below
// strict includes: strictNullChecks, strictFunctionTypes,
// strictBindCallApply, strictPropertyInitialization,
// noImplicitAny, noImplicitThis, alwaysStrict, useUnknownInCatchVariables
// ── Additional strict flags (NOT included in "strict") ──
"noUncheckedIndexedAccess": true, // obj[key] is T | undefined, not T
"exactOptionalPropertyTypes": true, // { x?: string } !== { x: string | undefined }
"noFallthroughCasesInSwitch": true, // switch fall-through is an error
"noPropertyAccessFromIndexSignature": true, // forces bracket notation for index sigs
"forceConsistentCasingInFileNames": true, // prevents case-sensitivity bugs on macOS/Win
// ── Module system ────────────────────────────────────────
"module": "ESNext",
"moduleResolution": "bundler",
"verbatimModuleSyntax": true, // forces `import type` for type-only imports
"isolatedModules": true, // safe for esbuild / swc / Bun transpilation
"esModuleInterop": true,
"resolveJsonModule": true,
// ── Target ───────────────────────────────────────────────
"target": "ESNext",
"lib": ["ESNext"],
// ── Emit ─────────────────────────────────────────────────
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"outDir": "dist",
"rootDir": "src",
// ── Performance ──────────────────────────────────────────
"skipLibCheck": true, // skip checking .d.ts files for speed
"incremental": true
},
"include": ["src"],
"exclude": ["node_modules", "dist"]
}
```
### What each extra flag catches
| Flag | What it prevents |
|---|---|
| `noUncheckedIndexedAccess` | `arr[0]` is `T \| undefined`, not `T`. Forces you to check before using. |
| `exactOptionalPropertyTypes` | `{ x?: string }` means "missing or string", NOT "string \| undefined". Assigns `undefined` explicitly? Type error. |
| `noFallthroughCasesInSwitch` | Forgetting `break` / `return` in a switch case. |
| `noPropertyAccessFromIndexSignature` | `obj.foo` on `Record<string, X>` is an error. Use `obj["foo"]`. |
| `verbatimModuleSyntax` | Forces `import type { X }` for type-only imports. Prevents runtime import of types. |
### Bun-specific additions
For Bun projects, add to `compilerOptions`:
```jsonc
{
"types": ["bun-types"],
"moduleDetection": "force"
}
```
---
## biome.jsonc
```jsonc
{
"$schema": "https://biomejs.dev/schemas/2.0.6/schema.json",
"organizeImports": {
"enabled": true
},
"linter": {
"enabled": true,
"rules": {
"recommended": true,
"suspicious": {
"noExplicitAny": "error",
"noConfusingVoidType": "error",
"noFallthroughSwitchClause": "error"
},
"style": {
"noDefaultExport": "error",
"useImportType": "error",
"noNonNullAssertion": "error",
"useEnumInitializers": "off",
"noParameterAssign": "error"
},
"correctness": {
"noUnusedVariables": "error",
"noUnusedImports": "error"
},
"complexity": {
"noBannedTypes": "error"
}
}
},
"formatter": {
"enabled": true,
"indentStyle": "space",
"indentWidth": 2,
"lineWidth": 100
},
"javascript": {
"formatter": {
"quoteStyle": "double",
"semicolons": "asNeeded"
}
},
"files": {
"ignore": ["node_modules", "dist", "build", ".next", ".nuxt", "coverage"]
}
}
```
### Key Biome rules
| Rule | What |
|---|---|
| `noExplicitAny` | `any` in annotations is an error |
| `noNonNullAssertion` | `x!` is an error |
| `noDefaultExport` | Forces named exports |
| `useImportType` | Forces `import type` for type-only imports |
| `noParameterAssign` | No mutation of function parameters |
---
## CI gate
```bash
bunx biome check .
bunx tsc --noEmit
bun test
```
---
## Sources
- TypeScript: [tsconfig reference](https://www.typescriptlang.org/tsconfig)
- Biome: [configuration](https://biomejs.dev/reference/configuration/)
- Total TypeScript: [tsconfig cheat sheet](https://www.totaltypescript.com/tsconfig-cheat-sheet)
packages/shared-skills/skills/programming/references/typescript/type-patterns.md
+196
View File
@@ -0,0 +1,196 @@
# Type Patterns
How to use TypeScript's type system to catch bugs at compile time.
---
## Branded types — distinct primitives
Same runtime type, different meaning. The compiler prevents mixing.
```typescript
declare const brand: unique symbol
type Brand<T, B extends string> = T & { readonly [brand]: B }
type UserId = Brand<string, "UserId">
type OrderId = Brand<string, "OrderId">
type Milliseconds = Brand<number, "Milliseconds">
type Seconds = Brand<number, "Seconds">
function UserId(value: string): UserId { return value as UserId }
function OrderId(value: string): OrderId { return value as OrderId }
function getUser(id: UserId): User { ... }
getUser(UserId("abc")) // OK
getUser(OrderId("abc")) // type error: OrderId is not UserId
getUser("abc") // type error: string is not UserId
```
With Zod (preferred at boundaries):
```typescript
import { z } from "zod"
const UserIdSchema = z.string().uuid().brand("UserId")
type UserId = z.infer<typeof UserIdSchema>
```
**Use when**: IDs, indices, units of measurement — any pair where swapping is a bug.
---
## as const — literal types from values
Freezes a value to its narrowest possible type. The foundation for enum-free TypeScript.
```typescript
const ROLES = ["admin", "user", "guest"] as const
type Role = (typeof ROLES)[number] // "admin" | "user" | "guest"
const HTTP_STATUS = {
OK: 200,
NOT_FOUND: 404,
INTERNAL: 500,
} as const
type HttpStatus = (typeof HTTP_STATUS)[keyof typeof HTTP_STATUS] // 200 | 404 | 500
```
**Use when**: fixed set of constants. Replaces `enum` entirely.
**Skip when**: the set is open-ended or user-defined.
---
## satisfies — validate without widening
Type-checks a value against a type while preserving the literal type. Best of both worlds.
```typescript
type Config = Record<string, string | number>
// BAD — widens to Record<string, string | number>
const config: Config = { api: "https://api.example.com", timeout: 30 }
config.api // string | number — lost the narrowing
// GOOD — validates AND preserves literal types
const config = {
api: "https://api.example.com",
timeout: 30,
} satisfies Config
config.api // string (narrowed)
config.timeout // number (narrowed)
```
**Use when**: you want type validation on a value without losing narrowing.
---
## Discriminated unions — algebraic data types
Model every outcome as a type. Force the caller to handle all cases.
```typescript
type GetUserResult =
| { readonly kind: "found"; readonly user: User }
| { readonly kind: "not_found"; readonly id: UserId }
| { readonly kind: "forbidden"; readonly reason: string }
```
The `kind` field (or `type`, `status`, `_tag`) is the discriminant. TypeScript narrows on it automatically.
---
## Exhaustive switch — assertNever
Every switch on a discriminated union ends with a default that calls `assertNever`.
```typescript
function assertNever(x: never): never {
throw new Error(`Unexpected value: ${JSON.stringify(x)}`)
}
function handleResult(result: GetUserResult): string {
switch (result.kind) {
case "found":
return result.user.name
case "not_found":
return `No user ${result.id}`
case "forbidden":
return `Denied: ${result.reason}`
default:
return assertNever(result)
}
}
```
Add a new variant to `GetUserResult`? The compiler errors on the `assertNever` call until you handle it.
---
## Narrowing — let the compiler follow your logic
TypeScript narrows types through `typeof`, `instanceof`, `in`, equality checks, and discriminants.
```typescript
function process(value: string | number | null): string {
if (value === null) return "nothing"
// compiler knows: string | number
if (typeof value === "string") return value.toUpperCase()
// compiler knows: number
return String(value * 2)
}
```
### Custom type guards
```typescript
function isNonNull<T>(value: T | null | undefined): value is T {
return value != null
}
const items = [1, null, 2, undefined, 3]
const clean = items.filter(isNonNull) // number[]
```
---
## import type — separate values from types
Always use `import type` for type-only imports. Enforced by `verbatimModuleSyntax`.
```typescript
import type { User, Config } from "./types" // erased at runtime
import { createUser } from "./services" // kept at runtime
```
For mixed imports:
```typescript
import { createUser, type User } from "./users"
```
---
## Utility types — quick reference
| Need | Use |
|---|---|
| All properties readonly | `Readonly<T>` |
| All properties optional | `Partial<T>` |
| All properties required | `Required<T>` |
| Pick specific properties | `Pick<T, "a" \| "b">` |
| Omit specific properties | `Omit<T, "a" \| "b">` |
| Key-value map | `Record<K, V>` |
| Extract from union | `Extract<T, U>` |
| Exclude from union | `Exclude<T, U>` |
| Return type of function | `ReturnType<typeof fn>` |
| Parameters of function | `Parameters<typeof fn>` |
| Awaited type | `Awaited<Promise<T>>` → `T` |
---
## Sources
- TypeScript Handbook: [Narrowing](https://www.typescriptlang.org/docs/handbook/2/narrowing.html)
- TypeScript Handbook: [Template Literal Types](https://www.typescriptlang.org/docs/handbook/2/template-literal-types.html)
- Total TypeScript: [as const](https://www.totaltypescript.com/as-const)
packages/shared-skills/skills/programming/scripts/go/check-no-excuse-rules.sh
Executable
+173
View File
@@ -0,0 +1,173 @@
#!/usr/bin/env bash
# No-excuse rule checker for Go files.
# Mirrors the philosophy of python-programmer / typescript-programmer / rust-programmer scripts:
# only rules that can be enforced via pure text matching live here.
# Everything semantic is on golangci-lint + nilaway + go test -race.
set -euo pipefail
if [ $# -eq 0 ]; then
echo "Usage: $0 <file.go> [file.go ...]" >&2
exit 2
fi
violations=0
report() {
local file="$1"
local line="$2"
local rule="$3"
local detail="$4"
echo "::error file=${file},line=${line}::[${rule}] ${detail}" >&2
violations=$((violations + 1))
}
is_test_file() {
case "$1" in
*_test.go) return 0 ;;
esac
return 1
}
is_generated_file() {
local file="$1"
case "$file" in
*.pb.go|*.connect.go|*.gen.go) return 0 ;;
*_string.go) return 0 ;;
esac
# First-line check for "Code generated ... DO NOT EDIT." (the official marker)
if [ -f "$file" ]; then
head -n 5 "$file" 2>/dev/null | grep -qE "^// Code generated .* DO NOT EDIT\.$" && return 0
fi
return 1
}
for file in "$@"; do
[ -f "$file" ] || continue
case "$file" in
*.go) ;;
*) continue ;;
esac
if is_generated_file "$file"; then
continue
fi
in_test=0
if is_test_file "$file"; then
in_test=1
fi
line_no=0
while IFS= read -r raw_line || [ -n "$raw_line" ]; do
line_no=$((line_no + 1))
line="$raw_line"
# Strip line comments before pattern checks
# (block comments are not handled — keep the rules robust to that limitation).
code_only="${line%%//*}"
# ── Exemption marker: // no-excuse-ok: <reason> ──────────────────
if [[ "$line" =~ //[[:space:]]*no-excuse-ok:[[:space:]]*.+ ]]; then
continue
fi
# ── Rule: no `_ = err` (silent error swallow) ────────────────────
# The errcheck linter catches most of these but the `_ = err` form
# specifically slips through if used with named returns.
if [[ "$code_only" =~ ^[[:space:]]*_[[:space:]]*=[[:space:]]*err[[:space:]]*$ ]] ||
[[ "$code_only" =~ ^[[:space:]]*_[[:space:]]*=[[:space:]]*err[[:space:]]*[^a-zA-Z0-9_].*$ ]]; then
if [ "$in_test" -eq 0 ]; then
report "$file" "$line_no" "silent-err" "discarding err with '_ = err' — handle the error"
fi
fi
# ── Rule: no `panic(` in non-test, non-main code ─────────────────
# Allowed in main(), allowed in tests, allowed with explicit marker.
if [[ "$code_only" =~ [^a-zA-Z0-9_]panic\( ]] || [[ "$code_only" =~ ^[[:space:]]*panic\( ]]; then
if [ "$in_test" -eq 0 ]; then
# main package main.go is the one exception
pkg_line=$(head -n 5 "$file" 2>/dev/null | grep -m1 "^package ")
if [[ "$pkg_line" != "package main" ]]; then
report "$file" "$line_no" "panic-in-lib" "panic outside main/test — return error instead"
fi
fi
fi
# ── Rule: no `log.Fatal` / `log.Panic` in library code ───────────
if [[ "$code_only" =~ log\.(Fatal|Panic)(f|ln)?\( ]]; then
if [ "$in_test" -eq 0 ]; then
pkg_line=$(head -n 5 "$file" 2>/dev/null | grep -m1 "^package ")
if [[ "$pkg_line" != "package main" ]]; then
report "$file" "$line_no" "log-fatal-in-lib" "log.Fatal/Panic outside main — return error"
fi
fi
fi
# ── Rule: no init() functions ─────────────────────────────────
# init() ruins testability and creates hidden global state.
# Exception: //go:build constraint files and generated code.
if [[ "$code_only" =~ ^func[[:space:]]+init\(\)[[:space:]]*\{ ]]; then
report "$file" "$line_no" "no-init-func" "init() ruins testability — use explicit constructor"
fi
# ── Rule: no `time.Sleep` in non-test code ──────────────────────
if [[ "$code_only" =~ time\.Sleep\( ]]; then
if [ "$in_test" -eq 0 ]; then
report "$file" "$line_no" "time-sleep" "time.Sleep in production code — use ticker/timer with ctx"
fi
fi
# ── Rule: no `context.Background()` inside functions (only in main/init/test) ──
if [[ "$code_only" =~ context\.Background\(\) ]]; then
if [ "$in_test" -eq 0 ]; then
pkg_line=$(head -n 5 "$file" 2>/dev/null | grep -m1 "^package ")
if [[ "$pkg_line" != "package main" ]]; then
report "$file" "$line_no" "ctx-background-in-lib" "context.Background() outside main — propagate ctx as parameter"
fi
fi
fi
# ── Rule: no `interface{}` (use `any`, the alias from Go 1.18+) ──
if [[ "$code_only" =~ interface\{\} ]]; then
report "$file" "$line_no" "old-interface-empty" "use 'any' instead of 'interface{}' (Go 1.18+)"
fi
# ── Rule: no bare `fmt.Println` for logging (use slog) ───────────
# Acceptable in main.go (CLI output) and tests. Reject in libraries.
if [[ "$code_only" =~ fmt\.(Print|Println|Printf)\( ]]; then
if [ "$in_test" -eq 0 ]; then
pkg_line=$(head -n 5 "$file" 2>/dev/null | grep -m1 "^package ")
if [[ "$pkg_line" != "package main" ]]; then
report "$file" "$line_no" "fmt-print-in-lib" "fmt.Print* in library — use slog for structured logs"
fi
fi
fi
# ── Rule: no `nolint` directive without reason ───────────────────
if [[ "$line" =~ //nolint(:|$| ) ]]; then
if ! [[ "$line" =~ //nolint:[a-zA-Z0-9_,-]+[[:space:]]+//[[:space:]]*[^[:space:]] ]]; then
report "$file" "$line_no" "nolint-no-reason" "//nolint requires a // reason after the linter list"
fi
fi
# ── Rule: no TODO / FIXME without an issue link or owner ─────────
# Check the full line — TODOs live in comments, which $code_only has stripped.
if echo "$line" | grep -qE '(TODO|FIXME|XXX)([[:space:]]|:)'; then
if ! echo "$line" | grep -qE '(TODO|FIXME|XXX).*[(@[]'; then
report "$file" "$line_no" "todo-no-owner" "TODO/FIXME requires (#issue) or @owner attribution"
fi
fi
done < "$file"
done
if [ "$violations" -gt 0 ]; then
echo "" >&2
echo "go-programmer: $violations violation(s). Run also:" >&2
echo " gofumpt -l ." >&2
echo " golangci-lint run --timeout 5m ./..." >&2
echo " nilaway ./..." >&2
echo " go test -race -shuffle=on -count=1 ./..." >&2
exit 1
fi
echo "go-programmer: no-excuse rules passed for $# file(s)."
packages/shared-skills/skills/programming/scripts/go/new-project.py
Executable
+138
View File
@@ -0,0 +1,138 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "typer",
# "rich",
# ]
# ///
# ─── How to run ───
# 1. Install uv (if not installed):
# curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run:
# uv run new-project.py myservice
# uv run new-project.py myservice --module github.com/your-org/myservice
# ──────────────────
#
# Creates a new Go project with the canonical strict layout:
# - go.mod with go 1.23
# - .golangci.yml (v2, strict bundle)
# - Taskfile.yml (fmt + lint + test + build)
# - cmd/server/main.go entrypoint
# - internal/{cmd,config,api,domain,obs} skeletons
# - .github/workflows/ci.yml
#
# Templates live in ./templates/ — keep this script under 250 pure LOC.
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
from string import Template
import typer
from rich.console import Console
console = Console(stderr=True)
TEMPLATES_DIR = Path(__file__).parent / "templates"
def _render(template_file: str, **subs: str) -> str:
"""Read a template file and apply $placeholder substitutions.
Uses string.Template ($name) so Go/YAML curly braces stay literal.
"""
raw = (TEMPLATES_DIR / template_file).read_text()
if not subs:
return raw
return Template(raw).substitute(**subs)
# (template-file → relative output path; is_format = .format() is run)
FILES: list[tuple[str, str, bool]] = [
(".golangci.yml", ".golangci.yml", False),
("Taskfile.yml", "Taskfile.yml", False),
(".editorconfig", ".editorconfig", False),
("gitignore", ".gitignore", False),
("ci.yml", ".github/workflows/ci.yml", False),
("run.go", "internal/cmd/run.go", False),
("config.go", "internal/config/config.go", False),
("main.go.tmpl", "cmd/server/main.go", True),
("AGENTS.md.tmpl", "AGENTS.md", True),
("README.md.tmpl", "README.md", True),
]
def _init_go_module(project_dir: Path, module: str) -> None:
try:
subprocess.run(
["go", "mod", "init", module],
cwd=project_dir,
check=True,
capture_output=True,
)
console.print(f" [dim]ran[/] go mod init {module}")
except (subprocess.CalledProcessError, FileNotFoundError) as e:
console.print(f" [yellow]warn[/] go mod init failed ({e}); writing fallback go.mod")
(project_dir / "go.mod").write_text(f"module {module}\n\ngo 1.23\n")
def _create_layout(project_dir: Path) -> None:
"""Create the canonical internal/ tree."""
subdirs = [
"cmd/server",
"internal/cmd",
"internal/config",
"internal/api",
"internal/domain",
"internal/obs",
".github/workflows",
]
for sd in subdirs:
(project_dir / sd).mkdir(parents=True)
def _write_files(project_dir: Path, name: str, module: str, purpose: str) -> None:
"""Render every template into the project tree."""
for tmpl_name, out_rel, is_format in FILES:
subs = (
{"name": name, "module": module, "short_purpose": purpose}
if is_format
else {}
)
content = _render(tmpl_name, **subs)
out_path = project_dir / out_rel
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_text(content)
console.print(f" [dim]wrote[/] {out_rel}")
def main(
name: str,
path: str = typer.Option(".", help="Parent dir"),
module: str = typer.Option("", help="Go module path; default: <name>"),
purpose: str = typer.Option("HTTP", help="Short purpose for AGENTS.md"),
) -> None:
"""Scaffold a new Go project with the strict toolchain."""
project_dir = Path(path) / name
if project_dir.exists():
console.print(f"[red]✗[/red] {project_dir} already exists")
sys.exit(1)
module_path = module or name
project_dir.mkdir(parents=True)
_create_layout(project_dir)
_init_go_module(project_dir, module_path)
_write_files(project_dir, name, module_path, purpose)
console.print(f"\n[bold green]Done![/] cd {project_dir}")
console.print(" go get github.com/caarlos0/env/v11")
console.print(" task # fmt + lint + test")
if __name__ == "__main__":
typer.run(main)
packages/shared-skills/skills/programming/scripts/go/templates/.editorconfig
+13
View File
@@ -0,0 +1,13 @@
root = true
[*]
indent_style = tab
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[*.{yml,yaml,json,md}]
indent_style = space
indent_size = 2
packages/shared-skills/skills/programming/scripts/go/templates/.golangci.yml
+95
View File
@@ -0,0 +1,95 @@
version: "2"
run:
timeout: 5m
tests: true
modules-download-mode: readonly
linters:
default: none
enable:
- govet
- staticcheck
- errcheck
- errorlint
- nilerr
- nilnil
- bodyclose
- rowserrcheck
- sqlclosecheck
- contextcheck
- fatcontext
- copyloopvar
- intrange
- usetesting
- testifylint
- gofumpt
- goimports
- whitespace
- misspell
- unconvert
- unparam
- ineffassign
- dupword
- gocognit
- gocyclo
- funlen
- lll
- nestif
- dupl
- revive
- unused
- exhaustive
- gosec
- sloglint
- perfsprint
- prealloc
- makezero
linters-settings:
errcheck:
check-type-assertions: true
check-blank: true
errorlint:
errorf: true
asserts: true
comparison: true
gocognit:
min-complexity: 25
gocyclo:
min-complexity: 15
funlen:
lines: 90
statements: 60
lll:
line-length: 120
tab-width: 4
nestif:
min-complexity: 4
exhaustive:
default-signifies-exhaustive: false
check: [switch, map]
sloglint:
no-mixed-args: true
attr-only: true
no-global: all
context: scope
static-msg: true
no-raw-keys: true
key-naming-case: snake
formatters:
enable:
- gofumpt
- goimports
issues:
max-issues-per-linter: 0
max-same-issues: 0
exclude-rules:
- path: _test\\.go
linters: [funlen, lll, dupl, gosec]
- path: \\.pb\\.go$
linters: [all]
- path: \\.connect\\.go$
linters: [all]
packages/shared-skills/skills/programming/scripts/go/templates/AGENTS.md.tmpl
+24
View File
@@ -0,0 +1,24 @@
# AGENTS.md
Go 1.23+ $short_purpose service.
## Commands
- `task` — fmt + lint + test
- `task build` — produce ./bin/server
- `task ci` — full CI pipeline locally
## Architecture
- `cmd/server/main.go` — entrypoint, ≤50 LOC
- `internal/cmd/` — root command, signal wiring
- `internal/api/` — HTTP handlers + middleware (gin)
- `internal/domain/` — smart-constructor types, no I/O
- `internal/store/` — DB layer (sqlc-generated, never hand-edited)
- `internal/config/` — env-driven Config
- `internal/obs/` — slog setup, observability
## Conventions
- `slog` for all logs; never `log.*`, never `fmt.Println` in libs
- `context.Context` first arg for every public function with I/O
- Errors wrapped with `%w`; check with `errors.Is/As`
- 250 pure LOC ceiling per file
- Tests follow Given/When/Then; less mock the better
packages/shared-skills/skills/programming/scripts/go/templates/README.md.tmpl
+12
View File
@@ -0,0 +1,12 @@
# $name
Bootstrapped with the `programming` skill's Go scaffold.
## Run
```bash
task # fmt + lint + test
task run # build + run server
```
See `AGENTS.md` for architecture conventions.
packages/shared-skills/skills/programming/scripts/go/templates/Taskfile.yml
+40
View File
@@ -0,0 +1,40 @@
version: '3'
vars:
BINARY: server
PKG: ./cmd/server
tasks:
default:
deps: [fmt, lint, test]
fmt:
cmds:
- gofumpt -w .
- goimports -w -local "$(go list -m)" .
lint:
cmds:
- golangci-lint run --timeout 5m ./...
- nilaway ./... || true
test:
cmds:
- go test -race -shuffle=on -count=1 ./...
test-cover:
cmds:
- go test -race -shuffle=on -count=1 -coverprofile=coverage.out ./...
- go tool cover -html=coverage.out -o coverage.html
build:
cmds:
- go build -trimpath -ldflags="-s -w" -o bin/{{.BINARY}} {{.PKG}}
run:
deps: [build]
cmds:
- ./bin/{{.BINARY}}
ci:
deps: [fmt, lint, test, build]
packages/shared-skills/skills/programming/scripts/go/templates/ci.yml
+37
View File
@@ -0,0 +1,37 @@
name: ci
on:
pull_request:
push:
branches: [main]
jobs:
ci:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.23'
cache: true
- name: Install tools
run: |
go install mvdan.cc/gofumpt@latest
go install github.com/golangci/golangci-lint/cmd/golangci-lint@v2.0.0
go install go.uber.org/nilaway/cmd/nilaway@latest
go install github.com/go-task/task/v3/cmd/task@latest
- name: Format check
run: gofumpt -l . | (! grep .)
- name: Lint
run: golangci-lint run --timeout 5m ./...
- name: Nilaway
run: nilaway ./... || true
- name: Test
run: go test -race -shuffle=on -count=1 ./...
- name: Build
run: go build -trimpath ./...
packages/shared-skills/skills/programming/scripts/go/templates/config.go
+24
View File
@@ -0,0 +1,24 @@
// Package config loads typed config from env.
package config
import (
"time"
"github.com/caarlos0/env/v11"
)
type Config struct {
Host string `env:"HOST" envDefault:"0.0.0.0"`
Port int `env:"PORT" envDefault:"8080"`
ShutdownTimeout time.Duration `env:"SHUTDOWN_TIMEOUT" envDefault:"20s"`
LogLevel string `env:"LOG_LEVEL" envDefault:"info"`
LogFormat string `env:"LOG_FORMAT" envDefault:"json"`
}
func Load() (Config, error) {
var cfg Config
if err := env.Parse(&cfg); err != nil {
return Config{}, err
}
return cfg, nil
}
packages/shared-skills/skills/programming/scripts/go/templates/gitignore
+15
View File
@@ -0,0 +1,15 @@
bin/
coverage.out
coverage.html
*.test
*.prof
.idea/
.vscode/
*.swp
.env
.env.local
*.pem
*.key
packages/shared-skills/skills/programming/scripts/go/templates/main.go.tmpl
+22
View File
@@ -0,0 +1,22 @@
package main
import (
"context"
"log/slog"
"os"
"os/signal"
"syscall"
"$module/internal/cmd"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGINT, syscall.SIGTERM)
defer stop()
if err := cmd.Execute(ctx); err != nil {
slog.Error("fatal", slog.Any("err", err))
os.Exit(1)
}
}
packages/shared-skills/skills/programming/scripts/go/templates/run.go
+15
View File
@@ -0,0 +1,15 @@
// Package cmd wires the root command and subcommands.
package cmd
import (
"context"
"log/slog"
"os"
)
// Execute runs the root command. Wire cobra/subcommands here.
func Execute(ctx context.Context) error {
slog.New(slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo}))
slog.InfoContext(ctx, "starting")
return nil
}
packages/shared-skills/skills/programming/scripts/python/check-no-excuse-rules.py
+687
View File
@@ -0,0 +1,687 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
# noqa: SIZE_OK — single self-contained checker, splitting adds import ceremony for no readability gain
"""Check Python files for no-excuse violations.
The python-programmer skill enforces these rules. Run after editing.
Rules:
cast-any - cast(Any, ...) / cast(typing.Any, ...) / typing.cast(Any, ...)
type-ignore - `# type: ignore` comments (any variant)
pyright-ignore - `# pyright: ignore` comments (any variant)
bare-except - `except:` with no class
silent-except - `except X: pass` or `except X: ...` (single statement)
no-asyncio - `import asyncio` / `from asyncio import ...`
Opt out per import line: trailing `# noqa: ANYIO_OK`
no-pandas - `import pandas` / `from pandas import ...`
Opt out per import line: trailing `# noqa: PANDAS_OK`
mutable-dataclass - @dataclass without frozen=True
Opt out: trailing `# noqa: MUTABLE_OK`
missing-slots - @dataclass without slots=True
Opt out: trailing `# noqa: SLOTS_OK`
raw-dict-return - function returns bare `dict` type
Opt out: trailing `# noqa: DICT_OK`
missing-assert-never - match statement without assert_never in default case
Opt out: `# noqa: MATCH_OK` on the match line
generic-exception - raise ValueError/TypeError/RuntimeError with bare string
Opt out: trailing `# noqa: GENERIC_ERR_OK`
no-object - `object` used as type annotation (param, return, variable)
Opt out: trailing `# noqa: OBJECT_OK`
if-elif-on-variant - isinstance/enum-comparison if/elif chain (should be match/case)
Opt out: trailing `# noqa: IF_VARIANT_OK`
oversized-module - file exceeds 250 pure LOC (non-blank, non-comment)
Opt out: `# noqa: SIZE_OK` in first 10 lines
broad-except - `except Exception` / `except BaseException` (too broad)
Opt out: trailing `# noqa: BROAD_EXCEPT_OK`
Usage:
check-no-excuse-rules.py <file-or-dir>...
Exit codes:
0 - no violations
1 - one or more violations
2 - input error (path missing, etc.)
"""
from __future__ import annotations
import ast
import io
import re
import sys
import tokenize
from collections.abc import Iterable
from dataclasses import dataclass
from pathlib import Path
EXCLUDED_DIRS = frozenset({
".git", ".hg", ".svn", ".venv", "venv", "env", ".env",
"__pycache__", ".tox", ".nox", "dist", "build", ".eggs",
".ruff_cache", ".mypy_cache", ".pytest_cache", ".basedpyright",
"node_modules",
})
SUPPRESSION_RE = re.compile(r"#\s*(type|pyright)\s*:\s*ignore\b")
ANYIO_OK_RE = re.compile(r"#\s*noqa:\s*ANYIO_OK\b")
PANDAS_OK_RE = re.compile(r"#\s*noqa:\s*PANDAS_OK\b")
BANNED_IMPORTS: dict[str, tuple[str, re.Pattern[str], str]] = {
"asyncio": (
"no-asyncio",
ANYIO_OK_RE,
"import asyncio - use anyio (opt out: trailing `# noqa: ANYIO_OK`)",
),
"pandas": (
"no-pandas",
PANDAS_OK_RE,
"import pandas - use polars (opt out: trailing `# noqa: PANDAS_OK`)",
),
}
# Opt-out patterns for new Rust-like rules
MUTABLE_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*MUTABLE_OK")
SLOTS_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*SLOTS_OK")
DICT_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*DICT_OK")
MATCH_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*MATCH_OK")
GENERIC_ERR_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*GENERIC_ERR_OK")
OBJECT_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*OBJECT_OK")
IF_VARIANT_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*IF_VARIANT_OK")
SIZE_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*SIZE_OK")
BROAD_EXCEPT_OK_RE: re.Pattern[str] = re.compile(r"#\s*noqa:\s*BROAD_EXCEPT_OK")
PURE_LOC_LIMIT: int = 250
@dataclass(frozen=True, slots=True)
class Violation:
rule: str
file: Path
line: int
col: int
message: str
def render(self) -> str:
return f"{self.file}:{self.line}:{self.col}: [{self.rule}] {self.message}"
def discover_files(inputs: Iterable[Path]) -> list[Path]:
seen: set[Path] = set()
for raw in inputs:
path = raw.resolve()
if not path.exists():
print(f"check-no-excuse-rules: input does not exist: {path}", file=sys.stderr)
sys.exit(2)
if path.is_file():
if path.suffix == ".py":
seen.add(path)
continue
for child in path.rglob("*.py"):
if any(part in EXCLUDED_DIRS for part in child.parts):
continue
seen.add(child)
return sorted(seen)
def is_any_node(node: ast.AST) -> bool:
if isinstance(node, ast.Name):
return node.id == "Any"
if isinstance(node, ast.Attribute):
return node.attr == "Any"
return False
def is_cast_callable(node: ast.AST) -> bool:
if isinstance(node, ast.Name):
return node.id == "cast"
if isinstance(node, ast.Attribute):
return node.attr == "cast"
return False
def find_node_violations(tree: ast.AST, file: Path) -> list[Violation]:
violations: list[Violation] = []
for node in ast.walk(tree):
if (
isinstance(node, ast.Call)
and is_cast_callable(node.func)
and node.args
and is_any_node(node.args[0])
):
violations.append(Violation(
rule="cast-any",
file=file,
line=node.lineno,
col=node.col_offset + 1,
message="cast(Any, ...) - narrow with isinstance/TypeGuard or use a Protocol/TypedDict",
))
if isinstance(node, ast.ExceptHandler):
if node.type is None:
violations.append(Violation(
rule="bare-except",
file=file,
line=node.lineno,
col=node.col_offset + 1,
message="bare `except:` - catch the narrowest exception you mean",
))
if len(node.body) != 1:
continue
body = node.body[0]
if isinstance(body, ast.Pass):
violations.append(Violation(
rule="silent-except",
file=file,
line=body.lineno,
col=body.col_offset + 1,
message="silent `except: pass` - log, re-raise, or actually handle the error",
))
elif (
isinstance(body, ast.Expr)
and isinstance(body.value, ast.Constant)
and body.value.value is Ellipsis
):
violations.append(Violation(
rule="silent-except",
file=file,
line=body.lineno,
col=body.col_offset + 1,
message="silent `except: ...` - log, re-raise, or actually handle the error",
))
return violations
def find_import_violations(tree: ast.AST, source_lines: list[str], file: Path) -> list[Violation]:
violations: list[Violation] = []
def line_text(lineno: int) -> str:
index = lineno - 1
return source_lines[index] if 0 <= index < len(source_lines) else ""
for node in ast.walk(tree):
if isinstance(node, ast.Import): # noqa: IF_VARIANT_OK — filtering walk, not closed union
for alias in node.names:
top = alias.name.split(".")[0]
if top not in BANNED_IMPORTS:
continue
rule, opt_re, message = BANNED_IMPORTS[top]
if opt_re.search(line_text(node.lineno)):
continue
violations.append(Violation(
rule=rule,
file=file,
line=node.lineno,
col=node.col_offset + 1,
message=message,
))
elif isinstance(node, ast.ImportFrom):
top = (node.module or "").split(".")[0]
if top not in BANNED_IMPORTS:
continue
rule, opt_re, message = BANNED_IMPORTS[top]
if opt_re.search(line_text(node.lineno)):
continue
violations.append(Violation(
rule=rule,
file=file,
line=node.lineno,
col=node.col_offset + 1,
message=message,
))
return violations
def find_comment_violations(source: str, file: Path) -> list[Violation]:
"""Use tokenize so we don't false-match `# type: ignore` inside string literals."""
violations: list[Violation] = []
try:
tokens = list(tokenize.generate_tokens(io.StringIO(source).readline))
except tokenize.TokenError as exc:
print(f"check-no-excuse-rules: tokenize failed for {file}: {exc}", file=sys.stderr)
return violations
for tok in tokens:
if tok.type != tokenize.COMMENT:
continue
match = SUPPRESSION_RE.search(tok.string)
if not match:
continue
kind = match.group(1)
rule = "type-ignore" if kind == "type" else "pyright-ignore"
violations.append(Violation(
rule=rule,
file=file,
line=tok.start[0],
col=tok.start[1] + match.start() + 1,
message=f"`# {kind}: ignore` - fix the underlying type instead",
))
return violations
# ─────────────────────────────────────────────────────────────────
# Rust-like pattern checks
# ─────────────────────────────────────────────────────────────────
def _has_keyword(decorator_node: ast.Call, keyword: str) -> bool | None:
"""Check if a decorator call has a specific keyword argument.
Returns True if keyword is True, False if keyword is False or absent, None if not a Call.
"""
for kw in decorator_node.keywords:
if kw.arg == keyword and isinstance(kw.value, ast.Constant):
return bool(kw.value.value)
return False
def _is_dataclass_decorator(node: ast.expr) -> tuple[bool, ast.Call | None]:
"""Return (is_dataclass, call_node_or_None)."""
if isinstance(node, ast.Name) and node.id == "dataclass":
return True, None
if isinstance(node, ast.Attribute) and node.attr == "dataclass":
return True, None
if isinstance(node, ast.Call):
inner, _ = _is_dataclass_decorator(node.func)
if inner:
return True, node
return False, None
def find_dataclass_violations(
tree: ast.Module, source_lines: list[str], file: Path,
) -> list[Violation]:
"""Check @dataclass decorators for frozen=True and slots=True."""
violations: list[Violation] = []
for node in ast.walk(tree):
if not isinstance(node, ast.ClassDef):
continue
for dec in node.decorator_list:
is_dc, call_node = _is_dataclass_decorator(dec)
if not is_dc:
continue
# Get the line of the decorator for opt-out check
dec_line = source_lines[dec.lineno - 1] if dec.lineno <= len(source_lines) else ""
if call_node is not None:
has_frozen = _has_keyword(call_node, "frozen")
has_slots = _has_keyword(call_node, "slots")
else:
# bare @dataclass with no arguments
has_frozen = False
has_slots = False
if not has_frozen and not MUTABLE_OK_RE.search(dec_line):
violations.append(Violation(
rule="mutable-dataclass",
file=file,
line=dec.lineno,
col=dec.col_offset + 1,
message=f"class {node.name}: @dataclass without frozen=True",
))
if not has_slots and not SLOTS_OK_RE.search(dec_line):
violations.append(Violation(
rule="missing-slots",
file=file,
line=dec.lineno,
col=dec.col_offset + 1,
message=f"class {node.name}: @dataclass without slots=True",
))
return violations
def find_dict_return_violations(
tree: ast.Module, source_lines: list[str], file: Path,
) -> list[Violation]:
"""Check for functions returning bare `dict` type."""
violations: list[Violation] = []
for node in ast.walk(tree):
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
continue
ret = node.returns
if ret is None:
continue
# Check for bare `dict` return annotation
is_bare_dict = (
(isinstance(ret, ast.Name) and ret.id == "dict")
or (isinstance(ret, ast.Attribute) and ret.attr == "dict")
)
if not is_bare_dict:
continue
func_line = source_lines[node.lineno - 1] if node.lineno <= len(source_lines) else ""
if DICT_OK_RE.search(func_line):
continue
violations.append(Violation(
rule="raw-dict-return",
file=file,
line=node.lineno,
col=node.col_offset + 1,
message=f"`{node.name}` returns bare dict - use TypedDict/dataclass/Pydantic model",
))
return violations
def find_match_violations(
tree: ast.Module, source_lines: list[str], file: Path,
) -> list[Violation]:
"""Check match statements for assert_never in default case."""
violations: list[Violation] = []
for node in ast.walk(tree):
if not isinstance(node, ast.Match):
continue
match_line = source_lines[node.lineno - 1] if node.lineno <= len(source_lines) else ""
if MATCH_OK_RE.search(match_line):
continue
has_assert_never = False
for case in node.cases:
# Wildcard: `case _:` -> MatchAs(pattern=None, name=None)
# `case _ as x:` -> MatchAs(pattern=MatchAs(pattern=None, name=None), name="x")
pattern = case.pattern
is_wildcard = (
isinstance(pattern, ast.MatchAs)
and (
pattern.pattern is None
or (
isinstance(pattern.pattern, ast.MatchAs)
and pattern.pattern.pattern is None
and pattern.pattern.name is None
)
)
)
if not is_wildcard:
continue
# Check if body contains assert_never call
for stmt in case.body:
if isinstance(stmt, ast.Expr) and isinstance(stmt.value, ast.Call):
func = stmt.value.func
if (
(isinstance(func, ast.Name) and func.id == "assert_never")
or (isinstance(func, ast.Attribute) and func.attr == "assert_never")
):
has_assert_never = True
break
if not has_assert_never:
violations.append(Violation(
rule="missing-assert-never",
file=file,
line=node.lineno,
col=node.col_offset + 1,
message="match without `case _: assert_never(x)` default",
))
return violations
def find_generic_exception_violations(
tree: ast.Module, source_lines: list[str], file: Path,
) -> list[Violation]:
"""Check for raise ValueError/TypeError/RuntimeError with bare string or f-string."""
GENERIC_EXCEPTIONS = {"ValueError", "TypeError", "RuntimeError", "KeyError"}
violations: list[Violation] = []
for node in ast.walk(tree):
if not isinstance(node, ast.Raise) or node.exc is None:
continue
exc = node.exc
# Match: raise SomeError("string literal")
if not isinstance(exc, ast.Call):
continue
func = exc.func
exc_name: str | None = None
if isinstance(func, ast.Name) and func.id in GENERIC_EXCEPTIONS:
exc_name = func.id
elif isinstance(func, ast.Attribute) and func.attr in GENERIC_EXCEPTIONS:
exc_name = func.attr
if exc_name is None:
continue
# Check if all arguments are string literals or f-strings
if not exc.args:
continue
all_str = all(
(isinstance(arg, ast.Constant) and isinstance(arg.value, str))
or isinstance(arg, ast.JoinedStr)
for arg in exc.args
)
if not all_str:
continue
raise_line = source_lines[node.lineno - 1] if node.lineno <= len(source_lines) else ""
if GENERIC_ERR_OK_RE.search(raise_line):
continue
violations.append(Violation(
rule="generic-exception",
file=file,
line=node.lineno,
col=node.col_offset + 1,
message=f"`raise {exc_name}(\"...\")` - define a typed error class instead",
))
return violations
def _is_isinstance_test(node: ast.expr) -> bool:
"""Check if node is an isinstance() call."""
return (
isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "isinstance"
)
def _is_enum_comparison(node: ast.expr) -> bool:
"""Check if node is `x == Enum.VALUE` or `x is Enum.VALUE`."""
if isinstance(node, ast.Compare) and len(node.ops) == 1:
op = node.ops[0]
if isinstance(op, (ast.Eq, ast.Is)):
comparator = node.comparators[0]
# x == Enum.VALUE (attribute access on the right)
if isinstance(comparator, ast.Attribute):
return True
# Enum.VALUE == x (attribute access on the left)
if isinstance(node.left, ast.Attribute):
return True
return False
def find_object_annotation_violations(
tree: ast.Module, source_lines: list[str], file: Path,
) -> list[Violation]:
"""Check for `object` used as a type annotation."""
violations: list[Violation] = []
def _check_annotation(ann: ast.expr | None) -> None:
if ann is None:
return
for child in ast.walk(ann):
if isinstance(child, ast.Name) and child.id == "object":
line = source_lines[child.lineno - 1] if child.lineno <= len(source_lines) else ""
if OBJECT_OK_RE.search(line):
return
violations.append(Violation(
rule="no-object",
file=file,
line=child.lineno,
col=child.col_offset + 1,
message="`object` as type annotation \u2014 use Protocol, TypeVar, or union",
))
for node in ast.walk(tree):
match node: # noqa: MATCH_OK — filtering walk, not discriminating a closed union
case ast.FunctionDef() | ast.AsyncFunctionDef():
all_args = (
node.args.args
+ node.args.posonlyargs
+ node.args.kwonlyargs
)
for arg in all_args:
_check_annotation(arg.annotation)
if node.args.vararg:
_check_annotation(node.args.vararg.annotation)
if node.args.kwarg:
_check_annotation(node.args.kwarg.annotation)
_check_annotation(node.returns)
case ast.AnnAssign():
_check_annotation(node.annotation)
return violations
def find_if_elif_variant_violations(
tree: ast.Module, source_lines: list[str], file: Path,
) -> list[Violation]:
"""Check for if/elif chains on isinstance or enum comparison."""
violations: list[Violation] = []
for node in ast.walk(tree):
if not isinstance(node, ast.If):
continue
line = source_lines[node.lineno - 1] if node.lineno <= len(source_lines) else ""
if IF_VARIANT_OK_RE.search(line):
continue
is_variant_test = _is_isinstance_test(node.test) or _is_enum_comparison(node.test)
if not is_variant_test:
continue
# Must have at least one elif that is also a variant test
orelse = node.orelse
while orelse and len(orelse) == 1 and isinstance(orelse[0], ast.If):
elif_node = orelse[0]
if _is_isinstance_test(elif_node.test) or _is_enum_comparison(elif_node.test):
violations.append(Violation(
rule="if-elif-on-variant",
file=file,
line=node.lineno,
col=node.col_offset + 1,
message="isinstance/enum if/elif chain \u2014 use match/case + assert_never",
))
break
orelse = elif_node.orelse
return violations
def find_broad_except_violations(
tree: ast.Module, source_lines: list[str], file: Path,
) -> list[Violation]:
"""Check for except Exception / except BaseException (too broad)."""
BROAD_EXCEPTIONS = {"Exception", "BaseException"}
violations: list[Violation] = []
for node in ast.walk(tree):
if not isinstance(node, ast.ExceptHandler):
continue
if node.type is None:
continue # already caught by bare-except
exc_name: str | None = None
if isinstance(node.type, ast.Name) and node.type.id in BROAD_EXCEPTIONS:
exc_name = node.type.id
elif isinstance(node.type, ast.Attribute) and node.type.attr in BROAD_EXCEPTIONS:
exc_name = node.type.attr
if exc_name is None:
continue
line = source_lines[node.lineno - 1] if node.lineno <= len(source_lines) else ""
if BROAD_EXCEPT_OK_RE.search(line):
continue
violations.append(Violation(
rule="broad-except",
file=file,
line=node.lineno,
col=node.col_offset + 1,
message=f"`except {exc_name}` is too broad \u2014 catch the specific exception you expect",
))
return violations
def find_oversized_module_violations(
source_lines: list[str], file: Path,
) -> list[Violation]:
"""Check if file exceeds 250 pure LOC (non-blank, non-comment)."""
# File-level opt-out in first 10 lines (shebang + script metadata can push it down)
for line in source_lines[:10]:
if SIZE_OK_RE.search(line):
return []
pure_loc = sum(
1 for line in source_lines
if line.strip() and not line.strip().startswith("#")
)
if pure_loc > PURE_LOC_LIMIT:
return [Violation(
rule="oversized-module",
file=file,
line=1,
col=1,
message=f"{pure_loc} pure LOC (limit: {PURE_LOC_LIMIT}) \u2014 split by responsibility",
)]
return []
def check_file(file: Path) -> list[Violation]:
source = file.read_text(encoding="utf-8")
try:
tree = ast.parse(source, filename=str(file))
except SyntaxError as exc:
return [Violation(
rule="syntax-error",
file=file,
line=exc.lineno or 1,
col=exc.offset or 1,
message=f"SyntaxError: {exc.msg}",
)]
source_lines = source.splitlines()
return [
*find_node_violations(tree, file),
*find_import_violations(tree, source_lines, file),
*find_comment_violations(source, file),
*find_dataclass_violations(tree, source_lines, file),
*find_dict_return_violations(tree, source_lines, file),
*find_match_violations(tree, source_lines, file),
*find_generic_exception_violations(tree, source_lines, file),
*find_object_annotation_violations(tree, source_lines, file),
*find_if_elif_variant_violations(tree, source_lines, file),
*find_oversized_module_violations(source_lines, file),
*find_broad_except_violations(tree, source_lines, file),
]
def main() -> int:
if len(sys.argv) < 2:
print("usage: check-no-excuse-rules.py <file-or-dir>...", file=sys.stderr)
return 2
files = discover_files(Path(arg) for arg in sys.argv[1:])
if not files:
print("check-no-excuse-rules: no .py files found", file=sys.stderr)
return 0
violations: list[Violation] = []
for file in files:
violations.extend(check_file(file))
if not violations:
print(f"no violations in {len(files)} file(s)")
return 0
for violation in violations:
print(violation.render(), file=sys.stderr)
print(
f"\n{len(violations)} violation(s) in {len(files)} file(s)",
file=sys.stderr,
)
return 1
if __name__ == "__main__":
sys.exit(main())
packages/shared-skills/skills/programming/scripts/python/new-project.py
+172
View File
@@ -0,0 +1,172 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "typer",
# "rich",
# ]
# ///
# ─── How to run ───
# 1. Install uv (if not installed):
# curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run:
# uv run new-project.py myproject
# uv run new-project.py myproject --path ./workspace
# uv run new-project.py myproject --lib # library (publishable)
# ──────────────────
"""Scaffold a new Python project with ultra-strict config from pyproject-strict.md.
Creates via `uv init`, then injects basedpyright + ruff ALL + pytest config.
"""
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
import typer
from rich import print as rprint
# ── Strict tool config (from pyproject-strict.md) ──
TOOL_CONFIG = '''
[dependency-groups]
dev = [
"basedpyright>=1.21",
"ruff>=0.8",
"pytest>=8",
"pytest-cov>=5",
]
[tool.basedpyright]
typeCheckingMode = "all"
pythonVersion = "3.13"
reportMissingTypeStubs = false
reportUnknownMemberType = false
reportUnknownArgumentType = false
reportUnknownVariableType = false
reportUnknownLambdaType = false
reportUnknownParameterType = false
reportMissingParameterType = false
reportUnnecessaryIsInstance = false
reportUnusedCallResult = false
reportImplicitOverride = false
[tool.ruff]
target-version = "py313"
line-length = 120
[tool.ruff.lint]
select = ["ALL"]
ignore = [
"COM812", # trailing comma (conflicts with formatter)
"ISC001", # single-line string concat (conflicts with formatter)
"D1", # undocumented-public-* (too noisy early on)
"ANN101", # deprecated: self annotation
"ANN102", # deprecated: cls annotation
"S101", # assert used (pytest needs it)
"PLR2004", # magic-value-comparison (test data)
"FBT", # boolean-trap (too strict for CLIs)
"TD", # flake8-todos (noisy)
"FIX", # fixme (noisy)
]
[tool.ruff.lint.per-file-ignores]
"tests/**/*.py" = ["S101", "PLR2004", "SLF001", "D", "ARG", "ANN"]
[tool.ruff.lint.pydocstyle]
convention = "google"
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-ra --strict-markers --strict-config"
'''
GITIGNORE = """\
__pycache__/
*.py[cod]
*.so
.venv/
dist/
*.egg-info/
.coverage
htmlcov/
.basedpyright/
.ruff_cache/
"""
def main(
name: str = typer.Argument(help="Project name"),
path: Path = typer.Option(Path("."), "--path", "-p", help="Parent directory"),
lib: bool = typer.Option(False, "--lib", help="Create as publishable library (uv init --lib)"),
) -> None:
"""Create a new Python project with ultra-strict config."""
project_dir = path / name
if project_dir.exists():
rprint(f"[red]Error:[/red] {project_dir} already exists")
raise SystemExit(1)
# Run uv init
cmd = ["uv", "init", "--lib" if lib else "--app", str(project_dir)]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
rprint(f"[red]uv init failed:[/red] {result.stderr}")
raise SystemExit(1)
# Read existing pyproject.toml
pyproject_path = project_dir / "pyproject.toml"
content = pyproject_path.read_text()
# Remove the default [dependency-groups] if uv init created one
# (we'll replace it with our strict version)
lines = content.splitlines(keepends=True)
filtered: list[str] = []
skip = False
for line in lines:
if line.strip().startswith("[dependency-groups]"):
skip = True
continue
if skip and line.strip().startswith("["):
skip = False
if not skip:
filtered.append(line)
content = "".join(filtered).rstrip("\n") + "\n"
# Append strict tool config
content += TOOL_CONFIG
pyproject_path.write_text(content)
# Add dev dependencies
subprocess.run(
["uv", "add", "--dev", "basedpyright", "ruff", "pytest", "pytest-cov"],
cwd=project_dir,
capture_output=True,
)
# Create tests directory
tests_dir = project_dir / "tests"
tests_dir.mkdir(exist_ok=True)
(tests_dir / "__init__.py").touch()
# Overwrite .gitignore
(project_dir / ".gitignore").write_text(GITIGNORE)
# Create py.typed marker for libraries
if lib:
src_dir = project_dir / "src" / name.replace("-", "_")
if src_dir.exists():
(src_dir / "py.typed").touch()
rprint(f"[green]✓[/green] Created: [bold]{project_dir}[/bold]")
rprint(f" cd {name} && uv sync && uv run basedpyright . && uv run ruff check .")
if __name__ == "__main__":
typer.run(main)
packages/shared-skills/skills/programming/scripts/python/new-script.py
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "typer",
# "rich",
# ]
# ///
# ─── How to run ───
# 1. Install uv (if not installed):
# curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run:
# uv run new-script.py my_tool
# uv run new-script.py my_tool --output ./scripts/my_tool.py
# uv run new-script.py my_tool --deps 'httpx2[http2,brotli,zstd]' --deps rich --deps polars
# uv run new-script.py my_tool --py 3.13
# ──────────────────
"""Generate a PEP 723 Python script with all boilerplate pre-filled.
Creates a new .py file with:
- uv shebang
- PEP 723 inline metadata (requires-python + dependencies)
- Mandatory "How to run" comment block
- from __future__ import annotations
- main() + if __name__ guard
By default writes to a temp directory and prints the path.
"""
from __future__ import annotations
import os
import stat
import sys
import tempfile
from pathlib import Path
import typer
from rich import print as rprint
TEMPLATE = '''\
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">={python_version}"
# dependencies = [
{deps_block}# ]
# ///
# ─── How to run ───
# 1. Install uv (if not installed):
# curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run directly (no venv, no pip install needed):
# uv run {filename} {args_hint}
# 3. Or make executable and run:
# chmod +x {filename} && ./{filename}
# ──────────────────
from __future__ import annotations
def main() -> None:
"""TODO: implement."""
if __name__ == "__main__":
main()
'''
def main(
name: str = typer.Argument(help="Script name (without .py extension)"),
output: Path | None = typer.Option(None, "--output", "-o", help="Output path. Default: OS temp directory."),
deps: list[str] = typer.Option([], "--deps", "-d", help="Dependencies to include (repeat --deps for each)."),
py: str = typer.Option("3.13", "--py", help="Minimum Python version."),
) -> None:
"""Generate a new PEP 723 script with all boilerplate pre-filled."""
filename = f"{name}.py" if not name.endswith(".py") else name
stem = filename.removesuffix(".py")
if output is not None:
dest = Path(output)
else:
tmp_dir = Path(tempfile.gettempdir()) / "uv-scripts"
tmp_dir.mkdir(exist_ok=True)
dest = tmp_dir / filename
dep_list = deps or []
if dep_list:
deps_block = "".join(f'# "{d}",\n' for d in dep_list)
else:
deps_block = '# # add deps here, e.g.: "httpx2[http2,brotli,zstd]"\n'
content = TEMPLATE.format(
python_version=py,
deps_block=deps_block,
filename=filename,
args_hint="",
)
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(content)
# Make executable on Unix
if sys.platform != "win32":
st = dest.stat()
dest.chmod(st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
rprint(f"[green]✓[/green] Created: [bold]{dest}[/bold]")
rprint(f" Run: [cyan]uv run {dest}[/cyan]")
if __name__ == "__main__":
typer.run(main)
packages/shared-skills/skills/programming/scripts/rust/check-no-excuse-rules.py
Executable
+296
View File
@@ -0,0 +1,296 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
#
# How to run:
# uv run --script check-no-excuse-rules.py src/lib.rs src/main.rs
# uv run --script check-no-excuse-rules.py src/ # recursively finds .rs files
# uv run --script check-no-excuse-rules.py . # entire tree
#
# No-excuse rule checker for Rust files — Python rewrite of check-no-excuse-rules.sh.
# Only rules enforceable via pure text matching live here.
# Everything semantic is on clippy + miri + nextest.
#
# Rules:
# unwrap .unwrap() outside tests without // SAFE-UNWRAP:
# expect .expect() outside tests without // SAFE-EXPECT:
# placeholder-macro todo!/unimplemented!/unreachable!/unreachable_unchecked! in committed code
# box-dyn-error Box<dyn Error> in non-test code
# lib-panic panic!() in library code
# unsafe-no-safety unsafe { without // SAFETY: in preceding 5 lines
# unjustified-clippy-allow #[allow(clippy::...)] without // CLIPPY-ALLOW:
# narrowing-as-cast possible narrowing 'as' cast
#
# Opt-out: place the appropriate comment on the previous line:
# // SAFE-UNWRAP: <reason>
# // SAFE-EXPECT: <reason>
# // SAFETY: <reason> (for unsafe blocks, within 5 lines above)
# // CLIPPY-ALLOW: <reason>
#
# Test paths (exempt from unwrap/expect/placeholder/box-dyn-error/lib-panic):
# tests/, benches/, examples/, build.rs, *_test.rs, #[cfg(test)] regions
from __future__ import annotations
import re
import sys
from pathlib import Path
# ---------------------------------------------------------------------------
# Patterns (compiled once)
# ---------------------------------------------------------------------------
RE_UNWRAP = re.compile(r"\.unwrap\(\)")
RE_EXPECT = re.compile(r"\.expect\(")
RE_PLACEHOLDER = re.compile(r"\b(todo!|unimplemented!|unreachable!|unreachable_unchecked!)")
RE_BOX_DYN_ERROR = re.compile(r"Box<dyn\s+Error")
RE_PANIC = re.compile(r"\bpanic!\(")
RE_UNSAFE_BLOCK = re.compile(r"\bunsafe\s*\{")
RE_CLIPPY_ALLOW = re.compile(r"#\[allow\(clippy::")
RE_CFG_TEST = re.compile(r"#\[cfg\(test\)\]")
RE_SAFE_UNWRAP = re.compile(r"//\s*SAFE-UNWRAP:")
RE_SAFE_EXPECT = re.compile(r"//\s*SAFE-EXPECT:")
RE_SAFETY = re.compile(r"//\s*SAFETY:")
RE_CLIPPY_ALLOW_JUST = re.compile(r"//\s*CLIPPY-ALLOW:")
# Narrowing cast: (wider) as (narrower)
# Wider types that lose bits when cast to narrower targets.
# No leading \b — must match e.g. `999u64 as u32` where a digit precedes the type.
_WIDER = r"(?:u16|u32|u64|u128|usize|i16|i32|i64|i128|isize)"
_NARROWER = r"(?:u8|u16|u32|i8|i16|i32)"
RE_NARROWING_CAST = re.compile(
rf"{_WIDER}\s+as\s+{_NARROWER}"
)
# Test-path fragments.
_TEST_PATH_PARTS = {"tests", "benches", "examples"}
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
violations = 0
def report(file: str, line: int, rule: str, detail: str) -> None:
"""Emit a GitHub-Actions-compatible error annotation to stderr."""
global violations
print(f"::error file={file},line={line}::[{rule}] {detail}", file=sys.stderr)
violations += 1
def is_test_path(path: Path) -> bool:
"""Return True if *path* is in a test/bench/example directory or is a test file."""
parts = path.parts
for part in parts:
if part in _TEST_PATH_PARTS:
return True
if path.name == "build.rs":
return True
if path.name.endswith("_test.rs"):
return True
return False
def is_lib_path(file: Path) -> bool:
"""Heuristic: is this file library code (not main.rs, not src/bin/*)."""
parts = path_parts_str(file)
# Must live under src/
if "src" not in parts:
return False
if file.name == "main.rs":
return False
# src/bin/* is binary code
try:
src_idx = parts.index("src")
if src_idx + 1 < len(parts) and parts[src_idx + 1] == "bin":
return False
except ValueError:
return False
return True
def path_parts_str(p: Path) -> list[str]:
return list(p.parts)
def strip_line_comment(line: str) -> str:
"""Return the portion of *line* before any ``//`` line comment.
This is a crude heuristic — it does not handle ``//`` inside string
literals, but matches the behaviour of the bash version.
"""
idx = line.find("//")
if idx == -1:
return line
return line[:idx]
def collect_rs_files(args: list[str]) -> list[Path]:
"""Expand CLI arguments: files are kept as-is, directories are walked."""
result: list[Path] = []
for arg in args:
p = Path(arg)
if p.is_file():
if p.suffix == ".rs":
result.append(p)
elif p.is_dir():
result.extend(sorted(p.rglob("*.rs")))
# Ignore non-existent / non-.rs
return result
# ---------------------------------------------------------------------------
# Main checker
# ---------------------------------------------------------------------------
def check_file(file: Path) -> None:
in_test_file = is_test_path(file)
in_cfg_test = False
cfg_test_brace_depth = 0
try:
lines = file.read_text(encoding="utf-8", errors="replace").splitlines()
except OSError as exc:
print(f"warning: cannot read {file}: {exc}", file=sys.stderr)
return
for line_no_0, raw_line in enumerate(lines):
line_no = line_no_0 + 1 # 1-indexed
# --- #[cfg(test)] region tracker ---
if RE_CFG_TEST.search(raw_line):
in_cfg_test = True
cfg_test_brace_depth = 0
if in_cfg_test:
opens = raw_line.count("{")
closes = raw_line.count("}")
cfg_test_brace_depth += opens - closes
if cfg_test_brace_depth <= 0 and not RE_CFG_TEST.search(raw_line):
in_cfg_test = False
exempt = in_test_file or in_cfg_test
code_only = strip_line_comment(raw_line)
if not exempt:
# .unwrap()
if RE_UNWRAP.search(code_only):
prev = lines[line_no_0 - 1] if line_no_0 > 0 else ""
if not RE_SAFE_UNWRAP.search(prev):
report(
str(file), line_no, "unwrap",
".unwrap() outside tests - use ? / ok_or / pattern match "
"or annotate previous line with // SAFE-UNWRAP: <reason>",
)
# .expect(...)
if RE_EXPECT.search(code_only):
prev = lines[line_no_0 - 1] if line_no_0 > 0 else ""
if not RE_SAFE_EXPECT.search(prev):
report(
str(file), line_no, "expect",
".expect() outside tests - use ? or annotate previous "
"line with // SAFE-EXPECT: <reason>",
)
# todo!/unimplemented!/unreachable!/unreachable_unchecked!
if RE_PLACEHOLDER.search(code_only):
report(
str(file), line_no, "placeholder-macro",
"todo!/unimplemented!/unreachable! in committed code",
)
# Box<dyn Error>
if RE_BOX_DYN_ERROR.search(code_only):
report(
str(file), line_no, "box-dyn-error",
"Box<dyn Error> in non-test code - use anyhow::Error (apps) "
"or thiserror enum (libs)",
)
# panic!() in library code
if is_lib_path(file) and RE_PANIC.search(code_only):
report(
str(file), line_no, "lib-panic",
"panic!() in library code - return Result",
)
# unsafe { without // SAFETY: — always enforced, even in tests
if RE_UNSAFE_BLOCK.search(code_only):
start = max(0, line_no_0 - 5)
window = "\n".join(lines[start : line_no_0 + 1])
if not RE_SAFETY.search(window):
report(
str(file), line_no, "unsafe-no-safety-comment",
"unsafe block without // SAFETY: comment in preceding 5 lines",
)
# #[allow(clippy::...)] without // CLIPPY-ALLOW: — always enforced
if RE_CLIPPY_ALLOW.search(code_only):
prev = lines[line_no_0 - 1] if line_no_0 > 0 else ""
if not RE_CLIPPY_ALLOW_JUST.search(prev):
report(
str(file), line_no, "unjustified-clippy-allow",
"#[allow(clippy::...)] without // CLIPPY-ALLOW: <reason> on "
"previous line",
)
# Narrowing numeric `as` casts
if RE_NARROWING_CAST.search(code_only):
report(
str(file), line_no, "narrowing-as-cast",
"possible narrowing 'as' cast - use TryFrom / try_into() for "
"fallible conversion",
)
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def main() -> None:
global violations
if len(sys.argv) < 2:
print(f"Usage: {sys.argv[0]} <file.rs|dir> [file.rs|dir ...]", file=sys.stderr)
sys.exit(2)
files = collect_rs_files(sys.argv[1:])
if not files:
print("warning: no .rs files found in the given arguments", file=sys.stderr)
sys.exit(0)
for f in files:
check_file(f)
if violations > 0:
print("", file=sys.stderr)
print(
f"rust-programmer: {violations} violation(s). Fix before declaring work done.",
file=sys.stderr,
)
print("", file=sys.stderr)
print("Then run the full toolchain gate:", file=sys.stderr)
print(" cargo +stable fmt --all -- --check", file=sys.stderr)
print(
" cargo +stable clippy --all-targets --all-features -- -D warnings",
file=sys.stderr,
)
print(" cargo nextest run --all-targets --all-features", file=sys.stderr)
print(
" cargo +nightly miri nextest run --all-features # if unsafe touched",
file=sys.stderr,
)
print(" cargo machete", file=sys.stderr)
print(" cargo deny check", file=sys.stderr)
sys.exit(1)
print(f"rust-programmer: no-excuse rules passed for {len(files)} file(s).")
if __name__ == "__main__":
main()
packages/shared-skills/skills/programming/scripts/rust/check-no-excuse-rules.sh
Executable
+158
View File
@@ -0,0 +1,158 @@
#!/usr/bin/env bash
# No-excuse rule checker for Rust files.
# Mirrors the philosophy of python-programmer / typescript-programmer scripts:
# only rules that can be enforced via pure text matching live here.
# Everything semantic is on clippy + miri + nextest.
set -euo pipefail
if [ $# -eq 0 ]; then
echo "Usage: $0 <file.rs> [file.rs ...]" >&2
exit 2
fi
violations=0
report() {
local file="$1"
local line="$2"
local rule="$3"
local detail="$4"
echo "::error file=${file},line=${line}::[${rule}] ${detail}" >&2
violations=$((violations + 1))
}
is_test_path() {
local path="$1"
case "$path" in
*/tests/*|*/benches/*|*/examples/*|*/build.rs|*_test.rs|tests/*|benches/*|examples/*) return 0 ;;
esac
# In-file #[cfg(test)] modules are handled per-line below.
return 1
}
for file in "$@"; do
[ -f "$file" ] || continue
case "$file" in
*.rs) ;;
*) continue ;;
esac
if is_test_path "$file"; then
# Test files are exempt from unwrap/expect/todo rules.
# Still enforce unsafe-comment, allow-comment, panic-in-lib rules below
# by setting a marker - keeping the loop unified.
in_test_file=1
else
in_test_file=0
fi
# Track #[cfg(test)] regions for per-line exemptions.
in_cfg_test=0
cfg_test_brace_depth=0
line_no=0
while IFS= read -r raw_line || [ -n "$raw_line" ]; do
line_no=$((line_no + 1))
line="$raw_line"
# Crude #[cfg(test)] region tracker: when we see #[cfg(test)] on a
# line followed by a mod with `{`, count braces until depth returns
# to zero. This is approximate but matches typical formatting.
if [[ "$line" =~ \#\[cfg\(test\)\] ]]; then
in_cfg_test=1
cfg_test_brace_depth=0
fi
if [ "$in_cfg_test" -eq 1 ]; then
opens=$(printf '%s' "$line" | tr -cd '{' | wc -c)
closes=$(printf '%s' "$line" | tr -cd '}' | wc -c)
cfg_test_brace_depth=$((cfg_test_brace_depth + opens - closes))
if [ "$cfg_test_brace_depth" -le 0 ] && [[ ! "$line" =~ \#\[cfg\(test\)\] ]]; then
in_cfg_test=0
fi
fi
exempt=0
[ "$in_test_file" -eq 1 ] && exempt=1
[ "$in_cfg_test" -eq 1 ] && exempt=1
# Strip line comments before pattern checks - so doc comments and
# explanatory prose do not trip the regexes.
code_only="${line%%//*}"
if [ "$exempt" -eq 0 ]; then
# .unwrap()
if [[ "$code_only" =~ \.unwrap\(\) ]]; then
# Allow if previous line had // SAFE-UNWRAP: comment
prev_line=$(sed -n "$((line_no - 1))p" "$file" 2>/dev/null || true)
if [[ ! "$prev_line" =~ //[[:space:]]*SAFE-UNWRAP: ]]; then
report "$file" "$line_no" "unwrap" ".unwrap() outside tests - use ? / ok_or / pattern match or annotate previous line with // SAFE-UNWRAP: <reason>"
fi
fi
# .expect("...")
if [[ "$code_only" =~ \.expect\( ]]; then
prev_line=$(sed -n "$((line_no - 1))p" "$file" 2>/dev/null || true)
if [[ ! "$prev_line" =~ //[[:space:]]*SAFE-EXPECT: ]]; then
report "$file" "$line_no" "expect" ".expect() outside tests - use ? or annotate previous line with // SAFE-EXPECT: <reason>"
fi
fi
# todo!() / unimplemented!() / unreachable!()
if [[ "$code_only" =~ (todo!|unimplemented!|unreachable!|unreachable_unchecked!) ]]; then
report "$file" "$line_no" "placeholder-macro" "todo!/unimplemented!/unreachable! in committed code"
fi
# Box<dyn Error
if [[ "$code_only" =~ Box\<dyn[[:space:]]+Error ]]; then
report "$file" "$line_no" "box-dyn-error" "Box<dyn Error> in non-test code - use anyhow::Error (apps) or thiserror enum (libs)"
fi
# panic!( in lib
if [[ "$file" == */src/lib.rs || "$file" == */src/*/mod.rs || ( "$file" == */src/*.rs && "$file" != */src/main.rs && "$file" != */src/bin/* ) ]]; then
if [[ "$code_only" =~ panic!\( ]]; then
report "$file" "$line_no" "lib-panic" "panic!() in library code - return Result"
fi
fi
fi
# unsafe { without preceding // SAFETY: in the last 5 lines (always enforced)
if [[ "$code_only" =~ unsafe[[:space:]]*\{ ]]; then
start=$((line_no > 5 ? line_no - 5 : 1))
window=$(sed -n "${start},${line_no}p" "$file" 2>/dev/null || true)
if [[ ! "$window" =~ //[[:space:]]*SAFETY: ]]; then
report "$file" "$line_no" "unsafe-no-safety-comment" "unsafe block without // SAFETY: comment in preceding 5 lines"
fi
fi
# #[allow(clippy::...)] without preceding // CLIPPY-ALLOW: justification
if [[ "$code_only" =~ \#\[allow\(clippy:: ]]; then
prev_line=$(sed -n "$((line_no - 1))p" "$file" 2>/dev/null || true)
if [[ ! "$prev_line" =~ //[[:space:]]*CLIPPY-ALLOW: ]]; then
report "$file" "$line_no" "unjustified-clippy-allow" "#[allow(clippy::...)] without // CLIPPY-ALLOW: <reason> on previous line"
fi
fi
# Narrowing numeric `as` casts - heuristic flag for human review.
# Catches the common shapes; precise type analysis belongs to clippy::cast_possible_truncation.
if [[ "$code_only" =~ as[[:space:]]+(u8|u16|u32|i8|i16|i32) ]] && \
[[ "$code_only" =~ (u16|u32|u64|u128|usize|i16|i32|i64|i128|isize)[[:space:]]+as[[:space:]]+(u8|u16|u32|i8|i16|i32) ]]; then
report "$file" "$line_no" "narrowing-as-cast" "possible narrowing 'as' cast - use TryFrom / try_into() for fallible conversion"
fi
done < "$file"
done
if [ "$violations" -gt 0 ]; then
echo "" >&2
echo "rust-programmer: ${violations} violation(s). Fix before declaring work done." >&2
echo "" >&2
echo "Then run the full toolchain gate:" >&2
echo " cargo +stable fmt --all -- --check" >&2
echo " cargo +stable clippy --all-targets --all-features -- -D warnings" >&2
echo " cargo nextest run --all-targets --all-features" >&2
echo " cargo +nightly miri nextest run --all-features # if unsafe touched" >&2
echo " cargo machete" >&2
echo " cargo deny check" >&2
exit 1
fi
echo "rust-programmer: no-excuse rules passed for $# file(s)."
packages/shared-skills/skills/programming/scripts/rust/new-project.py
Executable
+175
View File
@@ -0,0 +1,175 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "typer",
# "rich",
# ]
# ///
# ─── How to run ───
# 1. Install uv (if not installed):
# curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Run:
# uv run new-project.py myproject
# uv run new-project.py myproject --path ./workspace
# ──────────────────
#
# Creates a new Rust project with strict lints, deny.toml, rustfmt.toml,
# rust-toolchain.toml, and .cargo/config.toml pre-configured.
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
import typer
from rich.console import Console
console = Console(stderr=True)
# ── Embedded config contents ─────────────────────────────────────────────
RUST_TOOLCHAIN_TOML = """\
[toolchain]
channel = "stable"
components = ["rustfmt", "clippy", "rust-src"]
profile = "default"
"""
CARGO_TOML_LINTS = """
[lints.rust]
unsafe_op_in_unsafe_fn = "deny"
missing_docs = "warn"
missing_debug_implementations = "warn"
unreachable_pub = "warn"
unused_must_use = "deny"
elided_lifetimes_in_paths = "warn"
non_ascii_idents = "deny"
trivial_numeric_casts = "warn"
unused_lifetimes = "warn"
single_use_lifetimes = "warn"
[lints.clippy]
all = { level = "deny", priority = -1 }
pedantic = { level = "warn", priority = -1 }
nursery = { level = "warn", priority = -1 }
cargo = { level = "warn", priority = -1 }
undocumented_unsafe_blocks = "deny"
multiple_unsafe_ops_per_block = "deny"
unwrap_used = "deny"
expect_used = "deny"
panic = "deny"
todo = "deny"
unimplemented = "deny"
dbg_macro = "deny"
print_stdout = "warn"
print_stderr = "warn"
module_name_repetitions = { level = "allow" }
must_use_candidate = { level = "allow" }
missing_errors_doc = { level = "allow" }
missing_panics_doc = { level = "allow" }
"""
CARGO_CONFIG_TOML = """\
[build]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]
[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=lld"]
[target.aarch64-apple-darwin]
rustflags = []
"""
DENY_TOML = """\
[advisories]
vulnerability = "deny"
unmaintained = "warn"
yanked = "deny"
[licenses]
unlicensed = "deny"
allow = ["MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC", "Unicode-3.0", "Zlib"]
[bans]
multiple-versions = "warn"
wildcards = "deny"
[sources]
unknown-registry = "deny"
unknown-git = "deny"
"""
RUSTFMT_TOML = """\
edition = "2024"
max_width = 100
use_field_init_shorthand = true
use_try_shorthand = true
"""
# ── Main ─────────────────────────────────────────────────────────────────
app = typer.Typer(add_completion=False)
@app.command()
def main(
name: str = typer.Argument(help="Name of the new Rust project"),
path: Path = typer.Option(
Path.cwd(),
"--path",
"-p",
help="Parent directory where the project folder is created",
),
) -> None:
"""Scaffold a new Rust project with strict lints and tooling configs."""
project_dir = path / name
# ── cargo init ───────────────────────────────────────────────────
console.print(f"[bold green]Creating[/] project [cyan]{name}[/] at [dim]{project_dir}[/]")
try:
subprocess.run(
["cargo", "init", str(project_dir), "--name", name],
check=True,
capture_output=True,
text=True,
)
except FileNotFoundError:
console.print("[bold red]Error:[/] cargo not found. Install Rust via https://rustup.rs")
sys.exit(1)
except subprocess.CalledProcessError as exc:
console.print(f"[bold red]cargo init failed:[/]\n{exc.stderr}")
sys.exit(1)
# ── rust-toolchain.toml ──────────────────────────────────────────
(project_dir / "rust-toolchain.toml").write_text(RUST_TOOLCHAIN_TOML)
console.print(" [dim]wrote[/] rust-toolchain.toml")
# ── Append [lints] to Cargo.toml ─────────────────────────────────
cargo_toml = project_dir / "Cargo.toml"
with cargo_toml.open("a") as f:
f.write(CARGO_TOML_LINTS)
console.print(" [dim]appended[/] [lints] to Cargo.toml")
# ── .cargo/config.toml ───────────────────────────────────────────
cargo_config_dir = project_dir / ".cargo"
cargo_config_dir.mkdir(parents=True, exist_ok=True)
(cargo_config_dir / "config.toml").write_text(CARGO_CONFIG_TOML)
console.print(" [dim]wrote[/] .cargo/config.toml")
# ── deny.toml ────────────────────────────────────────────────────
(project_dir / "deny.toml").write_text(DENY_TOML)
console.print(" [dim]wrote[/] deny.toml")
# ── rustfmt.toml ─────────────────────────────────────────────────
(project_dir / "rustfmt.toml").write_text(RUSTFMT_TOML)
console.print(" [dim]wrote[/] rustfmt.toml")
console.print(f"\n[bold green]Done![/] cd {project_dir} && cargo check")
if __name__ == "__main__":
app()
packages/shared-skills/skills/programming/scripts/typescript/check-no-excuse-rules.ts
+282
View File
@@ -0,0 +1,282 @@
#!/usr/bin/env bun
/**
* Check TypeScript files for no-excuse violations.
*
* Rules:
* no-any-assertion - `as any`
* no-unknown-assertion - `as unknown`
* no-ts-ignore - `@ts-ignore` comments
* no-ts-expect-error - `@ts-expect-error` comments
* no-enum - `enum` declarations
* no-non-null-assertion - `x!` postfix operator
* no-throw-literal - `throw "string"` / `throw 123`
* no-mutable-export - `export let` / `export var`
* no-any-annotation - `: any` in annotations (opt out: `// no-excuse-ok: any`)
* no-explicit-any-return - `(): any` return types (opt out: `// no-excuse-ok: any`)
* empty-catch - `catch { }` or `catch (e) { }` with empty body
* catch-without-narrowing - catch block that uses error without instanceof narrowing
*
* Usage:
* bun run scripts/check-no-excuse-rules.ts <file-or-dir>...
*
* Exit codes:
* 0 - no violations
* 1 - violations found
* 2 - input error
*/
import fs from "node:fs"
import path from "node:path"
import process from "node:process"
import ts from "typescript"
type RuleId =
| "no-any-assertion"
| "no-unknown-assertion"
| "no-ts-ignore"
| "no-ts-expect-error"
| "no-enum"
| "no-non-null-assertion"
| "no-throw-literal"
| "no-mutable-export"
| "no-any-annotation"
| "no-explicit-any-return"
| "empty-catch"
| "catch-without-narrowing"
type Violation = {
readonly ruleId: RuleId
readonly filePath: string
readonly line: number
readonly column: number
readonly message: string
}
const INCLUDED_EXTENSIONS = new Set([".ts", ".tsx", ".mts", ".cts"])
const IGNORED_DIRECTORIES = new Set([
".git", ".next", ".nuxt", ".turbo", ".yarn",
"coverage", "dist", "build", "node_modules",
])
const OPT_OUT_RE = /\/\/\s*no-excuse-ok:\s*any/
const CATCH_OK_RE = /\/\/\s*no-excuse-ok:\s*catch/
function isIncludedFile(filePath: string): boolean {
return INCLUDED_EXTENSIONS.has(path.extname(filePath).toLowerCase())
}
function isDeclarationFile(filePath: string): boolean {
return filePath.endsWith(".d.ts") || filePath.endsWith(".d.mts") || filePath.endsWith(".d.cts")
}
function discoverFiles(inputs: string[]): string[] {
const files: string[] = []
for (const input of inputs) {
const resolved = path.resolve(input)
if (!fs.existsSync(resolved)) {
console.error(`Path does not exist: ${resolved}`)
process.exit(2)
}
if (fs.statSync(resolved).isFile()) {
if (isIncludedFile(resolved) && !isDeclarationFile(resolved)) files.push(resolved)
continue
}
const walk = (dir: string): void => {
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
if (entry.isDirectory()) {
if (!IGNORED_DIRECTORIES.has(entry.name)) walk(path.join(dir, entry.name))
} else if (isIncludedFile(entry.name) && !isDeclarationFile(entry.name)) {
files.push(path.join(dir, entry.name))
}
}
}
walk(resolved)
}
return files
}
function getLineText(sourceFile: ts.SourceFile, line: number): string {
const lineStarts = sourceFile.getLineStarts()
const start = lineStarts[line]
const end = line + 1 < lineStarts.length ? lineStarts[line + 1] : sourceFile.getEnd()
return sourceFile.text.slice(start, end)
}
function analyzeFile(filePath: string): Violation[] {
const source = fs.readFileSync(filePath, "utf-8")
const sourceFile = ts.createSourceFile(filePath, source, ts.ScriptTarget.Latest, true)
const violations: Violation[] = []
function pos(node: ts.Node): { line: number; column: number } {
const { line, character } = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile))
return { line: line + 1, column: character + 1 }
}
function lineHasOptOut(node: ts.Node): boolean {
const { line } = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile))
return OPT_OUT_RE.test(getLineText(sourceFile, line))
}
function visit(node: ts.Node): void {
// ── as any / as unknown ──
if (ts.isAsExpression(node)) {
const typeText = node.type.getText(sourceFile)
if (typeText === "any") {
const p = pos(node)
violations.push({ ruleId: "no-any-assertion", filePath, ...p, message: "`as any` — narrow with type guards or redesign the types" })
}
if (typeText === "unknown") {
const p = pos(node)
violations.push({ ruleId: "no-unknown-assertion", filePath, ...p, message: "`as unknown` — redesign the types" })
}
}
// ── enum ──
if (ts.isEnumDeclaration(node)) {
const p = pos(node)
violations.push({ ruleId: "no-enum", filePath, ...p, message: "`enum` — use `as const` object + literal union type" })
}
// ── x! non-null assertion ──
if (ts.isNonNullExpression(node)) {
const p = pos(node)
violations.push({ ruleId: "no-non-null-assertion", filePath, ...p, message: "`x!` — use narrowing or optional chaining" })
}
// ── throw "literal" ──
if (ts.isThrowStatement(node) && node.expression) {
const expr = node.expression
if (ts.isStringLiteral(expr) || ts.isNumericLiteral(expr) || ts.isNoSubstitutionTemplateLiteral(expr)) {
const p = pos(node)
violations.push({ ruleId: "no-throw-literal", filePath, ...p, message: "`throw literal` — throw an Error subclass" })
}
if (ts.isTemplateExpression(expr)) {
const p = pos(node)
violations.push({ ruleId: "no-throw-literal", filePath, ...p, message: "`throw template` — throw an Error subclass" })
}
}
// ── export let / export var ──
if (ts.isVariableStatement(node)) {
const hasExport = node.modifiers?.some((m) => m.kind === ts.SyntaxKind.ExportKeyword)
if (hasExport) {
const flags = node.declarationList.flags
if (!(flags & ts.NodeFlags.Const)) {
const p = pos(node)
violations.push({ ruleId: "no-mutable-export", filePath, ...p, message: "`export let/var` — use `export const`" })
}
}
}
// ── : any in annotations ──
if (ts.isTypeReferenceNode(node) || node.kind === ts.SyntaxKind.AnyKeyword) {
if (node.kind === ts.SyntaxKind.AnyKeyword && !lineHasOptOut(node)) {
const parent = node.parent
// Skip `as any` — already caught by no-any-assertion
if (parent && ts.isAsExpression(parent)) {
// already handled
} else if (parent && (
ts.isParameter(parent) ||
ts.isVariableDeclaration(parent) ||
ts.isPropertyDeclaration(parent) ||
ts.isPropertySignature(parent)
)) {
const p = pos(node)
violations.push({ ruleId: "no-any-annotation", filePath, ...p, message: "`: any` annotation — use `unknown` and narrow" })
} else if (parent && (
ts.isFunctionDeclaration(parent) ||
ts.isMethodDeclaration(parent) ||
ts.isArrowFunction(parent) ||
ts.isFunctionExpression(parent)
)) {
const p = pos(node)
violations.push({ ruleId: "no-explicit-any-return", filePath, ...p, message: "`(): any` return — use a specific type" })
}
}
}
// ── empty catch / catch without narrowing ──
if (ts.isCatchClause(node)) {
const catchLine = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile)).line
const catchLineText = getLineText(sourceFile, catchLine)
if (!CATCH_OK_RE.test(catchLineText)) {
const body = node.block
const stmts = body.statements
if (stmts.length === 0) {
// Empty catch — swallows everything silently
const p = pos(node)
violations.push({ ruleId: "empty-catch", filePath, ...p, message: "empty `catch` block — handle, re-throw, or remove the try/catch" })
} else if (node.variableDeclaration) {
// Has a bound variable — check if it's narrowed with instanceof
const varName = node.variableDeclaration.name.getText(sourceFile)
const blockText = body.getText(sourceFile)
const hasInstanceof = blockText.includes(`instanceof`)
const hasRethrow = blockText.includes(`throw ${varName}`) || blockText.includes(`throw new`)
if (!hasInstanceof && !hasRethrow) {
const p = pos(node)
violations.push({ ruleId: "catch-without-narrowing", filePath, ...p, message: "`catch` without `instanceof` narrowing or re-throw — narrow the error type or re-throw" })
}
}
}
}
ts.forEachChild(node, visit)
}
visit(sourceFile)
// ── @ts-ignore / @ts-expect-error in comments ──
const commentRanges = [
...(ts.getLeadingCommentRanges(source, 0) ?? []),
]
// Scan all comments via regex for reliability
const commentRegex = /\/\/\s*@ts-(ignore|expect-error)/g
let match: RegExpExecArray | null
while ((match = commentRegex.exec(source)) !== null) {
const { line, character } = sourceFile.getLineAndCharacterOfPosition(match.index)
const kind = match[1]
violations.push({
ruleId: kind === "ignore" ? "no-ts-ignore" : "no-ts-expect-error",
filePath,
line: line + 1,
column: character + 1,
message: `\`@ts-${kind}\` — fix the underlying type`,
})
}
return violations
}
function formatViolation(v: Violation): string {
return `${v.filePath}:${v.line}:${v.column}: [${v.ruleId}] ${v.message}`
}
function main(): void {
const args = process.argv.slice(2)
if (args.length === 0) {
console.error("usage: check-no-excuse-rules.ts <file-or-dir>...")
process.exit(2)
}
const files = discoverFiles(args)
if (files.length === 0) {
console.error("No TypeScript files found.")
process.exit(2)
}
const violations = files.flatMap((f) => analyzeFile(f))
if (violations.length === 0) {
console.log(`No violations in ${files.length} file(s).`)
return
}
for (const v of violations) {
console.error(formatViolation(v))
}
console.error(`\n${violations.length} violation(s) in ${files.length} file(s).`)
process.exit(1)
}
main()
packages/shared-skills/skills/programming/scripts/typescript/new-project.ts
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env bun
/**
* Scaffold a new TypeScript project with ultra-strict defaults.
*
* ─── How to run ───
* 1. Install Bun: curl -fsSL https://bun.sh/install | bash
* 2. Run:
* bun run scripts/new-project.ts my-api
* bun run scripts/new-project.ts my-api --path ./projects
* ──────────────────
*
* Creates:
* <name>/
* package.json (Bun + Hono + Zod + Drizzle + Biome)
* tsconfig.json (ultra-strict from tsconfig-strict.md)
* biome.json (strict from tsconfig-strict.md)
* src/index.ts (minimal Hono entrypoint)
* .gitignore
*/
import { mkdirSync, writeFileSync, existsSync } from "node:fs";
import { join, resolve } from "node:path";
import { parseArgs } from "node:util";
const { values, positionals } = parseArgs({
args: Bun.argv.slice(2),
options: {
path: { type: "string", default: "." },
help: { type: "boolean", short: "h", default: false },
},
allowPositionals: true,
strict: true,
});
if (values.help || positionals.length === 0) {
console.log(`Usage: bun run new-project.ts <name> [--path <dir>]
Arguments:
name Project directory name (kebab-case)
Options:
--path Parent directory (default: current dir)
-h, --help Show this help`);
process.exit(positionals.length === 0 ? 2 : 0);
}
const name = positionals[0]!;
const root = resolve(values.path!, name);
if (existsSync(root)) {
console.error(`Error: ${root} already exists`);
process.exit(1);
}
// ── Directory structure ──
mkdirSync(join(root, "src"), { recursive: true });
// ── package.json ──
const pkg = {
name,
version: "0.0.1",
private: true,
type: "module",
scripts: {
dev: "bun --hot src/index.ts",
start: "bun src/index.ts",
check: "bunx biome check . && bunx tsc --noEmit && bun test",
"check:fix": "bunx biome check --write .",
test: "bun test",
},
dependencies: {
hono: "^4.12.5",
zod: "^3.24.0",
},
devDependencies: {
"@biomejs/biome": "^1.9.0",
"@types/bun": "latest",
typescript: "^5.8.0",
},
};
writeFileSync(join(root, "package.json"), JSON.stringify(pkg, null, 2) + "\n");
// ── tsconfig.json (ultra-strict) ──
const tsconfig = {
compilerOptions: {
strict: true,
noUncheckedIndexedAccess: true,
exactOptionalPropertyTypes: true,
noFallthroughCasesInSwitch: true,
forceConsistentCasingInFileNames: true,
verbatimModuleSyntax: true,
isolatedModules: true,
esModuleInterop: true,
resolveJsonModule: true,
target: "ESNext",
lib: ["ESNext"],
declaration: true,
declarationMap: true,
sourceMap: true,
outDir: "dist",
rootDir: "src",
module: "ESNext",
moduleResolution: "bundler",
types: ["bun-types"],
skipLibCheck: true,
noEmit: true,
},
include: ["src/**/*.ts"],
exclude: ["node_modules", "dist"],
};
writeFileSync(
join(root, "tsconfig.json"),
JSON.stringify(tsconfig, null, 2) + "\n",
);
// ── biome.json (strict) ──
const biome = {
$schema: "https://biomejs.dev/schemas/1.9.0/schema.json",
organizeImports: { enabled: true },
formatter: {
enabled: true,
indentStyle: "space",
indentWidth: 2,
lineWidth: 100,
},
linter: {
enabled: true,
rules: {
recommended: true,
complexity: {
noBannedTypes: "error",
noExtraBooleanCast: "error",
noUselessConstructor: "error",
noUselessRename: "error",
noVoid: "error",
},
correctness: {
noUnusedVariables: "error",
noUnusedImports: "error",
useExhaustiveDependencies: "warn",
},
style: {
noNonNullAssertion: "error",
useConst: "error",
noParameterAssign: "error",
},
suspicious: {
noExplicitAny: "error",
noAssertion: "warn",
},
},
},
};
writeFileSync(join(root, "biome.json"), JSON.stringify(biome, null, 2) + "\n");
// ── src/index.ts ──
const indexTs = `import { Hono } from "hono";
const app = new Hono();
app.get("/", (c) => c.json({ status: "ok" }));
export default app;
`;
writeFileSync(join(root, "src/index.ts"), indexTs);
// ── .gitignore ──
const gitignore = `node_modules/
dist/
*.tsbuildinfo
.env
.env.*
`;
writeFileSync(join(root, ".gitignore"), gitignore);
console.log(`✓ Created: ${root}`);
console.log(` cd ${name} && bun install && bun run check`);