docs(omo-codex): batch 95 (14 files)
This commit is contained in:
@@ -0,0 +1,314 @@
|
|||||||
|
|
||||||
|
# Python Programmer
|
||||||
|
|
||||||
|
Modern Python. Type-strict, stack-first, async-correct.
|
||||||
|
|
||||||
|
## Philosophy
|
||||||
|
|
||||||
|
The type checker is your compiler. Make illegal states unrepresentable. Parse at boundaries. Own resources explicitly. Every function has a contract; the type system enforces it.
|
||||||
|
|
||||||
|
## Hard rules
|
||||||
|
|
||||||
|
These are deliberate project choices. Violations are always wrong, not "style preferences".
|
||||||
|
|
||||||
|
### Tooling
|
||||||
|
|
||||||
|
| Category | Use | Never |
|
||||||
|
|---|---|---|
|
||||||
|
| Package manager | `uv` | pip, poetry, conda, pipenv |
|
||||||
|
| Type checker | `basedpyright` (`typeCheckingMode = "all"`) | pyright, mypy |
|
||||||
|
| Linter + formatter | `ruff` (`select = ["ALL"]`) | flake8, black, isort, autopep8 |
|
||||||
|
| Async runtime | `anyio` | `import asyncio` |
|
||||||
|
| Data | `polars` + `duckdb` + `numpy` | pandas |
|
||||||
|
| Web framework | FastAPI + Pydantic v2 | Flask, Django REST |
|
||||||
|
| ORM | SQLAlchemy 2.x async | Django ORM, Tortoise |
|
||||||
|
| HTTP client | [`httpx2`](https://github.com/pydantic/httpx2) | requests, aiohttp, httpx |
|
||||||
|
| Testing | `pytest` | unittest |
|
||||||
|
| CLI | `typer` + `rich` | argparse, click, fire |
|
||||||
|
|
||||||
|
### The iron list
|
||||||
|
|
||||||
|
1. **Frozen by default** — `@dataclass(frozen=True, slots=True)`. Pydantic: `model_config = ConfigDict(frozen=True)`. Mutable only when mutation is the documented purpose.
|
||||||
|
2. **NewType for distinct IDs** — `UserId = NewType("UserId", int)`. Never pass raw `int` where a branded type exists.
|
||||||
|
3. **`match` only for variants, `if` only for booleans** — **NEVER** use `if/elif/else` to discriminate on type (`isinstance`), enum value, or literal variant. `match/case` is mandatory for these — non-negotiable. **ALWAYS** end with `case unreachable: assert_never(unreachable)` — bare `case _: pass` and `case _: raise ValueError` are banned (they silently swallow new variants). `if/else` is fine only for boolean expressions, range checks, and predicate calls that aren't variant discrimination. See "Why `if/elif` on variants is banned" below for examples.
|
||||||
|
4. **Protocol over ABC** — `typing.Protocol` for interfaces. ABC only when you need shared method implementation.
|
||||||
|
5. **No raw dicts in signatures** — params and returns use `TypedDict`, `dataclass`, or Pydantic model. Internal scratch dicts are fine.
|
||||||
|
6. **Parse, don't validate** — constructors produce typed objects or raise. Never pass unvalidated data deeper into the call stack.
|
||||||
|
7. **Typed errors** — error types are dataclasses or exceptions with typed fields. Never `raise ValueError("something")` with a bare string. Use union returns when the caller is within 1-2 call levels and must handle the outcome (repository → service). Use exceptions when the error should propagate up many layers to a boundary handler (service → HTTP handler).
|
||||||
|
8. **Final for constants** — module-level constants use `Final`. Mutable module globals are a code smell.
|
||||||
|
9. **Explicit None** — annotate `-> X | None`. Never return `None` from a function whose signature omits it.
|
||||||
|
10. **Context managers for resources** — files, DB connections, HTTP clients, locks. No manual `.close()`.
|
||||||
|
11. **No Any, no object** — both are banned as type annotations. `object` erases all structural information (zero callable attributes, zero narrowing). Use `Protocol` (structural typing), `TypeVar` (generic pass-through), explicit union (known variants), or `TypedDict` (dict shapes).
|
||||||
|
12. **No cast** — `cast()` is banned. Redesign the types.
|
||||||
|
13. **No type: ignore** — fix the type error. The checker is right; you are wrong.
|
||||||
|
14. **No broad except** — `except Exception` and `except BaseException` are banned. Catch the **specific** exception you expect. A broad catch swallows bugs you need to see — `KeyError`, `AttributeError`, `TypeError` all vanish silently. If you genuinely need a catch-all at a top-level boundary (CLI entry, HTTP handler), use `# noqa: BROAD_EXCEPT_OK` and log + re-raise.
|
||||||
|
|
||||||
|
### Typing and safety
|
||||||
|
|
||||||
|
- `basedpyright` in `typeCheckingMode = "all"`. Every public function has full annotations. Internal helpers: annotate return type; parameter types may be inferred.
|
||||||
|
- `ruff` with `select = ["ALL"]`. Override specific rules per project in `pyproject.toml`, never globally disable the strict baseline.
|
||||||
|
- Every new function must have a `docstring` unless its name + signature makes it completely obvious (e.g. `def full_name(first: str, last: str) -> str:`).
|
||||||
|
- Use `X | Y` union syntax (PEP 604), never `Union[X, Y]` or `Optional[X]`.
|
||||||
|
|
||||||
|
### Why `object` is banned
|
||||||
|
|
||||||
|
`object` pretends to be safe ("it's the top type!") but gives **zero** narrowing and **zero** attributes. Even `Any` is more honest — it admits the boundary is untyped.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BANNED
|
||||||
|
def process(data: object) -> object: ...
|
||||||
|
def store(items: list[object]) -> None: ...
|
||||||
|
results: dict[str, object] = {}
|
||||||
|
|
||||||
|
# GOOD — Protocol for structural typing
|
||||||
|
class Serializable(Protocol):
|
||||||
|
def serialize(self) -> bytes: ...
|
||||||
|
def process(data: Serializable) -> ProcessResult: ...
|
||||||
|
|
||||||
|
# GOOD — TypeVar for generic pass-through
|
||||||
|
def identity[T](x: T) -> T: ...
|
||||||
|
def first[T](items: Sequence[T]) -> T: ...
|
||||||
|
|
||||||
|
# GOOD — explicit union for known variants
|
||||||
|
def parse(raw: str | bytes) -> Document: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Why `if/elif` on variants is banned
|
||||||
|
|
||||||
|
`if/elif/else` chains on type, enum, or literal values lose compile-time exhaustiveness. When a new variant is added, nothing warns you. `match/case` + `assert_never` does.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BANNED — if/elif for type discrimination
|
||||||
|
if isinstance(event, Click):
|
||||||
|
handle_click(event.x, event.y)
|
||||||
|
elif isinstance(event, Scroll):
|
||||||
|
handle_scroll(event.delta)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unknown: {event}") # runtime bomb
|
||||||
|
|
||||||
|
# BANNED — if/elif for enum discrimination
|
||||||
|
if status == Status.PENDING:
|
||||||
|
start_review()
|
||||||
|
elif status == Status.ACTIVE:
|
||||||
|
continue_processing()
|
||||||
|
elif status == Status.CLOSED:
|
||||||
|
archive()
|
||||||
|
|
||||||
|
# BANNED — non-exhaustive match (swallows new variants)
|
||||||
|
match event:
|
||||||
|
case Click(x, y): handle_click(x, y)
|
||||||
|
case _: pass
|
||||||
|
|
||||||
|
# GOOD — exhaustive match with assert_never
|
||||||
|
match event:
|
||||||
|
case Click(x=x, y=y):
|
||||||
|
handle_click(x, y)
|
||||||
|
case Scroll(delta=delta):
|
||||||
|
handle_scroll(delta)
|
||||||
|
case unreachable:
|
||||||
|
assert_never(unreachable)
|
||||||
|
|
||||||
|
# GOOD — enum match
|
||||||
|
match status:
|
||||||
|
case Status.PENDING: start_review()
|
||||||
|
case Status.ACTIVE: continue_processing()
|
||||||
|
case Status.CLOSED: archive()
|
||||||
|
case unreachable: assert_never(unreachable)
|
||||||
|
```
|
||||||
|
|
||||||
|
`if/else` is fine for boolean conditions and range checks — things that aren't variant discrimination:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# FINE — boolean, not variant
|
||||||
|
if age >= 18:
|
||||||
|
grant_access()
|
||||||
|
else:
|
||||||
|
deny_access()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Why broad `except` is banned
|
||||||
|
|
||||||
|
`except Exception` catches **every** non-system exception — `KeyError`, `TypeError`, `AttributeError`, `ValueError` all vanish. You lose the stack trace that would have told you exactly what went wrong. The fix is always to name the exception you expect.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BANNED — swallows bugs
|
||||||
|
try:
|
||||||
|
result = api.fetch(url)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
# BANNED — catch-and-ignore
|
||||||
|
try:
|
||||||
|
parse(data)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# GOOD — catch what you expect
|
||||||
|
try:
|
||||||
|
result = api.fetch(url)
|
||||||
|
except httpx.HTTPStatusError as e:
|
||||||
|
logger.error("API %d: %s", e.response.status_code, e.request.url)
|
||||||
|
return None
|
||||||
|
except httpx.ConnectError:
|
||||||
|
raise ServiceUnavailableError(service="api") from None
|
||||||
|
|
||||||
|
# GOOD — top-level boundary (only place broad catch is acceptable)
|
||||||
|
def main() -> int: # noqa: BROAD_EXCEPT_OK
|
||||||
|
try:
|
||||||
|
return run()
|
||||||
|
except Exception:
|
||||||
|
logger.exception("unhandled error")
|
||||||
|
return 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Async
|
||||||
|
|
||||||
|
- `import asyncio` is **BANNED**. Use `import anyio`.
|
||||||
|
- For background tasks, use `anyio.create_task_group`. Never fire-and-forget with `asyncio.create_task`.
|
||||||
|
- For concurrency gates, use `anyio.CapacityLimiter` (not `asyncio.Semaphore`).
|
||||||
|
- Load `async-anyio.md` when writing async code for the full pattern library.
|
||||||
|
|
||||||
|
### Data modeling — which container, when
|
||||||
|
|
||||||
|
All model fields carry type annotations. No `Any`, no untyped dicts in public APIs.
|
||||||
|
Use `polars` + `duckdb` for data. pandas is never the right answer in this stack.
|
||||||
|
|
||||||
|
| Situation | Use |
|
||||||
|
|---|---|
|
||||||
|
| User input, API request/response | `Pydantic BaseModel (frozen=True)` |
|
||||||
|
| Internal value object (no I/O) | `@dataclass(frozen=True, slots=True)` |
|
||||||
|
| Function with multiple outcomes | Union of frozen dataclasses + `match` |
|
||||||
|
| Dict shape for JSON compat / `**kwargs` | `TypedDict` |
|
||||||
|
| Fixed constants | `StrEnum` / `IntEnum` |
|
||||||
|
| Distinct primitive (UserId vs MovieId) | `NewType` |
|
||||||
|
| Contract / capability | `Protocol` |
|
||||||
|
| Contract + shared implementation | `ABC` |
|
||||||
|
| ORM model (SQLAlchemy) | `Mapped[]` — inherently mutable, `# noqa: MUTABLE_OK` |
|
||||||
|
| Config from env vars | `pydantic-settings BaseSettings` |
|
||||||
|
|
||||||
|
**The one rule**: data crosses trust boundary → Pydantic. Everything else → dataclass.
|
||||||
|
|
||||||
|
Load `data-modeling.md` for the full decision flowchart and comparison matrix.
|
||||||
|
|
||||||
|
### When frozen=True does not apply
|
||||||
|
|
||||||
|
- **ORM models** — SQLAlchemy `Mapped[]` requires mutation. Use `# noqa: MUTABLE_OK`.
|
||||||
|
- **Builder / accumulator** — object exists to be mutated (counter, buffer, state machine). Docstring must explain why.
|
||||||
|
- **Pydantic Settings** — tests override fields. Mutable is acceptable.
|
||||||
|
|
||||||
|
If you need `# noqa: MUTABLE_OK`, the class docstring must say why mutation is required.
|
||||||
|
|
||||||
|
### Libraries
|
||||||
|
|
||||||
|
Canonical defaults (override only if `pyproject.toml` explicitly picks something else):
|
||||||
|
|
||||||
|
| Domain | Library | Reason |
|
||||||
|
|---|---|---|
|
||||||
|
| CLI | `typer` | Type-annotated CLI from function sigs |
|
||||||
|
| Pretty output | `rich` | Tables, progress, tracebacks, markdown |
|
||||||
|
| HTTP client | [`httpx2`](https://github.com/pydantic/httpx2) | Next-gen HTTP client (Pydantic stewardship), HTTP/2, brotli+zstd. Always `httpx2[http2,brotli,zstd]`. See `httpx2-optimization.md` |
|
||||||
|
| Validation | `pydantic` v2 | Fast native validator, JSON Schema |
|
||||||
|
| Web API | `fastapi` | Async, Pydantic-native, OpenAPI |
|
||||||
|
| ORM | `sqlalchemy` 2.x async | `Mapped[]` types, async sessions |
|
||||||
|
| DB driver (Postgres) | `asyncpg` (via SQLAlchemy) | Fastest PG driver |
|
||||||
|
| AI agents | `pydantic-ai` | Typed deps, structured output |
|
||||||
|
| TUI | `textual` | Rich-based, CSS layout, widgets |
|
||||||
|
| Logging | `rich.logging.RichHandler` | Pretty; swap to `structlog` in prod |
|
||||||
|
|
||||||
|
## pyproject.toml — the one true config
|
||||||
|
|
||||||
|
Scaffold a new project with all strict defaults pre-configured:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run ../../scripts/python/new-project.py myproject
|
||||||
|
uv run ../../scripts/python/new-project.py myproject --path ./workspace
|
||||||
|
uv run ../../scripts/python/new-project.py myproject --lib # publishable library
|
||||||
|
```
|
||||||
|
|
||||||
|
Creates via `uv init`, then injects basedpyright `typeCheckingMode = "all"` + ruff `select = ["ALL"]` + pytest strict. Cross-platform (macOS, Linux, Windows).
|
||||||
|
|
||||||
|
For manual setup: `uv init --app myproject`, then load `pyproject-strict.md`.
|
||||||
|
|
||||||
|
## PEP 723 — inline script metadata (mandatory for ALL scripts)
|
||||||
|
|
||||||
|
Every `.py` script — even throwaway — MUST use PEP 723 inline metadata with the `# ─── How to run ───` comment block. No venv, no `requirements.txt`. The script IS the environment spec. A script without the usage comment block is incomplete.
|
||||||
|
|
||||||
|
Scaffold with: `uv run ../../scripts/python/new-script.py <name> --deps "httpx2[http2,brotli,zstd]"` (writes to temp dir by default, `--output` for specific path).
|
||||||
|
|
||||||
|
Load `one-liners.md` for full patterns, examples, and anti-patterns.
|
||||||
|
|
||||||
|
## Reference loading
|
||||||
|
|
||||||
|
Load on demand — not all at once.
|
||||||
|
|
||||||
|
| Need | Load |
|
||||||
|
|---|---|
|
||||||
|
| Full pyproject.toml config | `pyproject-strict.md` |
|
||||||
|
| Type patterns (NewType, Final, enums, narrowing) | `type-patterns.md` |
|
||||||
|
| Data modeling (container choice, frozen, parse-don't-validate) | `data-modeling.md` |
|
||||||
|
| Error handling (typed errors, union returns, exhaustive match) | `error-handling.md` |
|
||||||
|
| Async patterns (anyio) | `async-anyio.md` |
|
||||||
|
| Data processing (polars / duckdb) | `data-processing.md` |
|
||||||
|
| FastAPI + SQLAlchemy stack | `fastapi-stack.md` |
|
||||||
|
| Library decision tree | `libraries.md` |
|
||||||
|
| **httpx2 optimization** (MUST load for any network code) | `httpx2-optimization.md` |
|
||||||
|
| **orjson** (when JSON is in the hot path; FastAPI/Pydantic v2 integration) | `orjson-stack.md` |
|
||||||
|
| One-liner scripts (PEP 723) | `one-liners.md` |
|
||||||
|
| PydanticAI agents | `pydantic-ai.md` |
|
||||||
|
| Textual TUI | `textual-tui.md` |
|
||||||
|
|
||||||
|
## httpx2 — mandatory for ALL network requests
|
||||||
|
|
||||||
|
Every outgoing HTTP call MUST use [`httpx2`](https://github.com/pydantic/httpx2) (`httpx2[http2,brotli,zstd]`). Never `requests`, never `aiohttp`, never the original `httpx`.
|
||||||
|
|
||||||
|
**ALL optimizations are ON by default — not optional, not progressive, not "nice to have".** A bare `httpx2.AsyncClient()` is a bug — treat it like a lint violation. The correct way is the factory pattern in `httpx2-optimization.md` with: HTTP/2 enabled, tuned connection pool (200/40/30s), split timeouts (5/30/10/10), transport retries (3), TCP_NODELAY, follow_redirects, and event hooks for observability.
|
||||||
|
|
||||||
|
When writing or reviewing ANY network code, **ALWAYS load `httpx2-optimization.md`** and use the factory pattern verbatim. No exceptions.
|
||||||
|
|
||||||
|
## No-excuse audit
|
||||||
|
|
||||||
|
Violations caught by `../../scripts/python/check-no-excuse-rules.py`. Run after every edit session.
|
||||||
|
|
||||||
|
| Rule ID | Catches | Opt-out |
|
||||||
|
|---|---|---|
|
||||||
|
| `cast-any` | `cast(Any, ...)` | None — redesign types |
|
||||||
|
| `type-ignore` | `# type: ignore` | None — fix the type |
|
||||||
|
| `pyright-ignore` | `# pyright: ignore` | None — fix the type |
|
||||||
|
| `bare-except` | `except:` with no class | None — name the exception |
|
||||||
|
| `silent-except` | `except X: pass` / `except X: ...` | None — handle or re-raise |
|
||||||
|
| `no-asyncio` | `import asyncio` | `# noqa: ANYIO_OK` |
|
||||||
|
| `no-pandas` | `import pandas` | `# noqa: PANDAS_OK` |
|
||||||
|
| `mutable-dataclass` | `@dataclass` without `frozen=True` | `# noqa: MUTABLE_OK` |
|
||||||
|
| `missing-slots` | `@dataclass` without `slots=True` | `# noqa: SLOTS_OK` |
|
||||||
|
| `raw-dict-return` | `-> dict` in function return type | `# noqa: DICT_OK` |
|
||||||
|
| `missing-assert-never` | `match` block without `assert_never` default | `# noqa: MATCH_OK` |
|
||||||
|
| `generic-exception` | `raise ValueError("...")` / `raise TypeError("...")` with bare string | `# noqa: GENERIC_ERR_OK` |
|
||||||
|
| `no-object` | `object` used as type annotation (param, return, generic arg) | `# noqa: OBJECT_OK` |
|
||||||
|
| `if-elif-on-variant` | `if isinstance()`/`if x == Enum.V` chain that should be `match/case` | `# noqa: IF_VARIANT_OK` |
|
||||||
|
| `oversized-module` | File exceeds 250 pure LOC (non-blank, non-comment) | `# noqa: SIZE_OK` |
|
||||||
|
| `broad-except` | `except Exception` / `except BaseException` (too broad) | `# noqa: BROAD_EXCEPT_OK` |
|
||||||
|
|
||||||
|
Fix every violation before declaring work done. basedpyright + ruff strict config catches the rest.
|
||||||
|
|
||||||
|
## In tests
|
||||||
|
|
||||||
|
Tests are strict too, with these exceptions (already configured in `pyproject.toml` per-file-ignores):
|
||||||
|
|
||||||
|
| In tests you may | Why |
|
||||||
|
|---|---|
|
||||||
|
| Use `assert` | That's how pytest works (`S101` ignored) |
|
||||||
|
| Use magic numbers | Test data (`PLR2004` ignored) |
|
||||||
|
| Access `_private` members | Testing internals (`SLF001` ignored) |
|
||||||
|
| Skip docstrings | Test names are the docs (`D` ignored) |
|
||||||
|
| Have unused function args | Fixtures (`ARG` ignored) |
|
||||||
|
|
||||||
|
Tests still follow the iron list — frozen dataclasses, typed errors, exhaustive match. If test fixtures need mutable state, use `# noqa: MUTABLE_OK` on the fixture class.
|
||||||
|
|
||||||
|
## Existing codebases
|
||||||
|
|
||||||
|
When editing an existing file that doesn't follow these rules: **write new code in strict style, don't refactor existing code in the same change.** Mixing feature work with style migration makes reviews harder and bugs likelier.
|
||||||
|
|
||||||
|
## Activation
|
||||||
|
|
||||||
|
This skill activates whenever you are writing or modifying any `.py` file. Even one-off scripts get the strict treatment — that is the whole point of PEP 723 + uv: production hygiene with throwaway ergonomics.
|
||||||
@@ -0,0 +1,442 @@
|
|||||||
|
# AnyIO Reference: Replacing asyncio Idioms
|
||||||
|
|
||||||
|
> **Skill mandate**: `import asyncio` is BANNED. Use `import anyio` exclusively.
|
||||||
|
> This reference targets AnyIO 4.x (2026 Python projects).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Task Groups (The Core Primitive)
|
||||||
|
|
||||||
|
AnyIO uses **structured concurrency** via task groups. A task group is an async context manager that guarantees all child tasks finish before the block exits.
|
||||||
|
|
||||||
|
### `start_soon` — fire-and-forget
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
async def worker(n: int) -> None:
|
||||||
|
await anyio.sleep(1)
|
||||||
|
print(f"task {n} done")
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
async with anyio.create_task_group() as tg:
|
||||||
|
for i in range(3):
|
||||||
|
tg.start_soon(worker, i)
|
||||||
|
print("all tasks finished")
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Signature**: `tg.start_soon(func, *args, name=None)`
|
||||||
|
- `func` must be a **coroutine function** (not a coroutine object).
|
||||||
|
- `name` is optional, for introspection/debugging.
|
||||||
|
- No return value; exceptions propagate as `ExceptionGroup` on exit.
|
||||||
|
|
||||||
|
### `start` — wait for ready signal
|
||||||
|
|
||||||
|
Use when a task must initialize before the caller proceeds (e.g., starting a server and then connecting to it).
|
||||||
|
|
||||||
|
```python
|
||||||
|
from anyio import TASK_STATUS_IGNORED, create_task_group, run
|
||||||
|
from anyio.abc import TaskStatus
|
||||||
|
|
||||||
|
async def start_server(port: int, *, task_status: TaskStatus[None] = TASK_STATUS_IGNORED) -> None:
|
||||||
|
listener = await anyio.create_tcp_listener(local_host="127.0.0.1", local_port=port)
|
||||||
|
task_status.started() # unblocks tg.start()
|
||||||
|
await listener.serve(handler)
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
async with create_task_group() as tg:
|
||||||
|
await tg.start(start_server, 8080) # blocks until task_status.started()
|
||||||
|
# server is guaranteed ready here
|
||||||
|
async with await anyio.connect_tcp("127.0.0.1", 8080) as client:
|
||||||
|
...
|
||||||
|
|
||||||
|
run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rule of thumb**:
|
||||||
|
- Use `start_soon` when you don't need to know when the task is ready.
|
||||||
|
- Use `start` when the task must signal readiness before you continue.
|
||||||
|
|
||||||
|
### `create_task` — retrieving return values (AnyIO 4.14+)
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def add(x: int, y: int) -> int:
|
||||||
|
return x + y
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
async with anyio.create_task_group() as tg:
|
||||||
|
handle = tg.create_task(add(2, 4))
|
||||||
|
result = await handle # == 6
|
||||||
|
print(handle.return_value) # also 6
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Signature**: `tg.create_task(coro, *, name=None, context=None) -> TaskHandle[T]`
|
||||||
|
- Returns a `TaskHandle` you can `await` for the result.
|
||||||
|
- If the task raises, awaiting raises `TaskFailed` (or `TaskCancelled`).
|
||||||
|
- This is the canonical replacement for `asyncio.gather` when you need results.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. asyncio → anyio Cheat Sheet
|
||||||
|
|
||||||
|
| asyncio | anyio | Notes |
|
||||||
|
|---------|-------|-------|
|
||||||
|
| `asyncio.gather(a, b, c)` | `tg.create_task(a); tg.create_task(b); tg.create_task(c); results = [await h for h in handles]` | No direct gather; structured concurrency requires explicit task group scope. For fire-and-forget, use `tg.start_soon`. |
|
||||||
|
| `asyncio.create_task(coro)` | `tg.start_soon(func, *args)` or `tg.create_task(coro)` | `start_soon` takes a coroutine **function** + args. `create_task` takes a coroutine **object** and returns a handle. |
|
||||||
|
| `asyncio.sleep(n)` | `anyio.sleep(n)` | Identical semantics. |
|
||||||
|
| `asyncio.wait_for(coro, timeout)` | `with anyio.fail_after(timeout): await coro` | Raises `TimeoutError`. Use `move_on_after` for silent timeout. |
|
||||||
|
| `asyncio.Event()` | `anyio.Event()` | AnyIO events are **not reusable**; create a new one instead of `.clear()`. |
|
||||||
|
| `asyncio.Lock()` | `anyio.Lock()` | Use `async with lock:`. Pass `fast_acquire=True` if performance-critical. |
|
||||||
|
| `asyncio.Semaphore(n)` | `anyio.Semaphore(n)` | Same. Pass `fast_acquire=True` if performance-critical. |
|
||||||
|
| `asyncio.Condition()` | `anyio.Condition()` | Same semantics. |
|
||||||
|
| `asyncio.run(main())` | `anyio.run(main)` | Backend-agnostic entry point. |
|
||||||
|
| `asyncio.Queue(maxsize=N)` | `anyio.create_memory_object_stream[T](max_buffer_size=N)` | Returns `(send_stream, receive_stream)`. Supports `async for` on receive end. |
|
||||||
|
| `asyncio.to_thread(fn, *args)` | `anyio.to_thread.run_sync(fn, *args)` | Supports `abandon_on_cancel=True` and custom `limiter`. |
|
||||||
|
| `asyncio.run_coroutine_threadsafe(coro, loop)` | `anyio.from_thread.run(func, *args)` | Call async code from a worker thread. |
|
||||||
|
| `loop.call_soon_threadsafe(callback)` | `anyio.from_thread.run_sync(func, *args)` | Call sync code in event loop thread from worker thread, **with return value**. |
|
||||||
|
| `asyncio.shield(coro)` | `with anyio.CancelScope(shield=True): ...` | AnyIO shielding does not orphan tasks. |
|
||||||
|
| `asyncio.timeout(delay)` | `with anyio.fail_after(delay): ...` | AnyIO uses level cancellation, not edge cancellation. |
|
||||||
|
| `asyncio.CancelledError` | `anyio.get_cancelled_exc_class()` | Use this to catch cancellation portably across backends. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Cancellation & CancelScope
|
||||||
|
|
||||||
|
AnyIO uses **level cancellation** (inspired by Trio), not asyncio's **edge cancellation**.
|
||||||
|
|
||||||
|
- **Edge cancellation** (asyncio): A `CancelledError` is injected once. If caught and not re-raised, the task keeps running.
|
||||||
|
- **Level cancellation** (anyio): As long as a task is inside an effectively cancelled scope, every yield point raises a new cancellation exception.
|
||||||
|
|
||||||
|
### Basic CancelScope
|
||||||
|
|
||||||
|
```python
|
||||||
|
from anyio import CancelScope, create_task_group, get_cancelled_exc_class, sleep, run
|
||||||
|
|
||||||
|
async def worker() -> None:
|
||||||
|
try:
|
||||||
|
await sleep(10)
|
||||||
|
except get_cancelled_exc_class():
|
||||||
|
print("cancelled!")
|
||||||
|
raise # ALWAYS re-raise cancellation exceptions
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
async with create_task_group() as tg:
|
||||||
|
tg.start_soon(worker)
|
||||||
|
await sleep(0.1)
|
||||||
|
tg.cancel_scope.cancel() # cancels all children
|
||||||
|
|
||||||
|
run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Shielding
|
||||||
|
|
||||||
|
Shield a block from external cancellation. Essential for cleanup.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from anyio import CancelScope, create_task_group, sleep, run
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
async with create_task_group() as tg:
|
||||||
|
with CancelScope(shield=True):
|
||||||
|
tg.start_soon(some_task)
|
||||||
|
tg.cancel_scope.cancel() # shielded block is protected
|
||||||
|
await sleep(1) # this still runs
|
||||||
|
|
||||||
|
run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Combine with timeouts for graceful shutdown**:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from anyio import CancelScope, move_on_after
|
||||||
|
|
||||||
|
async def do_something(resource) -> None:
|
||||||
|
try:
|
||||||
|
await run_async_stuff()
|
||||||
|
except BaseException:
|
||||||
|
# Allow up to 10s for cleanup, then move on
|
||||||
|
with move_on_after(10, shield=True):
|
||||||
|
await resource.aclose()
|
||||||
|
raise
|
||||||
|
```
|
||||||
|
|
||||||
|
### Structured Concurrency Guarantee
|
||||||
|
|
||||||
|
A task group contains its own `CancelScope`. If any child task raises an exception:
|
||||||
|
1. The task group's cancel scope is cancelled.
|
||||||
|
2. All other child tasks receive cancellation.
|
||||||
|
3. The task group waits for all children to finish.
|
||||||
|
4. The original exception (wrapped in `ExceptionGroup` if multiple) is re-raised.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Timeouts
|
||||||
|
|
||||||
|
Two context managers. Both create a `CancelScope` internally.
|
||||||
|
|
||||||
|
### `fail_after` — raises on timeout
|
||||||
|
|
||||||
|
```python
|
||||||
|
from anyio import fail_after, sleep, run
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
try:
|
||||||
|
with fail_after(5) as scope:
|
||||||
|
await sleep(10)
|
||||||
|
except TimeoutError:
|
||||||
|
print("timed out")
|
||||||
|
print(scope.cancelled_caught) # True
|
||||||
|
|
||||||
|
run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
### `move_on_after` — silent timeout
|
||||||
|
|
||||||
|
```python
|
||||||
|
from anyio import move_on_after, sleep, run
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
with move_on_after(5) as scope:
|
||||||
|
await sleep(10)
|
||||||
|
print("this never prints")
|
||||||
|
|
||||||
|
print("exited scope, cancelled =", scope.cancelled_caught)
|
||||||
|
|
||||||
|
run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Combined with shielding
|
||||||
|
|
||||||
|
```python
|
||||||
|
from anyio import move_on_after
|
||||||
|
|
||||||
|
# Give cleanup 10 seconds, but don't let outer cancellation interrupt it
|
||||||
|
with move_on_after(10, shield=True):
|
||||||
|
await resource.aclose()
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Memory Object Streams (Queue Replacement)
|
||||||
|
|
||||||
|
Replaces `asyncio.Queue` with a safer, typed, structured-concurrency-friendly construct.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from anyio import create_task_group, create_memory_object_stream, run
|
||||||
|
from anyio.streams.memory import MemoryObjectReceiveStream
|
||||||
|
|
||||||
|
async def consumer(stream: MemoryObjectReceiveStream[str]) -> None:
|
||||||
|
async with stream: # closes receive end on exit
|
||||||
|
async for item in stream:
|
||||||
|
print("received", item)
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
# Type-annotated stream creation (AnyIO 4+ syntax)
|
||||||
|
send_stream, receive_stream = create_memory_object_stream[str](max_buffer_size=10)
|
||||||
|
|
||||||
|
async with create_task_group() as tg:
|
||||||
|
tg.start_soon(consumer, receive_stream)
|
||||||
|
async with send_stream:
|
||||||
|
for i in range(5):
|
||||||
|
await send_stream.send(f"item {i}")
|
||||||
|
# send_stream closed → consumer's async for loop exits naturally
|
||||||
|
|
||||||
|
run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key differences from `asyncio.Queue`**:
|
||||||
|
- **Bounded by default**: `max_buffer_size=0` means send blocks until a receiver is ready.
|
||||||
|
- **Cloneable**: Each producer/consumer can close its own clone. The stream only ends when **all** clones of one end are closed.
|
||||||
|
- **Async iterable**: `async for item in receive_stream:` works out of the box.
|
||||||
|
- **Type-safe**: Generic `create_memory_object_stream[T]()`.
|
||||||
|
- **Synchronous close**: Both `close()` and `async with` work.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Backend Selection
|
||||||
|
|
||||||
|
AnyIO is backend-agnostic. Code written against AnyIO APIs runs on both asyncio and Trio.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
print("running on", anyio.current_async_library())
|
||||||
|
await anyio.sleep(1)
|
||||||
|
|
||||||
|
# Default backend (asyncio)
|
||||||
|
anyio.run(main)
|
||||||
|
|
||||||
|
# Explicit backend
|
||||||
|
anyio.run(main, backend="trio")
|
||||||
|
anyio.run(main, backend="asyncio", backend_options={"debug": True})
|
||||||
|
```
|
||||||
|
|
||||||
|
**Library design rule**: Never hardcode a backend. Let the application choose via `anyio.run()`. Libraries should only import `anyio` and avoid backend-specific APIs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Compatibility with asyncio-only libraries
|
||||||
|
|
||||||
|
### Using asyncio libraries under the asyncio backend
|
||||||
|
|
||||||
|
If a third-party library exposes only an asyncio interface (returns asyncio coroutine objects), it works directly under the asyncio backend because AnyIO runs on top of asyncio's event loop:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
import some_asyncio_only_lib # returns asyncio.Future/coroutine objects
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
# This works because under the asyncio backend, await passes through
|
||||||
|
result = await some_asyncio_only_lib.fetch_data()
|
||||||
|
|
||||||
|
anyio.run(main, backend="asyncio")
|
||||||
|
```
|
||||||
|
|
||||||
|
**Important**: This only works on the `asyncio` backend. On the `trio` backend, asyncio-native objects will not work.
|
||||||
|
|
||||||
|
### When you MUST use asyncio APIs
|
||||||
|
|
||||||
|
Some APIs have no AnyIO equivalent and require direct event loop access:
|
||||||
|
|
||||||
|
| Scenario | asyncio API | AnyIO approach |
|
||||||
|
|----------|-------------|----------------|
|
||||||
|
| Signal handlers | `loop.add_signal_handler()` | `anyio.open_signal_receiver()` |
|
||||||
|
| Custom protocols | `asyncio.Protocol` | Use AnyIO streams / sockets |
|
||||||
|
| Direct Future manipulation | `asyncio.Future` | Avoid; use AnyIO primitives |
|
||||||
|
| Eager task factories | `asyncio.eager_task_factory` | Experimental in AnyIO; avoid |
|
||||||
|
|
||||||
|
If you absolutely need the running loop:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
# ... do something loop-specific ...
|
||||||
|
# WARNING: this breaks backend-agnosticism
|
||||||
|
|
||||||
|
anyio.run(main, backend="asyncio")
|
||||||
|
```
|
||||||
|
|
||||||
|
**Best practice**: Wrap asyncio-only code in a backend-agnostic facade, and document that the feature requires the asyncio backend.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Idiomatic Code Snippets
|
||||||
|
|
||||||
|
### Snippet 1: Parallel HTTP requests with timeout and cleanup
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
async def fetch(url: str) -> bytes:
|
||||||
|
await anyio.sleep(0.5) # simulate
|
||||||
|
return b"data"
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
urls = ["a", "b", "c"]
|
||||||
|
async with anyio.create_task_group() as tg:
|
||||||
|
with anyio.move_on_after(5):
|
||||||
|
for url in urls:
|
||||||
|
tg.start_soon(fetch, url)
|
||||||
|
# All tasks are cancelled on timeout; task group waits for cleanup
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Snippet 2: Producer-consumer with memory object stream
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
from anyio.streams.memory import MemoryObjectReceiveStream
|
||||||
|
|
||||||
|
async def producer(send_stream: anyio.streams.memory.MemoryObjectSendStream[int]) -> None:
|
||||||
|
async with send_stream:
|
||||||
|
for i in range(100):
|
||||||
|
await send_stream.send(i)
|
||||||
|
|
||||||
|
async def consumer(receive_stream: MemoryObjectReceiveStream[int]) -> None:
|
||||||
|
async with receive_stream:
|
||||||
|
async for item in receive_stream:
|
||||||
|
print(f"consumed {item}")
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
send, receive = anyio.create_memory_object_stream[int](max_buffer_size=5)
|
||||||
|
async with anyio.create_task_group() as tg:
|
||||||
|
tg.start_soon(producer, send)
|
||||||
|
tg.start_soon(consumer, receive)
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Snippet 3: Calling sync code from async
|
||||||
|
|
||||||
|
```python
|
||||||
|
import time
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
# Run blocking function in worker thread
|
||||||
|
result = await anyio.to_thread.run_sync(time.sleep, 2)
|
||||||
|
print("done")
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Snippet 4: Calling async code from a worker thread
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
def blocking_callback() -> None:
|
||||||
|
# Inside a worker thread, call back into the event loop
|
||||||
|
anyio.from_thread.run(anyio.sleep, 1)
|
||||||
|
anyio.from_thread.run_sync(print, "hello from thread")
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
await anyio.to_thread.run_sync(blocking_callback)
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Snippet 5: Graceful shutdown with shielded cleanup
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
async def worker() -> None:
|
||||||
|
try:
|
||||||
|
await anyio.sleep_forever()
|
||||||
|
except anyio.get_cancelled_exc_class():
|
||||||
|
with anyio.CancelScope(shield=True):
|
||||||
|
await anyio.sleep(0.5) # cleanup
|
||||||
|
print("cleaned up")
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
async with anyio.create_task_group() as tg:
|
||||||
|
tg.start_soon(worker)
|
||||||
|
await anyio.sleep(1)
|
||||||
|
tg.cancel_scope.cancel()
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- AnyIO Documentation (stable): https://anyio.readthedocs.io/en/stable/
|
||||||
|
- AnyIO GitHub (HEAD `cb245dba`): https://github.com/agronholm/anyio
|
||||||
|
- Task Groups: https://anyio.readthedocs.io/en/stable/tasks.html
|
||||||
|
- Cancellation & Timeouts: https://anyio.readthedocs.io/en/stable/cancellation.html
|
||||||
|
- Streams: https://anyio.readthedocs.io/en/stable/streams.html
|
||||||
|
- Synchronization: https://anyio.readthedocs.io/en/stable/synchronization.html
|
||||||
|
- Threads: https://anyio.readthedocs.io/en/stable/threads.html
|
||||||
|
- Basics / Backends: https://anyio.readthedocs.io/en/stable/basics.html
|
||||||
|
- Design Rationale (why asyncio is problematic): https://anyio.readthedocs.io/en/stable/why.html
|
||||||
@@ -0,0 +1,233 @@
|
|||||||
|
# Data Modeling
|
||||||
|
|
||||||
|
Which container to use, how to structure data, and why frozen is the default.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision flowchart
|
||||||
|
|
||||||
|
```
|
||||||
|
Is it a fixed set of named constants?
|
||||||
|
YES → StrEnum / IntEnum
|
||||||
|
NO ↓
|
||||||
|
Is it just branding a primitive (int, str, float)?
|
||||||
|
YES → NewType("X", base)
|
||||||
|
NO ↓
|
||||||
|
Is it an interface / contract ("this thing can do X")?
|
||||||
|
├─ Shape only, no shared code → Protocol
|
||||||
|
└─ Shared method implementation needed → ABC
|
||||||
|
NO ↓
|
||||||
|
Does the data cross a trust boundary (user input, API, file, external DB)?
|
||||||
|
YES → pydantic.BaseModel (frozen=True) — validates + serializes
|
||||||
|
NO ↓
|
||||||
|
Is it a dict shape needed for JSON compat / **kwargs typing?
|
||||||
|
YES → TypedDict
|
||||||
|
NO ↓
|
||||||
|
Is it structured data with named fields?
|
||||||
|
YES → @dataclass(frozen=True, slots=True)
|
||||||
|
NO ↓
|
||||||
|
Is it a tuple with positional semantics (x, y coords / DB row)?
|
||||||
|
YES → NamedTuple
|
||||||
|
NO → you probably don't need a new type
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Container reference
|
||||||
|
|
||||||
|
### @dataclass — internal value object
|
||||||
|
|
||||||
|
The default for structured data inside your codebase. Zero overhead, no framework coupling.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import NewType
|
||||||
|
|
||||||
|
UserId = NewType("UserId", int)
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class User:
|
||||||
|
id: UserId
|
||||||
|
name: str
|
||||||
|
email: str
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Point:
|
||||||
|
x: float
|
||||||
|
y: float
|
||||||
|
```
|
||||||
|
|
||||||
|
Always `frozen=True, slots=True`. Mutable only when mutation is the documented purpose — opt out with `# noqa: MUTABLE_OK`.
|
||||||
|
|
||||||
|
### Pydantic BaseModel — trust boundary guardian
|
||||||
|
|
||||||
|
Use when data enters or leaves your system. Validates at construction, serializes to JSON, generates OpenAPI schema.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic import BaseModel, ConfigDict, EmailStr
|
||||||
|
|
||||||
|
class CreateUserRequest(BaseModel):
|
||||||
|
model_config = ConfigDict(frozen=True)
|
||||||
|
|
||||||
|
name: str
|
||||||
|
email: EmailStr
|
||||||
|
age: int
|
||||||
|
|
||||||
|
class UserResponse(BaseModel):
|
||||||
|
model_config = ConfigDict(frozen=True)
|
||||||
|
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
email: str
|
||||||
|
```
|
||||||
|
|
||||||
|
**The one rule**: data crosses a trust boundary → Pydantic. Everything else → dataclass.
|
||||||
|
Never use Pydantic for internal-only data just because it's convenient. The validation cost is real.
|
||||||
|
|
||||||
|
### TypedDict — dict that knows its shape
|
||||||
|
|
||||||
|
Use when the value must stay a `dict` at runtime — JSON blobs, `**kwargs`, third-party APIs expecting dicts.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import TypedDict, NotRequired
|
||||||
|
|
||||||
|
class Headers(TypedDict):
|
||||||
|
content_type: str
|
||||||
|
authorization: NotRequired[str]
|
||||||
|
|
||||||
|
def make_request(url: str, headers: Headers) -> None: ...
|
||||||
|
|
||||||
|
make_request("https://api.example.com", {"content_type": "application/json"})
|
||||||
|
```
|
||||||
|
|
||||||
|
### Protocol — structural interface
|
||||||
|
|
||||||
|
"Anything that has method X" — no inheritance required.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
class Renderable(Protocol):
|
||||||
|
def render(self) -> str: ...
|
||||||
|
|
||||||
|
class Saveable(Protocol):
|
||||||
|
async def save(self) -> None: ...
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class MarkdownDoc:
|
||||||
|
content: str
|
||||||
|
def render(self) -> str:
|
||||||
|
return self.content
|
||||||
|
|
||||||
|
def publish(doc: Renderable) -> None:
|
||||||
|
print(doc.render()) # MarkdownDoc works — no inheritance needed
|
||||||
|
```
|
||||||
|
|
||||||
|
Default to Protocol for interfaces. ABC only when you need shared method implementations.
|
||||||
|
|
||||||
|
### ABC — interface with shared code
|
||||||
|
|
||||||
|
Only when Protocol isn't enough.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
|
||||||
|
class BaseRepository(ABC):
|
||||||
|
@abstractmethod
|
||||||
|
async def get(self, id: int) -> Model | None: ...
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
async def save(self, model: Model) -> None: ...
|
||||||
|
|
||||||
|
async def get_or_raise(self, id: int) -> Model:
|
||||||
|
result = await self.get(id)
|
||||||
|
if result is None:
|
||||||
|
msg = f"{type(self).__name__}: id {id} not found"
|
||||||
|
raise LookupError(msg)
|
||||||
|
return result
|
||||||
|
```
|
||||||
|
|
||||||
|
### NamedTuple — positional + named (rare)
|
||||||
|
|
||||||
|
Only when you need tuple protocol (unpacking, indexing).
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import NamedTuple
|
||||||
|
|
||||||
|
class Coordinate(NamedTuple):
|
||||||
|
x: float
|
||||||
|
y: float
|
||||||
|
|
||||||
|
x, y = Coordinate(1.0, 2.0) # tuple unpacking
|
||||||
|
```
|
||||||
|
|
||||||
|
99% of the time, `@dataclass(frozen=True, slots=True)` is better.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick lookup
|
||||||
|
|
||||||
|
| Situation | Use | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| User input, API request/response | `Pydantic BaseModel` | Validation, JSON schema, serialization |
|
||||||
|
| DB row ↔ Python (ORM) | SQLAlchemy `Mapped[]` model | ORM integration, async session |
|
||||||
|
| Internal value object | `@dataclass(frozen=True, slots=True)` | Zero overhead, no validation needed |
|
||||||
|
| Multiple outcomes from function | Union of frozen dataclasses | Distinct types for `match` |
|
||||||
|
| Dict shape for JSON / `**kwargs` | `TypedDict` | Stays a dict at runtime |
|
||||||
|
| Fixed constants | `StrEnum` / `IntEnum` | Exhaustive match, no typos |
|
||||||
|
| Distinct primitive | `NewType("X", int)` | Zero runtime cost, type-level only |
|
||||||
|
| Contract / capability | `Protocol` | Structural typing, no inheritance |
|
||||||
|
| Contract + shared impl | `ABC` | When Protocol isn't enough |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Comparison matrix
|
||||||
|
|
||||||
|
| Feature | dataclass | Pydantic | TypedDict | Protocol | NamedTuple | NewType | Enum |
|
||||||
|
|---|---|---|---|---|---|---|---|
|
||||||
|
| Validation | - | ✓ | - | - | - | - | - |
|
||||||
|
| JSON serialization | manual | built-in | native dict | - | - | - | `.value` |
|
||||||
|
| Immutable | frozen=True | frozen=True | - (dict) | N/A | always | N/A | always |
|
||||||
|
| Runtime cost | ~zero | validation | zero | zero | ~zero | zero | ~zero |
|
||||||
|
| `match` support | ✓ | ✓ | - | - | ✓ | - | ✓ |
|
||||||
|
| `slots` support | ✓ | - | - | - | - | - | - |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Parse, don't validate
|
||||||
|
|
||||||
|
Validate at the boundary. Inside the boundary, types are proof of validity.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BAD — validate then pass raw data
|
||||||
|
def process_email(email: str) -> None:
|
||||||
|
if "@" not in email:
|
||||||
|
raise ValueError("invalid email")
|
||||||
|
# still a raw str everywhere downstream
|
||||||
|
|
||||||
|
# GOOD — parse into typed value at boundary
|
||||||
|
from typing import NewType
|
||||||
|
|
||||||
|
Email = NewType("Email", str)
|
||||||
|
|
||||||
|
def parse_email(raw: str) -> Email:
|
||||||
|
if "@" not in raw or "." not in raw.split("@")[1]:
|
||||||
|
msg = f"invalid email: {raw}"
|
||||||
|
raise ValueError(msg)
|
||||||
|
return Email(raw.lower().strip())
|
||||||
|
|
||||||
|
# Downstream only sees Email, never raw str
|
||||||
|
def send_welcome(email: Email) -> None: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
With Pydantic this happens automatically — `EmailStr` is already a parsed type. Once constructed, `.email` is always valid. No re-validation needed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- Python docs: [dataclasses](https://docs.python.org/3/library/dataclasses.html)
|
||||||
|
- Pydantic v2: [docs.pydantic.dev](https://docs.pydantic.dev/latest/)
|
||||||
|
- Python docs: [typing — Protocol](https://docs.python.org/3/library/typing.html#typing.Protocol)
|
||||||
|
- Python docs: [typing — TypedDict](https://docs.python.org/3/library/typing.html#typing.TypedDict)
|
||||||
|
- Alexis King: [Parse, don't validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/)
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
# Data Processing — Polars + DuckDB
|
||||||
|
|
||||||
|
## The rule
|
||||||
|
|
||||||
|
NEVER pandas. Polars (with numpy) plus DuckDB. Pandas is 10-50x slower, has weaker types, and the modern Python data ecosystem has moved on.
|
||||||
|
|
||||||
|
## Quick decision tree
|
||||||
|
|
||||||
|
| Operation | Use | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| `.csv` / `.parquet` / `.json` direct query | DuckDB | Zero memory load, SQL ergonomics |
|
||||||
|
| `.duckdb` file | DuckDB | Native format |
|
||||||
|
| Filter (any size) | Polars | 128x faster than DuckDB for filtering |
|
||||||
|
| Sort | Polars | 12x faster |
|
||||||
|
| Multi-table join | DuckDB | 3x faster, more join types |
|
||||||
|
| Heavy GROUP BY aggregation | DuckDB | 4x faster on large datasets |
|
||||||
|
| Window function | Polars | 3-5x faster |
|
||||||
|
| Pivot / melt / string ops | Polars | 2x faster |
|
||||||
|
| Larger than RAM | Polars streaming or DuckDB out-of-core | Both handle OOM |
|
||||||
|
| Mixed pipeline | Hybrid (zero-copy via Arrow) | Use each tool's strengths |
|
||||||
|
|
||||||
|
For the deep version (per-operation benchmarks, OOM strategies, full execution templates), load the **`data-scientist`** skill - it lives in this same skill set and is the source of truth for performance numbers.
|
||||||
|
|
||||||
|
## Standard imports
|
||||||
|
|
||||||
|
```python
|
||||||
|
import numpy as np
|
||||||
|
import polars as pl
|
||||||
|
import duckdb
|
||||||
|
```
|
||||||
|
|
||||||
|
## DuckDB direct file query (zero memory load)
|
||||||
|
|
||||||
|
```python
|
||||||
|
result = duckdb.sql("""
|
||||||
|
SELECT category, SUM(amount) AS total
|
||||||
|
FROM 'data.csv'
|
||||||
|
WHERE date >= '2026-01-01'
|
||||||
|
GROUP BY category
|
||||||
|
ORDER BY total DESC
|
||||||
|
""").pl() # zero-copy → Polars DataFrame
|
||||||
|
```
|
||||||
|
|
||||||
|
`.pl()` returns Polars; `.df()` would return pandas - never use `.df()`.
|
||||||
|
|
||||||
|
## Polars lazy pipeline
|
||||||
|
|
||||||
|
```python
|
||||||
|
result = (
|
||||||
|
pl.scan_csv("data.csv") # lazy, no read yet
|
||||||
|
.filter(pl.col("amount") > 1000)
|
||||||
|
.filter(pl.col("status") == "active")
|
||||||
|
.sort("amount", descending=True)
|
||||||
|
.head(100)
|
||||||
|
.collect() # execute optimised plan
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
`scan_*` over `read_*` for files; `lazy()` then `collect()` for in-memory frames. Polars optimises the entire plan before execution (predicate pushdown, projection pushdown, common subexpression elimination).
|
||||||
|
|
||||||
|
## Streaming for OOM data
|
||||||
|
|
||||||
|
```python
|
||||||
|
result = (
|
||||||
|
pl.scan_csv("huge.csv")
|
||||||
|
.filter(pl.col("active"))
|
||||||
|
.group_by("category")
|
||||||
|
.agg([
|
||||||
|
pl.len().alias("count"),
|
||||||
|
pl.sum("amount").alias("total"),
|
||||||
|
])
|
||||||
|
.collect(streaming=True)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Hybrid pipeline (most realistic shape)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Phase 1: DuckDB for the join (3x faster)
|
||||||
|
joined = duckdb.sql("""
|
||||||
|
SELECT o.*, c.region, p.category
|
||||||
|
FROM 'orders.parquet' o
|
||||||
|
JOIN 'customers.parquet' c ON o.customer_id = c.id
|
||||||
|
JOIN 'products.parquet' p ON o.product_id = p.id
|
||||||
|
""").pl()
|
||||||
|
|
||||||
|
# Phase 2: Polars for filtering and transformation (128x + 2x faster)
|
||||||
|
processed = (
|
||||||
|
joined
|
||||||
|
.filter(pl.col("amount") > 100)
|
||||||
|
.with_columns([
|
||||||
|
(pl.col("amount") * 1.1).alias("amount_with_tax"),
|
||||||
|
])
|
||||||
|
)
|
||||||
|
|
||||||
|
# Phase 3: DuckDB for final aggregation (4x faster) - register Polars frame by name
|
||||||
|
duckdb.register("processed", processed)
|
||||||
|
final = duckdb.sql("""
|
||||||
|
SELECT region, category, SUM(amount_with_tax) AS revenue
|
||||||
|
FROM processed
|
||||||
|
GROUP BY region, category
|
||||||
|
ORDER BY revenue DESC
|
||||||
|
""").pl()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Type safety with Polars
|
||||||
|
|
||||||
|
Polars supports schema overrides at read time, and `.cast()` for explicit conversion. Avoid implicit coercion in hot paths.
|
||||||
|
|
||||||
|
```python
|
||||||
|
schema = {"id": pl.Int64, "amount": pl.Float64, "date": pl.Date}
|
||||||
|
df = pl.read_csv("data.csv", schema_overrides=schema)
|
||||||
|
```
|
||||||
|
|
||||||
|
basedpyright understands `polars-stubs`, which ship with polars itself. No extra type stubs to install.
|
||||||
|
|
||||||
|
## Things you might miss from pandas (and how to do them in Polars)
|
||||||
|
|
||||||
|
| pandas | polars |
|
||||||
|
|---|---|
|
||||||
|
| `df.iloc[5]` | `df.row(5)` (named tuple) or `df[5]` (single-row frame) |
|
||||||
|
| `df.loc[df["x"] > 5]` | `df.filter(pl.col("x") > 5)` |
|
||||||
|
| `df["x"].apply(fn)` | `df["x"].map_elements(fn)` (slow path) or use native expressions |
|
||||||
|
| `df.merge(...)` | `df.join(other, on="key")` |
|
||||||
|
| `df.groupby(...).agg(...)` | `df.group_by(...).agg(...)` |
|
||||||
|
| `pd.read_csv(...).dtypes` | `pl.read_csv(...).schema` |
|
||||||
|
| `df.to_dict("records")` | `df.to_dicts()` |
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- Polars docs: <https://docs.pola.rs>
|
||||||
|
- DuckDB Python API: <https://duckdb.org/docs/api/python/overview>
|
||||||
|
- Cross-reference - this skill set's `data-scientist` skill (load it for the deep version)
|
||||||
@@ -0,0 +1,218 @@
|
|||||||
|
# Error Handling
|
||||||
|
|
||||||
|
Typed errors, exhaustive matching, union returns, and resource safety.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Typed errors — no bare strings
|
||||||
|
|
||||||
|
Error types carry structured data. Pattern matching works. Callers know exactly what can go wrong.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import NewType
|
||||||
|
|
||||||
|
UserId = NewType("UserId", int)
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class UserNotFoundError(Exception):
|
||||||
|
user_id: UserId
|
||||||
|
|
||||||
|
def __str__(self) -> str: # REQUIRED — see note below
|
||||||
|
return f"user {self.user_id} not found"
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class PermissionDeniedError(Exception):
|
||||||
|
user_id: UserId
|
||||||
|
required_role: str
|
||||||
|
|
||||||
|
def __str__(self) -> str:
|
||||||
|
return f"user {self.user_id} needs role {self.required_role}"
|
||||||
|
```
|
||||||
|
|
||||||
|
**`__str__` is mandatory** on dataclass exceptions. `@dataclass` replaces `Exception.__init__`, so `self.args` is always `()`. Without `__str__`, `str(e)` returns an empty string and logging/monitoring breaks.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BAD
|
||||||
|
raise ValueError("user not found")
|
||||||
|
raise ValueError("permission denied")
|
||||||
|
|
||||||
|
# GOOD
|
||||||
|
raise UserNotFoundError(user_id=uid)
|
||||||
|
raise PermissionDeniedError(user_id=uid, required_role="admin")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Union returns — expected failures without exceptions
|
||||||
|
|
||||||
|
For failures that are **expected** (not found, validation error, permission denied), return a union instead of raising. Exceptions are for **unexpected** failures (network down, OOM, corrupted data).
|
||||||
|
|
||||||
|
### Define the outcome types
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class User:
|
||||||
|
id: UserId
|
||||||
|
name: str
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class UserNotFound:
|
||||||
|
id: UserId
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class PermissionDenied:
|
||||||
|
id: UserId
|
||||||
|
reason: str
|
||||||
|
|
||||||
|
type GetUserResult = User | UserNotFound | PermissionDenied
|
||||||
|
```
|
||||||
|
|
||||||
|
### Handle exhaustively
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import assert_never
|
||||||
|
|
||||||
|
def handle_result(result: GetUserResult) -> str:
|
||||||
|
match result:
|
||||||
|
case User(name=name):
|
||||||
|
return f"Found: {name}"
|
||||||
|
case UserNotFound(id=uid):
|
||||||
|
return f"No user with id {uid}"
|
||||||
|
case PermissionDenied(reason=reason):
|
||||||
|
return f"Denied: {reason}"
|
||||||
|
case _ as unreachable:
|
||||||
|
assert_never(unreachable)
|
||||||
|
```
|
||||||
|
|
||||||
|
`assert_never` in the default case: if you add a new variant to `GetUserResult` without handling it here, the type checker errors. No silent fall-through.
|
||||||
|
|
||||||
|
### When to use which
|
||||||
|
|
||||||
|
**The heuristic**: caller is 1-2 levels away and MUST handle it → union return. Error should propagate up many layers to a boundary → exception.
|
||||||
|
|
||||||
|
| Scenario | Pattern | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| Repository → service (caller handles it) | Union return (`User \| UserNotFound`) | Caller is right there, must handle both |
|
||||||
|
| Validation at boundary (parsing input) | Exception (typed, with fields) | Propagates up to HTTP/CLI handler |
|
||||||
|
| Infrastructure failure (network, OOM) | Exception | Can't handle locally, must propagate |
|
||||||
|
| Service → service (deep internal) | Exception (typed) | Union boilerplate across many layers is worse than exceptions |
|
||||||
|
| HTTP handler → response | Catch exceptions, convert to response | Boundary code catches and translates |
|
||||||
|
|
||||||
|
**Practical tradeoff**: union returns are safest (type checker forces handling) but create boilerplate when every caller in a chain must `match`. If the error would just propagate through 3+ layers unchanged, use a typed exception instead.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exhaustive match — every match needs a default
|
||||||
|
|
||||||
|
Every `match` statement ends with `case _: assert_never(x)`. No exceptions.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from enum import StrEnum
|
||||||
|
from typing import assert_never
|
||||||
|
|
||||||
|
class Status(StrEnum):
|
||||||
|
PENDING = "pending"
|
||||||
|
ACTIVE = "active"
|
||||||
|
DELETED = "deleted"
|
||||||
|
|
||||||
|
def describe(status: Status) -> str:
|
||||||
|
match status:
|
||||||
|
case Status.PENDING:
|
||||||
|
return "waiting"
|
||||||
|
case Status.ACTIVE:
|
||||||
|
return "live"
|
||||||
|
case Status.DELETED:
|
||||||
|
return "gone"
|
||||||
|
case _ as unreachable:
|
||||||
|
assert_never(unreachable)
|
||||||
|
```
|
||||||
|
|
||||||
|
Add a new enum member? The type checker tells you every `match` that needs updating.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context managers — resource safety
|
||||||
|
|
||||||
|
If it has `.close()`, `.shutdown()`, `.disconnect()`, or `.release()`, wrap it in `with`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BAD
|
||||||
|
f = open("data.txt")
|
||||||
|
data = f.read()
|
||||||
|
f.close() # forgotten? leaked
|
||||||
|
|
||||||
|
# GOOD
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
data = Path("data.txt").read_text()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Async resources
|
||||||
|
|
||||||
|
```python
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
async def fetch_users() -> list[User]:
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
response = await client.get("https://api.example.com/users")
|
||||||
|
response.raise_for_status()
|
||||||
|
return [User(**u) for u in response.json()]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom context manager
|
||||||
|
|
||||||
|
```python
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def managed_connection(url: str) -> AsyncIterator[Connection]:
|
||||||
|
conn = await connect(url)
|
||||||
|
try:
|
||||||
|
yield conn
|
||||||
|
finally:
|
||||||
|
await conn.close()
|
||||||
|
|
||||||
|
async with managed_connection("postgres://...") as conn:
|
||||||
|
await conn.execute("SELECT 1")
|
||||||
|
# conn is closed here, guaranteed
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exception hierarchy — when you do raise
|
||||||
|
|
||||||
|
Keep exception hierarchies shallow and specific.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class AppError(Exception):
|
||||||
|
"""Base for all application errors."""
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class NotFoundError(AppError):
|
||||||
|
entity: str
|
||||||
|
id: int
|
||||||
|
|
||||||
|
def __str__(self) -> str:
|
||||||
|
return f"{self.entity} {self.id} not found"
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ConflictError(AppError):
|
||||||
|
entity: str
|
||||||
|
field: str
|
||||||
|
value: str
|
||||||
|
|
||||||
|
def __str__(self) -> str:
|
||||||
|
return f"{self.entity}.{self.field} = {self.value!r} already exists"
|
||||||
|
```
|
||||||
|
|
||||||
|
Callers catch `AppError` at the boundary, or specific subtypes where they can do something useful.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- Python docs: [typing — assert_never](https://docs.python.org/3/library/typing.html#typing.assert_never)
|
||||||
|
- Python docs: [contextlib](https://docs.python.org/3/library/contextlib.html)
|
||||||
|
- Python docs: [match statement](https://docs.python.org/3/reference/compound_stmts.html#the-match-statement)
|
||||||
@@ -0,0 +1,316 @@
|
|||||||
|
# FastAPI + SQLAlchemy 2.x async + Postgres + Pydantic v2
|
||||||
|
|
||||||
|
The canonical web API stack. Async end-to-end, type-safe end-to-end, OpenAPI-generated end-to-end.
|
||||||
|
|
||||||
|
## Project layout
|
||||||
|
|
||||||
|
```
|
||||||
|
myapi/
|
||||||
|
├── pyproject.toml
|
||||||
|
├── alembic.ini
|
||||||
|
├── migrations/
|
||||||
|
│ └── env.py
|
||||||
|
├── src/
|
||||||
|
│ └── myapi/
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── main.py # FastAPI app + lifespan
|
||||||
|
│ ├── config.py # pydantic-settings
|
||||||
|
│ ├── db.py # engine, session factory, dependency
|
||||||
|
│ ├── models.py # SQLAlchemy declarative models
|
||||||
|
│ ├── schemas.py # Pydantic request/response models
|
||||||
|
│ └── routers/
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ └── users.py
|
||||||
|
└── tests/
|
||||||
|
├── conftest.py
|
||||||
|
└── test_users.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv add fastapi 'sqlalchemy[asyncio]>=2.0' asyncpg 'pydantic[email]>=2' pydantic-settings 'uvicorn[standard]' orjson
|
||||||
|
uv add --dev httpx pytest alembic
|
||||||
|
```
|
||||||
|
|
||||||
|
`orjson` is mandatory: set `default_response_class=ORJSONResponse` on the FastAPI app. Pydantic-typed responses bypass it (Pydantic v2's `model_dump_json` is already Rust-backed); raw `dict` / `list` returns are accelerated. For SSE / NDJSON streams, call `orjson.dumps(...)` per chunk inside `StreamingResponse`. See `orjson-stack.md` for the decision tree, flag reference, and benchmarks.
|
||||||
|
|
||||||
|
## Configuration (`config.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from functools import lru_cache
|
||||||
|
|
||||||
|
from pydantic import Field, PostgresDsn
|
||||||
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||||
|
|
||||||
|
|
||||||
|
class Settings(BaseSettings):
|
||||||
|
model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPI_")
|
||||||
|
|
||||||
|
database_url: PostgresDsn
|
||||||
|
debug: bool = False
|
||||||
|
cors_origins: list[str] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache
|
||||||
|
def get_settings() -> Settings:
|
||||||
|
return Settings() # type: ignore[call-arg] # pydantic populates from env
|
||||||
|
```
|
||||||
|
|
||||||
|
Wait — that comment violates the no-excuse rule. Use proper field defaults instead. Real version:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class Settings(BaseSettings):
|
||||||
|
model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPI_")
|
||||||
|
database_url: PostgresDsn
|
||||||
|
debug: bool = False
|
||||||
|
cors_origins: list[str] = Field(default_factory=list)
|
||||||
|
```
|
||||||
|
|
||||||
|
Construct via `Settings(_env_file=".env")` if needed in tests; in production it reads from env.
|
||||||
|
|
||||||
|
## Database (`db.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from typing import Annotated
|
||||||
|
|
||||||
|
from fastapi import Depends
|
||||||
|
from sqlalchemy.ext.asyncio import (
|
||||||
|
AsyncEngine,
|
||||||
|
AsyncSession,
|
||||||
|
async_sessionmaker,
|
||||||
|
create_async_engine,
|
||||||
|
)
|
||||||
|
|
||||||
|
from myapi.config import get_settings
|
||||||
|
|
||||||
|
|
||||||
|
def make_engine() -> AsyncEngine:
|
||||||
|
settings = get_settings()
|
||||||
|
return create_async_engine(
|
||||||
|
str(settings.database_url),
|
||||||
|
echo=settings.debug,
|
||||||
|
pool_pre_ping=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_engine = make_engine()
|
||||||
|
_SessionFactory = async_sessionmaker(_engine, expire_on_commit=False)
|
||||||
|
|
||||||
|
|
||||||
|
async def get_session() -> AsyncIterator[AsyncSession]:
|
||||||
|
async with _SessionFactory() as session:
|
||||||
|
yield session
|
||||||
|
|
||||||
|
|
||||||
|
SessionDep = Annotated[AsyncSession, Depends(get_session)]
|
||||||
|
```
|
||||||
|
|
||||||
|
`expire_on_commit=False` is essential for FastAPI - otherwise attribute access after commit triggers an implicit refresh and errors out under async.
|
||||||
|
|
||||||
|
## Models (`models.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from datetime import datetime, UTC
|
||||||
|
from sqlalchemy import DateTime, String, func
|
||||||
|
from sqlalchemy.orm import (
|
||||||
|
DeclarativeBase,
|
||||||
|
Mapped,
|
||||||
|
MappedAsDataclass,
|
||||||
|
mapped_column,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class Base(MappedAsDataclass, DeclarativeBase):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class User(Base):
|
||||||
|
__tablename__ = "users"
|
||||||
|
|
||||||
|
id: Mapped[int] = mapped_column(primary_key=True, init=False)
|
||||||
|
email: Mapped[str] = mapped_column(String(255), unique=True, index=True)
|
||||||
|
name: Mapped[str] = mapped_column(String(100))
|
||||||
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True),
|
||||||
|
server_default=func.now(),
|
||||||
|
init=False,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
`MappedAsDataclass` makes `User(email=..., name=...)` work as a real dataclass constructor. `init=False` excludes the auto-generated columns (`id`, `created_at`) from `__init__`.
|
||||||
|
|
||||||
|
## Schemas (`schemas.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from datetime import datetime
|
||||||
|
from pydantic import BaseModel, ConfigDict, EmailStr
|
||||||
|
|
||||||
|
|
||||||
|
class UserCreate(BaseModel):
|
||||||
|
email: EmailStr
|
||||||
|
name: str
|
||||||
|
|
||||||
|
|
||||||
|
class UserRead(BaseModel):
|
||||||
|
model_config = ConfigDict(from_attributes=True) # SQLAlchemy → Pydantic
|
||||||
|
|
||||||
|
id: int
|
||||||
|
email: EmailStr
|
||||||
|
name: str
|
||||||
|
created_at: datetime
|
||||||
|
```
|
||||||
|
|
||||||
|
Always have a separate `*Create` (input) and `*Read` (output) model. Never expose your ORM model as the API model.
|
||||||
|
|
||||||
|
## Routers (`routers/users.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from fastapi import APIRouter, HTTPException, status
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from myapi.db import SessionDep
|
||||||
|
from myapi.models import User
|
||||||
|
from myapi.schemas import UserCreate, UserRead
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/users", tags=["users"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("", response_model=UserRead, status_code=status.HTTP_201_CREATED)
|
||||||
|
async def create_user(payload: UserCreate, session: SessionDep) -> User:
|
||||||
|
user = User(email=payload.email, name=payload.name)
|
||||||
|
session.add(user)
|
||||||
|
await session.commit()
|
||||||
|
await session.refresh(user)
|
||||||
|
return user
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{user_id}", response_model=UserRead)
|
||||||
|
async def get_user(user_id: int, session: SessionDep) -> User:
|
||||||
|
result = await session.execute(select(User).where(User.id == user_id))
|
||||||
|
user = result.scalar_one_or_none()
|
||||||
|
if user is None:
|
||||||
|
raise HTTPException(status.HTTP_404_NOT_FOUND, "User not found")
|
||||||
|
return user
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("", response_model=list[UserRead])
|
||||||
|
async def list_users(session: SessionDep, limit: int = 100) -> list[User]:
|
||||||
|
result = await session.execute(select(User).limit(limit))
|
||||||
|
return list(result.scalars().all())
|
||||||
|
```
|
||||||
|
|
||||||
|
## Application (`main.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
|
||||||
|
from fastapi import FastAPI
|
||||||
|
|
||||||
|
from myapi.config import get_settings
|
||||||
|
from myapi.routers import users
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(_: FastAPI) -> AsyncIterator[None]:
|
||||||
|
# Startup: warm up engine pool, run migrations check, etc.
|
||||||
|
yield
|
||||||
|
# Shutdown: close engine
|
||||||
|
from myapi.db import _engine
|
||||||
|
await _engine.dispose()
|
||||||
|
|
||||||
|
|
||||||
|
def create_app() -> FastAPI:
|
||||||
|
settings = get_settings()
|
||||||
|
app = FastAPI(
|
||||||
|
title="My API",
|
||||||
|
debug=settings.debug,
|
||||||
|
lifespan=lifespan,
|
||||||
|
)
|
||||||
|
app.include_router(users.router)
|
||||||
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
app = create_app()
|
||||||
|
```
|
||||||
|
|
||||||
|
Run with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run uvicorn myapi.main:app --host 0.0.0.0 --port 8000 --reload
|
||||||
|
```
|
||||||
|
|
||||||
|
## Migrations (Alembic + async)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run alembic init -t async migrations
|
||||||
|
```
|
||||||
|
|
||||||
|
In `migrations/env.py` replace the `target_metadata` line:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from myapi.models import Base
|
||||||
|
target_metadata = Base.metadata
|
||||||
|
```
|
||||||
|
|
||||||
|
Set `sqlalchemy.url` in `alembic.ini` to your async URL or override via `env.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from myapi.config import get_settings
|
||||||
|
config.set_main_option("sqlalchemy.url", str(get_settings().database_url))
|
||||||
|
```
|
||||||
|
|
||||||
|
Generate and apply:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run alembic revision --autogenerate -m "create users"
|
||||||
|
uv run alembic upgrade head
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tests (`tests/test_users.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
from httpx import ASGITransport, AsyncClient
|
||||||
|
|
||||||
|
from myapi.main import app
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.anyio
|
||||||
|
async def test_create_and_get_user() -> None:
|
||||||
|
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||||
|
create_response = await client.post(
|
||||||
|
"/users",
|
||||||
|
json={"email": "alice@example.com", "name": "Alice"},
|
||||||
|
)
|
||||||
|
assert create_response.status_code == 201
|
||||||
|
user_id = create_response.json()["id"]
|
||||||
|
|
||||||
|
get_response = await client.get(f"/users/{user_id}")
|
||||||
|
assert get_response.status_code == 200
|
||||||
|
assert get_response.json()["email"] == "alice@example.com"
|
||||||
|
```
|
||||||
|
|
||||||
|
For database-backed tests, run a Postgres container in CI (`testcontainers-python` or `docker-compose`) and apply migrations against a test schema. SQLite-as-test-db breaks once you use Postgres-specific types (`JSONB`, `tsvector`, arrays).
|
||||||
|
|
||||||
|
## Common pitfalls
|
||||||
|
|
||||||
|
| Pitfall | Fix |
|
||||||
|
|---|---|
|
||||||
|
| `MissingGreenlet` exception when accessing relationships after commit | `expire_on_commit=False` on the session factory |
|
||||||
|
| Connection pool exhausted under load | Set `pool_size`, `max_overflow` in `create_async_engine` |
|
||||||
|
| Pydantic v1 syntax (`from pydantic import ...; class X(BaseModel): class Config: orm_mode = True`) | v2 uses `model_config = ConfigDict(from_attributes=True)` |
|
||||||
|
| Returning ORM objects without `response_model` | FastAPI serialises with `from_attributes=True` automatically; declare `response_model` so OpenAPI is correct |
|
||||||
|
| `await session.execute(...)` returning Sequence | Wrap with `list(result.scalars().all())` to satisfy strict types |
|
||||||
|
| `func.now()` returning naive datetime | Use `DateTime(timezone=True)` and `created_at: Mapped[datetime]` with `UTC`-aware default |
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- FastAPI: <https://fastapi.tiangolo.com>
|
||||||
|
- SQLAlchemy 2.x async: <https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html>
|
||||||
|
- SQLAlchemy MappedAsDataclass: <https://docs.sqlalchemy.org/en/20/orm/dataclasses.html>
|
||||||
|
- asyncpg: <https://magicstack.github.io/asyncpg/current/>
|
||||||
|
- Pydantic v2 migration: <https://docs.pydantic.dev/latest/migration/>
|
||||||
|
- Alembic async: <https://alembic.sqlalchemy.org/en/latest/cookbook.html#using-asyncio-with-alembic>
|
||||||
+360
@@ -0,0 +1,360 @@
|
|||||||
|
# httpx2 — Production Defaults
|
||||||
|
|
||||||
|
> **Source**: [pydantic/httpx2](https://github.com/pydantic/httpx2) — next-generation HTTP client for Python 3, continuation of HTTPX under Pydantic stewardship.
|
||||||
|
>
|
||||||
|
> **Rule**: Every network request MUST use `httpx2`. **ALL optimizations below are ON by default** — HTTP/2, brotli+zstd, tuned connection pool, fine-grained timeouts, transport retries, TCP_NODELAY. This is the baseline, not a stretch goal. A bare `httpx2.AsyncClient()` is a bug.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Installation — all extras, always
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# pyproject.toml
|
||||||
|
dependencies = [
|
||||||
|
"httpx2[http2,brotli,zstd]",
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Extra | What it enables | Why it's mandatory |
|
||||||
|
|-------|----------------|--------------------|
|
||||||
|
| `http2` | HTTP/2 multiplexing via `h2` | Single TCP connection handles concurrent requests; eliminates head-of-line blocking |
|
||||||
|
| `brotli` | Brotli content decoding (`br`) | ~20% smaller payloads than gzip for text/JSON |
|
||||||
|
| `zstd` | Zstandard content decoding | Faster decompression than brotli at similar ratios; stdlib in Python ≥ 3.14 |
|
||||||
|
| `socks` | SOCKS5 proxy support via `socksio` | Install only if you route through SOCKS proxies |
|
||||||
|
|
||||||
|
All three core extras (`http2,brotli,zstd`) are non-negotiable. Omitting any is leaving performance on the table.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. The canonical defaults — ALL ON
|
||||||
|
|
||||||
|
These are not "optimizations to consider". These are **the correct defaults** that every httpx2 client must use.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import socket
|
||||||
|
import httpx2
|
||||||
|
|
||||||
|
# ── These are the STANDARD values. Use them verbatim. ──
|
||||||
|
|
||||||
|
LIMITS = httpx2.Limits(
|
||||||
|
max_connections=200, # library default 100 is too conservative
|
||||||
|
max_keepalive_connections=40, # library default 20 wastes reconnects
|
||||||
|
keepalive_expiry=30.0, # library default 5s kills warm connections too fast
|
||||||
|
)
|
||||||
|
|
||||||
|
TIMEOUT = httpx2.Timeout(
|
||||||
|
connect=5.0, # TCP + TLS handshake budget
|
||||||
|
read=30.0, # time to receive a response chunk
|
||||||
|
write=10.0, # time to send a request chunk
|
||||||
|
pool=10.0, # time to acquire a connection from pool
|
||||||
|
)
|
||||||
|
|
||||||
|
SOCKET_OPTIONS: list[tuple[int, int, int]] = [
|
||||||
|
(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1), # disable Nagle — no 40ms delay
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Why each knob is set this way
|
||||||
|
|
||||||
|
| Setting | Library default | Our default | Why |
|
||||||
|
|---------|----------------|-------------|-----|
|
||||||
|
| `http2` | `False` | **`True`** | HTTP/2 multiplexing is strictly superior for any modern API |
|
||||||
|
| `max_connections` | `100` | `200` | Headroom for fan-out; prevents pool exhaustion under load |
|
||||||
|
| `max_keepalive_connections` | `20` | `40` | Keeps warm connections alive; fewer TLS handshakes |
|
||||||
|
| `keepalive_expiry` | `5.0s` | `30.0s` | 5s is too aggressive — kills connections between burst requests |
|
||||||
|
| `Timeout(5.0)` uniform | `5.0` all | Split | Uniform 5s is too tight for reads, too loose for connects |
|
||||||
|
| `read` timeout | `5.0` | `30.0` | Slow APIs and streaming need breathing room |
|
||||||
|
| `pool` timeout | `5.0` | `10.0` | Explicit — hitting this means `max_connections` needs raising |
|
||||||
|
| `TCP_NODELAY` | off | **on** | Eliminates Nagle's 40ms coalescing delay for small payloads |
|
||||||
|
| `retries` | `0` | `3` | Retries on `ConnectError`/`ConnectTimeout` only — safe and resilient |
|
||||||
|
| `follow_redirects` | `False` | **`True`** | Most APIs redirect; failing on 3xx is wrong default behavior |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Factory functions — the ONE correct way to create clients
|
||||||
|
|
||||||
|
Copy this into your project. This is the canonical pattern.
|
||||||
|
|
||||||
|
```python
|
||||||
|
"""httpx2 client factory. Always use create_client() / create_async_client()."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import socket
|
||||||
|
import typing
|
||||||
|
|
||||||
|
import httpx2
|
||||||
|
|
||||||
|
_LIMITS = httpx2.Limits(
|
||||||
|
max_connections=200,
|
||||||
|
max_keepalive_connections=40,
|
||||||
|
keepalive_expiry=30.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
_TIMEOUT = httpx2.Timeout(
|
||||||
|
connect=5.0,
|
||||||
|
read=30.0,
|
||||||
|
write=10.0,
|
||||||
|
pool=10.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
_SOCKET_OPTIONS: list[tuple[int, int, int]] = [
|
||||||
|
(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def create_async_client(
|
||||||
|
*,
|
||||||
|
base_url: str = "",
|
||||||
|
http2: bool = True,
|
||||||
|
retries: int = 3,
|
||||||
|
limits: httpx2.Limits = _LIMITS,
|
||||||
|
timeout: httpx2.Timeout = _TIMEOUT,
|
||||||
|
headers: dict[str, str] | None = None,
|
||||||
|
event_hooks: dict[str, list[typing.Callable[..., typing.Any]]] | None = None,
|
||||||
|
**kwargs: typing.Any,
|
||||||
|
) -> httpx2.AsyncClient:
|
||||||
|
transport = httpx2.AsyncHTTPTransport(
|
||||||
|
http2=http2,
|
||||||
|
retries=retries,
|
||||||
|
limits=limits,
|
||||||
|
socket_options=_SOCKET_OPTIONS,
|
||||||
|
)
|
||||||
|
return httpx2.AsyncClient(
|
||||||
|
transport=transport,
|
||||||
|
timeout=timeout,
|
||||||
|
base_url=base_url,
|
||||||
|
headers=headers or {},
|
||||||
|
event_hooks=event_hooks or {},
|
||||||
|
follow_redirects=True,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def create_client(
|
||||||
|
*,
|
||||||
|
base_url: str = "",
|
||||||
|
http2: bool = True,
|
||||||
|
retries: int = 3,
|
||||||
|
limits: httpx2.Limits = _LIMITS,
|
||||||
|
timeout: httpx2.Timeout = _TIMEOUT,
|
||||||
|
headers: dict[str, str] | None = None,
|
||||||
|
event_hooks: dict[str, list[typing.Callable[..., typing.Any]]] | None = None,
|
||||||
|
**kwargs: typing.Any,
|
||||||
|
) -> httpx2.Client:
|
||||||
|
transport = httpx2.HTTPTransport(
|
||||||
|
http2=http2,
|
||||||
|
retries=retries,
|
||||||
|
limits=limits,
|
||||||
|
socket_options=_SOCKET_OPTIONS,
|
||||||
|
)
|
||||||
|
return httpx2.Client(
|
||||||
|
transport=transport,
|
||||||
|
timeout=timeout,
|
||||||
|
base_url=base_url,
|
||||||
|
headers=headers or {},
|
||||||
|
event_hooks=event_hooks or {},
|
||||||
|
follow_redirects=True,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Async — the common case
|
||||||
|
async with create_async_client(base_url="https://api.example.com") as client:
|
||||||
|
r = await client.get("/users")
|
||||||
|
|
||||||
|
# Sync
|
||||||
|
with create_client() as client:
|
||||||
|
r = client.get("https://api.example.com/health")
|
||||||
|
```
|
||||||
|
|
||||||
|
**If you are NOT using this factory pattern, you are doing it wrong.** A bare `httpx2.AsyncClient()` leaves HTTP/2 off, retries off, TCP_NODELAY off, keepalive too short, and timeouts too uniform.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Special case overrides
|
||||||
|
|
||||||
|
The factory defaults cover 95% of use cases. Override only when you have a specific reason:
|
||||||
|
|
||||||
|
| Scenario | Override |
|
||||||
|
|----------|----------|
|
||||||
|
| LLM streaming endpoints | `timeout=httpx2.Timeout(connect=10.0, read=None, write=10.0, pool=10.0)` — no read timeout on streaming |
|
||||||
|
| Single-host API with low concurrency | `limits=httpx2.Limits(max_connections=50, max_keepalive_connections=20, keepalive_expiry=60.0)` |
|
||||||
|
| Ephemeral short-lived requests | `keepalive_expiry=5.0` — don't hold connections |
|
||||||
|
| Unix domain sockets | `httpx2.AsyncHTTPTransport(uds="/path/to/socket", ...)` |
|
||||||
|
| mTLS / client certs | Pass `verify=ssl_ctx` with `ctx.load_cert_chain(certfile=...)` |
|
||||||
|
| SOCKS proxy | `httpx2[socks]`, `proxy="socks5://..."` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Event hooks — always wire observability
|
||||||
|
|
||||||
|
This is not optional. Every production client should log requests.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
async def log_request(request: httpx2.Request) -> None:
|
||||||
|
request.extensions["request_start"] = time.perf_counter()
|
||||||
|
|
||||||
|
async def log_response(response: httpx2.Response) -> None:
|
||||||
|
start = response.request.extensions.get("request_start", 0)
|
||||||
|
elapsed = time.perf_counter() - start
|
||||||
|
logger.info(
|
||||||
|
"HTTP %s %s → %d (%.3fs, %s)",
|
||||||
|
response.request.method,
|
||||||
|
response.request.url,
|
||||||
|
response.status_code,
|
||||||
|
elapsed,
|
||||||
|
response.http_version,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Sync versions for Client
|
||||||
|
def log_request_sync(request: httpx2.Request) -> None:
|
||||||
|
request.extensions["request_start"] = time.perf_counter()
|
||||||
|
|
||||||
|
def log_response_sync(response: httpx2.Response) -> None:
|
||||||
|
start = response.request.extensions.get("request_start", 0)
|
||||||
|
elapsed = time.perf_counter() - start
|
||||||
|
logger.info(
|
||||||
|
"HTTP %s %s → %d (%.3fs, %s)",
|
||||||
|
response.request.method,
|
||||||
|
response.request.url,
|
||||||
|
response.status_code,
|
||||||
|
elapsed,
|
||||||
|
response.http_version,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
For auto `raise_for_status()`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def raise_on_error(response: httpx2.Response) -> None:
|
||||||
|
response.raise_for_status()
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Verification script — confirm your setup is fully optimized
|
||||||
|
|
||||||
|
Run this against your target endpoint to **verify** (not decide) that all optimizations are active:
|
||||||
|
|
||||||
|
```python
|
||||||
|
"""Verify httpx2 is fully optimized against a target endpoint."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import socket
|
||||||
|
import time
|
||||||
|
|
||||||
|
import anyio
|
||||||
|
import httpx2
|
||||||
|
|
||||||
|
|
||||||
|
TARGET_URL = "https://api.example.com/health"
|
||||||
|
ITERATIONS = 30
|
||||||
|
|
||||||
|
|
||||||
|
async def bench(label: str, client: httpx2.AsyncClient, url: str, n: int) -> float:
|
||||||
|
for _ in range(3): # warmup
|
||||||
|
await client.get(url)
|
||||||
|
start = time.perf_counter()
|
||||||
|
for _ in range(n):
|
||||||
|
r = await client.get(url)
|
||||||
|
assert r.status_code == 200
|
||||||
|
elapsed = time.perf_counter() - start
|
||||||
|
avg_ms = (elapsed / n) * 1000
|
||||||
|
print(f" {label}: {avg_ms:.1f}ms avg ({n} reqs in {elapsed:.2f}s)")
|
||||||
|
return avg_ms
|
||||||
|
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
results: dict[str, float] = {}
|
||||||
|
|
||||||
|
# BAD: bare defaults (this is what we're proving is worse)
|
||||||
|
async with httpx2.AsyncClient() as c:
|
||||||
|
results["BAD-bare-defaults"] = await bench("BAD-bare-defaults", c, TARGET_URL, ITERATIONS)
|
||||||
|
|
||||||
|
# GOOD: full production defaults (this is what we always use)
|
||||||
|
limits = httpx2.Limits(max_connections=200, max_keepalive_connections=40, keepalive_expiry=30.0)
|
||||||
|
timeout = httpx2.Timeout(connect=5.0, read=30.0, write=10.0, pool=10.0)
|
||||||
|
transport = httpx2.AsyncHTTPTransport(
|
||||||
|
http2=True, retries=3, limits=limits,
|
||||||
|
socket_options=[(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)],
|
||||||
|
)
|
||||||
|
async with httpx2.AsyncClient(transport=transport, timeout=timeout, follow_redirects=True) as c:
|
||||||
|
results["GOOD-full-production"] = await bench("GOOD-full-production", c, TARGET_URL, ITERATIONS)
|
||||||
|
|
||||||
|
print("\n--- Proof ---")
|
||||||
|
baseline = results["BAD-bare-defaults"]
|
||||||
|
for label, avg in results.items():
|
||||||
|
delta = ((avg - baseline) / baseline) * 100
|
||||||
|
print(f" {label}: {avg:.1f}ms ({delta:+.1f}% vs bare)")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Quick reference — all knobs
|
||||||
|
|
||||||
|
### `httpx2.AsyncClient` / `httpx2.Client`
|
||||||
|
|
||||||
|
| Parameter | Type | Library Default | **Our Default** |
|
||||||
|
|-----------|------|-----------------|-----------------|
|
||||||
|
| `http1` | `bool` | `True` | `True` |
|
||||||
|
| `http2` | `bool` | `False` | **`True`** |
|
||||||
|
| `verify` | `ssl.SSLContext \| str \| bool` | `True` | `True` |
|
||||||
|
| `cert` | `CertTypes \| None` | `None` | `None` |
|
||||||
|
| `proxy` | `str \| Proxy \| None` | `None` | `None` |
|
||||||
|
| `mounts` | `dict[str, Transport]` | `None` | `None` |
|
||||||
|
| `timeout` | `Timeout \| float \| None` | `Timeout(5.0)` | **Split: 5/30/10/10** |
|
||||||
|
| `limits` | `Limits` | `Limits(100, 20, 5.0)` | **`Limits(200, 40, 30.0)`** |
|
||||||
|
| `follow_redirects` | `bool` | `False` | **`True`** |
|
||||||
|
| `max_redirects` | `int` | `20` | `20` |
|
||||||
|
| `event_hooks` | `dict` | `{}` | **Wire logging** |
|
||||||
|
| `base_url` | `str` | `""` | Set for single-API clients |
|
||||||
|
| `trust_env` | `bool` | `True` | `True` |
|
||||||
|
| `default_encoding` | `str \| Callable` | `"utf-8"` | `"utf-8"` |
|
||||||
|
|
||||||
|
### `httpx2.AsyncHTTPTransport` / `httpx2.HTTPTransport`
|
||||||
|
|
||||||
|
| Parameter | Type | Library Default | **Our Default** |
|
||||||
|
|-----------|------|-----------------|-----------------|
|
||||||
|
| `http1` | `bool` | `True` | `True` |
|
||||||
|
| `http2` | `bool` | `False` | **`True`** |
|
||||||
|
| `retries` | `int` | `0` | **`3`** |
|
||||||
|
| `limits` | `Limits` | `Limits(100, 20, 5.0)` | **`Limits(200, 40, 30.0)`** |
|
||||||
|
| `uds` | `str \| None` | `None` | `None` |
|
||||||
|
| `local_address` | `str \| None` | `None` | `None` |
|
||||||
|
| `socket_options` | `Iterable[SOCKET_OPTION]` | `None` | **`[TCP_NODELAY]`** |
|
||||||
|
| `proxy` | `str \| Proxy \| None` | `None` | `None` |
|
||||||
|
|
||||||
|
### `httpx2.Timeout`
|
||||||
|
|
||||||
|
| Parameter | Library Default | **Our Default** |
|
||||||
|
|-----------|-----------------|-----------------|
|
||||||
|
| `connect` | `5.0` | `5.0` |
|
||||||
|
| `read` | `5.0` | **`30.0`** |
|
||||||
|
| `write` | `5.0` | **`10.0`** |
|
||||||
|
| `pool` | `5.0` | **`10.0`** |
|
||||||
|
|
||||||
|
### `httpx2.Limits`
|
||||||
|
|
||||||
|
| Parameter | Library Default | **Our Default** |
|
||||||
|
|-----------|-----------------|-----------------|
|
||||||
|
| `max_connections` | `100` | **`200`** |
|
||||||
|
| `max_keepalive_connections` | `20` | **`40`** |
|
||||||
|
| `keepalive_expiry` | `5.0` | **`30.0`** |
|
||||||
|
|
||||||
|
### Async backend (httpcore2)
|
||||||
|
|
||||||
|
httpcore2 uses `anyio` by default (works with both asyncio and trio). No extra config needed if you're already on the anyio stack. For trio, install `httpcore2[trio]`.
|
||||||
@@ -0,0 +1,307 @@
|
|||||||
|
# Library Defaults — Decision Tree
|
||||||
|
|
||||||
|
For each domain, the canonical 2026 choice, why, and the canonical usage snippet. The skill enforces these unless the project's `pyproject.toml` explicitly says otherwise.
|
||||||
|
|
||||||
|
## CLI — typer
|
||||||
|
|
||||||
|
`typer` builds a CLI from type-annotated function signatures. argparse needs 5x the code; click ignores type annotations; fire is magic that breaks at scale.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import typer
|
||||||
|
from rich import print as rprint
|
||||||
|
|
||||||
|
app = typer.Typer()
|
||||||
|
|
||||||
|
@app.command()
|
||||||
|
def greet(name: str, count: int = 1, shout: bool = False) -> None:
|
||||||
|
"""Print a greeting `count` times."""
|
||||||
|
message = f"Hello, {name}!" if not shout else f"HELLO, {name.upper()}!"
|
||||||
|
for _ in range(count):
|
||||||
|
rprint(message)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
app()
|
||||||
|
```
|
||||||
|
|
||||||
|
For a single-function script, `typer.run(main)` skips the `Typer()` boilerplate. Subcommands use `@app.command()`.
|
||||||
|
|
||||||
|
## Terminal output — rich
|
||||||
|
|
||||||
|
`rich` produces tables, progress bars, syntax highlighting, traceback rendering. Use it for any structured output. Plain `print` is acceptable for non-interactive log lines (and even those are usually better via `rich.console.Console(stderr=True).log(...)`).
|
||||||
|
|
||||||
|
```python
|
||||||
|
from rich.console import Console
|
||||||
|
from rich.table import Table
|
||||||
|
|
||||||
|
console = Console()
|
||||||
|
|
||||||
|
table = Table(title="Users")
|
||||||
|
table.add_column("ID", style="cyan")
|
||||||
|
table.add_column("Name", style="magenta")
|
||||||
|
table.add_row("1", "Alice")
|
||||||
|
console.print(table)
|
||||||
|
|
||||||
|
# Rich tracebacks (call once at process start)
|
||||||
|
from rich.traceback import install
|
||||||
|
install(show_locals=True)
|
||||||
|
```
|
||||||
|
|
||||||
|
## HTTP client — [httpx2](https://github.com/pydantic/httpx2)
|
||||||
|
|
||||||
|
Next-generation HTTP client under Pydantic stewardship. Sync and async in one library, HTTP/2 native, brotli + zstd content decoding, real type stubs. Replaces `requests` (sync only), `aiohttp` (async only), and the original `httpx`.
|
||||||
|
|
||||||
|
**Install**: `httpx2[http2,brotli,zstd]` — always include all three extras, no exceptions.
|
||||||
|
|
||||||
|
**A bare `httpx2.AsyncClient()` / `httpx2.Client()` is a bug.** Always use the factory pattern from `references/httpx2-optimization.md` with ALL optimizations enabled by default:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import socket
|
||||||
|
import httpx2
|
||||||
|
|
||||||
|
# ── Production defaults — ALL ON, always. ──
|
||||||
|
_LIMITS = httpx2.Limits(max_connections=200, max_keepalive_connections=40, keepalive_expiry=30.0)
|
||||||
|
_TIMEOUT = httpx2.Timeout(connect=5.0, read=30.0, write=10.0, pool=10.0)
|
||||||
|
_SOCKET_OPTS: list[tuple[int, int, int]] = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)]
|
||||||
|
|
||||||
|
# Async (the common case)
|
||||||
|
transport = httpx2.AsyncHTTPTransport(http2=True, retries=3, limits=_LIMITS, socket_options=_SOCKET_OPTS)
|
||||||
|
async with httpx2.AsyncClient(transport=transport, timeout=_TIMEOUT, follow_redirects=True) as client:
|
||||||
|
response = await client.get("https://api.example.com/users")
|
||||||
|
response.raise_for_status()
|
||||||
|
users = response.json()
|
||||||
|
|
||||||
|
# Sync
|
||||||
|
transport = httpx2.HTTPTransport(http2=True, retries=3, limits=_LIMITS, socket_options=_SOCKET_OPTS)
|
||||||
|
with httpx2.Client(transport=transport, timeout=_TIMEOUT, follow_redirects=True) as client:
|
||||||
|
response = client.get("https://api.example.com/users")
|
||||||
|
response.raise_for_status()
|
||||||
|
users = response.json()
|
||||||
|
```
|
||||||
|
|
||||||
|
See `references/httpx2-optimization.md` for the full factory functions (`create_client()` / `create_async_client()`), event hooks, and the rationale behind every setting. **Load that reference whenever you write ANY network code.**
|
||||||
|
|
||||||
|
## JSON — stdlib `json` (default) or `orjson` (hot paths)
|
||||||
|
|
||||||
|
Stdlib `json` is fine for cold paths and configs. **Reach for `orjson` when JSON is in the hot path** — cache layers, queue payloads, streaming responses, structured logs, FastAPI endpoints returning raw `dict` / `list`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import orjson
|
||||||
|
|
||||||
|
# orjson.dumps returns bytes, not str
|
||||||
|
raw: bytes = orjson.dumps(
|
||||||
|
payload,
|
||||||
|
option=orjson.OPT_NAIVE_UTC | orjson.OPT_UTC_Z | orjson.OPT_SERIALIZE_DATACLASS,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Critical 2026 fact**: with Pydantic v2, `model.model_dump_json()` is backed by pydantic-core (Rust) and is faster than `orjson + default=` bridge for Pydantic-shaped responses. **Use `model_dump_json()` for Pydantic; orjson for everything else.**
|
||||||
|
|
||||||
|
For FastAPI: `app = FastAPI(default_response_class=ORJSONResponse)`. Pydantic-typed responses bypass it (and that's correct — Pydantic's path is faster). Raw `dict`/`list` returns go through orjson.
|
||||||
|
|
||||||
|
See `references/orjson-stack.md` for the full decision tree, option flag reference, FastAPI integration, Redis/queue/logging patterns, and the `model_dump_json()` vs orjson benchmark.
|
||||||
|
|
||||||
|
## Validation — pydantic v2
|
||||||
|
|
||||||
|
Pydantic v2's core is in Rust (~10x faster than v1). It is the de-facto boundary validator. Use it for:
|
||||||
|
|
||||||
|
- HTTP request/response models (FastAPI uses pydantic natively)
|
||||||
|
- Config files (env vars via `pydantic-settings`)
|
||||||
|
- Anything entering the program from outside
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic import BaseModel, Field, EmailStr, field_validator
|
||||||
|
|
||||||
|
class User(BaseModel):
|
||||||
|
id: int = Field(ge=1)
|
||||||
|
email: EmailStr
|
||||||
|
name: str = Field(min_length=1, max_length=100)
|
||||||
|
age: int | None = Field(default=None, ge=0, le=150)
|
||||||
|
|
||||||
|
@field_validator("name")
|
||||||
|
@classmethod
|
||||||
|
def name_no_digits(cls, v: str) -> str:
|
||||||
|
if any(c.isdigit() for c in v):
|
||||||
|
raise ValueError("name cannot contain digits")
|
||||||
|
return v
|
||||||
|
|
||||||
|
# Inside the program, use the validated instance with confidence
|
||||||
|
user = User.model_validate({"id": 1, "email": "a@b.com", "name": "Alice"})
|
||||||
|
print(user.model_dump_json(indent=2))
|
||||||
|
```
|
||||||
|
|
||||||
|
`@dataclass` is fine for purely internal records (no validation needed). For anything crossing a process boundary, use Pydantic.
|
||||||
|
|
||||||
|
## Async — anyio
|
||||||
|
|
||||||
|
Full reference: [async-anyio.md](async-anyio.md). The summary:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
async def fetch(url: str) -> str:
|
||||||
|
await anyio.sleep(0.1)
|
||||||
|
return url
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
async with anyio.create_task_group() as tg:
|
||||||
|
for url in ["a", "b", "c"]:
|
||||||
|
tg.start_soon(fetch, url)
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
Never `import asyncio` directly. The third-party libraries you call are free to use asyncio internally.
|
||||||
|
|
||||||
|
## Web framework — fastapi
|
||||||
|
|
||||||
|
Type-hint-driven HTTP framework. Pydantic models become OpenAPI schemas automatically.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
app = FastAPI()
|
||||||
|
|
||||||
|
class CreateUser(BaseModel):
|
||||||
|
name: str
|
||||||
|
email: str
|
||||||
|
|
||||||
|
class User(BaseModel):
|
||||||
|
id: int
|
||||||
|
name: str
|
||||||
|
email: str
|
||||||
|
|
||||||
|
@app.post("/users", response_model=User)
|
||||||
|
async def create_user(payload: CreateUser) -> User:
|
||||||
|
return User(id=1, **payload.model_dump())
|
||||||
|
```
|
||||||
|
|
||||||
|
Full stack with database: [fastapi-stack.md](fastapi-stack.md).
|
||||||
|
|
||||||
|
## ORM — sqlalchemy 2.x async
|
||||||
|
|
||||||
|
SQLAlchemy 2.x finally has a real async API. Use the modern declarative `MappedAsDataclass` style with type annotations.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sqlalchemy import String
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker
|
||||||
|
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, MappedAsDataclass
|
||||||
|
|
||||||
|
class Base(MappedAsDataclass, DeclarativeBase):
|
||||||
|
pass
|
||||||
|
|
||||||
|
class User(Base):
|
||||||
|
__tablename__ = "users"
|
||||||
|
id: Mapped[int] = mapped_column(primary_key=True, init=False)
|
||||||
|
name: Mapped[str] = mapped_column(String(100))
|
||||||
|
email: Mapped[str] = mapped_column(String(255), unique=True)
|
||||||
|
|
||||||
|
engine = create_async_engine("postgresql+asyncpg://localhost/myapp")
|
||||||
|
SessionFactory = async_sessionmaker(engine, expire_on_commit=False)
|
||||||
|
```
|
||||||
|
|
||||||
|
Full pattern with FastAPI integration: [fastapi-stack.md](fastapi-stack.md).
|
||||||
|
|
||||||
|
## Database — postgres + asyncpg
|
||||||
|
|
||||||
|
For new applications, default to Postgres. SQLite for tests is fine; SQLite for production is not.
|
||||||
|
|
||||||
|
asyncpg is the fastest Python Postgres driver, native to SQLAlchemy 2.x async, native to FastAPI's lifespan model. URL: `postgresql+asyncpg://user:pass@host:5432/db`.
|
||||||
|
|
||||||
|
For migrations, use Alembic with `[alembic.context]` configured to use the async engine. Single-step:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv add alembic
|
||||||
|
uv run alembic init -t async migrations
|
||||||
|
```
|
||||||
|
|
||||||
|
## TUI — textual
|
||||||
|
|
||||||
|
Textual builds rich, mouse-aware, mobile-style TUIs on the rich rendering engine. See [textual-tui.md](textual-tui.md).
|
||||||
|
|
||||||
|
## AI agents — pydantic-ai
|
||||||
|
|
||||||
|
The agent framework from the Pydantic team. Type-strict, structured outputs are first-class, model-agnostic. See [pydantic-ai.md](pydantic-ai.md).
|
||||||
|
|
||||||
|
## DataFrames — polars + numpy
|
||||||
|
|
||||||
|
Polars is 10-50x faster than pandas, has a real type system, and supports lazy evaluation. Numpy stays in the toolbox for arrays. See [data-processing.md](data-processing.md).
|
||||||
|
|
||||||
|
## OLAP / SQL — duckdb
|
||||||
|
|
||||||
|
DuckDB is the SQL engine for analytical workloads. Query CSV/Parquet/JSON files directly without loading into memory; perform joins and aggregations 3-4x faster than Polars; zero-copy interchange with Polars via Arrow. See [data-processing.md](data-processing.md).
|
||||||
|
|
||||||
|
## Tests — pytest
|
||||||
|
|
||||||
|
Plain `unittest` is fine for stdlib; everything else uses pytest. Conventions:
|
||||||
|
|
||||||
|
- File names `test_*.py`, function names `test_*`.
|
||||||
|
- Fixtures via `@pytest.fixture`. Async fixtures are anyio-aware (`@pytest.fixture` on an async function works under `pytest-anyio` which is bundled with anyio).
|
||||||
|
- Parametrise with `@pytest.mark.parametrize`.
|
||||||
|
- Mark async tests with `@pytest.mark.anyio` (provided by anyio's pytest plugin).
|
||||||
|
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
import anyio
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_user() -> dict[str, str]:
|
||||||
|
return {"name": "Alice", "email": "a@b.com"}
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("count,expected", [(1, "Hello"), (2, "Hello, Hello")])
|
||||||
|
def test_greet(count: int, expected: str) -> None:
|
||||||
|
result = ", ".join(["Hello"] * count)
|
||||||
|
assert result == expected
|
||||||
|
|
||||||
|
@pytest.mark.anyio
|
||||||
|
async def test_async_fetch() -> None:
|
||||||
|
await anyio.sleep(0)
|
||||||
|
assert True
|
||||||
|
```
|
||||||
|
|
||||||
|
`pyproject.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
minversion = "8.0"
|
||||||
|
testpaths = ["tests"]
|
||||||
|
addopts = ["-ra", "--strict-config", "--strict-markers"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Settings / config — pydantic-settings
|
||||||
|
|
||||||
|
Loads env vars and `.env` files into a Pydantic model. Replaces ad-hoc `os.environ.get(...)` everywhere.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic import Field
|
||||||
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||||
|
|
||||||
|
class Settings(BaseSettings):
|
||||||
|
model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPP_")
|
||||||
|
|
||||||
|
database_url: str
|
||||||
|
api_key: str = Field(min_length=1)
|
||||||
|
debug: bool = False
|
||||||
|
|
||||||
|
settings = Settings() # loads at import time; raises if any required var is missing
|
||||||
|
```
|
||||||
|
|
||||||
|
## Logging — stdlib logging + rich handler
|
||||||
|
|
||||||
|
Stdlib `logging` is fine; it gets a face-lift from `rich.logging.RichHandler`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import logging
|
||||||
|
from rich.logging import RichHandler
|
||||||
|
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(message)s",
|
||||||
|
datefmt="[%X]",
|
||||||
|
handlers=[RichHandler(rich_tracebacks=True, show_path=False)],
|
||||||
|
)
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
log.info("ready")
|
||||||
|
```
|
||||||
|
|
||||||
|
For structured logging in production, swap to `structlog` (separate dep). Don't roll your own.
|
||||||
@@ -0,0 +1,268 @@
|
|||||||
|
# One-liner Scripts (PEP 723 + uv)
|
||||||
|
|
||||||
|
Self-contained Python scripts with declared dependencies, run with no environment setup. The combination eliminates the historical reason to write small tools in Go or Bash.
|
||||||
|
|
||||||
|
**Rule: EVERY `.py` script — even throwaway — MUST use PEP 723 inline metadata with the usage comment block.** No venv, no requirements.txt, no setup.py. The script IS the environment spec.
|
||||||
|
|
||||||
|
## The two patterns
|
||||||
|
|
||||||
|
### Pattern 1: inline `uv run` invocation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run --with httpx2 --with rich python -c "
|
||||||
|
import httpx2
|
||||||
|
from rich import print
|
||||||
|
print(httpx2.get('https://api.github.com').json())
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
Use for terminal one-shots that you don't want to save. `--with PKG` may be repeated.
|
||||||
|
|
||||||
|
### Pattern 2: PEP 723 script with shebang (THE CANONICAL PATTERN)
|
||||||
|
|
||||||
|
A regular `.py` file with metadata in a comment block. uv reads the metadata, materialises a disposable venv (cached), and runs the script.
|
||||||
|
|
||||||
|
```python
|
||||||
|
#!/usr/bin/env -S uv run --script
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.13"
|
||||||
|
# dependencies = [
|
||||||
|
# "httpx2[http2,brotli,zstd]",
|
||||||
|
# "rich",
|
||||||
|
# ]
|
||||||
|
# ///
|
||||||
|
|
||||||
|
# ─── How to run ───
|
||||||
|
# 1. Install uv (if not installed):
|
||||||
|
# curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||||
|
# 2. Run directly (no venv, no pip install needed):
|
||||||
|
# uv run my_script.py
|
||||||
|
# 3. Or make executable and run:
|
||||||
|
# chmod +x my_script.py && ./my_script.py
|
||||||
|
# ──────────────────
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import httpx2
|
||||||
|
from rich import print as rprint
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
with httpx2.Client(http2=True, follow_redirects=True) as client:
|
||||||
|
resp = client.get("https://api.github.com")
|
||||||
|
resp.raise_for_status()
|
||||||
|
rprint(resp.json())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Mandatory elements
|
||||||
|
|
||||||
|
Every PEP 723 script MUST include these, in order:
|
||||||
|
|
||||||
|
1. **Shebang**: `#!/usr/bin/env -S uv run --script`
|
||||||
|
2. **PEP 723 metadata block**: `# /// script` ... `# ///` with `requires-python` and `dependencies`
|
||||||
|
3. **Usage comment block**: How to install uv + how to run the script. Copy the template above verbatim.
|
||||||
|
4. **`from __future__ import annotations`**: Always first import.
|
||||||
|
5. **`if __name__ == "__main__": main()`**: Entry point guard.
|
||||||
|
|
||||||
|
### The usage comment block (NON-NEGOTIABLE)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# ─── How to run ───
|
||||||
|
# 1. Install uv (if not installed):
|
||||||
|
# curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||||
|
# 2. Run directly (no venv, no pip install needed):
|
||||||
|
# uv run <SCRIPT_NAME>.py [ARGS]
|
||||||
|
# 3. Or make executable and run:
|
||||||
|
# chmod +x <SCRIPT_NAME>.py && ./<SCRIPT_NAME>.py
|
||||||
|
# ──────────────────
|
||||||
|
```
|
||||||
|
|
||||||
|
Replace `<SCRIPT_NAME>` with the actual filename. Add argument descriptions if the script takes CLI args. This block goes immediately after the `# ///` closing line, before any imports.
|
||||||
|
|
||||||
|
**Why mandatory**: Anyone who receives this script — colleague, CI, future you — must know how to run it without reading docs. The comment IS the docs.
|
||||||
|
|
||||||
|
## Template generator
|
||||||
|
|
||||||
|
Use `scripts/new-script.py` to scaffold a new PEP 723 script with all boilerplate pre-filled:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate to temp directory (default)
|
||||||
|
uv run scripts/new-script.py my_tool
|
||||||
|
|
||||||
|
# Generate to specific path
|
||||||
|
uv run scripts/new-script.py my_tool --output ./scripts/my_tool.py
|
||||||
|
|
||||||
|
# With extra dependencies
|
||||||
|
uv run scripts/new-script.py my_tool --deps "polars" "duckdb" "rich"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common dependency sets
|
||||||
|
|
||||||
|
| Use case | Dependencies line |
|
||||||
|
|---|---|
|
||||||
|
| API client | `"httpx2[http2,brotli,zstd]"` |
|
||||||
|
| Data processing | `"polars"`, `"duckdb"` |
|
||||||
|
| CLI tool | `"typer"`, `"rich"` |
|
||||||
|
| Web scraping | `"httpx2[http2,brotli,zstd]"`, `"selectolax"` |
|
||||||
|
| File watcher | `"watchfiles"` |
|
||||||
|
| JSON pretty | `"rich"` |
|
||||||
|
| AI / LLM | `"pydantic-ai"`, `"httpx2[http2,brotli,zstd]"` |
|
||||||
|
|
||||||
|
## Real-world examples
|
||||||
|
|
||||||
|
### Fetch + print JSON
|
||||||
|
|
||||||
|
```python
|
||||||
|
#!/usr/bin/env -S uv run --script
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.13"
|
||||||
|
# dependencies = [
|
||||||
|
# "httpx2[http2,brotli,zstd]",
|
||||||
|
# "rich",
|
||||||
|
# ]
|
||||||
|
# ///
|
||||||
|
|
||||||
|
# ─── How to run ───
|
||||||
|
# 1. Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||||
|
# 2. Run: uv run fetch_json.py https://api.github.com/repos/pydantic/httpx2
|
||||||
|
# ──────────────────
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import httpx2
|
||||||
|
from rich import print as rprint
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
url = sys.argv[1] if len(sys.argv) > 1 else "https://api.github.com"
|
||||||
|
with httpx2.Client(http2=True, follow_redirects=True) as client:
|
||||||
|
resp = client.get(url)
|
||||||
|
resp.raise_for_status()
|
||||||
|
rprint(resp.json())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
```
|
||||||
|
|
||||||
|
### CSV → Parquet conversion
|
||||||
|
|
||||||
|
```python
|
||||||
|
#!/usr/bin/env -S uv run --script
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.13"
|
||||||
|
# dependencies = [
|
||||||
|
# "polars",
|
||||||
|
# "typer",
|
||||||
|
# "rich",
|
||||||
|
# ]
|
||||||
|
# ///
|
||||||
|
|
||||||
|
# ─── How to run ───
|
||||||
|
# 1. Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||||
|
# 2. Run: uv run csv2parquet.py input.csv output.parquet
|
||||||
|
# ──────────────────
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import polars as pl
|
||||||
|
import typer
|
||||||
|
from rich import print as rprint
|
||||||
|
|
||||||
|
|
||||||
|
def main(input_path: Path, output_path: Path | None = None) -> None:
|
||||||
|
"""Convert CSV to Parquet."""
|
||||||
|
out = output_path or input_path.with_suffix(".parquet")
|
||||||
|
df = pl.read_csv(input_path)
|
||||||
|
df.write_parquet(out)
|
||||||
|
rprint(f"[green]✓[/green] {input_path} → {out} ({len(df)} rows)")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
typer.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Quick benchmark
|
||||||
|
|
||||||
|
```python
|
||||||
|
#!/usr/bin/env -S uv run --script
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.13"
|
||||||
|
# dependencies = [
|
||||||
|
# "httpx2[http2,brotli,zstd]",
|
||||||
|
# "rich",
|
||||||
|
# "anyio",
|
||||||
|
# ]
|
||||||
|
# ///
|
||||||
|
|
||||||
|
# ─── How to run ───
|
||||||
|
# 1. Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||||
|
# 2. Run: uv run bench.py https://api.example.com/health 50
|
||||||
|
# ──────────────────
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
import anyio
|
||||||
|
import httpx2
|
||||||
|
from rich import print as rprint
|
||||||
|
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
url = sys.argv[1] if len(sys.argv) > 1 else "https://api.github.com"
|
||||||
|
n = int(sys.argv[2]) if len(sys.argv) > 2 else 20
|
||||||
|
|
||||||
|
limits = httpx2.Limits(max_connections=200, max_keepalive_connections=40, keepalive_expiry=30.0)
|
||||||
|
timeout = httpx2.Timeout(connect=5.0, read=30.0, write=10.0, pool=10.0)
|
||||||
|
transport = httpx2.AsyncHTTPTransport(
|
||||||
|
http2=True, retries=3, limits=limits,
|
||||||
|
socket_options=[(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)],
|
||||||
|
)
|
||||||
|
|
||||||
|
async with httpx2.AsyncClient(transport=transport, timeout=timeout, follow_redirects=True) as client:
|
||||||
|
# warmup
|
||||||
|
for _ in range(3):
|
||||||
|
await client.get(url)
|
||||||
|
|
||||||
|
start = time.perf_counter()
|
||||||
|
for _ in range(n):
|
||||||
|
r = await client.get(url)
|
||||||
|
assert r.status_code == 200
|
||||||
|
elapsed = time.perf_counter() - start
|
||||||
|
|
||||||
|
avg_ms = (elapsed / n) * 1000
|
||||||
|
rprint(f"[bold]{url}[/bold]: {avg_ms:.1f}ms avg over {n} requests ({elapsed:.2f}s total, {r.http_version})")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Anti-patterns
|
||||||
|
|
||||||
|
| ❌ Don't | ✅ Do |
|
||||||
|
|---|---|
|
||||||
|
| `pip install httpx2 && python script.py` | `uv run script.py` |
|
||||||
|
| `requirements.txt` alongside script | PEP 723 inline metadata |
|
||||||
|
| `python -m venv .venv && ...` | `uv run --script` handles it |
|
||||||
|
| Script without usage comment | Always include the "How to run" block |
|
||||||
|
| `import asyncio; asyncio.run(main())` | `import anyio; anyio.run(main)` |
|
||||||
|
| Bare `httpx2.AsyncClient()` | Full production defaults (see `references/httpx2-optimization.md`) |
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- PEP 723 - Inline script metadata: <https://peps.python.org/pep-0723/>
|
||||||
|
- uv `run --script` docs: <https://docs.astral.sh/uv/guides/scripts/>
|
||||||
|
- Original article: <https://www.cottongeeks.com/articles/2025-06-24-fun-with-uv-and-pep-723>
|
||||||
|
- Simon Willison on one-shot Python tools: <https://simonwillison.net/2024/Dec/19/one-shot-python-tools/>
|
||||||
@@ -0,0 +1,378 @@
|
|||||||
|
# orjson — When to Use, How to Integrate
|
||||||
|
|
||||||
|
`orjson` is the fastest JSON library on PyPI — written in Rust, 6–11× faster than stdlib `json` on serialization, 1.5–4× faster on deserialization. It also supports types the stdlib refuses to serialize: `datetime`, `date`, `UUID`, `numpy` arrays, `dataclass`, Pydantic models (via a small bridge).
|
||||||
|
|
||||||
|
This document covers the production patterns. **Not every project needs orjson.** The decision tree is in §1.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Decision tree — should you adopt orjson?
|
||||||
|
|
||||||
|
```
|
||||||
|
Are you serializing/deserializing JSON in a hot path?
|
||||||
|
├─ NO → stdlib `json` is fine. Stop here.
|
||||||
|
└─ YES ↓
|
||||||
|
|
||||||
|
Is the project FastAPI?
|
||||||
|
├─ YES ↓
|
||||||
|
│
|
||||||
|
│ Is your response body fully described by a Pydantic v2 model?
|
||||||
|
│ ├─ YES → Use FastAPI's default JSON response (uses Pydantic's
|
||||||
|
│ │ Rust-backed serializer; orjson saves nothing in this path).
|
||||||
|
│ │ Adopt orjson only for *non-Pydantic* responses below.
|
||||||
|
│ └─ NO → Use `ORJSONResponse` for endpoints that return dicts,
|
||||||
|
│ lists, or arbitrary structures.
|
||||||
|
│
|
||||||
|
└─ NOT FastAPI ↓
|
||||||
|
|
||||||
|
Are you serializing Pydantic v2 models repeatedly?
|
||||||
|
├─ YES → Use `model.model_dump_json()` directly — backed by pydantic-core
|
||||||
|
│ (Rust), within ~10% of orjson on the same payload, and respects
|
||||||
|
│ every Pydantic feature (computed fields, aliases, validators).
|
||||||
|
└─ NO ↓
|
||||||
|
|
||||||
|
Are you serializing dicts / lists / dataclasses / datetime / UUID?
|
||||||
|
├─ YES → orjson is the right answer.
|
||||||
|
└─ NO → stdlib `json`.
|
||||||
|
```
|
||||||
|
|
||||||
|
**The crucial 2026 fact**: with Pydantic v2's `model_dump_json()`, **Pydantic-shaped responses no longer need orjson**. Adopt orjson where you are still going through `dict` / `list` / `dataclass`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Install
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# pyproject.toml
|
||||||
|
dependencies = [
|
||||||
|
"orjson>=3.10",
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
orjson wheels are published for every major CPython version and platform (macOS, Linux glibc/musl, Windows, ARM64). No compilation step on install.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Basic usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
import orjson
|
||||||
|
|
||||||
|
# Serialization — returns bytes, not str
|
||||||
|
raw: bytes = orjson.dumps({"hello": "world", "ts": datetime.now(UTC)})
|
||||||
|
|
||||||
|
# Deserialization
|
||||||
|
data = orjson.loads(raw)
|
||||||
|
```
|
||||||
|
|
||||||
|
Two things to internalize:
|
||||||
|
|
||||||
|
1. **`orjson.dumps` returns `bytes`**, not `str`. Stdlib `json.dumps` returns `str`. This is by design — most JSON destinations (sockets, files in binary mode, HTTP bodies) want bytes anyway, and skipping the encode/decode round trip is part of the speedup.
|
||||||
|
2. **No `indent` arg.** orjson supports `OPT_INDENT_2` (and only 2-space indent) via flags. If you need other indentation, use stdlib `json`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. The option flags you actually use
|
||||||
|
|
||||||
|
```python
|
||||||
|
import orjson
|
||||||
|
|
||||||
|
orjson.dumps(
|
||||||
|
payload,
|
||||||
|
option=(
|
||||||
|
orjson.OPT_NAIVE_UTC # treat naive datetimes as UTC (recommended)
|
||||||
|
| orjson.OPT_UTC_Z # render UTC as "...Z" instead of "+00:00"
|
||||||
|
| orjson.OPT_SERIALIZE_NUMPY # serialize numpy arrays natively
|
||||||
|
| orjson.OPT_SERIALIZE_DATACLASS # serialize @dataclass instances
|
||||||
|
| orjson.OPT_NON_STR_KEYS # allow int / UUID / datetime dict keys
|
||||||
|
# | orjson.OPT_SORT_KEYS # only when you need deterministic output
|
||||||
|
# | orjson.OPT_INDENT_2 # only for human-readable output (slower)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Each flag is opt-in for a reason — orjson defaults to spec-strict JSON.
|
||||||
|
|
||||||
|
The flag combination above is a sensible "production default" for application code. The `OPT_NAIVE_UTC | OPT_UTC_Z` pair is especially important: it produces RFC 3339 timestamps that every parser on earth accepts.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. orjson + FastAPI
|
||||||
|
|
||||||
|
### 5.1 The legacy pattern: `ORJSONResponse`
|
||||||
|
|
||||||
|
```python
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.responses import ORJSONResponse
|
||||||
|
|
||||||
|
app = FastAPI(default_response_class=ORJSONResponse)
|
||||||
|
|
||||||
|
@app.get("/items")
|
||||||
|
async def get_items() -> dict[str, list[dict[str, int]]]:
|
||||||
|
return {"items": [{"id": i, "qty": i * 2} for i in range(1000)]}
|
||||||
|
```
|
||||||
|
|
||||||
|
`default_response_class=ORJSONResponse` swaps the global JSON encoder for orjson. **This affects only the response body serialization**, not request parsing — for request parsing, FastAPI still uses Pydantic.
|
||||||
|
|
||||||
|
### 5.2 The 2026 reality — Pydantic v2 vs orjson
|
||||||
|
|
||||||
|
With FastAPI 0.100+ on Pydantic v2:
|
||||||
|
|
||||||
|
- If your response is annotated with a Pydantic model, FastAPI calls `model_dump_json()` directly. **orjson is bypassed** even with `default_response_class=ORJSONResponse`, because the Pydantic serializer is already Rust-backed.
|
||||||
|
- If your response is a raw `dict` / `list` / Python object, `ORJSONResponse` does kick in and saves real time.
|
||||||
|
|
||||||
|
The benchmark in `tiangolo/fastapi#11728` (Apr 2024) showed `model_dump_json()` is ~10–15% faster than `ORJSONResponse + model_dump()` for Pydantic-shaped responses. The shape of the data matters; on mixed-shape APIs, keep `ORJSONResponse` as the default and trust Pydantic's path for typed responses.
|
||||||
|
|
||||||
|
### 5.3 Recommended setup
|
||||||
|
|
||||||
|
```python
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.responses import ORJSONResponse
|
||||||
|
|
||||||
|
app = FastAPI(
|
||||||
|
default_response_class=ORJSONResponse, # benefits dict/list returns
|
||||||
|
# Pydantic-typed returns automatically use pydantic-core serialization
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Do NOT** wrap Pydantic models manually:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BAD — defeats Pydantic's optimized path
|
||||||
|
@app.get("/users/{id}", response_class=ORJSONResponse)
|
||||||
|
async def get_user(id: int) -> ORJSONResponse:
|
||||||
|
user = await fetch_user(id)
|
||||||
|
return ORJSONResponse(content=user.model_dump()) # extra dict trip
|
||||||
|
|
||||||
|
# GOOD — let FastAPI serialize the model
|
||||||
|
@app.get("/users/{id}")
|
||||||
|
async def get_user(id: int) -> User:
|
||||||
|
return await fetch_user(id)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.4 Streaming responses
|
||||||
|
|
||||||
|
`ORJSONResponse` does not stream — it buffers the whole response. For SSE, NDJSON, or chunked JSON, use `StreamingResponse` and call `orjson.dumps` per chunk:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from fastapi.responses import StreamingResponse
|
||||||
|
import orjson
|
||||||
|
|
||||||
|
async def ndjson_stream():
|
||||||
|
async for row in fetch_rows():
|
||||||
|
yield orjson.dumps(row) + b"\n"
|
||||||
|
|
||||||
|
@app.get("/export")
|
||||||
|
async def export():
|
||||||
|
return StreamingResponse(ndjson_stream(), media_type="application/x-ndjson")
|
||||||
|
```
|
||||||
|
|
||||||
|
This is where orjson shines — per-chunk serialization in a tight loop, zero buffering.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. orjson + Pydantic v2 (no FastAPI)
|
||||||
|
|
||||||
|
When you have a Pydantic model and want orjson's output for non-FastAPI contexts:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic import BaseModel
|
||||||
|
import orjson
|
||||||
|
|
||||||
|
class User(BaseModel):
|
||||||
|
id: int
|
||||||
|
email: str
|
||||||
|
created: datetime
|
||||||
|
|
||||||
|
user = User(id=1, email="a@b.com", created=datetime.now(UTC))
|
||||||
|
|
||||||
|
# Option A — Pydantic's built-in Rust serializer (USE THIS by default)
|
||||||
|
raw: bytes = user.model_dump_json().encode()
|
||||||
|
# 2026: ~1.2× faster than orjson on the same payload, supports
|
||||||
|
# every Pydantic feature (aliases, computed fields, json_schema_extra, etc.)
|
||||||
|
|
||||||
|
# Option B — orjson bridge for cases Pydantic does not cover
|
||||||
|
raw: bytes = orjson.dumps(
|
||||||
|
user,
|
||||||
|
default=lambda obj: obj.model_dump() if isinstance(obj, BaseModel) else None,
|
||||||
|
)
|
||||||
|
# Useful when serializing nested non-Pydantic structures that contain
|
||||||
|
# BaseModels — e.g. a list of dicts that each may contain a BaseModel.
|
||||||
|
```
|
||||||
|
|
||||||
|
For routine "serialize one Pydantic model to JSON", `model_dump_json()` wins on speed AND feature parity. Reach for orjson only at the *container* level (a dict of mixed types).
|
||||||
|
|
||||||
|
### Custom `default=` callback — the universal extension point
|
||||||
|
|
||||||
|
```python
|
||||||
|
import orjson
|
||||||
|
from decimal import Decimal
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
def _default(obj):
|
||||||
|
if isinstance(obj, BaseModel):
|
||||||
|
return obj.model_dump()
|
||||||
|
if isinstance(obj, Decimal):
|
||||||
|
return str(obj)
|
||||||
|
if isinstance(obj, set):
|
||||||
|
return list(obj)
|
||||||
|
raise TypeError(f"orjson: cannot serialize {type(obj).__name__}")
|
||||||
|
|
||||||
|
orjson.dumps(payload, default=_default, option=orjson.OPT_NAIVE_UTC | orjson.OPT_UTC_Z)
|
||||||
|
```
|
||||||
|
|
||||||
|
The `default=` callback runs once per unrecognized type, then orjson caches the path. Performance impact on subsequent calls is negligible.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Caching, queues, logging — the prime orjson use cases
|
||||||
|
|
||||||
|
These are where orjson pays off most clearly because there is no Pydantic in the loop:
|
||||||
|
|
||||||
|
### Redis cache
|
||||||
|
|
||||||
|
```python
|
||||||
|
import orjson
|
||||||
|
import redis.asyncio as redis
|
||||||
|
|
||||||
|
r = redis.from_url("redis://localhost")
|
||||||
|
|
||||||
|
async def set_cache(key: str, value: dict) -> None:
|
||||||
|
await r.set(key, orjson.dumps(value), ex=3600)
|
||||||
|
|
||||||
|
async def get_cache(key: str) -> dict | None:
|
||||||
|
raw = await r.get(key)
|
||||||
|
return orjson.loads(raw) if raw else None
|
||||||
|
```
|
||||||
|
|
||||||
|
`orjson` over stdlib `json` here saves ~5–10× on the serialize step for typical cache payloads. Multiply by request rate.
|
||||||
|
|
||||||
|
### Task queue payloads (Celery, RQ, dramatiq)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Celery custom serializer
|
||||||
|
from kombu.serialization import register
|
||||||
|
import orjson
|
||||||
|
|
||||||
|
def _orjson_dumps(obj):
|
||||||
|
return orjson.dumps(obj, option=orjson.OPT_NAIVE_UTC | orjson.OPT_UTC_Z).decode()
|
||||||
|
|
||||||
|
def _orjson_loads(s):
|
||||||
|
return orjson.loads(s)
|
||||||
|
|
||||||
|
register("orjson", _orjson_dumps, _orjson_loads,
|
||||||
|
content_type="application/x-orjson",
|
||||||
|
content_encoding="utf-8")
|
||||||
|
```
|
||||||
|
|
||||||
|
Same speedup, applied to every task payload encode/decode.
|
||||||
|
|
||||||
|
### Structured logging (structlog, custom slog)
|
||||||
|
|
||||||
|
```python
|
||||||
|
import structlog
|
||||||
|
import orjson
|
||||||
|
|
||||||
|
structlog.configure(
|
||||||
|
processors=[
|
||||||
|
structlog.processors.TimeStamper(fmt="iso"),
|
||||||
|
structlog.processors.add_log_level,
|
||||||
|
structlog.processors.JSONRenderer(serializer=orjson.dumps),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
structlog's `JSONRenderer` accepts any callable; orjson is the obvious default. Logging hot paths benefit dramatically — every log line at info level becomes ~5× cheaper to render.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Gotchas
|
||||||
|
|
||||||
|
### `orjson.dumps` returns bytes, not str
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BAD — concatenating bytes and str
|
||||||
|
log.info("payload: " + orjson.dumps(data)) # TypeError
|
||||||
|
|
||||||
|
# GOOD
|
||||||
|
log.info("payload: %s", orjson.dumps(data).decode())
|
||||||
|
# or
|
||||||
|
log.info("payload: %s", orjson.dumps(data)) # let the formatter handle it
|
||||||
|
```
|
||||||
|
|
||||||
|
### No `cls=` argument for custom encoders
|
||||||
|
|
||||||
|
orjson uses `default=` only. If you have a custom `JSONEncoder` subclass from stdlib `json`, port its `default()` method to a `default=` callable.
|
||||||
|
|
||||||
|
### Subclasses of `dict` / `list` are NOT serialized as their parent
|
||||||
|
|
||||||
|
```python
|
||||||
|
class StrictDict(dict): ...
|
||||||
|
d = StrictDict({"k": "v"})
|
||||||
|
|
||||||
|
import json
|
||||||
|
json.dumps(d) # OK — stdlib walks subclasses
|
||||||
|
orjson.dumps(d) # TypeError — orjson is strict by design
|
||||||
|
orjson.dumps(d, option=orjson.OPT_PASSTHROUGH_SUBCLASS) # then route via default=
|
||||||
|
```
|
||||||
|
|
||||||
|
Set `OPT_PASSTHROUGH_SUBCLASS` and handle the subclass in `default=`. The design discourages accidental subclass usage that breaks elsewhere.
|
||||||
|
|
||||||
|
### `int` overflow
|
||||||
|
|
||||||
|
orjson refuses to encode integers larger than 2⁵³ - 1 by default (the IEEE-754 double-precision safe-integer limit — what JavaScript can round-trip). For larger ints, opt in:
|
||||||
|
|
||||||
|
```python
|
||||||
|
orjson.dumps(huge_int, option=orjson.OPT_STRICT_INTEGER) # error
|
||||||
|
orjson.dumps(huge_int) # default — int is encoded as JSON number
|
||||||
|
# JavaScript clients lose precision past 2^53; consider sending as string
|
||||||
|
```
|
||||||
|
|
||||||
|
This is more spec-strict than stdlib `json`, which silently emits ints of any size.
|
||||||
|
|
||||||
|
### Timezone-naive datetimes
|
||||||
|
|
||||||
|
By default, orjson treats naive `datetime` as the system local timezone — almost never what you want. **Always set `OPT_NAIVE_UTC`** to treat naive datetimes as UTC, or use timezone-aware datetimes (which is the better long-term habit).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Benchmark — should I actually adopt this?
|
||||||
|
|
||||||
|
The numbers below are 2024–2026 averages from `tiangolo/fastapi#11728` and orjson's own benchmark suite, on Python 3.13, modern x86_64:
|
||||||
|
|
||||||
|
| Payload | stdlib `json` | `orjson` | `model_dump_json()` (Pydantic v2) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Small dict (100 fields) | 1.0× | **8×** | n/a |
|
||||||
|
| List of 10k dicts | 1.0× | **11×** | n/a |
|
||||||
|
| Pydantic model with 20 fields | 1.0× (after `model_dump()`) | 5× (with `default=` bridge) | **6×** |
|
||||||
|
| Datetime-heavy payload | 1.0× (after manual ISO conv) | **9×** | 6× |
|
||||||
|
| numpy array (1M floats) | impossible without manual conv | **20×** vs json+tolist | n/a |
|
||||||
|
|
||||||
|
The takeaways:
|
||||||
|
|
||||||
|
- For raw dict/list/datetime, **orjson is dramatically faster**.
|
||||||
|
- For Pydantic models, **`model_dump_json()` is already faster than orjson+bridge**.
|
||||||
|
- For numpy, orjson is the only sane choice.
|
||||||
|
|
||||||
|
In production, the actual measured win on a FastAPI app with mixed payloads is typically 5–15% reduction in p99 latency. Worth the one-line `default_response_class=ORJSONResponse` switch.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. When NOT to adopt orjson
|
||||||
|
|
||||||
|
- The codebase is small, JSON is not a bottleneck, and you have no measured perf concern.
|
||||||
|
- You depend on stdlib `json`'s `cls=` arg or its lax tolerance for non-spec input (NaN, Infinity, comments).
|
||||||
|
- You need pretty-printed JSON with custom indent — orjson only supports 2-space indent via the flag.
|
||||||
|
- You need pure-Python portability (e.g., MicroPython, no-wheel platforms) — orjson is a compiled Rust extension.
|
||||||
|
|
||||||
|
If the choice is "add a dependency that does 5–10× the speed on serialization for free", the answer is almost always yes. The "almost" is in the bullets above.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- orjson: https://github.com/ijl/orjson
|
||||||
|
- Pydantic v2 `model_dump_json`: https://docs.pydantic.dev/latest/concepts/serialization/#modelmodel_dump_json
|
||||||
|
- FastAPI `ORJSONResponse`: https://fastapi.tiangolo.com/advanced/custom-response/#use-orjsonresponse
|
||||||
|
- "FastAPI + orjson vs Pydantic v2" benchmark: https://github.com/fastapi/fastapi/discussions/11728
|
||||||
|
- structlog JSON rendering: https://www.structlog.org/en/stable/api.html#structlog.processors.JSONRenderer
|
||||||
@@ -0,0 +1,285 @@
|
|||||||
|
# PydanticAI Reference (v1.x, 2026)
|
||||||
|
|
||||||
|
> Canonical patterns for wiring PydanticAI agents. Target: production usage, late-2025 / 2026.
|
||||||
|
> Source: [ai.pydantic.dev](https://ai.pydantic.dev) and [pydantic/pydantic-ai@`cad9569`](https://github.com/pydantic/pydantic-ai/blob/cad956910079737ea0886b50cef15777208f92e6).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Agent Constructor
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic_ai import Agent
|
||||||
|
|
||||||
|
agent = Agent(
|
||||||
|
'openai:gpt-5.2', # model (str | Model | None)
|
||||||
|
output_type=MyOutputModel, # structured output type; default=str
|
||||||
|
instructions='You are a...', # static or callable instructions
|
||||||
|
system_prompt='Be concise.', # static system prompt(s)
|
||||||
|
deps_type=MyDeps, # dependency type for type-checking only
|
||||||
|
name='my-agent', # optional, inferred from var name if omitted
|
||||||
|
retries=1, # default retries for tools + output validation
|
||||||
|
output_retries=None, # override retries for output validation only
|
||||||
|
tools=[my_tool], # list of Tool objects or plain functions
|
||||||
|
defer_model_check=False, # set True to skip env-var check at init time
|
||||||
|
end_strategy='early', # 'early' | 'graceful' | 'exhaustive'
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Breaking change (v1.88.0)**: `result_type` was renamed to `output_type`. Use `output_type`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Model Strings
|
||||||
|
|
||||||
|
Format: `provider:model-name`. The framework infers the provider from the prefix.
|
||||||
|
|
||||||
|
| Provider prefix | Example |
|
||||||
|
|---|---|
|
||||||
|
| `openai:` | `'openai:gpt-5.2'`, `'openai:gpt-4o'` |
|
||||||
|
| `anthropic:` | `'anthropic:claude-sonnet-4-6'`, `'anthropic:claude-opus-4-1'` |
|
||||||
|
| `google-gla:` | `'google-gla:gemini-3-flash-preview'` |
|
||||||
|
| `google-vertex:` | `'google-vertex:gemini-3-pro-preview'` |
|
||||||
|
| `bedrock:` | `'bedrock:anthropic.claude-sonnet-4-6'` |
|
||||||
|
| `xai:` / `grok:` | `'xai:grok-3'`, `'grok:grok-3-fast'` |
|
||||||
|
| `deepseek:` | `'deepseek:deepseek-chat'` |
|
||||||
|
| `cohere:` | `'cohere:command-r-08-2024'` |
|
||||||
|
| `gateway/...` | `'gateway/openai:gpt-5.2'` (PydanticAI Gateway) |
|
||||||
|
|
||||||
|
Model can also be omitted at construction and passed per-run: `agent.run(prompt, model='openai:gpt-5.2')`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Tools
|
||||||
|
|
||||||
|
### Decorator syntax
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic_ai import Agent, RunContext
|
||||||
|
|
||||||
|
agent = Agent('openai:gpt-5.2', deps_type=str)
|
||||||
|
|
||||||
|
@agent.tool # default: receives RunContext as first arg
|
||||||
|
async def greet(ctx: RunContext[str], name: str) -> str:
|
||||||
|
return f"Hello {ctx.deps}, {name}!"
|
||||||
|
|
||||||
|
@agent.tool_plain # no context needed
|
||||||
|
async def roll_dice(sides: int) -> int:
|
||||||
|
import random
|
||||||
|
return random.randint(1, sides)
|
||||||
|
```
|
||||||
|
|
||||||
|
### `RunContext[Deps]`
|
||||||
|
|
||||||
|
First parameter of `@agent.tool` functions. Carries:
|
||||||
|
|
||||||
|
- `ctx.deps` — the dependency instance
|
||||||
|
- `ctx.model` — the model being used
|
||||||
|
- `ctx.usage` — token usage so far
|
||||||
|
- `ctx.messages` — conversation history
|
||||||
|
- `ctx.retry` / `ctx.max_retries` — current retry count
|
||||||
|
- `ctx.agent` — the running agent instance
|
||||||
|
|
||||||
|
Use `@agent.tool_plain` when the tool does **not** need any of the above.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Structured Output
|
||||||
|
|
||||||
|
Pass a Pydantic `BaseModel` (or `bool`, `int`, `list[str]`, etc.) as `output_type`. The result is accessed via `.output`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from pydantic_ai import Agent
|
||||||
|
|
||||||
|
class City(BaseModel):
|
||||||
|
name: str
|
||||||
|
country: str
|
||||||
|
population_millions: float
|
||||||
|
|
||||||
|
agent = Agent('openai:gpt-5.2', output_type=City)
|
||||||
|
result = agent.run_sync('Tell me about Tokyo')
|
||||||
|
print(result.output) # City(name='Tokyo', country='Japan', ...)
|
||||||
|
print(result.output.name) # 'Tokyo'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note**: `result.data` was renamed; the canonical accessor is `result.output`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Async vs Sync
|
||||||
|
|
||||||
|
| Method | Mode | Returns |
|
||||||
|
|---|---|---|
|
||||||
|
| `await agent.run(prompt, ...)` | async | `AgentRunResult[OutputDataT]` |
|
||||||
|
| `agent.run_sync(prompt, ...)` | sync | `AgentRunResult[OutputDataT]` |
|
||||||
|
| `async with agent.run_stream(prompt, ...) as response:` | async streaming | `StreamedRunResult` |
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Sync
|
||||||
|
result = agent.run_sync('What is the capital of Italy?')
|
||||||
|
print(result.output)
|
||||||
|
|
||||||
|
# Async
|
||||||
|
result = await agent.run('What is the capital of France?')
|
||||||
|
print(result.output)
|
||||||
|
|
||||||
|
# Streaming
|
||||||
|
async with agent.run_stream('What is the capital of the UK?') as response:
|
||||||
|
async for text in response.stream_text():
|
||||||
|
print(text, end='')
|
||||||
|
# After streaming finishes:
|
||||||
|
print(response.output)
|
||||||
|
```
|
||||||
|
|
||||||
|
`run_sync()` is a convenience wrapper over `loop.run_until_complete(self.run(...))`. Do not use it inside an active async context.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Dependencies
|
||||||
|
|
||||||
|
Use a `@dataclass` container, pass the **type** to `deps_type`, and pass an **instance** to `deps` at run time.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from dataclasses import dataclass
|
||||||
|
import httpx
|
||||||
|
from pydantic_ai import Agent, RunContext
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Deps:
|
||||||
|
api_key: str
|
||||||
|
http_client: httpx.AsyncClient
|
||||||
|
|
||||||
|
agent = Agent(
|
||||||
|
'openai:gpt-5.2',
|
||||||
|
deps_type=Deps,
|
||||||
|
)
|
||||||
|
|
||||||
|
@agent.tool
|
||||||
|
async def fetch_data(ctx: RunContext[Deps], endpoint: str) -> str:
|
||||||
|
r = await ctx.deps.http_client.get(
|
||||||
|
endpoint,
|
||||||
|
headers={'Authorization': f'Bearer {ctx.deps.api_key}'},
|
||||||
|
)
|
||||||
|
r.raise_for_status()
|
||||||
|
return r.text
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
deps = Deps(api_key='sk-...', http_client=client)
|
||||||
|
result = await agent.run('Get /users', deps=deps)
|
||||||
|
print(result.output)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Error Types & Retrying from a Tool
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic_ai import Agent, ModelRetry, UnexpectedModelBehavior, capture_run_messages
|
||||||
|
|
||||||
|
agent = Agent('openai:gpt-5.2', retries=3)
|
||||||
|
|
||||||
|
@agent.tool_plain
|
||||||
|
def calc_volume(size: int) -> int:
|
||||||
|
if size == 42:
|
||||||
|
return size ** 3
|
||||||
|
raise ModelRetry('Please try again with size 42.')
|
||||||
|
|
||||||
|
with capture_run_messages() as messages:
|
||||||
|
try:
|
||||||
|
result = agent.run_sync('Get the volume of a box with size 6.')
|
||||||
|
except UnexpectedModelBehavior as e:
|
||||||
|
print('Error:', e) # "Tool 'calc_volume' exceeded max retries count of 3"
|
||||||
|
print('Cause:', e.__cause__) # ModelRetry('Please try again...')
|
||||||
|
print('Messages:', messages)
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`ModelRetry`** — raise from a tool, output validator, or capability hook to ask the model to retry.
|
||||||
|
- **`UnexpectedModelBehavior`** — raised when the retry limit is exceeded or the model API returns an unrecoverable error.
|
||||||
|
- **`capture_run_messages()`** — context manager that records all messages exchanged during a run for debugging.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Logfire Integration
|
||||||
|
|
||||||
|
One-line setup if the `logfire` extra is installed (included in the default `pydantic-ai` package):
|
||||||
|
|
||||||
|
```python
|
||||||
|
import logfire
|
||||||
|
|
||||||
|
logfire.configure() # reads token from .logfire directory
|
||||||
|
logfire.instrument_pydantic_ai() # auto-traces all agent runs
|
||||||
|
```
|
||||||
|
|
||||||
|
Alternatively, set `instrument=True` on the agent:
|
||||||
|
|
||||||
|
```python
|
||||||
|
agent = Agent('openai:gpt-5.2', instrument=True)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Minimal Complete Snippets
|
||||||
|
|
||||||
|
### (a) Basic agent with structured output
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from pydantic_ai import Agent
|
||||||
|
|
||||||
|
class City(BaseModel):
|
||||||
|
name: str
|
||||||
|
country: str
|
||||||
|
|
||||||
|
agent = Agent('openai:gpt-5.2', output_type=City)
|
||||||
|
result = agent.run_sync('Tell me about Paris')
|
||||||
|
print(result.output) # City(name='Paris', country='France')
|
||||||
|
```
|
||||||
|
|
||||||
|
### (b) Agent with tools and dependencies
|
||||||
|
|
||||||
|
```python
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pydantic_ai import Agent, RunContext
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Deps:
|
||||||
|
api_key: str
|
||||||
|
|
||||||
|
agent = Agent('openai:gpt-5.2', deps_type=Deps)
|
||||||
|
|
||||||
|
@agent.tool
|
||||||
|
async def get_secret(ctx: RunContext[Deps], code: str) -> str:
|
||||||
|
if code == '1234':
|
||||||
|
return f'secret-for-{ctx.deps.api_key}'
|
||||||
|
return 'wrong code'
|
||||||
|
|
||||||
|
result = agent.run_sync('My code is 1234', deps=Deps(api_key='sk-abc'))
|
||||||
|
print(result.output)
|
||||||
|
```
|
||||||
|
|
||||||
|
### (c) Async streaming
|
||||||
|
|
||||||
|
```python
|
||||||
|
import anyio
|
||||||
|
from pydantic_ai import Agent
|
||||||
|
|
||||||
|
agent = Agent('openai:gpt-5.2')
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
async with agent.run_stream('Write a haiku about Python') as response:
|
||||||
|
async for text in response.stream_text():
|
||||||
|
print(text, end='')
|
||||||
|
print('\n---')
|
||||||
|
print('Final:', response.output)
|
||||||
|
|
||||||
|
anyio.run(main)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Version Notes
|
||||||
|
|
||||||
|
- **V1** reached API stability in September 2025. Breaking changes are reserved for V2 (earliest April 2026).
|
||||||
|
- **v1.88.0** renamed `result_type` → `output_type` and `result_tool_name` / `result_tool_description` were removed. Use `output_type`.
|
||||||
|
- The canonical accessor for run results is `result.output` (not `result.data`).
|
||||||
@@ -0,0 +1,232 @@
|
|||||||
|
# Strict pyproject.toml (basedpyright + ruff + uv)
|
||||||
|
|
||||||
|
The canonical "super strict but sane" config for modern Python projects. Copy-paste, then add your own dependencies.
|
||||||
|
|
||||||
|
## Bootstrap
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Application
|
||||||
|
uv init --app myproject
|
||||||
|
cd myproject
|
||||||
|
|
||||||
|
# Library (publishable to PyPI)
|
||||||
|
uv init --lib mylibrary
|
||||||
|
cd mylibrary
|
||||||
|
|
||||||
|
# Add dev tools
|
||||||
|
uv add --dev basedpyright ruff pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
`uv init` creates `pyproject.toml`, `.python-version`, and `src/` layout. Replace its `pyproject.toml` `[tool.*]` sections with the block below.
|
||||||
|
|
||||||
|
## The full pyproject.toml
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[project]
|
||||||
|
name = "myproject"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "..."
|
||||||
|
readme = "README.md"
|
||||||
|
requires-python = ">=3.13"
|
||||||
|
dependencies = []
|
||||||
|
|
||||||
|
[dependency-groups]
|
||||||
|
dev = [
|
||||||
|
"basedpyright>=1.21",
|
||||||
|
"ruff>=0.8",
|
||||||
|
"pytest>=8",
|
||||||
|
"pytest-cov>=5",
|
||||||
|
]
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────────
|
||||||
|
# basedpyright - typeCheckingMode = "all" sets every report flag to error
|
||||||
|
# Source: https://docs.basedpyright.com/latest/configuration/config-files/
|
||||||
|
# ─────────────────────────────────────────────────────────────────
|
||||||
|
[tool.basedpyright]
|
||||||
|
typeCheckingMode = "all"
|
||||||
|
pythonVersion = "3.13"
|
||||||
|
pythonPlatform = "All" # default in basedpyright; explicit for clarity
|
||||||
|
include = ["src", "tests"]
|
||||||
|
exclude = ["**/__pycache__", "**/.venv", "**/build", "**/dist"]
|
||||||
|
|
||||||
|
# Strict enforcement extras (most are already "error" under "all" mode,
|
||||||
|
# but listing them explicitly documents the intent)
|
||||||
|
reportUnusedCallResult = "warning" # flag ignored return values
|
||||||
|
reportUnnecessaryTypeIgnoreComment = "error" # stale type: ignore comments must die
|
||||||
|
reportUnusedVariable = "error" # unused variables are errors
|
||||||
|
reportMissingParameterType = "error" # every parameter must have a type
|
||||||
|
reportMissingReturnType = "error" # every function must declare its return type
|
||||||
|
reportPrivateUsage = "error" # respect _private convention
|
||||||
|
|
||||||
|
# Optional: gradual adoption baseline
|
||||||
|
# baselineFile = "./.basedpyright/baseline.json"
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────────
|
||||||
|
# ruff - select = ["ALL"] enables every rule, then we ignore the
|
||||||
|
# small set that conflicts with the formatter or is not useful.
|
||||||
|
# Source: https://docs.astral.sh/ruff/linter/#rule-selection
|
||||||
|
# ─────────────────────────────────────────────────────────────────
|
||||||
|
[tool.ruff]
|
||||||
|
target-version = "py313"
|
||||||
|
line-length = 88 # ruff/black default; 100 or 120 also fine
|
||||||
|
src = ["src", "tests"]
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
|
select = ["ALL"]
|
||||||
|
ignore = [
|
||||||
|
# Formatter conflicts (ruff itself tells you to ignore these)
|
||||||
|
"COM812", # missing trailing comma
|
||||||
|
"ISC001", # implicit string concat
|
||||||
|
# Docstyle conflicts (pick D211 over D203, D212 over D213)
|
||||||
|
"D203",
|
||||||
|
"D213",
|
||||||
|
# Project-specific noise
|
||||||
|
"CPY001", # missing copyright notice
|
||||||
|
"FBT001", # boolean positional arg in def
|
||||||
|
"FBT002", # boolean positional default in def
|
||||||
|
"TD002", # missing TODO author
|
||||||
|
"TD003", # missing TODO link
|
||||||
|
"FIX002", # line contains TODO (TODOs are allowed)
|
||||||
|
]
|
||||||
|
fixable = ["ALL"]
|
||||||
|
unfixable = []
|
||||||
|
|
||||||
|
[tool.ruff.lint.per-file-ignores]
|
||||||
|
"tests/**/*.py" = [
|
||||||
|
"S101", # `assert` is the entire point of pytest
|
||||||
|
"ARG", # unused args (fixtures appear unused)
|
||||||
|
"PLR2004", # magic numbers in test data
|
||||||
|
"SLF001", # tests need access to private members
|
||||||
|
"D", # docstrings not required in tests
|
||||||
|
]
|
||||||
|
"scripts/**/*.py" = [
|
||||||
|
"T201", # `print` allowed in scripts
|
||||||
|
"INP001", # implicit namespace package
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.ruff.lint.pydocstyle]
|
||||||
|
convention = "google" # or "numpy" / "pep257"
|
||||||
|
|
||||||
|
[tool.ruff.lint.flake8-bugbear]
|
||||||
|
# typer / fastapi rely on call-as-default for parameter metadata.
|
||||||
|
# Without this, ruff B008 ("function call in default") fires on every typer/fastapi route.
|
||||||
|
extend-immutable-calls = [
|
||||||
|
"typer.Argument",
|
||||||
|
"typer.Option",
|
||||||
|
"fastapi.Depends",
|
||||||
|
"fastapi.Query",
|
||||||
|
"fastapi.Path",
|
||||||
|
"fastapi.Body",
|
||||||
|
"fastapi.Header",
|
||||||
|
"fastapi.Cookie",
|
||||||
|
"fastapi.File",
|
||||||
|
"fastapi.Form",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.ruff.format]
|
||||||
|
quote-style = "double"
|
||||||
|
indent-style = "space"
|
||||||
|
docstring-code-format = true
|
||||||
|
docstring-code-line-length = "dynamic"
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────────
|
||||||
|
# pytest
|
||||||
|
# ─────────────────────────────────────────────────────────────────
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
minversion = "8.0"
|
||||||
|
testpaths = ["tests"]
|
||||||
|
addopts = [
|
||||||
|
"-ra",
|
||||||
|
"--strict-config",
|
||||||
|
"--strict-markers",
|
||||||
|
]
|
||||||
|
filterwarnings = ["error"]
|
||||||
|
|
||||||
|
# ─────────────────────────────────────────────────────────────────
|
||||||
|
# coverage
|
||||||
|
# ─────────────────────────────────────────────────────────────────
|
||||||
|
[tool.coverage.run]
|
||||||
|
source = ["src"]
|
||||||
|
branch = true
|
||||||
|
|
||||||
|
[tool.coverage.report]
|
||||||
|
exclude_lines = [
|
||||||
|
"pragma: no cover",
|
||||||
|
"if TYPE_CHECKING:",
|
||||||
|
"if typing.TYPE_CHECKING:",
|
||||||
|
"raise NotImplementedError",
|
||||||
|
"@(abc\\.)?abstractmethod",
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Why these settings
|
||||||
|
|
||||||
|
### basedpyright `typeCheckingMode = "all"`
|
||||||
|
|
||||||
|
basedpyright's modes, strictest first:
|
||||||
|
|
||||||
|
| Mode | Behavior |
|
||||||
|
|---|---|
|
||||||
|
| `"all"` | Every diagnostic at `error` |
|
||||||
|
| `"recommended"` | Same rules; less severe ones at `warning`; `failOnWarnings = true` makes CI still fail |
|
||||||
|
| `"strict"` | pyright's strict mode |
|
||||||
|
| `"standard"` | Default |
|
||||||
|
| `"basic"` / `"off"` | Loose / disabled |
|
||||||
|
|
||||||
|
`"all"` enables basedpyright-exclusive rules pyright lacks: `reportImplicitOverride`, `reportImplicitStringConcatenation`, `reportIncompatibleUnannotatedOverride`, `reportUnannotatedClassAttribute`. No need to opt-in to additional flags.
|
||||||
|
|
||||||
|
`pythonPlatform = "All"` is basedpyright's default (better than pyright's host-OS default) - it errors on platform-specific imports that fail on other OSes.
|
||||||
|
|
||||||
|
### ruff `select = ["ALL"]`
|
||||||
|
|
||||||
|
The official docs say *"Use ALL with discretion. Enabling ALL will implicitly enable new rules whenever you upgrade."* For a strict skill that is the intended behavior - every new ruff rule should be considered an error until you justify ignoring it.
|
||||||
|
|
||||||
|
The minimal ignore set:
|
||||||
|
|
||||||
|
| Rule | Reason |
|
||||||
|
|---|---|
|
||||||
|
| `COM812`, `ISC001` | Conflict with `ruff format` (ruff itself documents this) |
|
||||||
|
| `D203` vs `D211`, `D213` vs `D212` | Mutually-exclusive docstring conventions; pick the modern one |
|
||||||
|
| `CPY001` | Most projects don't need a copyright header on every file |
|
||||||
|
| `FBT001`, `FBT002` | Boolean flags are ergonomic for CLI/typer; ban makes typer awkward |
|
||||||
|
| `TD002`, `TD003`, `FIX002` | TODOs without a JIRA link are fine in solo / internal code |
|
||||||
|
|
||||||
|
`ANN101` and `ANN102` were **removed in ruff 0.8.0** (Nov 2024). Do NOT include them in `ignore` - ruff errors on unknown rule codes.
|
||||||
|
|
||||||
|
`per-file-ignores` for `tests/**` is the standard pattern from real-world repos like `community-of-python/auto-typing-final` and `Preston-Landers/concurrent-log-handler`.
|
||||||
|
|
||||||
|
## CI gate
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In CI, fail on any violation:
|
||||||
|
uv run basedpyright
|
||||||
|
uv run ruff check
|
||||||
|
uv run ruff format --check
|
||||||
|
uv run pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
A single `make ci` target combining the four works fine.
|
||||||
|
|
||||||
|
## Enforcement summary
|
||||||
|
|
||||||
|
The config above, combined with `scripts/check-no-excuse-rules.py`, enforces:
|
||||||
|
|
||||||
|
| What | How |
|
||||||
|
|---|---|
|
||||||
|
| Exhaustive match | basedpyright `all` mode + `assert_never` |
|
||||||
|
| No `Any` | basedpyright `all` mode + script `cast-any` rule |
|
||||||
|
| Ignored return values | `reportUnusedCallResult = "warning"` |
|
||||||
|
| Immutable default | Script `mutable-dataclass` + `missing-slots` rules |
|
||||||
|
| No null surprise | basedpyright strict `None` analysis |
|
||||||
|
| Constants are const | basedpyright catches `Final` reassignment |
|
||||||
|
| Unused variables | `reportUnusedVariable = "error"` |
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- basedpyright modes: <https://docs.basedpyright.com/latest/configuration/config-files/#type-check-diagnostics-settings>
|
||||||
|
- basedpyright `"all"` vs `"recommended"`: <https://docs.basedpyright.com/latest/configuration/config-files/#recommended-and-all>
|
||||||
|
- basedpyright better defaults: <https://docs.basedpyright.com/latest/benefits-over-pyright/better-defaults/>
|
||||||
|
- ruff rule selection: <https://docs.astral.sh/ruff/linter/#rule-selection>
|
||||||
|
- ruff ANN101/ANN102 removed: <https://github.com/astral-sh/ruff/pull/14384>
|
||||||
|
- Real-world ALL config: <https://github.com/community-of-python/auto-typing-final/blob/main/pyproject.toml>
|
||||||
|
- PEP 735 dependency-groups: <https://peps.python.org/pep-0735/>
|
||||||
@@ -0,0 +1,201 @@
|
|||||||
|
# Textual TUI
|
||||||
|
|
||||||
|
Textual builds rich, mouse-aware, scrollable, mobile-style TUIs on top of `rich`. Replaces curses, urwid, blessed.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv add textual
|
||||||
|
uv add --dev textual-dev # textual console + run --dev for hot reload
|
||||||
|
```
|
||||||
|
|
||||||
|
## Minimal app
|
||||||
|
|
||||||
|
```python
|
||||||
|
from textual.app import App, ComposeResult
|
||||||
|
from textual.widgets import Header, Footer, Button, Label
|
||||||
|
from textual.containers import Vertical
|
||||||
|
|
||||||
|
|
||||||
|
class CounterApp(App[None]):
|
||||||
|
"""A trivial counter app."""
|
||||||
|
|
||||||
|
BINDINGS = [("q", "quit", "Quit")]
|
||||||
|
CSS = """
|
||||||
|
#count {
|
||||||
|
height: 3;
|
||||||
|
content-align: center middle;
|
||||||
|
background: $boost;
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
|
||||||
|
count: int = 0
|
||||||
|
|
||||||
|
def compose(self) -> ComposeResult:
|
||||||
|
yield Header()
|
||||||
|
with Vertical():
|
||||||
|
yield Label("0", id="count")
|
||||||
|
yield Button("Increment", id="inc", variant="primary")
|
||||||
|
yield Button("Reset", id="reset", variant="warning")
|
||||||
|
yield Footer()
|
||||||
|
|
||||||
|
def on_button_pressed(self, event: Button.Pressed) -> None:
|
||||||
|
if event.button.id == "inc":
|
||||||
|
self.count += 1
|
||||||
|
elif event.button.id == "reset":
|
||||||
|
self.count = 0
|
||||||
|
self.query_one("#count", Label).update(str(self.count))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
CounterApp().run()
|
||||||
|
```
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python counter.py
|
||||||
|
```
|
||||||
|
|
||||||
|
For hot reload during development:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run textual run --dev counter.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Reactive attributes
|
||||||
|
|
||||||
|
Textual's `reactive()` descriptor turns a class attribute into something that watches assignments and re-renders automatically. Replaces the manual `query_one` + `update` dance.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from textual.app import App, ComposeResult
|
||||||
|
from textual.reactive import reactive
|
||||||
|
from textual.widgets import Label
|
||||||
|
|
||||||
|
|
||||||
|
class CountWidget(Label):
|
||||||
|
count: reactive[int] = reactive(0)
|
||||||
|
|
||||||
|
def render(self) -> str:
|
||||||
|
return f"Count: {self.count}"
|
||||||
|
|
||||||
|
|
||||||
|
class CounterApp(App[None]):
|
||||||
|
def compose(self) -> ComposeResult:
|
||||||
|
yield CountWidget()
|
||||||
|
|
||||||
|
def on_key(self, event) -> None:
|
||||||
|
if event.key == "space":
|
||||||
|
self.query_one(CountWidget).count += 1
|
||||||
|
```
|
||||||
|
|
||||||
|
`reactive()` triggers `render()` (or `watch_<attr>` and `validate_<attr>` callbacks if defined). Use `recompose=True` if you need to call `compose()` again on change.
|
||||||
|
|
||||||
|
## Async work — workers
|
||||||
|
|
||||||
|
NEVER block the event loop. For network/disk/CPU work, use `@work` (creates a worker) or `run_worker`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
import httpx
|
||||||
|
from textual.app import App, ComposeResult
|
||||||
|
from textual.widgets import Input, Static
|
||||||
|
from textual.work import work
|
||||||
|
|
||||||
|
|
||||||
|
class FetchApp(App[None]):
|
||||||
|
def compose(self) -> ComposeResult:
|
||||||
|
yield Input(placeholder="URL", id="url")
|
||||||
|
yield Static(id="result")
|
||||||
|
|
||||||
|
@work(exclusive=True)
|
||||||
|
async def fetch(self, url: str) -> None:
|
||||||
|
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||||
|
response = await client.get(url)
|
||||||
|
self.query_one("#result", Static).update(f"{response.status_code} - {len(response.text)} bytes")
|
||||||
|
|
||||||
|
def on_input_submitted(self, event: Input.Submitted) -> None:
|
||||||
|
self.fetch(event.value)
|
||||||
|
```
|
||||||
|
|
||||||
|
`exclusive=True` cancels the previous worker if the user submits a new URL before the first finishes. Workers integrate with Textual's lifecycle - they're cancelled when the app exits.
|
||||||
|
|
||||||
|
`@work` is asyncio-flavoured under the hood. That is fine - it does not violate the no-asyncio rule because you are calling Textual's API, not importing asyncio yourself. Inside the worker body, use `httpx.AsyncClient` and other anyio-friendly libraries.
|
||||||
|
|
||||||
|
## Action handlers
|
||||||
|
|
||||||
|
Bind keys to method calls via `BINDINGS` and `action_*` methods.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class App(App):
|
||||||
|
BINDINGS = [
|
||||||
|
("ctrl+s", "save", "Save"),
|
||||||
|
("ctrl+r", "reload", "Reload"),
|
||||||
|
]
|
||||||
|
|
||||||
|
def action_save(self) -> None:
|
||||||
|
# Called on ctrl+s
|
||||||
|
...
|
||||||
|
|
||||||
|
def action_reload(self) -> None:
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
Bindings can also include the `priority=True` flag to fire before children get a chance.
|
||||||
|
|
||||||
|
## CSS
|
||||||
|
|
||||||
|
Textual's CSS supports selectors, variables (`$primary`, `$boost`), animations. Inline via `CSS = "..."` or external via `CSS_PATH = "app.tcss"`.
|
||||||
|
|
||||||
|
```css
|
||||||
|
Screen {
|
||||||
|
background: $surface;
|
||||||
|
color: $text;
|
||||||
|
layout: vertical;
|
||||||
|
}
|
||||||
|
|
||||||
|
#sidebar {
|
||||||
|
width: 30;
|
||||||
|
background: $boost;
|
||||||
|
}
|
||||||
|
|
||||||
|
Button.danger {
|
||||||
|
background: $error;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Reload with `r` in dev mode (`textual run --dev`).
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```python
|
||||||
|
import pytest
|
||||||
|
from myapp import CounterApp
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.anyio
|
||||||
|
async def test_counter_increments() -> None:
|
||||||
|
app = CounterApp()
|
||||||
|
async with app.run_test() as pilot:
|
||||||
|
await pilot.click("#inc")
|
||||||
|
await pilot.click("#inc")
|
||||||
|
assert app.count == 2
|
||||||
|
```
|
||||||
|
|
||||||
|
`pilot.click(selector)`, `pilot.press("q")`, `pilot.pause()` for waiting on the next frame.
|
||||||
|
|
||||||
|
## When NOT to use Textual
|
||||||
|
|
||||||
|
| Need | Use |
|
||||||
|
|---|---|
|
||||||
|
| One-off CLI with structured output | typer + rich |
|
||||||
|
| Progress bar in a script | rich.progress |
|
||||||
|
| Tabular display of query results | rich.table |
|
||||||
|
| Full-screen app with state, input, mouse | Textual |
|
||||||
|
|
||||||
|
A pretty CLI is not a TUI. Reach for Textual when the user expects to navigate a UI, not when you want colours.
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- Textual docs: <https://textual.textualize.io>
|
||||||
|
- Textual tutorial: <https://textual.textualize.io/tutorial/>
|
||||||
|
- API reference: <https://textual.textualize.io/api/>
|
||||||
@@ -0,0 +1,176 @@
|
|||||||
|
# Type Patterns
|
||||||
|
|
||||||
|
How to use Python's type system to catch bugs at check time, not runtime.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## NewType — distinct primitives
|
||||||
|
|
||||||
|
Same runtime type, different meaning. The type checker prevents mixing.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import NewType
|
||||||
|
|
||||||
|
UserId = NewType("UserId", int)
|
||||||
|
MovieId = NewType("MovieId", int)
|
||||||
|
Email = NewType("Email", str)
|
||||||
|
Seconds = NewType("Seconds", float)
|
||||||
|
Milliseconds = NewType("Milliseconds", float)
|
||||||
|
|
||||||
|
def get_user(user_id: UserId) -> User: ...
|
||||||
|
def get_movie(movie_id: MovieId) -> Movie: ...
|
||||||
|
def sleep(duration: Seconds) -> None: ...
|
||||||
|
|
||||||
|
uid = UserId(42)
|
||||||
|
mid = MovieId(42)
|
||||||
|
|
||||||
|
get_user(uid) # OK
|
||||||
|
get_user(mid) # type error: MovieId is not UserId
|
||||||
|
get_user(42) # type error: int is not UserId
|
||||||
|
sleep(Milliseconds(100.0)) # type error
|
||||||
|
```
|
||||||
|
|
||||||
|
**Use when**: IDs, indices, keys, units of measurement — any pair where swapping is a bug.
|
||||||
|
**Skip when**: ephemeral local math where branding adds noise with zero safety gain.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Final — constants are const
|
||||||
|
|
||||||
|
Module-level constants declare their intent. Reassignment is a type error.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import Final
|
||||||
|
|
||||||
|
MAX_RETRIES: Final = 3
|
||||||
|
API_BASE_URL: Final = "https://api.example.com"
|
||||||
|
DEFAULT_TIMEOUT: Final = 30.0
|
||||||
|
|
||||||
|
MAX_RETRIES = 5 # type error: cannot assign to Final
|
||||||
|
```
|
||||||
|
|
||||||
|
If it changes at runtime, it's not a constant — make it a function parameter or config field.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TypeAlias — name complex types
|
||||||
|
|
||||||
|
If a union or generic appears more than once, give it a name.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Python 3.12+
|
||||||
|
type JsonValue = str | int | float | bool | None | list["JsonValue"] | dict[str, "JsonValue"]
|
||||||
|
type Headers = dict[str, str]
|
||||||
|
type Middleware = Callable[[Request], Awaitable[Response]]
|
||||||
|
|
||||||
|
# Pre-3.12
|
||||||
|
from typing import TypeAlias
|
||||||
|
|
||||||
|
JsonValue: TypeAlias = str | int | float | bool | None | list["JsonValue"] | dict[str, "JsonValue"]
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## StrEnum / IntEnum — closed sets
|
||||||
|
|
||||||
|
Any fixed set of known values. No string literals scattered through code.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from enum import StrEnum, IntEnum, unique
|
||||||
|
|
||||||
|
@unique
|
||||||
|
class Role(StrEnum):
|
||||||
|
ADMIN = "admin"
|
||||||
|
USER = "user"
|
||||||
|
GUEST = "guest"
|
||||||
|
|
||||||
|
@unique
|
||||||
|
class HttpStatus(IntEnum):
|
||||||
|
OK = 200
|
||||||
|
NOT_FOUND = 404
|
||||||
|
INTERNAL_ERROR = 500
|
||||||
|
|
||||||
|
# BAD
|
||||||
|
def check_role(role: str) -> bool: ...
|
||||||
|
|
||||||
|
# GOOD
|
||||||
|
def check_role(role: Role) -> bool: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
`StrEnum` when values serialize as strings (API, DB). `IntEnum` for numeric codes. Plain `Enum` for pure labels.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Type narrowing — let the checker follow your logic
|
||||||
|
|
||||||
|
`isinstance`, `is None`, and `match` narrow types automatically. Use them instead of `cast`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
def process(value: str | int | None) -> str:
|
||||||
|
if value is None:
|
||||||
|
return "nothing"
|
||||||
|
# checker knows: str | int
|
||||||
|
|
||||||
|
if isinstance(value, str):
|
||||||
|
return value.upper()
|
||||||
|
# checker knows: int
|
||||||
|
|
||||||
|
return str(value * 2)
|
||||||
|
```
|
||||||
|
|
||||||
|
### TypeGuard for custom narrowing
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import TypeGuard
|
||||||
|
|
||||||
|
def is_valid_email(value: str) -> TypeGuard[Email]:
|
||||||
|
return "@" in value and "." in value.split("@")[1]
|
||||||
|
|
||||||
|
def send(addr: str) -> None:
|
||||||
|
if not is_valid_email(addr):
|
||||||
|
raise ValueError(addr)
|
||||||
|
# checker knows: addr is Email
|
||||||
|
deliver(addr)
|
||||||
|
```
|
||||||
|
|
||||||
|
### TypeIs (Python 3.13+) — the strict version
|
||||||
|
|
||||||
|
`TypeIs` is stricter than `TypeGuard` — it narrows in both `if` and `else` branches.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import TypeIs
|
||||||
|
|
||||||
|
def is_str(value: str | int) -> TypeIs[str]:
|
||||||
|
return isinstance(value, str)
|
||||||
|
|
||||||
|
def handle(v: str | int) -> None:
|
||||||
|
if is_str(v):
|
||||||
|
print(v.upper()) # checker knows: str
|
||||||
|
else:
|
||||||
|
print(v + 1) # checker knows: int
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Union syntax
|
||||||
|
|
||||||
|
Always `X | Y`. Never `Union[X, Y]` or `Optional[X]`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BAD
|
||||||
|
from typing import Union, Optional
|
||||||
|
def f(x: Optional[int]) -> Union[str, int]: ...
|
||||||
|
|
||||||
|
# GOOD
|
||||||
|
def f(x: int | None) -> str | int: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
- Python docs: [typing — NewType](https://docs.python.org/3/library/typing.html#newtype)
|
||||||
|
- Python docs: [typing — Final](https://docs.python.org/3/library/typing.html#typing.Final)
|
||||||
|
- Python docs: [typing — TypeGuard](https://docs.python.org/3/library/typing.html#typing.TypeGuard)
|
||||||
|
- PEP 604: [Union syntax X | Y](https://peps.python.org/pep-0604/)
|
||||||
|
- PEP 742: [TypeIs](https://peps.python.org/pep-0742/)
|
||||||
Reference in New Issue
Block a user