docs(omo-codex): batch 95 (14 files)

2026-05-30 19:12:18 +09:00
parent 2a0d8822eb
commit c4273fe98f
14 changed files with 3863 additions and 0 deletions
@@ -0,0 +1,133 @@
+# Data Processing — Polars + DuckDB
+
+## The rule
+
+NEVER pandas. Polars (with numpy) plus DuckDB. Pandas is 10-50x slower, has weaker types, and the modern Python data ecosystem has moved on.
+
+## Quick decision tree
+
+| Operation | Use | Why |
+|---|---|---|
+| `.csv` / `.parquet` / `.json` direct query | DuckDB | Zero memory load, SQL ergonomics |
+| `.duckdb` file | DuckDB | Native format |
+| Filter (any size) | Polars | 128x faster than DuckDB for filtering |
+| Sort | Polars | 12x faster |
+| Multi-table join | DuckDB | 3x faster, more join types |
+| Heavy GROUP BY aggregation | DuckDB | 4x faster on large datasets |
+| Window function | Polars | 3-5x faster |
+| Pivot / melt / string ops | Polars | 2x faster |
+| Larger than RAM | Polars streaming or DuckDB out-of-core | Both handle OOM |
+| Mixed pipeline | Hybrid (zero-copy via Arrow) | Use each tool's strengths |
+
+For the deep version (per-operation benchmarks, OOM strategies, full execution templates), load the **`data-scientist`** skill - it lives in this same skill set and is the source of truth for performance numbers.
+
+## Standard imports
+
+```python
+import numpy as np
+import polars as pl
+import duckdb
+```
+
+## DuckDB direct file query (zero memory load)
+
+```python
+result = duckdb.sql("""
+    SELECT category, SUM(amount) AS total
+    FROM 'data.csv'
+    WHERE date >= '2026-01-01'
+    GROUP BY category
+    ORDER BY total DESC
+""").pl()  # zero-copy → Polars DataFrame
+```
+
+`.pl()` returns Polars; `.df()` would return pandas - never use `.df()`.
+
+## Polars lazy pipeline
+
+```python
+result = (
+    pl.scan_csv("data.csv")              # lazy, no read yet
+    .filter(pl.col("amount") > 1000)
+    .filter(pl.col("status") == "active")
+    .sort("amount", descending=True)
+    .head(100)
+    .collect()                            # execute optimised plan
+)
+```
+
+`scan_*` over `read_*` for files; `lazy()` then `collect()` for in-memory frames. Polars optimises the entire plan before execution (predicate pushdown, projection pushdown, common subexpression elimination).
+
+## Streaming for OOM data
+
+```python
+result = (
+    pl.scan_csv("huge.csv")
+    .filter(pl.col("active"))
+    .group_by("category")
+    .agg([
+        pl.len().alias("count"),
+        pl.sum("amount").alias("total"),
+    ])
+    .collect(streaming=True)
+)
+```
+
+## Hybrid pipeline (most realistic shape)
+
+```python
+# Phase 1: DuckDB for the join (3x faster)
+joined = duckdb.sql("""
+    SELECT o.*, c.region, p.category
+    FROM 'orders.parquet' o
+    JOIN 'customers.parquet' c ON o.customer_id = c.id
+    JOIN 'products.parquet'  p ON o.product_id  = p.id
+""").pl()
+
+# Phase 2: Polars for filtering and transformation (128x + 2x faster)
+processed = (
+    joined
+    .filter(pl.col("amount") > 100)
+    .with_columns([
+        (pl.col("amount") * 1.1).alias("amount_with_tax"),
+    ])
+)
+
+# Phase 3: DuckDB for final aggregation (4x faster) - register Polars frame by name
+duckdb.register("processed", processed)
+final = duckdb.sql("""
+    SELECT region, category, SUM(amount_with_tax) AS revenue
+    FROM processed
+    GROUP BY region, category
+    ORDER BY revenue DESC
+""").pl()
+```
+
+## Type safety with Polars
+
+Polars supports schema overrides at read time, and `.cast()` for explicit conversion. Avoid implicit coercion in hot paths.
+
+```python
+schema = {"id": pl.Int64, "amount": pl.Float64, "date": pl.Date}
+df = pl.read_csv("data.csv", schema_overrides=schema)
+```
+
+basedpyright understands `polars-stubs`, which ship with polars itself. No extra type stubs to install.
+
+## Things you might miss from pandas (and how to do them in Polars)
+
+| pandas | polars |
+|---|---|
+| `df.iloc[5]` | `df.row(5)` (named tuple) or `df[5]` (single-row frame) |
+| `df.loc[df["x"] > 5]` | `df.filter(pl.col("x") > 5)` |
+| `df["x"].apply(fn)` | `df["x"].map_elements(fn)` (slow path) or use native expressions |
+| `df.merge(...)` | `df.join(other, on="key")` |
+| `df.groupby(...).agg(...)` | `df.group_by(...).agg(...)` |
+| `pd.read_csv(...).dtypes` | `pl.read_csv(...).schema` |
+| `df.to_dict("records")` | `df.to_dicts()` |
+
+## Sources
+
+- Polars docs: <https://docs.pola.rs>
+- DuckDB Python API: <https://duckdb.org/docs/api/python/overview>
+- Cross-reference - this skill set's `data-scientist` skill (load it for the deep version)