Local archives & crawlers

Peter Steinberger’s (steipete, PSPDFKit founder turned “Clawdfather” of the OpenClaw ecosystem) family of local-first crawler CLIs — one per platform, all built on the same bet: an agent works best over data it owns locally, in SQLite/Markdown, not over APIs it doesn’t control. The pattern per tool: crawl the platform (web, desktop app, or API) → store in a local archive (usually SQLite, git-backed where history matters) → serve the agent with clean CLI queries. This is the read-side complement to CLIs for agents: agent-accessible products for platforms that aren’t agent-accessible.

The family (all captured)

The substrate:

• crawlkit (OSS) (63★, Go) — the shared Go infrastructure for all of them: local-first crawler archive plumbing. Factored out so each crawler doesn’t reinvent storage/sync.

Chat & social (the archaeology set):

• discrawl (OSS) (853★, Go) — Discord → SQLite; “mirror Discord into SQLite; search server history locally”
• slacrawl (OSS) (236★, Go) — Slack history clawed into SQLite (the read-side sibling of agent-slack’s write side)
• birdclaw (OSS) (1.7k★, TS) — Twitter archive archaeology with git-backed memory (“stores all your tweets nicely claw-able for agents”)
• telecrawl (OSS) (85★, Go) — Telegram Desktop archive CLI
• wacrawl (OSS) (202★, Go) — WhatsApp read-only desktop-archive extractor with encrypted receipts (“WhatsApp archaeology”)
• clawdex (OSS) (55★, Go) — contacts crawler into markdown (the address-book layer every other archive wants to join against)

Work platforms:

• gitcrawl (OSS) (123★, Go) — local-first GitHub issue/PR crawler for maintainer triage (feeds his ClawSweeper triage bot)
• notcrawl (OSS) (85★→134★, Go) — Notion → SQLite + normalized Markdown

Why this pattern matters

• Local-first beats API-first for agents: platform APIs rate-limit, change, and disappear; a SQLite file is yours, queryable at disk speed, and greppable by qmd-style hybrid search. The crawler pays the API pain once; the agent never does.
• Git-backed archives = time-traveling memory (birdclaw): history as commits, diffs as change detection — the same trick as this vault’s manifest-pinned clippings.
• SQLite as the universal agent exchange format: every tool converges on it independently (discrawl, slacrawl, notcrawl) — structured, typed, zero-deployment, Embedded databases-grade. Markdown beside it for prose-shaped data.
• One platform = one tool, composable via CLI — the Unix shape: the agent joins archives with sqlite3 and pipes, no monolith needed.
• KRZ kin: the oncall bundle lane (fetch→scp→process writes local files agents then query), Agent Reach (the same mission, web-scale), and the garden itself (clippings as crawled archives, hash-pinned).

Related: CLIs for agents (the design rules these follow) · Agent-accessible product (what platforms should do vs what these tools do to them) · Agent-Reach (OSS) · Embedded databases (SQLite as exchange format)

Memory lane kin: cortexdb (the Go one)

cortexdb (OSS) (270★ MIT — the liliang-cn Go namesake, NOT cortexdb.ai) — AI memory + knowledge graph in one SQLite file: HNSW+FTS5+RDF/SPARQL/SHACL (W3C-suite-passing), 80+ MCP tools, embedding-optional (lexical mode w/o a model), shared-brain gRPC for several agents. Strengthens this note’s SQLite-as-universal-agent-exchange thesis; homelab test candidate (single binary, loopback gRPC, no cloud). Compare CortexDB for the closed-source .ai doppelgänger behind OpenHuman Memory v2.

ctrl+f for the screenshot archive: gyotaku

gyotaku (OSS) (172★ GPL-3.0 Rust; via @xevrion_the1, 1,132❤ — “i screenshot everything and can never find anything. so i built ctrl+f for my screenshots folder”) — OCR-indexed screenshot search, fully offline (PaddleOCR + SQLite + gpui): type any word that was in an image, matches appear with the words highlighted in place; open one to copy text, selected lines, or the image. Doesn’t take screenshots — indexes the folders your existing tool saves to. The numbers are the pitch: 1–10 ms per keystroke at 5,000+ screenshots, 0.77 s saved-to-searchable, 37 MB idle, ~30 MB disk per 1,000 shots, GPU-optional (software rendering; 144 fps with). Native Linux (curl installer or .deb/.rpm, no root) + Windows. Fits this note’s locally-owned-data lane perfectly — screenshots are the most under-indexed personal archive there is; kin to dnsglobe (OSS)-grade native TUIs and the CLIs for agents inventory thesis (an agent-queryable index of visual history).