How I orchestrate 6,300+ autonomous AI jobs

01

Architecture overview

The system follows a hub-and-spoke pattern. A central orchestration engine accepts work as jobs, routes them to the right model and worker, tracks dependencies between jobs, and handles failure recovery. Everything flows through a single SQLite database that serves as the system's source of truth.

Two layers keep things sane. Layer 1 is the engine: a silent async job queue that executes work without ever messaging me. Layer 2 is the judgment layer: it decides what I actually need to see. 95% of completed work produces zero notifications. The other 5% arrives as tappable decision cards on my phone.

I didn't plan it this way. The system started as a Telegram bot and a single Claude API call. Over 45 days it accumulated enough operational complexity that the architecture had to emerge or collapse. It emerged.

02

Worker system

Workers are long-lived agent sessions with configuration: a backend (Claude, Codex, Gemini, LLM), a model, a persona, loaded skills, and persistence rules. The system manages around 20 concurrent execution slots. Some workers are persistent - they handle email triage, morning briefings, and project-specific context. Others are ephemeral - spun up for a single job and discarded.

The critical insight that took weeks to learn: working directory IS session identity. Claude Code stores sessions under ~/.claude/projects/<cwd-hash>/<session-id>.jsonl. If you get the working directory wrong, your agents lose accumulated context. Every session restart, every container reboot, this is the thing that breaks first.

Session routing follows five paths in order: explicit assignment, project match, idle worker with matching backend, or auto-create ephemeral. Each session gets safety checks before reuse - file exists, size under limit, reuse count, age. Sessions that accumulate too much context get compacted by an overnight maintenance cron rather than abandoned.

03

Memory architecture

Most AI memory systems are "dump everything in a vector DB" or "keep a flat file." Neither works at scale. I built three tiers because each layer does a different job, and the system only works because those layers stay separate.

Tier 1: Core memory

A budgeted plaintext file that gets injected into every conversation. Identity, constraints, active projects, communication rules. It's capped at 80 lines because context window space is expensive and every token of core memory competes with the actual task. This gets edited by the system itself - not appended to, edited. The system rewrites its own identity document as it learns.

Tier 2: Archival memory (LanceDB)

A vector database with hybrid search: dense embeddings (text-embedding-3-small, 1536 dimensions) plus BM25 full-text, fused with reciprocal rank fusion. Stores facts, decisions, learned procedures, entity relationships, and episode summaries. Currently holds hundreds of entries across six categories.

The retrieval scoring formula took weeks to tune:

• Similarity: 0.45 - how relevant is this memory to the current context
• Recency: 0.25 - newer memories get preference
• Importance: 0.20 - some facts matter more than others
• Frequency: 0.10 - how often this memory gets recalled

The most important feature I added was anti-stale rotation. Without it, the system kept retrieving the same 20 memories regardless of context. Recent high-frequency memories got a slight penalty to force diversity. It's a hack, but it works.

Tier 3: Working memory

Session-scoped context that accumulates during a job: the current task, recent results, checkpoint files. This tier lives in the filesystem, not the database. When a session gets compacted (context window approaching limits), working memory gets summarized and the critical parts get promoted to archival. The rest gets discarded. An overnight maintenance cron reviews the day's sessions and extracts behavioral patterns into procedures.

04

Job engine

The engine isn't a task queue. It's a promise executor. Jobs are promises that return results. Promise.all() fans work across models in parallel. .then() chains outputs forward through template variables. Promise.race() takes the first model to finish. A 10-second event loop claims, dispatches, and resolves hundreds of jobs through dependency DAGs.

A pipeline compiler turns a JavaScript DSL into executable job DAGs with automatic cycle detection. Results flow between jobs via template variables - {{prev_result}}, {{all_results}}, {{index}}. Promise groups track completion combinators: all, race, allSettled.

Priority routing uses five lanes. P0-P1 are urgent (user-facing requests, blocking dependencies). P2 is normal work. P3-P5 are background (maintenance, research, improvement). The engine processes higher-priority work first, but never starves the background queue completely.

05

Operational data

The system has been running since February 18, 2026. These numbers come from the production database, not estimates.

06

Model routing

The routing layer sits between the job engine and the model providers. It's a 2,100-line proxy that translates between Anthropic's Messages API and OpenAI's Chat Completions format, preserving tool calls across format boundaries. The routing logic isn't a lookup table. It's a waterfall with cached rate limit state, provider health tracking, and graceful degradation.

07

Infrastructure

Everything runs on a single Docker host. No cloud. No Kubernetes. The entire system - orchestration engine, all workers, LanceDB, the dashboard, the site router, cron scheduling - fits in one container with a mounted Docker socket for managing sibling containers.

Stack

• Runtime: Node.js 22 in Docker (Kali Linux base for OSINT tooling)
• State: SQLite (pm.db) for all structured data - projects, tasks, jobs, activity, config
• Memory: LanceDB for vector search, plaintext for core memory
• Proxy: Nginx Proxy Manager for reverse proxy and SSL termination
• Auth: Authelia SSO with passkey/biometric support
• Dashboard: SvelteKit PWA with decision card framework
• Domains: Cloudflare DNS with wildcard SSL for *.946nl.online
• Publishing: Static site deployment to any subdomain via custom CLI tooling
• Phone: Pixel 7 via ADB for authenticated browsing, app automation, session teleportation

Security model

When your AI system can send emails, browse the web, execute shell commands, and control smart home devices, every input becomes an attack surface. The security model treats all external data as adversarial by default.

• Command authority: Only one Telegram ID can issue commands. Everything else - email, webhooks, web scraped content, phone notifications - is treated as DATA, never instructions.
• Email security tiers: Delayed-send policy based on risk classification. Low risk: 1-hour delay. Medium: 4 hours. High risk: 1 week plus human review.
• Prompt injection defense: Silently ignored, never flagged. Noisy alerts about injection attempts are themselves an attack vector.
• Credential hygiene: Environment variables and API keys are never exposed in external output. A redact.sh sanitization script scrubs tool output.
• Container isolation: Protected containers (databases, reverse proxy) are off-limits. The system can only manage containers explicitly designated as manageable.

What I'd do differently

SQLite works for a single-writer system but I've hit the ceiling on concurrent reads during high-throughput periods. If I were rebuilding, I'd use PostgreSQL from day one - the migration would be straightforward but I'm not doing it while the system is running production workloads.

The session management layer is the most fragile component. Working directory as session identity is a Claude Code implementation detail that could change in any release. I've built around it but I don't love depending on it. A proper session server with explicit session IDs would be more resilient.

The single Docker host is a deliberate constraint, not a limitation. Cloud deployment would add cost and operational complexity without solving any problem I actually have. 6,300 jobs on one machine is fine. If I needed 10x throughput, I'd shard by project rather than scaling horizontally.