Celery Migration — Platform Async Substrate

Every piece of background work — indexing, exports, check-ins, scheduled jobs — ran through a legacy XMLRPC / scheduler jobserver that was a single point of fragility. When it crashed it left no logs. Jobs got stuck in the queue with no retry or cancellation. A lost database connection could take it down, and it had no concept of multi-tenant context, so a job had to be trusted to run against the right tenant by convention alone.

1. 01

   Designed and authored a managed Celery framework end-to-end — sole author across 42 files and roughly 10,800 lines — rather than bolting a queue onto the old jobserver.

2. 02

   Built multi-tenant context restoration into the task layer so every job reconstructs and runs inside the correct tenant context automatically, instead of relying on convention.

3. 03

   Made the workers self-healing: automatic reconnection to PostgreSQL on connection loss, and autorestart across the worker / search / AI containers so a transient fault never strands the queue.

4. 04

   Added a real task-lifecycle layer — scheduled-task logging, retry and cancellation — turning fire-and-forget jobs into ones you can observe, retry and stop.

5. 05

   Turned the old "crash with no logs" failure mode into a debuggable one by overhauling job-server logging first, which unblocked all the observability work that followed.

6. 06

   Dockerised the workers and hardened the serialization edges (a bytes/int leak and an exception-formatting mismatch surfacing on the long-tail error stream) so the system stayed quiet under real production load.

7. 07

   Ported 50+ legacy jobs onto the new substrate incrementally, so the migration shipped piece by piece rather than as one risky cutover.

The legacy jobserver was decommissioned and the platform's entire async workload moved onto the framework I designed. Two years on, I'm still its runtime owner — it's the reliability backbone every background process runs through.