Background Jobs with Elixir: Oban in the Real World

Background jobs are a staple in any seasoned engineer’s toolbelt. But in an ecosystem where OTP is readily available and concurrency is a first-class citizen, where exactly does a library like Oban fit? And more importantly, what happens when you hit the “scaling wall” of 10 million jobs a day?

Thinking of background processing at this magnitude requires a shift in strategy. It’s no longer just about using the library, it’s about leveraging advanced database techniques and the full power of Distributed Elixir (Erlang). Drawing from a production-proven implementation, we will explore:

• The “Why”: A critical comparison of Oban vs. pure GenServers vs. Broadway. When is the overhead of a database-backed queue a feature, and when is it a bottleneck?
• The “How”: Solving real-world challenges including strict rate-limiting for brittle external APIs, managing job distribution across a multi-node cluster, and handling dynamic scheduling (from crontabs to self-enqueueing recursive workers).
• The “Cost”: Tactical patterns for maintaining Postgres health under high-volume job churn, focusing on index bloat, pruning, and transactional integrity.