At first sight, it seems that the way Sidekiq jobs work is straightforward, but the truth is that the whole process is more complex. The job can fall into a specific flow depending on the configuration and the job outcome.

This article introduces the Sidekiq job’s flow to help you understand what is happening with the job you queue when it’s successfully performed, or an error is raised during execution. Such knowledge will help you to design your systems better and use all features that Sidekiq offers.

The moment when it all starts

Before I describe each of the possible flows for Sidekiq’s job, it’s worth discussing the process of queueing a single job. It would call this step “queue job” and use this name later in the article just to not repeat over and over the same information that I will show here.

When you queue a job, there are three possible scenarios for it:

• delayed execution - the job will be executed as soon as there is a place in the queue
• scheduled execution - the job will be executed as soon as there is a place in the queue, but Sidekiq will perform the verification after some time that is specified when the job is queued
• immediate execution - Ruby won’t send the job to Sidekiq queue; it will be performed in the place where you execute it (for example, Rails console)

As you can see in the above image, the scheduled job will be placed into the Scheduled queue. When the exact time comes for that job, it will be moved to the Enqueued queue and then moved to the Busy queue as soon as there is a free space to process it.

It is impossible to add jobs directly to the Busy queue as you can’t exceed the concurrent jobs limit set in the configuration.

The success path

As you might expect, the success path is the shortest path for a job in Sidekiq. First, the job is queued, either starting from a scheduled or enqueued queue, and then it’s moved to the busy queue where it is executed. Finally, when the job is finished, the Processed counter is increased, and that’s all.

The success with errors path

It’s a perfect moment to tell you more about the retries queue. It works the same way as the scheduled queue; it contains jobs that have to be executed at a given time in the future. The main difference is that in retries queue, jobs are added by the Sidekiq, and the time when the job should be executed is calculated based on the retries counter. The logic for retrying job can be defined, or the default mechanism can be used.

When an error is raised, the Failed counter is increased, and the job is added to the retries queue. Then, it is moved again to the enqueued queue from this queue, and it’s waiting for the free spot. Finally, when the job is executed successfully, the processed counter is increased, and the job ends its cycle.

The failure path

In the previous chapter, I introduced the retries queue, and now it’s time to introduce the dead queue. The dead queue is where all failed jobs are stored but are not retried automatically. Instead, you can retry them manually, either from a web dashboard or using the API.

If you don’t want to retry the job, when it experiences an error, it’s moved directly to the dead queue. In such a situation, both processed and failed counters are increased.

If you want to retry the job when the error is raised, the failed counter is increased, and the job is moved to the retries queue. By default, with every retry, the time when the job will be executed again increases. When all defined retries are exhausted, the job is moved to the dead queue and won’t be queued again unless you retry it manually from there.

The next steps

The above flows are just basic ones. You can modify them with callbacks or the Sidekiq API, but a single job in Sidekiq will behave as I described unless you change the defaults.