Question:

I have a Celery cluster made up of machines with 8-core processors. Each machine has one worker that is set to a concurrency factor of 8 (-c8).

I often see nodes with a lot of reserved tasks, but only one or two are running simultaneously. My tasks are often long-running with a lot of compute and I/O.

Any ideas as to why this is happening, and what I can do to increase the number of tasks simultaneously running? Does celery throttle the number of active tasks based on system load? I looked through the documentation but came up short.

Thanks to <a href="https://stackoverflow.com/users/2801037/banana" rel="nofollow">banana</a>, I think I found the answer.

Some of my tasks were spawning subprocesses, which Celery counts in its concurrency.