43694

Celery not using all concurrent slots

Question:

I have a Celery cluster made up of machines with 8-core processors. Each machine has one worker that is set to a concurrency factor of 8 (-c8).

I often see nodes with a lot of reserved tasks, but only one or two are running simultaneously. My tasks are often long-running with a lot of compute and I/O.

Any ideas as to why this is happening, and what I can do to increase the number of tasks simultaneously running? Does celery throttle the number of active tasks based on system load? I looked through the documentation but came up short.

Answer1:

Thanks to <a href="https://stackoverflow.com/users/2801037/banana" rel="nofollow">banana</a>, I think I found the answer.

Some of my tasks were spawning subprocesses, which Celery counts in its concurrency.

Recommend

  • “This app could not be installed at this time” CFBundleIdentifier error
  • hiding rows in excel based on words in a cell
  • CKAN ERROR preview CSV , but OK for Download
  • Excel 2010 search for text in IF function - separate cell data
  • DELETE FROM … reporting syntax error at or near “.”
  • Error:Execution failed for task ':app:dexDebug'. when importing ParseLoginUI as a module
  • SQL Server 2005, Caches and all that jazz
  • Process.PrivateMemorySize64 returning committed memory instead of private
  • qt how to know that a pushbutton is clicked?
  • Attemping to change attributes of $(this) inside success function
  • Autohotkey script running program with command line arguments
  • get all processes in parallel
  • How to protect an asp:textbox from user input?
  • rapply over a nested list in R
  • Neo4j: Legacy Indexes and auto index vs new label bases schema indexes
  • Should I be afraid to use UDP to make a client/server broadcast talk?
  • How to add learning rate to summaries?
  • d3.js selection conditional rendering
  • Geom_jitter colour based on values
  • Regex for Specific Tag
  • Deploying a CodeRush plugin from the Community Site
  • Problem with Django using Apache2 (mod_wsgi), Occassionally is “unable to import from module” for no
  • How can we prepend rows to a react native list-view?
  • Shouldn't else be indented in the below code
  • How do I retrieve the user information of a user authenticated with Apache's mod_ldap?
  • Azure table store snapshot/backup capability
  • What's the purpose of QString?
  • How to run “Deployd” on port 80 instead of port 5000 in webserver.
  • Jackson Parser: ignore deserializing for type mismatch
  • With Hadoop, can I create a tasktracker on a machine that isn't running a datanode?
  • Content-Length header not returned from Pylons response
  • Play WS (2.2.1): post/put large request
  • Replace value with Factor in r data.table
  • Record samples being played with OpenAL
  • Avoid links criss cross / overlap in d3.js using force layout
  • How to access EntityManager inside Entity class in EJB3
  • vba code to select only visible cells in specific column except heading
  • ActionScript 2 vs ActionScript 3 performance
  • How can I estimate amount of memory left with calling System.gc()?
  • Apache 2.4 - remove | delete | uninstall