31423

Communication between workers in uwsgi

I'm relatively new to Python, coming from a .Net background.

The short version: How can I create an application-wide singleton or some other mechanism to allow multiple threads/processes to communicate with each other?

Maybe I'm spoilt but in .Net, I'd just create something in App_Start or something else at the "application level". How can I do the same in python/uwsgi?

The long version:

We've got a restful API written using Django.

Some of the calls require some pre-processing and then passing to a back-end system which performs long-running operations.

The flow at the moment looks something like...

    <li>Receive request to process all documents matching given criteria</li> <li>Api determines which documents match those criteria (on the order of 100,000 - takes 15-20s)</li> <li>Api generates a uuid for this batch request</li> <li>Api publishes a message to a back-end queue for each of those documents, referencing the batch id.</li> <li>Api listens on a different queue for "completed" messages and counts success/failures for each batch id (~1-15 minutes)</li> <li>While processing is happening, UI can request an update for a specific batch id</li> </ul>

    We need to listen to the response queue using a different thread to that which is used to serve pages since it's in a wait spin-loop...

    while True: self.channel.wait()

    I was handling this by getting a reference to a QueueManager which is a singleton. The manager fires off the initial request, records the batch id and then, in a second thread, monitors the queue and updates local state.

    We don't actually care about preserving the state long-term - If the messages are on the queue, the processing will be done by the back-end and the state monitoring is only a cue to the user that things are underway. If they browse away, they also lose access to the status (batch id is stored in-memory in JS)

    This had a couple of benefits - we avoided using a database for syncing information (and the associated cleanup). We were able to use a single thread as a message consumer and didn't have to worry about concurrency issues as only one thread will ever collect messages/update the local state.

    So... Now it's time to run it using uwsgi I've found a major issue. If I set the number of processes to 1, the the singleton works as expected, but all requests are blocked during the 15-20s that the api is processing data. Clearly that's unacceptable. Conversely, if I spin up multiple workers, each has its own singleton and its own message listener - so it's pretty much random if the publisher and consumer are the same process. And even if they are, the request for a status update probably won't end up at that same process.

    How can I swap state information between multiple workers? Is there a way to use multiple threads instead of multiple workers?

    It seems like I really need:

      <li>n threads, each serving requests</li> <li>1 thread listening on the queue</li> <li>some in-memory way of communicating between them</li> </ul>

      Note I've already got --enable-threads but that only seems to apply to new threads I spawn (no idea why that wouldn't be enabled by default)

      Answer1:

      To spawn multiple threads just add --threads N where N is the number of threads to spawn.

      Pay attention, it will works only if you have a single worker/process. Another common approach is using the uWSGI caching framework (the name is misleading, infact is a shared dictionary). It will allows you to share data between workers and threads.

Recommend

  • Service discovery using dns-sd
  • Task processing status with a message queue
  • How to override JAXB @XMLAccessorType(XMLAccessType.FIELD) specified at a Class level with @XMLEleme
  • Facebook Open Graph Story Custom Actions Keep Getting Rejected - Advice Please?
  • finding symmetric difference/unique elements in multiple arrays in javascript
  • Aspect advising other aspects
  • Mocking Non-Standard Events in F# Foq
  • Error “Reflect.defineMetadata” while trying to load a transient web worker
  • How to replace TouchesBegan with UIGestureRecognizer
  • Extracting individual digits from a float
  • Copying rows in a database when rows have children
  • Asynchronous Sockets - Handling false socket.AcceptAsync values
  • Spring Integration Bridge with poller not working as expected for JMS
  • Find longest path less than or equal to given value of an acyclic, directed graph in Python
  • Cuda Clang and OS X Mavericks
  • Exception gevent.hub.LoopExit: LoopExit('This operation would block forever',)
  • How to make Twilio api Post request with the help of AFNetworking?
  • Is there a way to do normal logging with EureakLog?
  • Replace value with Factor in r data.table
  • Could not find rake using whenever rails
  • NHibernate Validation Localization with S#arp Architecture
  • How can I send an e-mail from a vbs script
  • Splitting given String into two variables - php
  • Align navbar back button on right side
  • Accessing IRQ description array within a module and displaying action names
  • How to add a column to a Pandas dataframe made of arrays of the n-preceding values of another column
  • Menu Color Fade on Hover with Jquery
  • How to apply VCL Styles to DLL-based forms in Inno Setup?
  • Build own AppleScript numerical error handling
  • Comma separated Values
  • json Serialization in asp
  • Rails 2: use form_for to build a form covering multiple objects of the same class
  • How can I get HTML syntax highlighting in my editor for CakePHP?
  • How do I configure my settings file to work with unit tests?
  • Getting Messege Twice Using IMvxMessenger
  • How to stop GridView from loading again when I press back button?
  • Turn off referential integrity in Derby? is it possible?
  • Add sale price programmatically to product variations
  • Unable to use reactive element in my shiny app
  • How do I use LINQ to get all the Items that have a particular SubItem?