49556

How to handle AllServersUnavailable Exception

I wanted to do a simple write operation to a Cassandra instance (v1.1.10) on a single node. I just wanted to see how it handles constant writes and if it can keep up with the write speed.

pool = ConnectionPool('testdb') test_cf = ColumnFamily(pool,'test') test2_cf = ColumnFamily(pool,'test2') test3_cf = ColumnFamily(pool,'test3') test_batch = test_cf.batch(queue_size=1000) test2_batch = test2_cf.batch(queue_size=1000) test3_batch = test3_cf.batch(queue_size=1000) chars=string.ascii_uppercase counter = 0 while True: counter += 1 uid = uuid.uuid1() junk = ''.join(random.choice(chars) for x in range(50)) test_batch.insert(uid, {'junk':junk}) test2_batch.insert(uid, {'junk':junk}) test3_batch.insert(uid, {'junk':junk}) sys.stdout.write(str(counter)+'\n') pool.dispose()

The code keeps crushing after a long write (when the counter is around 10M+) with the following message

pycassa.pool.AllServersUnavailable: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was timeout: timed out

I set the queue_size=100 which didn't help. Also I fired up the cqlsh -3 console to truncate the table after the script crashed and got the following error:

Unable to complete request: one or more nodes were unavailable.

Tailing /var/log/cassandra/system.log gives no error sign but INFO on Compaction, FlushWriter and so on. What am I doing wrong?

Answer1:

I've had this problem too - as @tyler-hobbs suggested in his comment the node is likely overloaded (it was for me). A simple fix that I've used is to back-off and let the node catch up. I've rewritten your loop above to catch the error, sleep a while and try again. I've run this against a single node cluster and it works a treat - pausing (for a minute) and backing off periodically (no more than 5 times in a row). No data is missed using this script unless the error throws five times in a row (in which case you probably want to fail hard rather than return to the loop).

while True: counter += 1 uid = uuid.uuid1() junk = ''.join(random.choice(chars) for x in range(50)) tryCount = 5 # 5 is probably unnecessarily high while tryCount > 0: try: test_batch.insert(uid, {'junk':junk}) test2_batch.insert(uid, {'junk':junk}) test3_batch.insert(uid, {'junk':junk}) tryCount = -1 except pycassa.pool.AllServersUnavailable as e: print "Trying to insert [" + str(uid) + "] but got error " + str(e) + " (attempt " + str(tryCount) + "). Backing off for a minute to let Cassandra settle down" time.sleep(60) # A delay of 60s is probably unnecessarily high tryCount = tryCount - 1 sys.stdout.write(str(counter)+'\n')

I've added a complete gist here

Recommend

  • Change colour of vertices after they are uploaded using a VBO
  • Radio button show hover between two radio button using jquery
  • Django Rest Framework: getting lists from query_params without request.getlist
  • Simple command-line app I/O in Dart
  • Sweave + RweaveHTML: cat output does not appear in the output
  • Read stdin in chunks in Bash pipe
  • R h2o.glm - issue with max_active_predictors
  • get all processes in parallel
  • pygame.init() shows as undefined variable after installing Pygame
  • date format change with DT and shiny
  • Cannot invoke my method on the array type int[]
  • C function strchr - How to calculate the position of the character?
  • triggering user space with kernel
  • Bad interaction between Zope2 XML-RPC and AT Image mutator?
  • Problem with Django using Apache2 (mod_wsgi), Occassionally is “unable to import from module” for no
  • Getting short path in python
  • Redirect STDERR in OPEN pipe comand. Perl Linux
  • How VBA declared Volatility works
  • Calling Worksheet functions from vba in foreign language versions of Excel
  • Parse a date string in a specific locale (not timezone!)
  • Excel's Macro-Recorder usage
  • Transactional Create with Validation in ServiceStack Redis Client
  • Jetty Server not starting: Unable to establish loopback connection
  • Django: Count of Group Elements
  • Reading JSON from a file using C++ REST SDK (Casablanca)
  • Scrapy recursive link crawler
  • NetLogo BehaviorSpace - Measure runs using reporters
  • PHP - How to update data to MySQL when click a radio button
  • How to check if every primary key value is being referenced as foreign key in another table
  • Join two tables and save into third-sql
  • VBA Convert delimiter text file to Excel
  • How to get next/previous record number?
  • 0x202A in filename: Why?
  • using conditional logic : check if record exists; if it does, update it, if not, create it
  • sending mail using smtp is too slow
  • Busy indicator not showing up in wpf window [duplicate]
  • Why is Django giving me: 'first_name' is an invalid keyword argument for this function?
  • Reading document lines to the user (python)
  • How can I use `wmic` in a Windows PE script?
  • Python/Django TangoWithDjango Models and Databases