23311

Handling long queries without violating REST

We have a REST api, and we've done a pretty good job at sticking to the spirit of REST. However, we have an important consumer, and they're requesting a way to reconcile their datastore. The flow works like this:

<ol> <li>Consumer makes a GET call to retrieve all inventory objects created within a date range. Lets say this returns 1 million inventory VINs.</li> <li>Consumer compares the payload with their own datastore, see's that they're missing 5,000 inventory objects</li> <li>Consumer would like to make a request with the 5,000 VIN id's, and return those 5,000 objects.</li> </ol>

The problem is that the long query string (JSON array of vins) bumps into the query string length limits imposed by our server. Possbile ideas - make 5k separate calls (seems horrible), increase querystring length limit on server (would like not to do this), use POST instead (not RESTful?).

So, I'm wondering what Roy Fielding would do...

Answer1:

What about a POST submitting the JSON file with the id's list to a new resource, e.g. called /inventory/difference?

If the computation goes any long, you can answer with 202 Accepted and the id of the resource being generated, then point back to it at /inventory/difference/:id.

Answer2:

Somewhat similar to what moonwave99 suggested, but instead you create a resource called a "set".

You POST to /set a list of identifiers that you wish to be in the set. The result of the POST is a redirect URL to the resource that names the specific set.

So:

POST /set

Result:

301 Moved Permanently Location: /set/123

Then:

GET /set/123

Returns the list of items in the set.

Sets are orthogonal to the use case of "fetching differences", they're simply a compilation of items.

If the creation of a set takes a long time, and you consider the set itself to be a snapshot of the data, when the user tries to do the GET /set/123 can simply reply with a 202 Accepted until the actual dataset has been completed.

You can then use:

GET /set/123/identifiers

To get a collection of the actual identifiers in the set, for example, if you like.

You can do something like

POST /setfromquery

and send a list of criteria (name like "John*", city = "Los Angeles", etc.). This doesn't really need its own specific resource, just define your query "language" to include both simple lists of IDs as well as perhaps other filter criteria.

Set operations (unions, differences, etc.). Lots of powerful things can be done with a set resource.

Finally, of course, there's the ever popular:

DELETE /set/123

Answer3:

I don't think anyone would fault you in working around GET not accepting a request body by using POST for a request that needs a request body. You are just being pragmatic.

I agree, making 5000 individual requests or upping the query string limit are ugly. POST is the way forward.

Answer4:

Using a post without creating a resource just seemed too dirty for me. In the end, we made it so that there was a limit of 100 ids requested in a "chunk". In practice, these requests will rarely be > 100, so hacking REST principles to accomodate an edge case seemed like a bad idea. I made sure the limitation was clearly defined in our API docs, done and done...

Recommend

  • What is the meaning of the term “Mocking” in C#? [closed]
  • Quartz trigger state is not persisting on server start
  • understanding REST Response and HTTP status code
  • How to refresh DataTables sort and filter info
  • Java Jersey RESTful web service using https
  • Why is bash so painfully slow in boot2docker for Windows?
  • Shake iPhone to open my app. Possible?
  • Rails - Redundant RESTFUL Actions for map.resources? (new, create)
  • What if the best way to return Option types by WCF service
  • How to add new collection criteria in Plone 5?
  • Process.StartTime Access Denied
  • Flask-Restful error: “as_view” method not inherited
  • Getting JavaScript runtime error: irrationalPath, what does it mean?
  • proxy request in node.js / express
  • OpenGL - Object Transformations and VBOs
  • What is this strange character in chrome's resource css viewer?
  • How to repeat sections of a SQL query across UNIONs? (DRY in SQL)
  • Write output of for loop to multiple files
  • Plotting densities in R
  • Consuming a WCF service in a Java Client using wsHttpBinding
  • How can I run DataNucleus Bytecode Enhancer from SBT?
  • How can I replace the server in Web Component Tester
  • Magento Fatal error: Maximum execution error solution, on WAMP
  • Unity3D & Android: Difference between “UnityMain” and “main” threads?
  • Using $this when not in object context
  • Fetching methods from BroadcastReceiver to update UI
  • Does CUDA 5 support STL or THRUST inside the device code?
  • How to limit post in wp_query
  • Jquery - Jquery Wysiwyg return html as a string
  • Delete MySQLi record without showing the id in the URL
  • PHP: When would you need the self:: keyword?
  • Python: how to group similar lists together in a list of lists?
  • Proper folder structure for lots of source files
  • Free memory of cv::Mat loaded using FileStorage API
  • Hits per day in Google Big Query
  • Understanding cpu registers
  • Memory offsets in inline assembly
  • Turn off referential integrity in Derby? is it possible?
  • How to get Windows thread pool to call class member function?
  • jQuery Masonry / Isotope and fluid images: Momentary overlap on window resize