74099

Can BigQuery be fast enough for real-time onsite request

<h3>Question</h3>

I'm looking into the possibility to use BigQuery and its API to do on-site queries depending on content that is viewed by our visitors. Therefore the response time is crucial. I have loaded a very simple structured dataset of 10k rows (4columns) and run a very simple query and that takes between 1 and 2 seconds. My questions is hopefully pretty simple to answer, will I ever be able to get a <1sec response time with the BQ API by optimising the data in someway or not?

Thanks a lot in advance!

Regards, Arjen


<h3>Answer1:</h3>

BigQuery is not an online transaction processing (OLTP) database but an interactive analysis database making possible to scan terabytes of data within seconds. While the query time is pretty consistent, since it is a shared service, the query time is not guaranteed, i.e. query running for 2 seconds might run 1.5 seconds or 3 seconds at different periods of time. Due to the nature and internals of BigQuery, query time of < 1s is not realistic as of today.

One of the popular design patterns is to let BigQuery do the heavy lifting of complex analysis of your data and then storing results in OLTP (like mySQL) or even in-memory (like Redis) database and serve the results to clients from there. You can periodically update the data by running the queries in the background.


<h3>Answer2:</h3>

BigQuery is designed for interactive analysis of MASSIVELY large datasets.
Processing time is expected within the seconds.
At the same time, expecting that running query against smaller dataset will bring the runtime under the second – is not realistic just because it does not design this way.
No matter what data size you have - you still “pay” all expenses as if you would process big dataset.

So, short answer unfortunatelly:
Today – Not! Tomorrow – Not! Day after – you never know, but I don’t think so, unless BigQuery will evolve into something else

来源:https://stackoverflow.com/questions/36228532/can-bigquery-be-fast-enough-for-real-time-onsite-request

Recommend

  • Using the universal selector (*) in a click event to select only that clicked and not parents
  • Can BigQuery be fast enough for real-time onsite request
  • Android/Cyanogenmod building: remove-project from roomservice.xml
  • Laravel phpunit returns 404
  • Play 2.2 application crashes on Heroku
  • Azure: Assign Roles via ARM Template to storage container
  • Run out of storage on Service Fabric scale set
  • Error: Command failed: C:\\Windows\\system32\\cmd.exe /s /c “rm ./mean/package.json”
  • Is it possible to specify 16 colums to bootstrap 3
  • How to do multi-class image classification in keras?
  • How to install PHP pthreads in cpanel?
  • How to extract details from the xml files using java?
  • Images tile on Google map in android
  • Android BottomNavigationView with Badge [closed]
  • How can I extend a Retrofit 2.0 Call?
  • Can the use of C++11's 'auto' deteriorate performance or even break the code?
  • Java anti fillRect (fill everything outside of said rectangle)
  • Is there a way to convert all existing table data to UTF8 collation?
  • How do I construct a local movie URL properly?
  • how to reload page after uploading images
  • iOS: Address Book
  • jQuery how to translate livequery to on?
  • Python List of Tuples (Find value with key + check if exist)
  • How to delete first 7 characters of folder name by using batch script?
  • Sorting Custom Listview Items Using Spinner Android
  • No Gradle file syntax highlighting in Eclipse Mars.2
  • How to convert days into months using datetime in Python3?
  • Creating 2d platforms using JavaScript
  • Cloud Code: Creating a Parse.File from URL
  • How to use FirstOrDefault inside Include
  • PHP Permalinks.. how to change?
  • Running R's aov() mixed effects model from Python using rpy2
  • ReferenceError: TextEncoder is not defined
  • Create/delete users from text file using Bash script