57409

Simple DB query on Google App Engine taking a lot of CPU time

Question:

I'm fairly new to Google App Engine and Python, but I did just release my first real-world site with it. But now I'm getting problems with one path that is using significantly more CPU (and API CPU) time than the other paths. I've narrowed it down to a single datastore fetch that's causing the problem: Carvings.all().fetch(1000)

Under the App Engine dashboard it's reporting "1040cpu_ms 846api_cpu_ms" pretty reliably for each request to that path. It has seemed like this may be the source to some unresponsiveness that my client has experienced with the site in general.

So I can't figure out what is so expensive about this query. Here is the related data model:

class Carving(db.Model): title = db.StringProperty(required=True) reference_number = db.StringProperty() main_category = db.StringProperty() sub_category = db.StringProperty() image = db.ReferenceProperty(CarvingImage) description = db.TextProperty() price = db.FloatProperty() size = db.StringProperty() material = db.StringProperty() added_at = db.DateTimeProperty(auto_now_add=True) modified_at = db.DateTimeProperty(auto_now=True)

In other places in the app when I pull this model from the datastore I do more filtering and I guess that's why they aren't causing any troubles. But the total number of entities for this model is just above 90 and I just can't imagine why this is so expensive.

Answer1:

<ul><li>

Memcache, if you haven't already, and especially if the same carvings are going to be fetched again and again. If you only have 90 total, I would imagine they would all be in the cache pretty quickly, and then you should be golden.

</li> <li>

Do you need all the properties of the Carvings? For example, if you're just displaying a list of carvings, you could have a separate Entity that was something like CarvingSummary that only had a few properties. This would mean your schema was denormalized, but sometimes that's the price you pay for speed.

</li> </ul>

Also, I'm assuming this is not the first page the user will always hit? If that were the case it could be the cloud spinning up a a new instance.

Answer2:

Sometimes you'll get better performance if you do an indexed query, rather than a query of "all" elements in the model.

Also, consider using memcache.

Answer3:

Do you actually need 1000 entities? CPU time goes up more or less linearly with the number of results retrieved, so if you don't actually need all the results, you may be wasting a lot of time fetching and decoding them.

Answer4:

It could be the image (and/or Text property) that is taking time to load & marshall into objects, depending on how big those properties are.

First prize: just use the memcache as others say. Then the overhead is incurred only on the first hit.

Second prize: I'm not sure how often your images are being changed and how many you might have, but you could consider uploading them as static files and simply linking to them in your HTML. Then it'd be just an HTTP GET from the browser - much lower overhead.

Recommend

  • Using embeds on a child association to improve per
  • Tableview ChangeListener Conflict
  • how to add the slugified field
  • SelfReferenceProperty question
  • Do Vue watched properties cache just like the computed properties?
  • Is array in java virtually sequential memory data structure? or physically sequential?
  • Getting The Most Recent Data Item - Google App Engine - Python
  • TSQL remap big number to smaller, but keep Identity
  • Optimize sum of powers with constant double exponent
  • Order of data in bigquery repeated records
  • Jquery filter with multiple criteria
  • Optimizing big import in php
  • Is there a more efficient LINQ statement to reverse-search for a condition in a List?
  • Why we should close the connection in JDBC? If we don't do it, what will happen
  • Fastest way to decode a hexadecimal digit
  • what is for…in statement in javascript
  • GData Youtube : obtaining thumbnails
  • Cache is client side or server side?
  • Exception handling as per java coding standards
  • Show records ordered with maximum price first in PHP & MySQL
  • Open Existing DB in MySQL WorkBench
  • How can I extract results of aggregate queries in slick?
  • Needing to do .toArray() to get output of mongodb .find() on key name not value
  • MongoDB in PHP using aggregate to group by _id is null not working
  • Display issues when we change from one jquery mobile page to another in firefox
  • java.lang.NoClassDefFoundError: com.parse.Parse$Configuration$Builder on below Lollipop versions
  • Resize panoramic image to fixed size
  • To display the title for the current loaction in map in iphone
  • Codeigniter doesn't let me update entry, because some fields must be unique
  • trying to dynamically update Highchart column chart but series undefined
  • How to stop GridView from loading again when I press back button?
  • Turn off referential integrity in Derby? is it possible?
  • Bitwise OR returns boolean when one of operands is nil
  • sending mail using smtp is too slow
  • MATLAB: Piecewise function in curve fitting toolbox using fittype
  • Does armcc optimizes non-volatile variables with -O0?
  • Busy indicator not showing up in wpf window [duplicate]
  • Why is Django giving me: 'first_name' is an invalid keyword argument for this function?
  • Binding checkboxes to object values in AngularJs
  • How can I use `wmic` in a Windows PE script?