9183

GCP-Storage: do files appear before upload is complete

<h3>Question</h3>

I want to transfer files into a VM whenever there is a new file added to storage, the problem is that i want the transfer to be done only when the upload is complete

So my question is : Do files appear even when the upload is still going on ? which means if I build a program that looks for new files every second, would it transfer the files from gcs to VM even if the upload is incomplete or the transfer would start whenever the upload is complete and not while it is uploading ?


<h3>Answer1:</h3>

Google Cloud Storage uploads are strongly consistent for object uploads. This means that the object is not visible until the object is 100% uploaded and any Cloud Storage housekeeping (such as replication) is complete. You cannot see nor access an object until the upload has completed and your software/tool receives a success response.

Google Cloud Storage Consistency

<blockquote>

Do files appear even when the upload is still going on ? which means if I build a program that looks for new files every second, would it transfer the files from gcs to VM even if the upload is incomplete or the transfer would start whenever the upload is complete and not while it is uploading ?

</blockquote>

No, your program will not see new objects until the new objects are 100% available. In Google Cloud Storage there are not partial uploads.


<h3>Answer2:</h3>

Files do not appear in the UI of Cloud Storage until the file is completely upload it to the specified bucket by the user.

I attached you how Google Cloud Platform manage the consistency in Google Cloud Storage Buckets here.

You could use gsutil to list all the files in one of your Cloud Storage Buckets at any moment, as stated here.

As for the application you are trying to develop, I highly suggest you to use Google Cloud Functions, in conjunction with triggers.

In this case, you could use google.storage.object.finalize trigger in order to execute your function every time a new object is uploaded to one of your buckets. You can see examples of this application here.

The Cloud Function will ensure that your bucket is correctly uploaded to the bucket before attempting to transfer the object to your GCE instance.

Therefore, after completing the upload, the only thing left will be to execute gcloud compute scp to copy files to your Google Compute Engine via scp, as stated here.

来源:https://stackoverflow.com/questions/59455769/gcp-storage-do-files-appear-before-upload-is-complete

Recommend

  • Loading multiple images in same UIImageView flashes last image before showing new image
  • c++ rvalue of moveable type in constructor
  • java.lang.NoClassDefFoundError: Failed resolution of: Lcom/Google/Android/gms/common/API/API$zzf;
  • React.default.memo is not a function (React-Native) wrapWithConnect
  • Why is [Windows::Foundation::Metadata::WebHostHidden] added by default in custom WinRT C++/CX contro
  • Azure DocumentDB Decimal Truncation
  • Boo to the Chocolatey Gods
  • How can I use Powershell as my tramp-encoding-shell?
  • Equivalent of “@section” in ASP.NET Core MVC?
  • What will be the best way to select Drop-down item in dojo based HTML in Java- Web-driver?
  • Transforming Calendar Quarter to Financial Quarter
  • Close expandable ListView on child click
  • Landscape mode in Qt-Android application
  • Making map! enumerator do what I want
  • How well does Entity Framework 6 support .NET 4.0?
  • Add chapter information to existing video with AVFoundation
  • How to find the corresponding key value from the string inserted into a QMap?
  • How to download bytea column as file using Java
  • forwarding MailItem Outlook Addinn issue
  • How to Enable Systemd service in openshift/jenkins-1-centos7 docker container?
  • Radio button in xamarin.ios
  • How to run python3.7 based flask web api on azure
  • SSRS Subscription to an email group (distribution list) is not working
  • Objects Sorting With date ,Time Problem in Array(Iphone Development)
  • Window-select multiple artists and drag them on canvas
  • Fixing corrupt encoding (with Python)
  • Transfer file trough SSH tunnel with Java
  • How to add multiple legend titles (columns) in ggplot
  • spring security manual login best practice
  • Misplaced CAGradientLayer on iPhone 6s
  • Wireshark Display Filter for Unique Source/Destination IP and Protocol
  • When to use the tag in the head and body section of a html page? [duplicate]
  • Multiplying polynomials/simplifying like terms
  • How to define something in JavaScript [closed]
  • Neo4j…how to get a visual representation of my data?
  • Support of :after in IE7
  • Amazon Elastick BeanStalk error: Failed to create the AWS Elastic Beanstalk application version
  • Cloud Code: Creating a Parse.File from URL
  • How to decleare char *const argv[] in swift [duplicate]
  • Grails - How to implement a foreign key relationship not using an id column?