54954

Unable to create Pipeline with PubsubIO.Read, Set up of resource … failed

Question:

My test pipeline is really simple. It attempts to read from a topic created via the console.

public static void main(String[] args) throws IOException { Options options = PipelineOptionsFactory.fromArgs(args). withValidation().as(Options.class); options.setStreaming(true); Pipeline pipeline = Pipeline.create(options); PCollection<String> input = pipeline .apply(PubsubIO.Read.topic(options.getPubsubTopic())) .apply(ParDo.of(new ExtractEvents())); pipeline.run(); }

When I attempt to execute this pipeline I get the following error:

<blockquote>

Workflow failed. Causes: (de5f777e2e08c1d9): Step setup_resource_additionaltopic.subscription-375367840492394866711: Set up of resource additionaltopic.subscription-3753678404923948667 failed

</blockquote>

The Dataflow console also reports an internal error:

<a href="https://i.stack.imgur.com/8NZJU.png" rel="nofollow"><img alt="Dataflow console error" class="b-lazy" data-src="https://i.stack.imgur.com/8NZJU.png" data-original="https://i.stack.imgur.com/8NZJU.png" src="https://etrip.eimg.top/images/2019/05/07/timg.gif" /></a>

I can't find anything in the documentation and my trial and error attempts at resolving this haven't been successful.

<strong>Solution</strong>

To run a Dataflow job, a project must enable the following Google Cloud Platform APIs:

<ul><li>Google Cloud Dataflow API </li> <li>Compute Engine API (Google Compute Engine)</li> <li>Google Cloud Logging API </li> <li>Google Cloud Storage </li> <li>Google Cloud Storage JSON API </li> <li>BigQuery API </li> <li>Google Cloud Pub/Sub </li> <li>Google Cloud Datastore API</li> </ul>

You can use the Google Cloud Platform Console to <a href="https://console.cloud.google.com/flows/enableapi?apiid=dataflow,compute_component,logging,storage_component,storage_api,bigquery,pubsub,datastore&redirect=https:%2F%2Fcloud.google.com%2Fdataflow%2Fgetting-started&showconfirmation=true" rel="nofollow">enable all</a> the required APIs at once.

Answer1:

It looks like your project doesn't have the Pubsub API enabled. Have you gone through the instructions in the getting started guide, especially the part on <a href="https://cloud.google.com/dataflow/getting-started#APIs" rel="nofollow">APIs</a>. There is a link there that should enable everything you need to get going.

You may also need to verify that the APIs and permissions are properly setup for the project that the Dataflow job is running in to access the Pubsub topic you are trying to subscribe to.

Recommend

  • Explode string except where surrounded by parentheses?
  • Constant embedded for loop condition optimization in C++ with gcc
  • Spark dataframe to nested map
  • ASP.NET MVC - Detect Time Spent on Page
  • How do I retrieve the user information of a user authenticated with Apache's mod_ldap?
  • Reduction and collapse clauses in OMP have some confusing points
  • MongoDb aggregation
  • How to use remove-erase idiom for removing empty vectors in a vector?
  • Java Scanner input dilemma. Automatically inputs without allowing user to type
  • How to show dropdown in excel using jrxml (jasper api)?
  • Importing jscolor library in angular 2
  • what is the difference between the asp.net mvc application and asp.net web application
  • Release, debug version and Authorization Google?
  • Websockets service method fails during R startup
  • Alternatives to the OPTIONAL fallback SPARQL pattern?
  • How to get next/previous record number?
  • Apache 2.4 - remove | delete | uninstall
  • Matrix multiplication with MKL
  • SQL merge duplicate rows and join values that are different
  • WPF Applying a trigger on binding failure
  • Proper way to use connect-multiparty with express.js?
  • Error creating VM instance in Google Compute Engine
  • Hits per day in Google Big Query
  • Trying to get generic when generic is not available
  • how does django model after text[] in postgresql [duplicate]
  • embed rChart in Markdown
  • How to get Windows thread pool to call class member function?
  • IndexOutOfRangeException on multidimensional array despite using GetLength check
  • LevelDB C iterator
  • Can't mass-assign protected attributes when import data from csv file
  • Busy indicator not showing up in wpf window [duplicate]
  • costura.fody for a dll that references another dll
  • Why is Django giving me: 'first_name' is an invalid keyword argument for this function?
  • Binding checkboxes to object values in AngularJs
  • Observable and ngFor in Angular 2
  • How to Embed XSL into XML
  • How can I use `wmic` in a Windows PE script?
  • UserPrincipal.Current returns apppool on IIS
  • Conditional In-Line CSS for IE and Others?
  • java string with new operator and a literal