55313

airflow: how can i put the method for read a json file in a local library

<h3>Question</h3>

I must generate some dag. I've saved the json table schema files on GCP bucket. The files on the GCP bucket associates to <em>composer</em> will be remapped on <em>/home/airflow/gcs/dags/</em>. If i define the method for read the json file, after the creation of the dag, all goes fine. But if I wish generate some "common code" (for put it on a library of mine), I can't access to FileSystem using the code in the library, in the specific I can't use the python json library.

The strange thing is that, I define the method out of the dag creation step, but I invoke it only after the dag creation!

To complete the discussion, i haven't problems if the code in the library uses only in memory objects.

I've this issue when i work with airflow (1.9 on GCP driver by composer)

This is my external library:

lib/ __init__.py bb_airflow_utils.py on external library

<pre class="lang-py prettyprint-override">def load_json_file(fname): #per far sì che il dag la veda with open(fname, 'r') as f: d = json.load(f) return d

on principal script

<pre class="lang-py prettyprint-override"> from lib.bb_airflow_utils import * ROOT_PATH = 'home/airflow/gcs/dags' IDCLI = 'goofy' ... ... with DAG(dag_id=dag_name, default_args=dag_args) as dag: filepath = path.join(ROOT_PATH, '{}-todwh.json'.format(IDCLI)) get_data = load_json_file(filepath) ..... task_dummy_start = DummyOperator(task_id='task_{}_start'.format(dag_name), dag=dag) .....

Airflow ignore the operator and by UI said that the dag has not SLA


<h3>Answer1:</h3>

Have a look at https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#install-local.

You can put common code in a separate file and put it in separate folder like the example below.

Place the dependencies within a subdirectory in the dags/ folder. To import a module from a subdirectory, each subdirectory in the module's path must contain a __init__.py package marker file.

In this example, the dependency is coin_module.py:

dags/ use_local_deps.py # A DAG file. dependencies/ __init__.py coin_module.py

Import the dependency from the DAG definition file.

For example:

from dependencies import coin_module

来源:https://stackoverflow.com/questions/55453398/airflow-how-can-i-put-the-method-for-read-a-json-file-in-a-local-library

Recommend

  • Issue a POST request with url_for in Flask
  • Neo4j Cypher WITH is required between CREATE and MATCH
  • web scraping in R
  • Spotfire Custom Expression : Calculate (Num/Den) Percentages
  • Eloquent Javascript: Can't understand how the number value is determined in the sum function
  • Putting multiple films in a circle in Raphael/Joint.js
  • Reading large text file very slow
  • Extract the data specified in brackets '[ ]' from a string message in python
  • Existing data serialized as hash produces error when upgrading to Rails 5
  • JSF validateLength question
  • Coin change recursive approach
  • In Moment.js, how do you get the date of the next occurrence of a specific month (ex: 'next Jan
  • What is the reason for using single-character generic type names in java?
  • Change selectOneMenu stylesheet (Primefaces)
  • Can I switch the 'connected' user within an sql script that is sourced by mysql?
  • iOS - open another application that exists on the phone?
  • Python tk scrollbar becomes inactive once text is outside the screen
  • Special characters (ë) in JSON-response
  • Finding all XML nodes between each two processing instructions
  • using JQuery autocomplete from database with PHP (CodeIgniter)
  • How to display youtube video on HTML page? [closed]
  • Auto send email based on the time and email address in database
  • Excel Reverse Match
  • Signed Java web start application with Glassfish 4.1 and Java7
  • How can I filter an array of dictionaries in 'updateSearchResultsForSearchController' to s
  • WiX ManagedBootstrapper SetDownloadSource confusion
  • GTK3 + OpenGL for windows: multiple GtkGLArea (ie. OpenGL contexts)
  • JQuery Mobile Ajax Navigation in Single-Page Template
  • How to use AJAX to upload large CSV file? [closed]
  • Creating 2d platforms using JavaScript
  • How to write seo friendly url's using htaccess?
  • VS2010 RDLC C#. How can I set a LocalReport object to a ReportViewer?
  • Is there a better way for handling SpatialPolygons that cross the antimeridian (date line)?
  • Google Spreadsheet Script to Blink a range of Cells
  • Cloud Code: Creating a Parse.File from URL
  • how to snap two objects in runtime in unity?
  • Another “Cannot make static reference…” Question
  • How to mutate multiple variables without repeating codes?
  • convert json to excel in java
  • How to check if object is null in Java?