python (12.9k questions)

javascript (9.2k questions)

reactjs (4.7k questions)

java (4.2k questions)

java (4.2k questions)

c# (3.5k questions)

c# (3.5k questions)

html (3.3k questions)

Questions - airflow

Does airflow support the cloud-sql-python-connector for connecting to CloudSQL Postgres?

Airflow uses SQLAlchemy to connect to the DB: https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#database-uri Google has official support for connecting to CloudSQL via t...
test-img

red888

python

airflow

google-cloud-sql

Votes: 0

Answers: 1

Latest Answer

To answer your question, yes, you can install cloud-sql-python-connector in airflow. Tried it on my end as well. Below is the sample configuration on how to setup sql_alchemy_conn. def init_conne...
test-img

Catherine O

Airflow: Convert one digit to two digit with execution_date

I am trying to convert my airflow date from one digit to two digit. At the moment when I use {{ execution_date.month }} my month becomes 1 but would like it to become 01. I have tried with {{ executio...
test-img

Alex K

python

date

airflow

Votes: 0

Answers: 1

Latest Answer

You can do: from datetime import datetime from airflow.models import DAG from airflow.operators.bash import BashOperator with DAG( dag_id="bash_example", schedule_interval=None, ...
test-img

Elad Kalif

How to rename multiple hdfs file to some other file name with sequence number and then move it to another location

hdfs dfs -ls user/unknown_test_11 -rw-r--r-- 3 svcusprd hdfs 0 2022-01-20 08:14 user/unknown_test_11/_SUCCESS -rw-r--r-- 3 svcusprd hdfs 727288997334 2022-01-20 08:14 user/unknown_test_...
test-img

cookie

python

bash

pyspark

hdfs

airflow

Votes: 0

Answers: 1

Latest Answer

Assuming newpath exists. Otherwise create it first by hand or with os.mkdir. import os oldpath= r'path_to_old_folder' newpath= r'path_to_new_folder' files = os.listdir(oldpath) cnt = 0 for i in fi...
test-img

EndorCartographer

Airflow wrong log_id format

I'm using Airflow v2.2.3 and apache-airflow-providers-elasticsearch==2.1.0, running in Kubernetes. Our logs are automatically shipped to Elasticsearch v7.6.2. I set the following configurations in Air...
test-img

ItayB

elasticsearch

airflow

airflow-scheduler

Votes: 0

Answers: 1

Latest Answer

I found the problem. I have a separate pod for airflow-scheduler and airflow-webserver. I added the AIRFLOW__ELASTICSEARCH__JSON_FORMAT=True only to the airflow-scheduler + the workers but not to the ...
test-img

ItayB

Posts

Questions

Blogs

Jobs