Airflow is a platform to programmatically author, schedule, and monitor workflows.
Questions tagged [apache-airflow]
220 questions
0
votes
1 answer
How do I raise an error with Bash?
I'm trying to raise an error with Bash, in a block that's wrapped inside of a Python Airflow script.
export PATH=/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
export rundate=`TZ='America/Los_Angeles' date…
simplycoding
- 2,256
- 8
- 34
- 70
0
votes
1 answer
Ensure sequential running of tasks (Apache Airflow)
Under a sequential executor, I have a DAG file where I specify three tasks that are needed to be run sequentially (t1-->t2-->t3):
default_args = {
'owner': 'airflow',
'start_date': datetime(2017, 6, 14, 23 , 20),
'email_on_failure':…
Vvan_G
- 155
- 1
- 3
- 12
0
votes
1 answer
on the means of deploying the Apache Airflow platform
Could anybody please share experience/views on the means of deploying the Apache Airflow platform (in its simpest standalone mode using sequential executor) on the production server?
Vvan_G
- 155
- 1
- 3
- 12
0
votes
2 answers
airflow webserver -p 8080 result in OSError: [Errno 13] Permission denied
After installing airflow package in AWS EC2 instance , i am trying to start the airflow webserver Its showing permission denied issue , i am not getting which file or folder its trying to create/modify to get this error.
[root@ip-172-31-62-1…
Sanjay
- 51
- 1
- 6
0
votes
0 answers
Airflow Controller triggers Target DAG with a future execution date; Target DAG stalls
I have a Controller DAG (SampleController) that will call a Target DAG (SampleWait), both with a start_date of datetime.now() and a schedule_interval of None.
I trigger the Controller DAG from command line or the webserver UI, and it will run right…
jastang
- 316
- 4
- 13
0
votes
1 answer
How to manually run Airflow DAG on a particular directory
I am evaluating whether Airflow is suitable for my needs (in bioinformatics). I am having some difficulty with the Airflow model. Specifically:
Where does the DAG file actually get executed? What is its context? How can I pass input data into…
Uri Laserson
- 2,069
- 4
- 26
- 38
0
votes
1 answer
How Airflow connects to Spark and Is it possible to set a different access right to users in Airflow?
Hi I am still new to Airflow. I would like to know how Airflow connects to Spark?
I have a server that is with different accounts (or profiles) to access the Spark cluster. I wonder Does Airflow have a fixed profile to connect the spark? or It…
Iamasupernoob
- 83
- 1
- 4
0
votes
1 answer
how to make sure only one bashoperator is retried but not other airflow
I have dag file as follows. Here I have no retries. However, I want to make sure particular files ( bash1, bash2) is should have retries 1. but not others file.
Following is default arguments.
default_args = {
'owner': 'airflow',
…
Kush Patel
- 2,901
- 4
- 32
- 53
0
votes
1 answer
Installing Apache Airflow on Amazon Lightsail Instance
So i figured out the basics of Apache Airflow and I can run dags/tasks on my computer (so sleek!). However, I want to be able to have these run when my computer's off - so I bought a $5/month Lightsail instance and tried to install Airflow on there…
nerves
- 93
- 1
- 5
0
votes
0 answers
apache airflow scheduler not working properly
I've created some DAGs, and I tried to put it on scheduler. I want to run all the tasks in the DAG after exact 24 hours.
I tried to do something like this.
DEFAULT_ARGS = {
'owner' : 'mubin',
'depends_on_past' : False,
'start_date' …
Mubin
- 3,597
- 4
- 28
- 49