apache airflow variables on startup

Question

I'm learning Airflow and am planning to set some variables to use across different tasks. These are in my dags folder, saved as configs.json, like so:

{
  "vars": {
    "task1_args": {
      "something": "This is task 1"
    },
    "task2_args": {
      "something": "this is task 2"
    }
  }
}

I get that we can enter Admin-->Variables--> upload the file. But I have 2 questions:

1. What if I want to adjust some of the variables while airflow is running? I can adjust my code easily and it updates in realtime but it doesn't seem like this works for variables.
2. Is there a way to just auto-import this specific file on startup? I don't want to have to add it every time I'm testing my project.

I don't see this mentioned in the docs but it seems like a pretty trivial thing to want.

Answer 1

What you are looking for is With code, how do you update an airflow variable?

---

Here's an untested snippet that should help

from airflow.models import Variable

Variable.set(key="my_key", value="my_value")

• So basically you can write a bootstrap python script to do this setup for you.
• In our team, we use such scripts to setup all Connections, and Pools too

---

In case you are wondering, here's the set(..) method from source

@classmethod
@provide_session
def set(
    cls,
    key: str,
    value: Any,
    serialize_json: bool = False,
    session: Session = None
):
    """
    Sets a value for an Airflow Variable with a given Key
    :param key: Variable Key
    :param value: Value to set for the Variable
    :param serialize_json: Serialize the value to a JSON string
    :param session: SQL Alchemy Sessions
    """

    if serialize_json:
        stored_value = json.dumps(value, indent=2)
    else:
        stored_value = str(value)

    Variable.delete(key, session=session)
    session.add(Variable(key=key, val=stored_value))
    session.flush()