Luigi Programmatic Configuration

Question

I was using a configuration file similar to the following for my luigi workflows:

# Luigi logging configuration
[logging]
version = 1
disable_existing_loggers = false

[logging.formatters.simple]
format = "{levelname:8} {asctime} {module}:{lineno} {message}"
style = "{"
datefmt = "%Y-%m-%d %H:%M:%S"

# Luigi contrib configurations
[postgres]
marker-table="luigi_table_updates"

And then invoking luigi from the command like like this:

LUIGI_CONFIG_PATH='path/to/luigi.toml' luigi ...

However, I would like to programmatically invoke my workflows. I can do this using the luigi.build method; however, I tried passing in my config path like this:

luigi.build(my_tasks, local_scheduler=True, LUIGI_CONFIG_PATH='path/to/luigi.toml')

And got the following error:

luigi.parameter.UnknownParameterException: core[args=(), kwargs={'local_scheduler': True, 'LUIGI_CONFIG_PATH': 'path/to/luigi.toml', 'no_lock': True}]: unknown parameter LUIGI_CONFIG_PATH

I assumed using it as a kwarg would load it into the environment (luigi.builds env_params) similar to how local_scheduler is loaded.

So, my question is, what is the appropriate way to configure luigi programmatically?

score 3 · Answer 1 · answered Apr 30 '19 at 18:21

I figured this out; not sure if there's a better way but I found a method call that loads the config:

import luigi

luigi.configuration.add_config_path('path/to/my-config.cfg')

I did not found a way to specify the parser, however, and after reading the the source I'm not sure its possible at the moment. So, I just switched from TOML to cfg and all was well.

Luigi Programmatic Configuration

1 Answers1