112

In my endless quest in over-complicating simple stuff, I am researching the most 'Pythonic' way to provide global configuration variables inside the typical 'config.py' found in Python egg packages.

The traditional way (aah, good ol' #define!) is as follows:

MYSQL_PORT = 3306
MYSQL_DATABASE = 'mydb'
MYSQL_DATABASE_TABLES = ['tb_users', 'tb_groups']

Therefore global variables are imported in one of the following ways:

from config import *
dbname = MYSQL_DATABASE
for table in MYSQL_DATABASE_TABLES:
    print table

or:

import config
dbname = config.MYSQL_DATABASE
assert(isinstance(config.MYSQL_PORT, int))

It makes sense, but sometimes can be a little messy, especially when you're trying to remember the names of certain variables. Besides, providing a 'configuration' object, with variables as attributes, might be more flexible. So, taking a lead from bpython config.py file, I came up with:

class Struct(object):

    def __init__(self, *args):
        self.__header__ = str(args[0]) if args else None

    def __repr__(self):
        if self.__header__ is None:
             return super(Struct, self).__repr__()
        return self.__header__

    def next(self):
        """ Fake iteration functionality.
        """
        raise StopIteration

    def __iter__(self):
        """ Fake iteration functionality.
        We skip magic attribues and Structs, and return the rest.
        """
        ks = self.__dict__.keys()
        for k in ks:
            if not k.startswith('__') and not isinstance(k, Struct):
                yield getattr(self, k)

    def __len__(self):
        """ Don't count magic attributes or Structs.
        """
        ks = self.__dict__.keys()
        return len([k for k in ks if not k.startswith('__')\
                    and not isinstance(k, Struct)])

and a 'config.py' that imports the class and reads as follows:

from _config import Struct as Section

mysql = Section("MySQL specific configuration")
mysql.user = 'root'
mysql.pass = 'secret'
mysql.host = 'localhost'
mysql.port = 3306
mysql.database = 'mydb'

mysql.tables = Section("Tables for 'mydb'")
mysql.tables.users = 'tb_users'
mysql.tables.groups =  'tb_groups'

and is used in this way:

from sqlalchemy import MetaData, Table
import config as CONFIG

assert(isinstance(CONFIG.mysql.port, int))

mdata = MetaData(
    "mysql://%s:%s@%s:%d/%s" % (
         CONFIG.mysql.user,
         CONFIG.mysql.pass,
         CONFIG.mysql.host,
         CONFIG.mysql.port,
         CONFIG.mysql.database,
     )
)

tables = []
for name in CONFIG.mysql.tables:
    tables.append(Table(name, mdata, autoload=True))

Which seems a more readable, expressive and flexible way of storing and fetching global variables inside a package.

Lamest idea ever? What is the best practice for coping with these situations? What is your way of storing and fetching global names and variables inside your package?

Rigel Di Scala
  • 2,728
  • 2
  • 15
  • 23
  • 3
    You already made a decision here that might or might not be good. The config itself can be stored in different ways, like JSON, XML, different grammars for *nixes and Windows and so on. Depending on who writes the config file ( a tool, a human, what background?) different grammars might be preferable. Most often it might not be a good idea to let the config file be written in the same language you use for your program, because it gives too much power to the user (what might be yourself, but you yourself might not remember everything that can go wrong some months ahead). – erikbwork Jun 01 '11 at 08:53
  • 4
    Often I end up writing a JSON config file. It can be read into python structures easily and also be created by a tool. It seems to have the most flexibility and the only cost are some braces that might be annoying to the user. I never wrote an Egg, though. Maybe that is the standard way. In that case just ignore my comment above. – erikbwork Jun 01 '11 at 08:55
  • 1
    You can use "vars(self)" instead of "self.__dict__.keys()" – Karlisson Dec 18 '13 at 15:19
  • 1
    Possible duplicate of [What's the best practice using a settings file in Python?](https://stackoverflow.com/questions/5055042/whats-the-best-practice-using-a-settings-file-in-python) They answer "Many ways are possible, and a bikeshed thread already exists. config.py is good unless you care about security." – Nikana Reklawyks Nov 21 '17 at 03:51
  • I ended up using `python-box`, see this [answer](https://stackoverflow.com/a/63071957/2137370) – evolved Jul 24 '20 at 10:58
  • I burst into laughter as I read "In my endless quest in over-complicating simple stuff..." – D_Serg Mar 04 '21 at 22:43

8 Answers8

61

How about just using the built-in types like this:

config = {
    "mysql": {
        "user": "root",
        "pass": "secret",
        "tables": {
            "users": "tb_users"
        }
        # etc
    }
}

You'd access the values as follows:

config["mysql"]["tables"]["users"]

If you are willing to sacrifice the potential to compute expressions inside your config tree, you could use YAML and end up with a more readable config file like this:

mysql:
  - user: root
  - pass: secret
  - tables:
    - users: tb_users

and use a library like PyYAML to conventiently parse and access the config file

ohaal
  • 5,050
  • 2
  • 31
  • 48
blubb
  • 8,562
  • 2
  • 36
  • 67
  • 1
    But normally you want to have different config files and thus not have any configuration data inside your code. So ´config´ would be an external JSON / YAML file which you have to load from disk every time you want to access it, in every single class. I believe the question is to "load once" and have global-like access to the loaded data. How would you do that with the solution you suggested? – omni Jun 28 '18 at 12:48
  • 3
    if just would something exist to keep the data in memory ^^ – cinatic Jul 31 '18 at 20:57
  • Do not forget to include the config file with something `from config import *` – Timo Dec 22 '20 at 20:14
21

I like this solution for small applications:

class App:
  __conf = {
    "username": "",
    "password": "",
    "MYSQL_PORT": 3306,
    "MYSQL_DATABASE": 'mydb',
    "MYSQL_DATABASE_TABLES": ['tb_users', 'tb_groups']
  }
  __setters = ["username", "password"]

  @staticmethod
  def config(name):
    return App.__conf[name]

  @staticmethod
  def set(name, value):
    if name in App.__setters:
      App.__conf[name] = value
    else:
      raise NameError("Name not accepted in set() method")

And then usage is:

if __name__ == "__main__":
   # from config import App
   App.config("MYSQL_PORT")     # return 3306
   App.set("username", "hi")    # set new username value
   App.config("username")       # return "hi"
   App.set("MYSQL_PORT", "abc") # this raises NameError

.. you should like it because:

  • uses class variables (no object to pass around/ no singleton required),
  • uses encapsulated built-in types and looks like (is) a method call on App,
  • has control over individual config immutability, mutable globals are the worst kind of globals.
  • promotes conventional and well named access / readability in your source code
  • is a simple class but enforces structured access, an alternative is to use @property, but that requires more variable handling code per item and is object-based.
  • requires minimal changes to add new config items and set its mutability.

--Edit--: For large applications, storing values in a YAML (i.e. properties) file and reading that in as immutable data is a better approach (i.e. blubb/ohaal's answer). For small applications, this solution above is simpler.

pds
  • 1,334
  • 1
  • 14
  • 15
12

How about using classes?

# config.py
class MYSQL:
    PORT = 3306
    DATABASE = 'mydb'
    DATABASE_TABLES = ['tb_users', 'tb_groups']

# main.py
from config import MYSQL

print(MYSQL.PORT) # 3306
Husky
  • 4,744
  • 2
  • 37
  • 38
  • I definitely prefer this approach as it works well with code completion in your IDE and you don't have to care about typos in string IDs. I'm not sure about the performance though. Works also pretty well when using inner classes to further separate entries. – Boketto Feb 03 '21 at 14:42
9

A small variation on Husky's idea that I use. Make a file called 'globals' (or whatever you like) and then define multiple classes in it, as such:

#globals.py

class dbinfo :      # for database globals
    username = 'abcd'
    password = 'xyz'

class runtime :
    debug = False
    output = 'stdio'

Then, if you have two code files c1.py and c2.py, both can have at the top

import globals as gl

Now all code can access and set values, as such:

gl.runtime.debug = False
print(gl.dbinfo.username)

People forget classes exist, even if no object is ever instantiated that is a member of that class. And variables in a class that aren't preceded by 'self.' are shared across all instances of the class, even if there are none. Once 'debug' is changed by any code, all other code sees the change.

By importing it as gl, you can have multiple such files and variables that lets you access and set values across code files, functions, etc., but with no danger of namespace collision.

This lacks some of the clever error checking of other approaches, but is simple and easy to follow.

eSurfsnake
  • 587
  • 5
  • 12
  • 1
    It's misadvised to name a module `globals`, since it's a built-in function which returns a dict with every symbol in the current global scope. In addition, PEP8 recommends CamelCase (with all capitals in acronyms) for classes (i.e. `DBInfo`) and uppercase with underscores for the so-called constants (i.e. `DEBUG`). – Nuno André Jun 29 '18 at 02:21
  • 1
    Thanks @NunoAndré for the comment, until I read it I was thinking that this answer does something strange with `globals`, author should change the name – oglop Mar 25 '20 at 15:57
  • This approach is my go to. I however see a lot of approaches that people say is "the best". Can you state some shortcomings to implementing config.py as this? – Yash Nag Apr 08 '20 at 17:45
  • This is [the global object pattern](https://python-patterns.guide/python/module-globals/), see also [the singleton pattern](https://python-patterns.guide/gang-of-four/singleton/) [as the article mentions the latter is less Pythonic] – Louis Maddox Apr 22 '21 at 13:33
9

Let's be honest, we should probably consider using a Python Software Foundation maintained library:

https://docs.python.org/3/library/configparser.html

Config example: (ini format, but JSON available)

[DEFAULT]
ServerAliveInterval = 45
Compression = yes
CompressionLevel = 9
ForwardX11 = yes

[bitbucket.org]
User = hg

[topsecret.server.com]
Port = 50022
ForwardX11 = no

Code example:

>>> import configparser
>>> config = configparser.ConfigParser()
>>> config.read('example.ini')
>>> config['DEFAULT']['Compression']
'yes'
>>> config['DEFAULT'].getboolean('MyCompression', fallback=True) # get_or_else

Making it globally-accessible:

import configpaser
class App:
 __conf = None

 @staticmethod
 def config():
  if App.__conf is None:  # Read only once, lazy.
   App.__conf = configparser.ConfigParser()
   App.__conf.read('example.ini')
  return App.__conf

if __name__ == '__main__':
 App.config()['DEFAULT']['MYSQL_PORT']
 # or, better:
 App.config().get(section='DEFAULT', option='MYSQL_PORT', fallback=3306)
 ....

Downsides:

  • Uncontrolled global mutable state.
pds
  • 1,334
  • 1
  • 14
  • 15
  • It's not useful to use .ini file if you need to apply if statements in your other files to change the configuration. It would be better to use config.py instead, but if the values don't change, and you just call and use it, I agree with the use of.ini file. – Lulu Dec 05 '19 at 01:08
  • Hmm, [configparser](https://docs.python.org/3/library/configparser.html) does have `read/write` and section `set/get` functionality. – pds Apr 22 '21 at 17:13
  • But yes, handling your config as a `dict` and then `json.dump(d)` and `write`, is (fewer steps, more common and) a more "modern" (current) way to manage config.... – pds Apr 22 '21 at 17:15
8

Similar to blubb's answer. I suggest building them with lambda functions to reduce code. Like this:

User = lambda passwd, hair, name: {'password':passwd, 'hair':hair, 'name':name}

#Col      Username       Password      Hair Color  Real Name
config = {'st3v3' : User('password',   'blonde',   'Steve Booker'),
          'blubb' : User('12345678',   'black',    'Bubb Ohaal'),
          'suprM' : User('kryptonite', 'black',    'Clark Kent'),
          #...
         }
#...

config['st3v3']['password']  #> password
config['blubb']['hair']      #> black

This does smell like you may want to make a class, though.

Or, as MarkM noted, you could use namedtuple

from collections import namedtuple
#...

User = namedtuple('User', ['password', 'hair', 'name']}

#Col      Username       Password      Hair Color  Real Name
config = {'st3v3' : User('password',   'blonde',   'Steve Booker'),
          'blubb' : User('12345678',   'black',    'Bubb Ohaal'),
          'suprM' : User('kryptonite', 'black',    'Clark Kent'),
          #...
         }
#...

config['st3v3'].password   #> passwd
config['blubb'].hair       #> black
Cory-G
  • 863
  • 12
  • 22
  • 3
    `pass` is an unfortunate variable name, since it is also a keyword. – Thomas Schreiter Dec 17 '14 at 00:21
  • Oh yeah... I just pulled together this dumb example. I will change the name – Cory-G Dec 18 '14 at 18:09
  • For this kind of approach, you might consider a class instead of the `mkDict` lambda. If we call our class `User`, your "config" dictionary keys would be initialized something like `{'st3v3': User('password','blonde','Steve Booker')}`. When your "user" is in a `user` variable, you can then access its properties as `user.hair`, etc. – Andrew Palmer Mar 29 '17 at 15:22
  • If you like this style you can also opt to use [collections.namedtuple](https://docs.python.org/3.6/library/collections.html#collections.namedtuple). `User = namedtuple('User', 'passwd hair name'); config = {'st3v3': User('password', 'blonde', 'Steve Booker')}` – MarkM May 01 '18 at 11:31
6

I did that once. Ultimately I found my simplified basicconfig.py adequate for my needs. You can pass in a namespace with other objects for it to reference if you need to. You can also pass in additional defaults from your code. It also maps attribute and mapping style syntax to the same configuration object.

Ali Nikneshan
  • 3,243
  • 23
  • 36
Keith
  • 37,985
  • 10
  • 48
  • 67
  • I know this is a few years old, but I'm a beginner and I think this config file is essentially what I am looking for (maybe too advanced), and I would like to understand it better. Do I just pass initialize `ConfigHolder` with a dict of configs I'd like to set and pass between modules? – Jinx Mar 04 '20 at 19:00
  • @Jinx At this point I would use (and am currently using) a YAML file and PyYAML for configuration. I also use a third-party module called `confit` and it supports merging multiple source. It's part of a new [devtest.config](https://github.com/kdart/devtest/blob/master/devtest/config.py) module. – Keith Mar 05 '20 at 01:50
3

please check out the IPython configuration system, implemented via traitlets for the type enforcement you are doing manually.

Cut and pasted here to comply with SO guidelines for not just dropping links as the content of links changes over time.

traitlets documentation

Here are the main requirements we wanted our configuration system to have:

Support for hierarchical configuration information.

Full integration with command line option parsers. Often, you want to read a configuration file, but then override some of the values with command line options. Our configuration system automates this process and allows each command line option to be linked to a particular attribute in the configuration hierarchy that it will override.

Configuration files that are themselves valid Python code. This accomplishes many things. First, it becomes possible to put logic in your configuration files that sets attributes based on your operating system, network setup, Python version, etc. Second, Python has a super simple syntax for accessing hierarchical data structures, namely regular attribute access (Foo.Bar.Bam.name). Third, using Python makes it easy for users to import configuration attributes from one configuration file to another. Fourth, even though Python is dynamically typed, it does have types that can be checked at runtime. Thus, a 1 in a config file is the integer ‘1’, while a '1' is a string.

A fully automated method for getting the configuration information to the classes that need it at runtime. Writing code that walks a configuration hierarchy to extract a particular attribute is painful. When you have complex configuration information with hundreds of attributes, this makes you want to cry.

Type checking and validation that doesn’t require the entire configuration hierarchy to be specified statically before runtime. Python is a very dynamic language and you don’t always know everything that needs to be configured when a program starts.

To acheive this they basically define 3 object classes and their relations to each other:

1) Configuration - basically a ChainMap / basic dict with some enhancements for merging.

2) Configurable - base class to subclass all things you'd wish to configure.

3) Application - object that is instantiated to perform a specific application function, or your main application for single purpose software.

In their words:

Application: Application

An application is a process that does a specific job. The most obvious application is the ipython command line program. Each application reads one or more configuration files and a single set of command line options and then produces a master configuration object for the application. This configuration object is then passed to the configurable objects that the application creates. These configurable objects implement the actual logic of the application and know how to configure themselves given the configuration object.

Applications always have a log attribute that is a configured Logger. This allows centralized logging configuration per-application. Configurable: Configurable

A configurable is a regular Python class that serves as a base class for all main classes in an application. The Configurable base class is lightweight and only does one things.

This Configurable is a subclass of HasTraits that knows how to configure itself. Class level traits with the metadata config=True become values that can be configured from the command line and configuration files.

Developers create Configurable subclasses that implement all of the logic in the application. Each of these subclasses has its own configuration information that controls how instances are created.

jLi
  • 123
  • 1
  • 7