Properly handling IOError thrown by logging.config.fileConfig?

Question

This may be an open ended or awkward question, but I find myself running into more and more exception handling concerns where I do not know the "best" approach in handling them.

Python's logging module raises an IOError if you try to configure a FileHandler with a file that does not exist. The module does not handle this exception, but simply raises it. Often times, it is that the path to the file does not exist (and therefore the file does not exist), so we must create the directories along the path if we want to handle the exception and continue.

I want my application to properly handle this error, as every user has asked why we don't make the proper directory for them.

The way I have decided to handle this can be seen below.

done = False
while not done:
    try:
        # Configure logging based on a config file
        # if a filehandler's full path to file does not exist, it raises an IOError
        logging.config.fileConfig(filename)

    except IOError as e:
        if e.args[0] == 2 and e.filename:
            # If we catch the IOError, we can see if it is a "does not exist" error
            # and try to recover by making the directories

            print "Most likely the full path to the file does not exist, so we can try and make it"
            fp = e.filename[:e.rfind("/")]

            # See http://stackoverflow.com/questions/273192/python-best-way-to-create-directory-if-it-doesnt-exist-for-file-write#273208 for why I don't just leap
            if not os.path.exists(fp):
                os.makedirs(fp)

        else:
            print "Most likely some other error...let's just reraise for now"
            raise
    else:
        done = True

I need to loop (or recurse I suppose) since there is N FileHandlers that need to be configured and therefore N IOErrors that need to be raised and corrected for this scenario.

Is this the proper way to do this? Is there a better, more Pythonic way, that I don't know of or may not understand?

I'm confused as to whether the non-existent file is the config file (passed to `fileConfig`, as in your source) or whether it's the name of the log file (as per the text of your question - "try to configure a FileHandler with a file that does not exist"). My answer was for the latter. Ordinarily, there's lots of things you need to ensure when specifying a file (whether a config file or a log file) - for example, a file may already exist but not be readable or writable because of permissions issues. There's no one-size-fits all solution, which is why you sometimes need to roll your own. — Vinay Sajip, Nov 15 '11 at 18:47
I am referring to the name of the log file, which I would need to fetch from the logging configuration file to check if it exists or not. It is inconvenient to have to fetch that information at this point in the code. I agree with you that doing this checks at start up would be ideal. Maybe my real question is why doesn't the logging module handle these types of exceptions for the user? — dicato, Nov 15 '11 at 18:59
Because the logging module expects the log file name it is given to be a writable file, and in a directory which exists. This behaviour is consistent with the principle of least surprise. A typical text editor (for example, Notepad on Windows or gedit on Linux), when invoked with a filename in a non-existent directory, will not automatically create the file for you. Notepad gives an error message; gedit does not save the file under that name, but prompts you for a location when you click "Save". — Vinay Sajip, Nov 15 '11 at 23:42

score 1 · Accepted Answer · answered Nov 14 '11 at 22:21

This is not something specific to the logging module: in general, Python code does not automatically create intermediate directories for you automatically; you need to do this explicitly using os.makedirs(), typically like this:

if not os.path.exists(dirname):
    os.makedirs(dirname)

You can replace the standard FileHandler provided by logging with a subclass which does the checks you need and, when necessary, creates the directory for the logging file using os.makedirs(). Then you can specify this handler in the config file instead of the standard handler.

So in order to handle an exception in the logging module properly, I need to subclass the module? While I agree the design is sound, it seems like a complicated way to handle an exception (or set of). — dicato, Nov 15 '11 at 14:14

score 0 · Answer 2 · answered Nov 14 '11 at 21:27

0

Assuming it only needs to be done once at the beginning of your app's execution, I would just os.makedirs() all the needed directories without checking for their existence first or even waiting for the logging module to raise an error. If you then you get an error trying to start a logger, you can just handle it the way you likely already did: print an error, disable the logger. You went above and beyond just by trying to create the directory. If the user gave you bogus information, you're no worse off than you are now, and you're better in the vast majority of cases.

answered Nov 14 '11 at 21:27

kindall

158,047
31
244
289

What concerns me with this approach is that I would need to parse, then iterate through the logging configuration file, check the args variable if it is a file handler, and take action. While this is not a challenge, it seems like a lengthy solution to a problem. Why should I need to do this? Why doesn't the logging module do this for me? – dicato Nov 14 '11 at 21:47
Hrm, yeah, I didn't notice at first that the filenames you were passing to the log file were those of logging config files rather than the log files. It looks like Vinay has a suggestion for doing this with a custom FileHandler which is probably a better way. – kindall Nov 14 '11 at 23:38
While Vinay's solution is certainly a good idea, do you feel that it is overkill for this problem? – dicato Nov 15 '11 at 15:11
No, it seems to me to be the way such an issue was intended to be handled. The file handler appears to be the class that's raising the exceptions you're catching in your example code, so if you could substitute your own file handler that created the directories as needed, that seems pretty surgical to me. – kindall Nov 15 '11 at 19:44
Surgical, yes, but think of a novice developer simply trying to use the logging module. That's a bit involved for a novice. Albiet, it will probably be the way I go. – dicato Nov 15 '11 at 19:54

Properly handling IOError thrown by logging.config.fileConfig?

2 Answers2