Import arbitrary python source file. (Python 3.3+)

Question

How can I import an arbitrary python source file (whose filename could contain any characters, and does not always ends with .py) in Python 3.3+?

I used imp.load_module as follows:

>>> import imp
>>> path = '/tmp/a-b.txt'
>>> with open(path, 'U') as f:
...     mod = imp.load_module('a_b', f, path, ('.py', 'U', imp.PY_SOURCE))
...
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>

It still works in Python 3.3, but according to imp.load_module documentation, it is deprecated:

Deprecated since version 3.3: Unneeded as loaders should be used to load modules and find_module() is deprecated.

and imp module documentation recommends to use importlib:

Note New programs should use importlib rather than this module.

What is the proper way to load an arbitrary python source file in Python 3.3+ without using the deprecated imp.load_module function?

Can I ask why you are doing this? I'm the maintainer of importlib and I have been trying to get answers from folks as to why they use `imp.load_module()` over a straight import statement. Do you expect to import the module by name later (e.g. `import a_b`)? Do you care that any custom importers won't be used in this approach? Do you expect the module to be full-featured (e.g. define `__name__` and `__loader__`)? — Brett Cannon, Mar 31 '14 at 15:49
@BrettCannon, A third-party program regularly (once a hour) modify a text file that contains python statements (mainly `THIS='blah'` like lines). The name of the file is not ended with `.py`. My program read that file. — falsetru, Mar 31 '14 at 15:58
@BrettCannon, I'm not aware of custom importers. I don't care the module to be full-featured. — falsetru, Mar 31 '14 at 15:59
IOW using Python as a really simple data structure format. Thanks for the info! — Brett Cannon, Mar 31 '14 at 16:33
@BrettCannon — I just ran into a case where I needed to import some Python code from within a directory which was named as a version number (e.g., "v1.0.2"). While possible, it would be highly undesirable to rename the directory. I wound up using stefan-scherfke's solution below. — Andrew Miner, Feb 27 '18 at 20:16
Related: https://stackoverflow.com/questions/2601047/import-a-python-module-without-the-py-extension/56090741#56090741 — Ciro Santilli新疆棉花TRUMP BAN BAD, May 11 '19 at 13:25
@BrettCannon: [Enaml](https://github.com/nucleic/enaml) is a use-case where one needs to use a different extension (in this case, `*.enaml`). Enaml is a superset of Python that allows for declarative markup (useful for creating responsive GUIs). Enaml runs with Python and has it's own import hooks (e.g., `enaml.import_hooks`) to allow for loading Enaml files in Python programs. Importing Enaml files typically requires doing so within an Enaml context manager (e.g., `with enaml.imports(): import ...`). However, sometimes we want to use `importlib` to load from an arbitrary Enaml source file. — Brad, Feb 19 '20 at 17:13
Does this answer your question? [Import a python module without the .py extension](https://stackoverflow.com/questions/2601047/import-a-python-module-without-the-py-extension) — Ciro Santilli新疆棉花TRUMP BAN BAD, Oct 22 '20 at 09:02
@CiroSantilli郝海东冠状病六四事件法轮功, Answers there mostly focused on python 2.x solution (imp), but I wanted solution that works Python 3.3+. Answers there that solve my question come after my own answer. (2013 vs 2016,2017,2019) [This one](https://stackoverflow.com/a/19011259/2225682) answered my question. — falsetru, Oct 22 '20 at 13:57

falsetru · Accepted Answer · 2020-01-17T03:08:47.170

89

Found a solution from importlib test code.

Using importlib.machinery.SourceFileLoader:

>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = loader.load_module()
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>

NOTE: only works in Python 3.3+.

UPDATE Loader.load_module is deprecated since Python 3.4. Use Loader.exec_module instead:

>>> import types
>>> import importlib.machinery
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> mod = types.ModuleType(loader.name)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b'>

>>> import importlib.machinery
>>> import importlib.util
>>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt')
>>> spec = importlib.util.spec_from_loader(loader.name, loader)
>>> mod = importlib.util.module_from_spec(spec)
>>> loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>

edited Jan 17 '20 at 03:08

answered Sep 25 '13 at 17:17

falsetru

314,667
49
610
551

26

Downvoter: How can I improve the answer? If you have a better way to accomplish what I want, please let me know. – falsetru Nov 08 '13 at 08:56
3

There's a helpful warning that `load_module` ignores via `warnings.catch_warnings`. If you instead use `mod = imp.load_source('a_b', '/tmp/a-b.txt')`, it raises the following warning (use `-Wall`): `DeprecationWarning: imp.load_source() is deprecated; use importlib.machinery.SourceFileLoader(name, pathname).load_module() instead`. – Eryk Sun Feb 18 '14 at 04:00
1

@eryksun, You're right. Thank you for the comment. BTW, Python 3.4(rc1) does not display the alternative usage unlike Python 3.3.x. – falsetru Feb 18 '14 at 07:46
What's the difference between the first and the second example at the bottom? – Matthew D. Scholefield May 11 '18 at 21:15
@MatthewD.Scholefield ways to get module objects are different. Using Module type directly or using utility. – falsetru May 12 '18 at 01:02
@falsetru - great work. Would be good to add that `SourceFileLoader` only understands pure Python source modules. For loading extension modules (.so / .dll files) with an arbitrary suffix, the sibling class `ExtensionFileLoader` is the way to go :) – mxxk Aug 26 '18 at 08:36
Howto pass arguments in module constructor? – e-info128 Jan 02 '19 at 18:52
@e-info128 What is module constructor? – falsetru Jan 06 '19 at 12:52
@falsetru I know this deviates a little from OP's question but, what if I have a (for example) class implementation that is held inside a string instead of in a file. Ex: `my_class = """class Test: def __init__(self): self.x = 5 def print_number(self): print(self.x)"""` Is there a way to programmatically import this class inside the string to the namespace? – ihavenoidea Nov 26 '20 at 16:00
1

@ihavenoidea, Please post a separated question, so that others can answer you, and other users can also read answers. – falsetru Nov 26 '20 at 16:24
1

@falsetru In fact I have [here](https://stackoverflow.com/questions/65009309/dynamically-import-module-from-memory-in-python-3-using-hooks), no answer so far. I posted here in the comments because I bumped into your answer after posting my question. If you know how to do this, I'd appreciate it! – ihavenoidea Nov 26 '20 at 16:39
I can't get this to load the source code from a class. Can you help me with my question here: https://stackoverflow.com/q/67663614/11343425 ? Thanks – SurpriseDog May 23 '21 at 19:52

score 26 · Answer 2 · edited May 30 '19 at 04:36

26

Shorter version of @falsetru 's solution:

>>> import importlib.util
>>> spec = importlib.util.spec_from_file_location('a_b', '/tmp/a-b.py')
>>> mod = importlib.util.module_from_spec(spec)
>>> spec.loader.exec_module(mod)
>>> mod
<module 'a_b' from '/tmp/a-b.txt'>

I tested it with Python 3.5 and 3.6.

According to the comments, it does not work with arbitrary file extensions.

edited May 30 '19 at 04:36

falsetru

314,667
49
610
551

answered Jan 11 '17 at 16:09

Stefan Scherfke

2,390
1
14
23

2

`importlib.util.spec_from_file_location(..)` returns `None` for me; causing an exception for the following `importlib.util.module_from_spec(..)` call. (See https://i.imgur.com/ZjyFhif.png) – falsetru Jan 12 '17 at 06:04
3

`importlib.util.spec_from_file_location` works for known file name extensions (`.py`, `.so`, ..), but not for others (`.txt`...) – falsetru Jan 13 '17 at 05:29
Oh, I’m using it only with Python files but modified my example to look like the one above and did not test it … I updated it. – Stefan Scherfke Jan 13 '17 at 06:32

score 12 · Answer 3 · answered Jul 28 '18 at 23:14

Similar to @falsetru but for Python 3.5+ and accounting for what the importlib doc states on using importlib.util.module_from_spec over types.ModuleType:

This function [importlib.util.module_from_spec] is preferred over using types.ModuleType to create a new module as spec is used to set as many import-controlled attributes on the module as possible.

We are able to import any file with importlib alone by modifying the importlib.machinery.SOURCE_SUFFIXES list.

import importlib

importlib.machinery.SOURCE_SUFFIXES.append('') # empty string to allow any file
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# if desired: importlib.machinery.SOURCE_SUFFIXES.pop()

Interestingly enough, while this hack of appending the empty string to the list of source suffixes works great for importing renamed Python source modules, the equivalent for importing renamed extension modules does not work... That is, using `importlib.machinery.EXTENSION_SUFFIXES.append('')` still makes `importlib.util.spec_from_file_location` return None. — mxxk, Aug 26 '18 at 08:28
presumably, `importlib.util.spec_from_file_location` should still work with extensions if you specify a loader — Alex Walczak, Dec 08 '18 at 22:05

Ciro Santilli新疆棉花TRUMP BAN BAD · Answer 4 · 2021-02-13T08:33:32.677

importlib helper function

Here is a convenient, ready-to-use helper to replace imp, with an example. The technique is the same as that of https://stackoverflow.com/a/19011259/895245 , this is just providing a more convenient function.

main.py

#!/usr/bin/env python3

import os
import importlib

def import_path(path):
    module_name = os.path.basename(path).replace('-', '_')
    spec = importlib.util.spec_from_loader(
        module_name,
        importlib.machinery.SourceFileLoader(module_name, path)
    )
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    sys.modules[module_name] = module
    return module

notmain = import_path('not-main')
print(notmain)
print(notmain.x)

not-main

x = 1

Run:

python3 main.py

Output:

<module 'not_main' from 'not-main'>
1

I replace - with _ because my importable Python executables without extension have hyphens as in my-cmd. This is not mandatory, but produces better module names like my_cmd.

This pattern is also mentioned in the docs at: https://docs.python.org/3.7/library/importlib.html#importing-a-source-file-directly

I ended up moving to it because after updating to Python 3.7, import imp prints:

DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

and I don't know how to turn that off, this was asked at:

Tested in Python 3.7.3.

Import arbitrary python source file. (Python 3.3+)

4 Answers4

Linked

Related