Python 3 - Modules and Packages Mess - Architecture

Question

Some months ago y started working in a small project that over the time became big. It started with a few python files:

project
│   program_1.py
│   program_2.py

We continue adding more programs, so in order to avoid code repetition I started extracting common functions to separate files.

project
│   program_1.py
│   program_2.py
│   program_3.py
│   shared_module_1.py
│   shared_module_2.py

We keep adding some more programs and they had a considerable size so I put them inside their own folders just like this. Also to avoid dropping more files inside the project folder (This is just a reduced example, folder has 15+ files):

project
│   program_1.py
│   program_2.py
│   program_3.py
│   shared_module_1.py
│   shared_module_2.py
└───program_4
    │   program_4.py
    │   program_4_aux.py

In order for the program_4.py to use the shared modules I had to use the following statement:

sys.path.append('../')

I am unable to keep up like this. Code is getting messy and hard to maintain. I have no idea how to clean this mess.

PD: I try using from .. import shared_module_1 but it tells me parent module '' not loaded cannot perform relative import

Thanks you very much for your time.

The project is a set of usefull automation tools for the company. Many are already in use. I would love a way of cleaning this mess without breaking all code. — NMO, Feb 21 '17 at 14:08
While I think this is a good question, it is a bit broad at the moment. Perhaps you can narrow it down to what you think could be a good layout and why it does not work? — kazemakase, Feb 21 '17 at 14:11
My idea was to have this big folder containing all this set of handy scripts and being able to reuse some logic between them. I want to know what are the best practices (following python principles) to achieve this. — NMO, Feb 21 '17 at 14:14

score 3 · Answer 1 · answered Feb 21 '17 at 14:36

Go and have a look at the way django is laid out.

Salient notes:

the root of the repository is not the main importable python module (i.e. there is no init.py file at the top-level). That is in the subdirectory ./django.
there should be no scripts, i.e. standalone python files that you can run inside the module. The manage_translations.py script uses some django functionality via import, but it lives in ./scripts. Also scripts should never use relative imports (because scripts are not in modules!)
tests also do not live inside the main module.
if you think you need to modify sys.path you are probably doing it wrong.
functionality within the django package is split logically based on its functionality. E.g. there is a submodule ./django/db for handling database related functionality. If this is done well then a generic submodule (if there are any) could easily be moved to a separate project entirely if that was helpful.

So a suggestion would be to move all your module code into a subdirectory separate from your program code, something like:

project
    module
        __init__.py
        module_1.py
        module_2
            __init__.py
            submodule_a.py
    scripts
        program1.py

Then you only need to ensure that project is in your PYTHONPATH, and you will be able to do e.g.

from module.module_1 import Module1Class

etc.

score -1 · Answer 2 · edited May 23 '17 at 11:46

The easiest solution is adding the directory where you have your programs to PYTHONPATH enviromental variable as in Permanently add a directory to PYTHONPATH

Another solution is to call the program from the directory where you shared modules are:

python program_4/program_4.py

You should have a look at init.py files too, despite not being a solution.

The third option would be to have the shared modules in the python library package.

Python 3 - Modules and Packages Mess - Architecture

2 Answers2