Setuptools installing code from git submodules

Question

I have a Python library I am trying to package for PyPi which contains several git submodules pointing to external repositories. Something like:

MyLibrary
 - setup.py
 - MyLibrary
   - __init__.py
   - my_code1.py
   - my_code2.py
   - submodules
     - __init__.py
     - git submodule https://www.github.com/external/repository1
        - repository1_code.py
     - git submodule https://www.github.com/external/repository2
        - repository2_code.py

My setup.py file looks something like:

from setuptools import setup, find_packages

setup(
    name='MyLibrary',
    ...
    packages=find_packages(),
    ...,
    install_requires=["numpy", "scipy"]
)

The problem I am having is when I use pip to install this library, all my code and subfolders get installed fine, but the submodules are missing; they are not initialized/pulled in and don't appear inside the installed folder, neither the submodule folders nor the inner code (e.g repository1_code.py).

Is there some way to instruct setup.py to expand the submodules, i.e. git submodule update --init --recursive before installing the code?

do your submodules have `__init__.py` at their root? (it's necessary for `find_packages` to descend into them) — Anthony Sottile, Sep 15 '20 at 00:00
You could probably give [`find_namespace_packages()`](https://setuptools.readthedocs.io/en/latest/setuptools.html#find-namespace-packages) a shot as well. It is less picky about `__init__.py` so it might work for your use case, but of course it has side effects as well. — sinoroc, Sep 15 '20 at 10:54

score -3 · Answer 1 · answered Sep 15 '20 at 02:22

One solution, thanks to the top comment to the question, is to fork all of the submodules and place an __init__.py in all of the folders needed. This isn't optimal though because I'd rather not have to fork and modify the submodules, but use the originals.

Another solution is instead of using find_packages(), to simply explicitly write out all of the folders in the submodules instead. This is a bit inconvenient but works very well, and seems to be the best solution I've found so far.

Setuptools installing code from git submodules

1 Answers1