0

I have a project that needs updating from python2 to python3. It currently uses pkg_resources, which has significant overhead (PyCon 2018 - Get Your Resources Faster with importlib.resources, by Barry Warsaw). It will be packaged into a zip file. The relevant portions of the package structure are:

+-- project/
|__+ __init__.py
|__+ main.py
|__resources/
|  |__+ images/
|  |  |__ # program images
|  |__+ logging/
|     |__+ custom_logger.py
|__log/
   |__+ project_log.log

Intent for Original Question

The intent for the original question was to ask "how does one get the absolute path to a subpackage from within another subpackage", but I did a poor job wording this. I thought since current working directory meant "path to where the cwd command is run from" that this could be used, but I later learned my understanding was incomplete. current working directory means (in layman's terms) "path to where the cwd command is run from on the commandline"). This is why I was having trouble. If I run my program from within the top-level package

C:\Users\usr\Destop\program>py main.py

and use pathlib.Path.cwd() in a .py file in any other subpackage (resources, logging, log), I will get the same answer:

C:\Users\usr\Destop\program

I learned this is proper behavior, but it means I cannot use relative paths to get from one subpackage to another (I need to use relative paths since I don't know where the end user will install this program on his/her machine).

Relevant 2nd Part to Question that was Not Well-Clarified

Since part of my need is to update to python3, both pkgutil and importlib.resources were investigated. Each of these require resources to be treated as packages, so to update to use either of them, I have to add __init__.py to the folders resources, logging, and log. Not a big deal thankfully as this isn't a large program.

Note, for backwards compatibility with minimal code reediting, a namespace renaming has been recommended and used by others (e.g. see (How to use importlib.resources.path(package, resource)?).:

try:
    import importlib.resources as pkg_resources
except ImportError:
    # Try backported to PY<37 `importlib_resources`.
    import importlib_resources as pkg_resources

Neither pgkutil nor importlib.resources has a method that directly returns "this subpackage" (in the way that pathlib.Path.cwd() "directly returns" the cwd without any extra code), but I think importlib.resources has a method I can use that will require the least amount of code and still be understandable to readers. Please note the ask is to get the absolute path to "this subpackage" and nothing else. I do not want to have to use a different approach.

The methods I would use is

try:
    import importlib.resources as pkg_resources
except ImportError:
    # Try backported to PY<37 `importlib_resources`.
    import importlib_resources as pkg_resources

with pkg_resources.path("logging", "__init__.py") as fl:
    path = fl.parent.resolve()

Conclusion

Beyond this, I have no questions, because it will probably be judged as an "opinion" question. Please note the __file__ method is unacceptable as this does not work with zip packages (see PyCon 2018 - Get Your Resources Faster with importlib.resources, by Barry Warsaw). If there is anything incorrect in this approach or if it can be improved, I would like to know.

A. Hendry
  • 1,521
  • 2
  • 11
  • 19
  • Do you want the answer to be 'project' in both the cases? I think that might require knowing how deeply nested the directory structure is. – mk03 Aug 19 '20 at 04:34
  • @mk03 I'm confused by what you mean. Are you talking about what `Path.cwd()` returns? – A. Hendry Aug 19 '20 at 05:04
  • I wanted to ask what is your desired output? – mk03 Aug 19 '20 at 05:59
  • 1
    Desired output would be absolute path to file. I might need to change to `cwd = str(fl.parent.resolve())` for this. – A. Hendry Aug 19 '20 at 20:08
  • I think that would work – mk03 Aug 20 '20 at 08:27
  • _CWD_ stands for current working directory, and usually (always?), yes it is the directory the user is currently browsing. That would explain the output of `Path.cwd()`. And what you are looking for is not the CWD, but something else. – sinoroc Aug 22 '20 at 20:32
  • @sinoroc Correct. This was not immediately obvious to me. – A. Hendry Aug 22 '20 at 21:05
  • 1
    Does this answer your question? [How to read a (static) file from inside a Python package?](https://stackoverflow.com/questions/6028000/how-to-read-a-static-file-from-inside-a-python-package) – sinoroc Aug 23 '20 at 08:46
  • No, @wim does not answer my question. – A. Hendry Aug 23 '20 at 19:18
  • Thanks for clarifying. I changed my vote. -- Something I would like to add, not sure if it's actually relevant, I haven't really tested it, but anyway... In case the project is _zipped_, then I believe you can't really have a path to a (sub)package from this project, only to a file. If I am not mistaken, functions such as `importlib.resources.path(...)` hide this from the user, in that they unzip the file in a temporary file (directory?), and thus the _path_ is only valid for the duration of the context manager and only for that file. Probably the rest of the package is not unzipped at all. – sinoroc Aug 24 '20 at 16:43
  • @sinoroc Thank you and very true. The path will be to the temp folder where items are unzipped. This will work if the goal is to do "pathlib.Path path dividing" in a for loop, but not if the goal is to get the actual path. If we need the actual path, we should use the `__file__` method as @Bobby_Ocean has kindly presented. Alternatively, `pkgutil` has a walk directory structure method (`pkgutil.walk_packages`) whereas `importlib.resources` does not. `importlib.resources` has the `contents` method, but it does not recurse into subdirectories. Very interesting point... – A. Hendry Aug 24 '20 at 18:00

1 Answers1

-2

Any python file can access it's own location with file. Hence for example you could simply do

pathlib.Path(__file__).parent

to access the directory of the file that runs that code. This is independent from where you run the program from and/or where you install the program.

Bobby Ocean
  • 2,205
  • 1
  • 3
  • 13
  • 1
    This does not work for code contained in zip files. (Can happen when `.egg`s are created or packaging for upload to PyPI) – A. Hendry Aug 22 '20 at 23:59
  • Weird reason to downvote. I don't recall zip files being mentioned by the OP. Additoanlly, acnt you just set the zip flag to not safe in setup.py? – Bobby Ocean Aug 23 '20 at 00:04
  • True, you can set `zip_safe` to `False`. Sorry for down voting. Only did so as answer could be improved since this does not work in all cases. – A. Hendry Aug 23 '20 at 00:15
  • Right, but again, that wasn't even apart of the OPs question. – Bobby Ocean Aug 23 '20 at 00:29
  • Unfortunately, Stackoverflow doesn't give me the option to upvote your answer unless you edit the question. If you add a comment about `zip_safe` to your answer as your edit, then I can upvote. Will that help? Zip safety is implied as part of the question. It asks for the correct way to get the `cwd` within a package structure in Python 3. Current best practice indicates accomodating for zip safety. No mention of this was given in your answer, hence it was downvoted as your answer was lacking and could be improved. However, i will upvote it back to 0 if you can edit your answer. – A. Hendry Aug 23 '20 at 01:40
  • Oh, I don't really care that much about the vote point. Just thought that was odd. I have worked on many packages where we set zip_safe to false all the time. In particular, I am not even sure the exact bug you are referring to. There are many built in attributes to python files, like file, all, etc. I would think the standard answer should involve those. If there is a special case that causes an error, then that is when a specialized package should be recommended. Also, I thought that __file__ was used by packages like importlib, inspect, etc. – Bobby Ocean Aug 23 '20 at 01:56
  • Also, we should point out that the OP does not mention a setup.py file nor pypi, nor installing their package. They only mention running it locally and want to know the simplest way to obtain the location of the file. – Bobby Ocean Aug 23 '20 at 01:59
  • I'm the OP. I do not want the simplest way. I state I want the _best_ way. Your way is not the best way as I've clearly stated because it does not cover all cases. Hence, it is appropriate that your answer is downvoted. – A. Hendry Aug 23 '20 at 03:25
  • 1
    No, `__file__` is not to be used by `importlib`. This is considered bad practice. Please see Barry Warsaw's PyCon 2018 video "Get your resources faster, with importlib.resources" . – A. Hendry Aug 23 '20 at 03:27