7

I've been trying to compile NumPy from source on Windows 10, with MSVC compiler and Intel MKL. I am running Windows 10.0.18363 with Microsoft Visual Studio 2019 (16.8.4) and Intel MKL 2017.8.275.

I managed to reproduce the issue with a minimal setup, using latest Python and NumPy.

  1. Download latest Python (3.9.1) and latest NumPy (1.20.1) source.

  2. Open a VS command prompt, unpack Python source, build with PCbuild\build.bat

  3. Run mklvars.bat intel64 to get the right environment variables set.

  4. Add the Intel compilers (needed for ifort) to PATH:

    set PATH=C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64;%PATH%

  5. Create a virtual env, copy a few files from the Python build and activate the virtual env:

    copy Python\PCbuild\amd64\python39.dll venv\Scripts
copy Python\PC\pyconfig.h venv\Include

  6. Build NumPy from source and install: pip install . -v

  7. Try to import NumPy: python -c "import numpy"

Error message

Traceback (most recent call last):
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py", line 22, in <module>
    from . import multiarray
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\core\multiarray.py", line 12, in <module>
    from . import overrides
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\core\overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\__init__.py", line 145, in <module>
    from . import core
  File "C:\cygwin\home\user\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "C:\cygwin\home\user\numpy_clean_env\venv\Scripts\python.exe"
  * The NumPy version is: "1.20.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.

Other information

The MKL libraries are picked up during compilation since it returns:

FOUND:
        libraries = ['mkl_rt']
        library_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib\\intel64']
        define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
        include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib']

I tried to analyze the DLL resolution with Dependencies (the newer version of Dependency Walker) but it seems that the MKL DLL loads fine. There are some DLLs that appear as not correctly loaded, but as far as I understand it is caused by the inspection software limit with Windows API sets, not by actual problems with this DLLs, so I think the system is correctly setup. Here is a list of the unique DLLs that cause a NOT_FOUND in Dependencies when analyzing _multiarray_umath.pyd:

EMCLIENT.dll (NOT_FOUND) :
HvsiFileTrust.dll (NOT_FOUND) :
UpdateAPI.dll (NOT_FOUND) :
api-ms-win-core-comm-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-psapi-ansi-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-psapi-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-stringansi-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-versionansi-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-versionansi-l1-1-1.dll (NOT_FOUND) :
api-ms-win-core-winrt-string-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-wow64-l1-1-0.dll (NOT_FOUND) :
api-ms-win-core-wow64-l1-1-1.dll (NOT_FOUND) :
api-ms-win-core-xstate-l2-1-0.dll (NOT_FOUND) :
api-ms-win-core-xstate-l2-1-1.dll (NOT_FOUND) :
api-ms-win-coreui-secruntime-l1-1-0.dll (NOT_FOUND) :
ext-ms-mf-pal-l2-1-0.dll (NOT_FOUND) :
ext-ms-onecore-appmodel-emclient-l1-1-0.dll (NOT_FOUND) :
ext-ms-onecore-defaultdiscovery-l1-1-0.dll (NOT_FOUND) :
ext-ms-onecore-orientation-l1-1-0.dll (NOT_FOUND) :
ext-ms-onecore-shellchromeapi-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-appmodel-deployment-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-appmodel-usercontext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-appmodel-viewscalefactor-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-audiocore-pal-l1-2-0.dll (NOT_FOUND) :
ext-ms-win-com-suspendresiliency-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-core-winrt-remote-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-dwmapi-ext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-dxcore-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-networking-wlanstorage-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-1.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-2.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-3.dll (NOT_FOUND) :
ext-ms-win-ntuser-window-l1-1-4.dll (NOT_FOUND) :
ext-ms-win-ras-tapi32-l1-1-1.dll (NOT_FOUND) :
ext-ms-win-ro-typeresolution-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-rtcore-minuser-input-l1-1-1.dll (NOT_FOUND) :
ext-ms-win-rtcore-minuser-private-ext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-security-capauthz-ext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-security-chambers-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell-knownfolderext-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell-shell32-l1-2-0.dll (NOT_FOUND) :
ext-ms-win-shell-shlwapi-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell-tabbedtitlebar-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell32-shellcom-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-shell32-shellfolders-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-ui-viewmanagement-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-uiacore-l1-1-0.dll (NOT_FOUND) :
ext-ms-win-wer-xbox-l1-1-0.dll (NOT_FOUND) :
ext-ms-win32-subsystem-query-l1-1-0.dll (NOT_FOUND) :
ext-ms-windowscore-deviceinfo-l1-1-0.dll (NOT_FOUND) :

I also tried to follow this guide and I can confirm that os.path.exists('_multiarray_umath.pyd') returns True, but ctypes.WinDLL('_multiarray_umath.pyd') fails to load the DLL (or one of its dependencies).

Working example

Same steps as above but without executing mklvars.bat, so without linking to Intel MKL.

In the past this setup used to work with Python 3.6, VS2015 and a similar version of Intel MKL. I am probably missing something, any suggestion is more than welcome.

cubanpit
  • 81
  • 6
  • What's wrong with *BLAS* library that "regular" *NumPy* uses? Are all the (recursively) required *.dll*s present (and built for the right architecture)? I noticed some *Cygwin* paths, are you mixing toolchains? Also in the setup that used to work, are versions the same? – CristiFati Feb 12 '21 at 09:05
  • @CristiFati Intel MKL is a requirement in my case, for speed and stability. DLLs for 64bit are available at the expected path and they appear as correctly linked in Dependencies (`mkl_rt` is listed and the path exists). The Cygwin path is in there just because the "non-minimal" setup is in Cygwin, but in this case I am just loading a VS2019 x64 Command Prompt. The setup that works has different versions of some tools (Python 3.6.2 > 3.9.1, VS2015 > VS2019); the newer setup fails the same way with NumPy 1.16, 1.17 and 1.20, and with Intel MKL 2017.2, 2017.8 and 2021. – cubanpit Feb 12 '21 at 14:36
  • Might want to check this out: https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy. – CristiFati Feb 12 '21 at 14:41
  • Thanks for the link, I can try the pre-compiled versions, but I use MKL to build other components and they all have to be shipped together. I think shipping MKL twice would require too much space, so I need to check if it is feasible to do otherwise. – cubanpit Feb 12 '21 at 15:31
  • @CristiFati I also notice that there is a very limited set of (Python version, NumPy version) combinations at that link, this could pose further problems with the rest of the building system, compiling it from source gives more flexibility. – cubanpit Feb 12 '21 at 15:41

1 Answers1

1

Solved by following a suggestion on Numpy mailing list (permalink).

Python >= 3.8 will no longer use PATH for resolving dependencies of extension modules. Use os.add_dll_directory(mkl_bin_path) https://docs.python.org/3/library/os.html#os.add_dll_directory in all your scripts before importing numpy or add the call to a _distributor_init.py file in the numpy package directory.

cubanpit
  • 81
  • 6
  • Hmm that was it? I wonder how come I missed it: https://stackoverflow.com/questions/58631512/pywin32-and-python-3-8-0. – CristiFati Feb 18 '21 at 01:20