Python subprocess run miniconda3/etc/profile.d/conda.sh from houdini

Question

I am stuck on a very weird behaviour in my eyes. I want to create a process from Houdini (https://www.sidefx.com/) and this process should init conda and active an environment. If I run my code from the terminal with python 2.7 everything works as excepted.

import os
import sys
import subprocess
conda_path = "/home/alelidis/miniconda3"

print("============================ Python version ============================")
print (sys.version)

my_env = {}
my_env["PYTHONHOME"] = conda_path + ":" + conda_path + "/envs/tflow1.15.0-cuda10-python3"
my_env["PATH"] = '/home/alelidis/miniconda3/bin:/opt/hfs18.0.566/bin:/opt/hfs18.0.566/houdini/sbin:/usr/bin'

script_cmd = 'source ' + os.path.join(conda_path, 'etc/profile.d/conda.sh')  + ' && ' 
script_cmd += 'conda activate tflow1.15.0-cuda10-python3' + ' && '
script_cmd += 'python --version' + ' && '
script_cmd += 'conda deactivate'


#print script_cmd
print("---------------------")
print ("PATH:")
# If shell=True, the command string is interpreted as a raw shell command.
print subprocess.check_output('echo $PATH', shell=True, env=my_env, executable='/bin/bash')
print ("PYTHONHOME:")
print subprocess.check_output('echo $PYTHONHOME', shell=True, env=my_env, executable='/bin/bash')
print ("python version:")
print subprocess.check_output('python --version', shell=True, env=my_env, executable='/bin/bash')
print ("python should be: 3.8.3")

print("++++++++++++++++++++++++++")
print subprocess.check_output(script_cmd, shell=True, executable='/bin/bash')

returns the following output:

============================ Python version ============================
2.7.15 |Anaconda, Inc.| (default, Dec 14 2018, 19:04:19) 
[GCC 7.3.0]
---------------------
PATH:
/home/alelidis/miniconda3/bin:/opt/hfs18.0.566/bin:/opt/hfs18.0.566/houdini/sbin:/usr/bin

PYTHONHOME:
/home/alelidis/miniconda3:/home/alelidis/miniconda3/envs/tflow1.15.0-cuda10-python3

python version:
Python 3.8.3

python should be: 3.8.3
++++++++++++++++++++++++++
source /home/alelidis/miniconda3/etc/profile.d/conda.sh && conda activate tflow1.15.0-cuda10-python3 && python --version && conda deactivate
++++++++++++++++++++++++++
Python 3.7.9

However, when I try to run the same code from Houdini I am getting the following error:

============================ Python version ============================
2.7.15 (default, Mar 11 2020, 10:04:04) 
[GCC 6.3.1 20170216 (Red Hat 6.3.1-3)]
---------------------
PATH:
/home/alelidis/miniconda3/bin:/opt/hfs18.0.566/bin:/opt/hfs18.0.566/houdini/sbin:/usr/bin

PYTHONHOME:
/home/alelidis/miniconda3/bin:/home/alelidis/miniconda3/envs/tflow1.15.0-cuda10-python3

python version:
Python 3.8.3

python should be: 3.8.3
++++++++++++++++++++++++++
source /home/alelidis/miniconda3/etc/profile.d/conda.sh && conda activate tflow1.15.0-cuda10-python3 && python --version && conda deactivate
++++++++++++++++++++++++++
Traceback (most recent call last):
  File "/home/alelidis/miniconda3/bin/conda", line 12, in <module>
    from conda.cli import main
  File "/home/alelidis/miniconda3/lib/python3.8/site-packages/conda/__init__.py", line 22, in <module>
    from .common.compat import text_type, iteritems
  File "/home/alelidis/miniconda3/lib/python3.8/site-packages/conda/common/compat.py", line 14, in <module>
    from tempfile import mkdtemp
  File "/home/alelidis/miniconda3/lib/python3.8/tempfile.py", line 45, in <module>
    from random import Random as _Random
  File "/home/alelidis/miniconda3/lib/python3.8/random.py", line 41, in <module>
    from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
ModuleNotFoundError: No module named 'math'

I have no clue why this happends! Any help or pointer would be very appreciated!

Answer 1

This could still be an environment issue. Try inheriting the current environment then editing it instead of creating a blank one. You're also completely overriding the PATH env var instead of adding/editing an entry which could cause issues. Aonther thing to try is setting working directory for the subprocess command. But still not sure why you're getting that error.

my_env = os.environ.copy()
my_env["PYTHONHOME"] = conda_path + ":" + conda_path + "/envs/tflow1.15.0-cuda10-python3"
my_env["PATH"] = '/home/alelidis/miniconda3/bin:/opt/hfs18.0.566/bin:/opt/hfs18.0.566/houdini/sbin:/usr/bin'

You could also try executing a python script instead of using standard I/O. I haven't tried Anaconda but I've successfully launched MayaPy from Houdini like this:

cmdArgs = [mayaPyPath, pyScriptPath]
output = subprocess.call(cmdArgs, cwd=mayaPyWorkDir, shell=False, env=curEnv)

Answer 2

I found out that Houdini is overwriting my environment variables, which I was setting with python. So after I removed the PYTHONHOME variable via internal GUI it worked as expected. To be honest I am still not entirely sure why this happened.