3

I need some help with building pythonservice.exe (PythonService.cpp) from https://github.com/kovidgoyal/pywin32/tree/master/win32/src.

It is said (https://github.com/kovidgoyal/pywin32) that VS2015 is required, so I downloaded the community version. I opened a new win32 console project and imported to source files only the PythonService.cpp.

In project settings I have:

Include directories: C:\Python27_32bit\include;C:\pywin32-master\win32\src;$(VC_IncludePath);$(WindowsSDK_IncludePath);

Library directories: C:\Python27\Lib\site-packages\win32\libs;C:\Python27_32bit\libs;C:\pywin32-master\win32\src;$(LibraryPath);C:\pywin32-master\lib\x32\win32\libs;

Now, when building I get the unresolved external symbol error:

1>PythonService.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) char * __cdecl GetPythonTraceback(struct _object *,struct _object *,struct _object *)" (__imp_?GetPythonTraceback@@YAPADPAU_object@@00@Z)

GetPythonTraceback is defined in PyWinTypesmodule.cpp but when I add it to my project it gives other errors requiring other unresolved functions. I don't want to compile the whole win32 project. All I need is pythonservices.exe. Is this possible to accomplish without compiling the whole project? Would be thankful for any help!

Thanks! Alexei

alexeik
  • 63
  • 7

2 Answers2

1

I eventually found a solution to my question (not without help from @CristiFati 's answer!). I downloaded https://github.com/mhammond/pywin32/tree/b222 and used the setup.py script to build the whole package. Also, I installed the required SDK package (for python 2.6+ Microsoft Windows SDK for Windows 7 and .NET Framework 4 (version 7.1)). After digging a little bit I found that the compilation and linkage are done with Visual C++ for Python 9.0 and below are the compilation and linkage recipes for pythonservice.pyd and pythonservice.exe (python lists for cmd parameter to calls of os.spawnv(os.P_WAIT, executable, cmd):

For pythonservice.pyd:

['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\WinSDK\\Bin\\x64\\mc.exe"','-h', 'win32/src', '-r', 'build\\temp.win-amd64-2.7\\Release\\win32/src', 'win32/src/PythonServiceMessages.mc']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\WinSDK\\Bin\\x64\\rc.exe"','/fobuild\\temp.win-amd64-2.7\\Release\\win32/src/PythonServiceMessages.res','build\\temp.win-amd64-2.7\\Release\\win32/src\\PythonServiceMessages.rc']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe"','/c', '/nologo', '/Ox','/W3','/GS', '/DNDEBUG', '/MD', '-DDISTUTILS_BUILD', '-D_CRT_SECURE_NO_WARNINGS','-Icom/win32com/src/include', '-Iwin32/src', '-IC:\\Python27\\include', '-IC:\\Python27\\PC','"-IC:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\include"', '/Tpwin32/src/PythonService.cpp','/Fobuild\\temp.win-amd64-2.7\\Release\\win32/src/PythonService.obj', '-DPYSERVICE_BUILD_DLL', '/Zi','/Fdbuild\\temp.win-amd64-2.7\\Release\\servicemanager_vc.pdb', '/EHsc', '/DUNICODE', '/D_UNICODE', '/DWINNT']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\link.exe"', '/DLL', '/nologo', '/INCREMENTAL:NO', '/LIBPATH:C:\\Python27\\libs', '/LIBPATH:C:\\Python27\\PCbuild\\amd64', '/LIBPATH:C:\\Python27\\PC\\VS9.0\\amd64', '/LIBPATH:build\\temp.win-amd64-2.7\\Release', '"/LIBPATH:C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\lib\\x64"', 'user32.lib', 'ole32.lib', 'advapi32.lib', 'shell32.lib', '/EXPORT:initservicemanager', 'build\\temp.win-amd64-2.7\\Release\\win32/src/PythonServiceMessages.res', 'build\\temp.win-amd64-2.7\\Release\\win32/src/PythonService.obj', '/OUT:build\\lib.win-amd64-2.7\\win32\\servicemanager.pyd', '/IMPLIB:build\\temp.win-amd64-2.7\\Release\\win32/src\\servicemanager.lib', '/MANIFEST:NO', '/MACHINE:amd64', '/BASE:0x1e7d0000', '/DEBUG', '/PDB:build\\temp.win-amd64-2.7\\Release\\servicemanager.pdb']

For pythonservice.exe:

['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe"','/c', '/nologo', '/Ox', '/W3', '/GS-', '/DNDEBUG', '/MD', '-DDISTUTILS_BUILD', '-D_CRT_SECURE_NO_WARNINGS','-Icom/win32com/src/include', '-Iwin32/src', '-IC:\\Python27\\include', '-IC:\\Python27\\PC','"-IC:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\include"','/Tpwin32\\src\\PythonService.cpp', '/Fobuild\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\PythonService.obj','/Zi', '/Fdbuild\\temp.win-amd64-2.7\\Release\\pythonservice_vc.pdb', '/EHsc', '/DUNICODE', '/D_UNICODE', '/DWINNT']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\WinSDK\\Bin\\x64\\rc.exe"','-DDISTUTILS_BUILD', '-D_CRT_SECURE_NO_WARNINGS', '-Icom/win32com/src/include', '-Iwin32/src', '-IC:\\Python27\\include','-IC:\\Python27\\PC', '"-IC:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\include"','/fobuild\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\PythonService.res', 'win32\\src\\PythonService.rc']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\link.exe"','/nologo', '/INCREMENTAL:NO', '/LIBPATH:C:\\Python27\\libs', '/LIBPATH:C:\\Python27\\PCbuild\\amd64','/LIBPATH:C:\\Python27\\PC\\VS9.0\\amd64', '/LIBPATH:build\\temp.win-amd64-2.7\\Release','"/LIBPATH:C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\lib\\x64"', 'user32.lib', 'advapi32.lib', 'ole32.lib','shell32.lib', 'build\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\PythonService.obj','build\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\PythonService.res','/OUT:build\\lib.win-amd64-2.7\\win32\\pythonservice.exe','/MANIFESTFILE:build\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\pythonservice.exe.manifest','/SUBSYSTEM:CONSOLE', '/MACHINE:amd64', '/DEBUG', '/PDB:build\\temp.win-amd64-2.7\\Release\\pythonservice.pdb']
['mt.exe', '-nologo', '-manifest','build\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\pythonservice.exe.manifest.orig','-outputresource:build\\lib.win-amd64-2.7\\win32\\pythonservice.exe;1']
['C:\\Python27\\python.exe', 'C:\\Python27\\Lib\\site-packages\\win32\\lib\\win32verstamp.py', '--version=2.7.222.1', '--comments=https://github.com/mhammond/pywin32', '--original-filename=pythonservice.exe','--product=PyWin32', '--quiet', 'build\\lib.win-amd64-2.7\\win32\\pythonservice.exe']
alexeik
  • 63
  • 7
0

The 1st thing that I see here is combining Python 2.7 and VStudio 2015. According to [Python.Wiki]: WindowsCompilers, for VStudion 2015 (or 14.0), you should use Python 3.5 or Python 3.6 (I have the former).
Check [SO]: Simstring (python) installation in windows (@CristiFati's answer) - (the 1st part), about compatibilities and restrictions in terms of:

  • Platform
  • Tools versions
  • Architecture

Also, some (basic) information about building C code on Win: [SO]: LNK2005 Error in CLR Windows Form (@CristiFati's answer)

  • Downloaded original PyWin32 from: [GitHub]: mhammond/pywin32 - (b222) (to pywin32-b222.zip). The "${PYWIN32_SRC_DIR}\win32\src\PythonService.cpp" is identical to its correspondent from the URL you mentioned
  • Create a dir and cd to it (should be empty). This will be the %ROOT_DIR%, and all the paths that I'm going to use will be relative to it (except of course for absolute ones), and this will be the default dir (when unspecified)

  • Unpack the downloaded .zip somewhere. We (directly) need from it:

    • "${PYWIN32_SRC_DIR}\win32\src\PythonService.cpp" - as you mentioned it, the source code
    • "${PYWIN32_SRC_DIR}\win32\src\PythonServiceMessages.mc" - message text file


    I copied them in a dir called src:

    E:\Work\Dev\StackOverflow\q048931801>dir /b
src

E:\Work\Dev\StackOverflow\q048931801>dir /b "src"
PythonService.cpp
PythonServiceMessages.mc
  • Since the build involves resources and Custom Build Steps (that I'm not very good at), and also it doesn't involve lots of files, I'm not going to use the IDE

  • The next best approach would be to use Makefiles, but since VStudio has a bug [MS.MSDN]: ntwin32.mak not found:

    • It affects VStudio 2015 as well
    • It requires a one time fix
    • I did the fix on my machine and it works, but you most likely didn't on yours, and I don't want to unnecessarily burden you with this step


    I am going to do everything manually

  • Setting up VStudio environment:

  • Compile the message file into a resource file - using mc.exe ([MS.Docs]: Message Compiler (MC.exe))

    E:\Work\Dev\StackOverflow\q048931801>mc "src\PythonServiceMessages.mc"
MC: Compiling src\PythonServiceMessages.mc

E:\Work\Dev\StackOverflow\q048931801>dir /b
MSG00001.bin
PythonServiceMessages.h
PythonServiceMessages.rc
src

    As seen, it generated 3 new files

  • Compile the resource file - using rc.exe ([MS.Docs]: Resource Compiler)

    E:\Work\Dev\StackOverflow\q048931801>rc /NOLOGO /r "PythonServiceMessages.rc"

E:\Work\Dev\StackOverflow\q048931801>dir /b
MSG00001.bin
PythonServiceMessages.h
PythonServiceMessages.rc
PythonServiceMessages.res
src

  • Done with the resources, going to the code (PythonService.cpp). As you already mentioned, it won't compile OOTB. I'm not going to list all the failed attempts, instead I'm going to list the steps I took, and only show the final (successful) compile. So, the file needs code from other files. The simplest way is to copy it from the other files and place it in this one. I chose to paste it around line #1530 (just before the "* Entry points" section):

    • PyWinTypesModule.cpp - GetPythonTraceback function (lines 1142 - 1235)
    • PyUnicode.cpp - PyWin_AutoFreeBstr::PyWin_AutoFreeBstr, PyWin_AutoFreeBstr::~PyWin_AutoFreeBstr, PyWin_AutoFreeBstr::SetBstr (lines 204 - 219)

    Either apply the changes manually, or save:

    --- PythonService.cpp.orig  2018-01-20 21:43:10.000000000 +0200
+++ PythonService.cpp   2018-02-23 20:02:20.186966800 +0200
@@ -1529,6 +1529,120 @@
     }
 }

+// @TODO: cfati - copied from PyUnicode.cpp (lines 204 - 219)
+PyWin_AutoFreeBstr::PyWin_AutoFreeBstr( BSTR bstr /*= NULL*/ )
+ : m_bstr(bstr)
+{
+   return;
+}
+
+PyWin_AutoFreeBstr::~PyWin_AutoFreeBstr()
+{
+   SysFreeString(m_bstr);
+}
+
+void PyWin_AutoFreeBstr::SetBstr( BSTR bstr )
+{
+   SysFreeString(m_bstr);
+   m_bstr = bstr;
+}
+
+
+// @TODO: cfati - copied from PyWinTypesModule.cpp (lines 1142 - 1235)
+// Function to format a python traceback into a character string.
+#define GPEM_ERROR(what) {errorMsg = "<Error getting traceback - " ## what ## ">";goto done;}
+char *GetPythonTraceback(PyObject *exc_type, PyObject *exc_value, PyObject *exc_tb)
+{
+   // Sleep (30000); // Time enough to attach the debugger (barely)
+   char *result = NULL;
+   char *errorMsg = NULL;
+   PyObject *modStringIO = NULL;
+   PyObject *modTB = NULL;
+   PyObject *obFuncStringIO = NULL;
+   PyObject *obStringIO = NULL;
+   PyObject *obFuncTB = NULL;
+   PyObject *argsTB = NULL;
+   PyObject *obResult = NULL;
+
+   /* Import the modules we need - cStringIO and traceback */
+#if (PY_VERSION_HEX < 0x03000000)
+   modStringIO = PyImport_ImportModule("cStringIO");
+#else
+   // In py3k, cStringIO is in "io"
+   modStringIO = PyImport_ImportModule("io");
+#endif
+
+   if (modStringIO==NULL) GPEM_ERROR("cant import cStringIO");
+   modTB = PyImport_ImportModule("traceback");
+   if (modTB==NULL) GPEM_ERROR("cant import traceback");
+
+   /* Construct a cStringIO object */
+   obFuncStringIO = PyObject_GetAttrString(modStringIO, "StringIO");
+   if (obFuncStringIO==NULL) GPEM_ERROR("cant find cStringIO.StringIO");
+   obStringIO = PyObject_CallObject(obFuncStringIO, NULL);
+   if (obStringIO==NULL) GPEM_ERROR("cStringIO.StringIO() failed");
+
+   /* Get the traceback.print_exception function, and call it. */
+   obFuncTB = PyObject_GetAttrString(modTB, "print_exception");
+   if (obFuncTB==NULL) GPEM_ERROR("cant find traceback.print_exception");
+   argsTB = Py_BuildValue("OOOOO"
+#if (PY_VERSION_HEX >= 0x03000000)
+       "i"     
+       // Py3k has added an undocumented 'chain' argument which defaults to True
+       //  and causes all kinds of exceptions while trying to print a goddam exception
+#endif
+       ,
+       exc_type ? exc_type : Py_None,
+       exc_value ? exc_value : Py_None,
+       exc_tb  ? exc_tb  : Py_None,
+       Py_None,    // limit
+       obStringIO
+#if (PY_VERSION_HEX >= 0x03000000)
+       ,0  // Goddam undocumented 'chain' param, which defaults to True
+#endif
+       );
+   if (argsTB==NULL) GPEM_ERROR("cant make print_exception arguments");
+
+   obResult = PyObject_CallObject(obFuncTB, argsTB);
+   if (obResult==NULL){
+       // Chain parameter when True causes traceback.print_exception to fail, leaving no
+       //  way to see what the original problem is, or even what error print_exc raises
+       // PyObject *t, *v, *tb;
+       // PyErr_Fetch(&t, &v, &tb);
+       // PyUnicodeObject *uo=(PyUnicodeObject *)v;
+       // DebugBreak();
+       GPEM_ERROR("traceback.print_exception() failed");
+       }
+   /* Now call the getvalue() method in the StringIO instance */
+   Py_DECREF(obFuncStringIO);
+   obFuncStringIO = PyObject_GetAttrString(obStringIO, "getvalue");
+   if (obFuncStringIO==NULL) GPEM_ERROR("cant find getvalue function");
+   Py_DECREF(obResult);
+   obResult = PyObject_CallObject(obFuncStringIO, NULL);
+   if (obResult==NULL) GPEM_ERROR("getvalue() failed.");
+
+   /* And it should be a string all ready to go - duplicate it. */
+   if (PyString_Check(obResult))
+       result = strdup(PyString_AsString(obResult));
+#if (PY_VERSION_HEX >= 0x03000000)
+   else if (PyUnicode_Check(obResult))
+       result = strdup(_PyUnicode_AsString(obResult));
+#endif
+   else
+       GPEM_ERROR("getvalue() did not return a string");
+
+done:
+   if (result==NULL && errorMsg != NULL)
+       result = strdup(errorMsg);
+   Py_XDECREF(modStringIO);
+   Py_XDECREF(modTB);
+   Py_XDECREF(obFuncStringIO);
+   Py_XDECREF(obStringIO);
+   Py_XDECREF(obFuncTB);
+   Py_XDECREF(argsTB);
+   Py_XDECREF(obResult);
+   return result;
+}

 /*************************************************************************
  *

    as "src\PythonService.diff" That is a diff. See [SO]: Run/Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (@CristiFati's answer) (Patching utrunner section) for how to apply patches on Win (basically, every line that starts with one "+" sign goes in, and every line that starts with one "-" sign goes out). I am using Cygwin:

    E:\Work\Dev\StackOverflow\q048931801>"c:\Install\x64\Cygwin\Cygwin\AllVers\bin\patch.exe" "src\PythonService.cpp" "src\PythonService.diff"
patching file 'src\PythonService.cpp'

  • Compile the code - using cl.exe ([MS.Docs]: Compiling a C/C++ Program)

    E:\Work\Dev\StackOverflow\q048931801>cl /GS /W1 /Zc:wchar_t /I"." /I"c:\Install\x64\Python\Python\3.5\include" /I"e:\Work\Dev\Fati\WinBuild\OPSWpython27\src\pywin32-b222\win32\src" /Gm- /O2 /Zc:inline /fp:precise /D "UNICODE" /D "_UNICODE" /D "WIN32" /D "WIN64" /D "NDEBUG" /D "BUILD_PYWINTYPES" /errorReport:prompt /WX- /Zc:forScope /Gd /MD /c "src\PythonService.cpp"
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

PythonService.cpp

E:\Work\Dev\StackOverflow\q048931801>dir /b "*.obj"
PythonService.obj

    Arguments are standard ones ([MS.Docs]: Compiler Options Listed Alphabetically), specific for this case:

    • /D "BUILD_PYWINTYPES" (macro definition)
    • /D "WIN64" - 64bit
    • Include paths (/I) - you'll have to adapt them to match your paths:
      1. "c:\Install\x64\Python\Python\3.5\include" (Python path)
      2. "e:\Work\Dev\Fati\WinBuild\OPSWpython27\src\pywin32-b222\win32\src" (pywin32 path)

  • Link everything together - using link.exe ([MS.Docs]: Linking)

    E:\Work\Dev\StackOverflow\q048931801>link /NOLOGO /MACHINE:X64 /SUBSYSTEM:CONSOLE /ERRORREPORT:PROMPT /LIBPATH:"c:\Install\x64\Python\Python\3.5\libs" /NXCOMPAT /DYNAMICBASE "kernel32.lib" "user32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "PythonService.obj" "PythonServiceMessages.res" /OUT:"PythonService.exe"
   Creating library PythonService.lib and object PythonService.exp

E:\Work\Dev\StackOverflow\q048931801>dir /b "*.exe"
PythonService.exe

    Again, arguments are standard ones ([MS.Docs]: Linker Options), specific for this case:

    • /MACHINE:X64 - 64bit
    • Library paths (/LIBPATH) - adapt it to match yours:
      1. "c:\Install\x64\Python\Python\3.5\libs (Python path)

  • One last step before running the service. Since it depends on python35.dll, the OS must know where to look for it. Simplest way is to add its path to %PATH% ([MS.Docs]: Dynamic-Link Library Search Order)

    E:\Work\Dev\StackOverflow\q048931801>set PATH=%PATH%;c:\Install\x64\Python\Python\3.5

E:\Work\Dev\StackOverflow\q048931801>PythonService.exe
P - Python Service Manager
Options:
 -register - register the EXE - this should generally not be necessary.
 -debug servicename [parms] - debug the Python service.

NOTE: You do not start the service using this program - start the
service using Control Panel, or 'net start service_name'
CristiFati
  • 28,721
  • 9
  • 41
  • 63
  • Thanks a lot for the great answer! I realized that I was missing a lot of stages in between (like compiling first the resource file etc.) I used the setup.py build script and extracted from it the relevant parts. – alexeik Feb 25 '18 at 10:54