We are running into issues with an old closed-source game engine failing to compile shaders when memory nears 2GB.
The issue is usually with D3DXCreateEffect
. Usually it returns HResult "out of memory", sometimes d3dx9_25.dll
prints random errors in a popup, or it just outright segfault.
I believe the issue is lack of Large Address Awareness: I noticed one of the d3dx9_25.dll
crashes doing something that would hint as such. It took a valid pointer that looked like 0x8xxxxxx3
, checked that bits 0x80000003
are lit and if yes, it bit inverts the pointer and derefs it. The resulting pointer pointed to unallocated memory. Forcing the engine to malloc 2GB before compilation makes the shaders fail to compile every time.
Unfortunately our knowledge of DX9 is very limited, I've seen that DX9 has a flag D3DXCONSTTABLE_LARGEADDRESSAWARE
but I'm not sure where exactly Its supposed to go. The only API call the game uses that I can find relies on it is D3DXGetShaderConstantTable
, but the issues happen before it is ever called. Injecting the flag (1 << 17) = 0x20000
to D3DXCreateEffect
makes the shader fail compilation in another way.
Is
D3DXCreateEffect
supposed to accept the Large Address Aware flag? I found a wine test using it, but digging into DX9 assembly, the error it throws is caused by an internal function returning HResult Invalid Call when any bit out ofFFFFF800
in flags is set, which leads me to believeCreateEffect
is not supposed to accept this flag.Is there anywhere else I should be injecting the Large Address Aware flag before this? I understand that a call to
D3DXGetShaderConstantTable
will need to be fixed to useD3DXGetShaderConstantTableEx
, but its not even reached yet.