Using VCL Styles gives a exception / crash in TOpenDialog

Question

I am experiencing a crash / exception when using a simple TOpenDialog with VCL Styles enabled.

Without the styles enabled, the dialog is of course working fine. The issue occurs with C++ Builder 10 and 10.1 Professional.

To reproduce:

create a simpe VCL Form that uses styles
add a TComboBox, a TButton and a TOpenDialog to the form

add the following code to the OnClick event for the button

OpenDialog1->Execute();
for(int i=0; i<100; i++)
  ComboBox1->Items->Add("test text");
ComboBox1->ItemIndex = 1;

run the application, click the button and select a file
For me, this yields to an exception 'Out of system resources'

This bug could be reproduced on Windows 7 Enterprise and Windows 8.1 Pro. Having the same issue with a TSaveDialog.

The weird thing for me is that regarding this exception, I can't find similar issues on the web. In my opinion, with only VCL styles and TOpenDialog required to have this, I would expect more information about this on the web.

I only found something remotely similar, but not an exception there and also no solution in terms of native VCL styles:

Using custom styles shows invalid characters when right-clicking a file in TOpenDialog

I tried also to disable SystemHooks shDialogs (please see screenshot) which I read somewhere regarding another problem with VCL styles, but to no avail.

* Edit 2016/05/26 *

Remy Lebeau is asking for a MCVE. I tried to put everything in the question, but for clarity here an abstract for a MCVE:

create new VCL project with default settings
drop a TComboBox, a TButton and a TOpenDialog into the form

add the following code to the OnClick event for the button:

void __fastcall TForm1::Button1Click(TObject *Sender) 
{
  OpenDialog1->Execute();
  for (int i=0; i<100; i++)
    ComboBox1->Items->Add(L"test text");
  ComboBox1->ItemIndex = 1;   // <- exception occurs here
}

enable a VCL style 'Smokey Quartz Kamri' in the project options
run program, press button, select any file and select OK in TOpenDialog

Here, this is really all it takes for the exception.

In my opinion, the call to Execute() messes up some VCL structures (only if styles are enabled) and then the access to another VCL item (combobox in my case) leads to the crash.

I am now aware that not every one has this crash. So forgive me if it is not a 100% verifiable example for each of you.

But me and my colleague can't be the only ones who have this crash (tested now on 4 different computers with 3 different OS versions), can we?

* Edit 2016/05/27 *

Regarding Tom Brunberg's request for single step, the exception occurs somewhere within WndProc, in the screenshot at address 005459F4 within the call.

If I step further, I am landing somewhere in TCustomCombo.WndProc. After that is is very hard to follow further because of repeating loops in WndProc, can't seem to reach the final place where the exception fires.

* Edit 2016/05/27 second *

OK I managed to pinpoint the exact location of the crash. Is in in the function CopyBitmap within VCL.Graphics. In the first screenshot, exception occurs at line:

Result := GDICheck(CreateCompatibleBitmap(ScreenDC, bmWidth, bmHeight))

In the function GDICheck() in the second screenshot, the variable Value is zero in the debugger, so in turn function GDIError is called. There, ErrorCode is zero as well, this leads to the call to OutOfResources.

Hope this helps to narrow it down further.

* Edit 2016/07/19 *

Since nobody here seemed to have the issue, we gave it a different try: A colleague of mine in the company did a fresh C++ Builder 10.1 Berlin install, in English (thought maybe the German IDE is the culprit), and first thing after the install, recreated the StylesCrashTest App. Result is the same, it crashes at once after selecting a file and hit 'open' in the dialog.

I have uploaded the test project here http://fboom.me/file/9904e22ddd22b/StylesCrashTest.zip

and our generated release exe here http://fboom.me/file/368d0b62cc1a7/StylesCrashTest.exe

The exe is tested with many antivirus scanners on virustotal.com. https://www.virustotal.com/de/file/e96f2e7eb80c162c2e5998decc15f26615c9fc76efec73379dd2e2140e4eba08/analysis/1468952442/

It would be helpful if you guys could test the exe and the test project and this could lead to separate the issue to either computer related or related to the installed IDE/generated exe. This of course only if someone can reproduce the issue.

With this exe, the app crashes here on two Windows 7 x64 Enterprise computers in a commercial environment. It does not, however, crash on my private computer with Windows 8.1 x64 Prof.

Right now I am at a dead end, nobody outside the town of Munich seems to be able to reproduce the issue, but we have it definitely on two different computers.

The issue is also filed with Embarcadero (login required): https://quality.embarcadero.com/browse/RSP-15019

Sadly, at the moment, this is a shop stopper for us for using VCL styles.

FWIW, using Delphi 10 Seattle on Windows 7 Pro, I can not reproduce the problem. Is the error message something like "Out of system resources"? On which line does it happen? — Tom Brunberg, May 25 '16 at 18:12
Unable to replicate in either C++ Builder or Delphi 10.1 Berlin or Seattle using the Amakrits or Aqua Graphite styles. — Ken White, May 25 '16 at 18:14
Please provide a [Minimal, Complete, and Verifiable example](http://stackoverflow.com/help/mcve). — Remy Lebeau, May 25 '16 at 18:38
This may be nothing but you are passing a char array to add function. try passing `L"test text"`. and see if it makes any different. — Sam, May 25 '16 at 21:15
@TomBrunberg Yes, execption is "Out of system resources". It happens on line 'ComboBox1->ItemIndex = 1;' I think this may be because the VCL styled Execute() messes up some internal VCL structures, in this case the combobox. — Tom Major, May 26 '16 at 09:16
@KenWhite This exception does not seem to occur on every computer. My tests so far: 2x PC Win7 Enterprise, commercial enviroment, always exception inside(Debugger) and outside IDE. PC Win 8.1 private enviroment, exception only inside IDE, running the exe outside is fine. — Tom Major, May 26 '16 at 09:20
@RemyLebeau Well I tried to provide a very minimal example, all you need is this code as postet in my question ` //--------------------------------------------------------------------------- void __fastcall TForm1::Button1Click(TObject *Sender) { // enable a VCL style in the project options, e.g. 'Smokey Quartz Kamri' OpenDialog1->Execute(); for (int i=0; i<100; i++) ComboBox1->Items->Add(L"test text"); ComboBox1->ItemIndex = 1; // — Tom Major, May 26 '16 at 09:23
@TomMajor don't put code in comments. Edit the question instead. And a MCVE would include things like project settings, which style is used, etc. Information needed to reproduce the problem. — Remy Lebeau, May 26 '16 at 14:47
@Sam changing `char*` to `wchar_t*` will not make any difference. `Add()` takes a `String` as input, and `String` has constructors for both `char*` and `wchar_t*`. — Remy Lebeau, May 26 '16 at 14:49
@RemyLebeau I have edited my question in the lower part to give a MCVE as best as I can. One problem with this crash seems to be the fact that not everyone does have it with my example. — Tom Major, May 26 '16 at 17:43
If you step into ( F7 ) `ComboBox1->ItemIndex = 1;` in the debugger, in which unit/line is the error triggered? — Tom Brunberg, May 26 '16 at 19:45
@TomBrunberg Please see my new edit above. Exception seems to occur somewhere in TCustomCombo.WndProc. Hard to narrow it down further. — Tom Major, May 27 '16 at 14:03
Have you tried to use VCL styles units from Rodrigo Ruz as a workaround (https://github.com/RRUZ/vcl-styles-utils)? They are an extension of the VCL styles and could not have this bug. To test them is very easy so it is worth to try — lechonex, May 31 '16 at 17:36
@lechonex Sorry, does not seem easy for me to test these. I added the VCL Styles Utils directory to the library path as described in the installation, but how to make a VCL C++ application which uses these VCL Styles Utils? The demo projects are Delphi projects and I have only C++ personality, I can't open them. I could not find a description on how to make a C++ VCL app which uses these VCL Styles Utils by Rodrigo Ruz. — Tom Major, Jun 08 '16 at 20:36
Add the *.pas files to the project and they will be compiled — lechonex, Jun 13 '16 at 14:58
Which *.pas files must I add to project? And how do I enable these extension styles with C++ code? There are no files on github which answer these questions on how to use them in a C++ project. — Tom Major, Jun 14 '16 at 21:09
Perhaps related, [Delphi EOutOfResources (GDIError)](http://stackoverflow.com/q/6045939/576719). — LU RD, Jul 19 '16 at 21:46

score 2 · Accepted Answer · answered Jul 25 '16 at 17:21

I found a fix to get rid of the crash. After some more debugging, it was discovered that the exception occurs every time TBitmap::SetSize is called with a negative parameter on the VCL styled TComboBox.

Please see the call stack in the attached screenshot:

TComboBoxStyleHook::ListBoxWndProc
TComboBoxStyleHook::DrawListBoxVertScroll
TBitmap::SetHeight
TBitmap::SetSize

After editing SetSize() in Vcl.Graphics.pas to exit on negative numbers the exceptions seems to be gone. This is no fix for the cause of the exception, because why the parameter is set to -1 in DrawListBoxVertScroll is still unknown (bug?, could also be the result of a lot more sub calls within this routine), but at least it is a fix to prevent the exception. Tested on all our machines where the exception took place with positive results.

Would love to hear the opinion of some real VCL experts like Remy Lebeau or even the VCL developers about it, though.

Again, I realize that not everybody can even reproduce the exception, but with the test project linked above, the exception is inevitable on our systems.

Using VCL Styles gives a exception / crash in TOpenDialog

1 Answers1

Linked