64bit application + out of process 32bit modal dialog

Question

We work on plugin extension (toolbar) for 3rd party application. New version of application is also 64bit - and here comes the problem. Our plugin is developed under VB6. In 32bit application works all fine.

Manufacturer of an application delivers also wrapper (with 64bit edition), that runs automatically for 32bit plugins, so our plugin runs "out of process". Generally it works fine, but it needs (obviously) some necessary workarounds.

Our plugin has modal forms only. When it runs "out of process" of 64bit application, needs set owner and parent like so

Declare Function FindWindow Lib "user32" Alias "FindWindowA" (ByVal lpClassName As String, ByVal lpWindowName As String) As Long
Declare Function BringWindowToTop Lib "user32" (ByVal hwnd As Long) As Long
Declare Function GetLastActivePopup Lib "user32" (ByVal hwnd As Long) As Long
Declare Function SetParent Lib "user32" (ByVal hwndC As Long, ByVal hwndP As Long) As Long
Declare Function GetCurrentProcessId Lib "kernel32" () As Long
Declare Function GetWindowLong Lib "user32" Alias "GetWindowLongA" (ByVal hwnd As Long, ByVal nIndex As Long) As Long
Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" (ByVal hwnd As Long, ByVal nIndex As Long, ByVal dwNewLong As Long) As Long

Public Sub PopupWindow(formHwnd As Long)
  '(Declarations here...)
  If (App.GetProcessId() <> GetCurrentProcessId()) Then
    hWndMain = FindWindow(vbNullString, App.Caption)
    hWndParent = GetLastActivePopup(hWndMain)
    CurrentStyle = GetWindowLong(formHwnd, GWL_STYLE)

    If (hWndParent = 0) Then
      hWndParent = hWndMain
      BringWindowToTop hWndParent
      SetWindowLong formHwnd, GWL_STYLE, CurrentStyle Or WS_POPUP
      SetParent formHwnd, hWndMain
    Else
      SetWindowLong formHwnd, GWL_STYLE, CurrentStyle Or WS_POPUP
      SetWindowLong formHwnd, GWL_HWNDPARENT, hWndParent
    End If
  End If
End Sub

Code for displaying Form is now:

PopupWindow formType.hwnd
formType.Show vbModal, [OwnerForm]

1) The first small problem is, that click to "parent" Application window causes losing focus of our window from plugin (but plugin window is still on top and parent Application has disabled controls). In 32bit edition it behaves standard way - plays "beep" and focus flash on modal dialog window - how get it done on "out of process"?

On The Old New Thing Blog is described this problem (yellow rectangle) and thereafter solution like

When you call DialogBox in process B, pass hwndA as the owner window

But in VB6 you have function Form.Show([Modal], [OwnerForm]), where OwnerForm means type, not hWnd.

Type of Appliacation Form is not available in its interface. I can not find solution using WinAPI.

2) The second problem (And I think the cause is the same) is, that when i have shown plugin modal dialog, then click "View Desktop" (Or Win+M), then I will go back to Application (click on icon in panel), it plays only "Beep" and do nothing. (solution is only kill the Application). Again, 32bit edition works fine.

Thank you for answers and appologize for my english. Yes, i know that VB6 is out of support now, the plugin is only in maintenance mode, development of the new version based on C# is in progress. Until it's done, VB6 must work.

EDIT:

In Spy++ difference in window property between running in 32bit App and 64bit App:

Plugin Window Styles differences under 32bit vs. 64bit App:

WS_CAPTION
*** WS_POPUP (only with 64bit App, because of code above)
WS_VISIBLE
WS_CLIPSBLINGS
WS_CLIPCHILDREN
WS_SYSMENU
WS_THICKFRAME
*** WS_OVERLAPPED (only with 32bit App)
WS_MINIMIZEBOX

Plugin Window Parent/Owner differences under 32bit vs. 64bit App:

*** Parent Window (32bit App None, 64bit App - Main Application hWnd) 
Owner Window (32bit and 64bit App - Main Application hWnd)

You'll also have to call EnableWindow to disable the parent window, fixes #1. If you don't then you get the deadlock from AttachThreadInput that Raymond warned about, fixes #2. — Hans Passant, Jul 23 '13 at 10:35
If I try to call `EnableWindow(hWndMain, 0)`, it returns 0, that means Main application window was **not** previously disabled - this is ok. But it still behaves the same way. — Ondrej, Jul 23 '13 at 11:50
You do have to make sure that you got the correct window handle. Use Spy++ to double-check. Odds that you get the right one from GetLastActivePopup() are certainly not great. — Hans Passant, Jul 23 '13 at 11:55
Yes, I have correct window handle (checked with Spy++). Code goes every time through `Else` statement (at least in debug mode). `hWndMain` equals `hWndParent` until it is not a new child dialog of plugin dialog. — Ondrej, Jul 23 '13 at 13:55
See edit of Question - isn't probem that under 64bit App has plugin dialog window both owner and parent? [A window can have a parent or an owner but not both](http://blogs.msdn.com/b/oldnewthing/archive/2010/03/15/9978691.aspx). I tried `SetParent(FormHwnd, 0)`, but Window parent was still there. — Ondrej, Jul 23 '13 at 14:23

64bit application + out of process 32bit modal dialog

0 Answers0