Restart Explorer.exe during MSI install doesn't work

Question

I have a Visual Studio Setup Project that creates an MSI to install my application. My application has a shell extension with a icon overlay handler so explorer.exe needs to be restarted in order for the icon overlay handler to start working. I have registered a custom action to run on commit that restarts explorer using the following method:

public static void restartExplorer()
{
    //stop the shell
    try
    {
       Process process1 = new Process();
       ProcessStartInfo startInfo = new ProcessStartInfo();
       startInfo.WindowStyle = ProcessWindowStyle.Hidden;
       startInfo.FileName = "CMD.exe";
       startInfo.Arguments = "/C \"taskkill /f /im explorer.exe\"";
       process1.StartInfo = startInfo;
       process1.Start();
       do
        {
          Thread.Sleep(100);
          if (process1.HasExited)
          {
               break;
          }
         } while (true);
        }
        catch (Exception e)
        {
            Log.Error("restart explorer", e.StackTrace);
        }

        //restart the shell
        string explorer = string.Format("{0}\\{1}", Environment.GetEnvironmentVariable("WINDIR"), "explorer.exe");
        Process process = new Process();
        process.StartInfo.FileName = explorer;
        process.StartInfo.UseShellExecute = true;
        process.Start();

    }

This method works well when I test it from visual studio and call it from command line but when it is called from the MSI during install or uninstall it stops explorer, but does not restart it.

Does anyone know why it would work in all circumstances except when called by the MSI during install or uninstall?

Does anyone have an alternate way to restart the explorer from an MSI during install/uninstall?

Answer 1

Killing the explorer is a bit rough... I suggest you use the Restart Manager API for that. The benefit is Explorer knows how to restart itself, and it will restore all opened windows after the restart. Here is a C# utility class that will do it for you. Just call this in you custom action:

...
var rm = new RestartManager();
rm.RestartExplorerProcesses();

...

/// <summary>
/// A utility class to restart programs the most gracefully possible. Wraps Windows <see href="https://msdn.microsoft.com/en-us/library/windows/desktop/cc948910.aspx">Restart Manager API</see>. This class cannot be inherited.
/// </summary>
public sealed class RestartManager
{
    /// <summary>
    /// The default kill timeout value (2000).
    /// </summary>
    public const int DefaultKillTimeout = 2000;

    /// <summary>
    /// The default retry count value (10).
    /// </summary>
    public const int DefaultRetryCount = 10;

    /// <summary>
    /// The default retry timeout value (100).
    /// </summary>
    public const int DefaultRetryTimeout = 100;

    /// <summary>
    /// Initializes a new instance of the <see cref="RestartManager"/> class.
    /// </summary>
    public RestartManager()
    {
        KillTimeout = DefaultKillTimeout;
        RetryCount = DefaultRetryCount;
        RetryTimeout = DefaultRetryTimeout;
    }

    /// <summary>
    /// Gets or sets the kill timeout in ms.
    /// </summary>
    /// <value>The kill timeout.</value>
    public int KillTimeout { get; set; }

    /// <summary>
    /// Gets or sets the retry count.
    /// </summary>
    /// <value>The retry count.</value>
    public int RetryCount { get; set; }

    /// <summary>
    /// Gets or sets the retry timeout in ms.
    /// </summary>
    /// <value>The retry timeout.</value>
    public int RetryTimeout { get; set; }

    /// <summary>
    /// Restarts the Windows Explorer processes.
    /// </summary>
    /// <param name="stoppedAction">The stopped action.</param>
    public void RestartExplorerProcesses() => RestartExplorerProcesses(null, false, out var error);

    /// <summary>
    /// Restarts the Windows Explorer processes.
    /// </summary>
    /// <param name="stoppedAction">The stopped action.</param>
    public void RestartExplorerProcesses(ContextCallback stoppedAction) => RestartExplorerProcesses(stoppedAction, false, out var error);

    /// <summary>
    /// Restarts the Windows Explorer processes.
    /// </summary>
    /// <param name="stoppedAction">The stopped action.</param>
    /// <param name="throwOnError">if set to <c>true</c> errors may be throw in case of Windows Restart Manager errors.</param>
    public void RestartExplorerProcesses(ContextCallback stoppedAction, bool throwOnError) => RestartExplorerProcesses(stoppedAction, throwOnError, out var error);

    /// <summary>
    /// Restarts the Windows Explorer processes.
    /// </summary>
    /// <param name="stoppedAction">The stopped action.</param>
    /// <param name="throwOnError">if set to <c>true</c> errors may be throw in case of Windows Restart Manager errors.</param>
    /// <param name="error">The error, if any.</param>
    public void RestartExplorerProcesses(ContextCallback stoppedAction, bool throwOnError, out Exception error)
    {
        var explorers = Process.GetProcessesByName("explorer").Where(p => IsExplorer(p)).ToArray();
        Restart(explorers, stoppedAction, throwOnError, out error);
    }

    /// <summary>
    /// Restarts the processes locking a specific file.
    /// </summary>
    /// <param name="path">The file path.</param>
    /// <param name="stoppedAction">The stopped action.</param>
    /// <param name="throwOnError">if set to <c>true</c> errors may be throw in case of Windows Restart Manager errors.</param>
    /// <param name="error">The error, if any.</param>
    /// <exception cref="ArgumentNullException">path is null.</exception>
    public void RestartProcessesLockingFile(string path, ContextCallback stoppedAction, bool throwOnError, out Exception error)
    {
        if (path == null)
            throw new ArgumentNullException(nameof(path));

        var lockers = GetLockingProcesses(path, false, throwOnError, out error);
        if (error != null)
            return;

        Restart(lockers, stoppedAction, throwOnError, out error);
    }

    /// <summary>
    /// Restarts the Windows Explorer processes locking a specific file.
    /// </summary>
    /// <param name="path">The file path.</param>
    /// <param name="stoppedAction">The stopped action.</param>
    /// <param name="throwOnError">if set to <c>true</c> errors may be throw in case of Windows Restart Manager errors.</param>
    /// <param name="error">The error, if any.</param>
    /// <exception cref="ArgumentNullException">path is null.</exception>
    public void RestartExplorerProcessesLockingFile(string path, ContextCallback stoppedAction, bool throwOnError, out Exception error)
    {
        if (path == null)
            throw new ArgumentNullException(nameof(path));

        var processes = GetLockingProcesses(path, false, throwOnError, out error);
        if (error != null)
            return;

        var explorers = processes.Where(p => IsExplorer(p)).ToArray();
        Restart(explorers, stoppedAction, throwOnError, out error);
    }

    /// <summary>
    /// Determines whether the specified process is Windows Explorer.
    /// </summary>
    /// <param name="process">The process.</param>
    /// <returns><c>true</c> if the specified process is Windows Explorer; otherwise, <c>false</c>.</returns>
    public static bool IsExplorer(Process process)
    {
        if (process == null)
            return false;

        string explorerPath = Path.Combine(Environment.GetEnvironmentVariable("windir"), "explorer.exe");
        return string.Compare(process.MainModule.FileName, explorerPath, StringComparison.OrdinalIgnoreCase) == 0;
    }

    /// <summary>
    /// Gets a list of processes locking a specific file.
    /// </summary>
    /// <param name="filePath">The file path.</param>
    /// <param name="onlyRestartable">if set to <c>true</c> list only restartable processes.</param>
    /// <param name="throwOnError">if set to <c>true</c> errors may be throw in case of Windows Restart Manager errors.</param>
    /// <param name="error">The error, if any.</param>
    /// <returns>A list of processes.</returns>
    /// <exception cref="ArgumentNullException">filePath is null.</exception>
    public IReadOnlyList<Process> GetLockingProcesses(string filePath, bool onlyRestartable, bool throwOnError, out Exception error)
    {
        if (filePath == null)
            throw new ArgumentNullException(nameof(filePath));

        return GetLockingProcesses(new[] { filePath }, onlyRestartable, throwOnError, out error);
    }

    // NOTE: file name comparison seems to be case insensitive
    /// <summary>
    /// Gets a list of processes locking a list of specific files.
    /// </summary>
    /// <param name="filePaths">The files paths.</param>
    /// <param name="onlyRestartable">if set to <c>true</c> list only restartable processes.</param>
    /// <param name="throwOnError">if set to <c>true</c> errors may be throw in case of Windows Restart Manager errors.</param>
    /// <param name="error">The error, if any.</param>
    /// <returns>A list of processes.</returns>
    /// <exception cref="ArgumentNullException">filePaths is null.</exception>
    public IReadOnlyList<Process> GetLockingProcesses(IEnumerable<string> filePaths, bool onlyRestartable, bool throwOnError, out Exception error)
    {
        if (filePaths == null)
            throw new ArgumentNullException(nameof(filePaths));

        var processes = new List<Process>();
        var paths = new List<string>(filePaths);
        var s = new StringBuilder(256);
        int hr = RmStartSession(out int session, 0, s);
        if (hr != 0)
        {
            error = new Win32Exception(hr);
            if (throwOnError)
                throw error;

            return processes;
        }

        try
        {
            hr = RmRegisterResources(session, paths.Count, paths.ToArray(), 0, null, 0, null);
            if (hr != 0)
            {
                error = new Win32Exception(hr);
                if (throwOnError)
                    throw error;

                return processes;
            }

            int procInfo = 0;
            int rebootReasons = RmRebootReasonNone;
            hr = RmGetList(session, out int procInfoNeeded, ref procInfo, null, ref rebootReasons);
            if (hr == 0)
            {
                error = null;
                return processes;
            }

            if (hr != ERROR_MORE_DATA)
            {
                error = new Win32Exception(hr);
                if (throwOnError)
                    throw error;

                return processes;
            }

            var processInfo = new RM_PROCESS_INFO[procInfoNeeded];
            procInfo = processInfo.Length;

            hr = RmGetList(session, out procInfoNeeded, ref procInfo, processInfo, ref rebootReasons);
            if (hr != 0)
            {
                error = new Win32Exception(hr);
                if (throwOnError)
                    throw error;

                return processes;
            }

            for (int i = 0; i < procInfo; i++)
            {
                try
                {
                    if (processInfo[i].bRestartable || !onlyRestartable)
                    {
                        var process = Process.GetProcessById(processInfo[i].Process.dwProcessId);
                        if (process != null)
                        {
                            processes.Add(process);
                        }
                    }
                }
                catch (Exception e)
                {
                    error = e;
                    // do nothing, fail silently
                    return processes;
                }
            }
            error = null;
            return processes;
        }
        finally
        {
            RmEndSession(session);
        }
    }

    /// <summary>
    /// Restarts the specified processes.
    /// </summary>
    /// <param name="processes">The processes.</param>
    /// <param name="stoppedAction">The stopped action.</param>
    /// <param name="throwOnError">if set to <c>true</c> errors may be throw in case of Windows Restart Manager errors.</param>
    /// <param name="error">The error, if any.</param>
    /// <exception cref="ArgumentNullException">processes is null.</exception>
    public void Restart(IEnumerable<Process> processes, ContextCallback stoppedAction, bool throwOnError, out Exception error)
    {
        if (processes == null)
            throw new ArgumentNullException(nameof(processes));

        if (processes.Count() == 0)
        {
            error = null;
            return;
        }

        var s = new StringBuilder(256);
        int hr = RmStartSession(out int session, 0, s);
        if (hr != 0)
        {
            error = new Win32Exception(hr);
            if (throwOnError)
                throw error;

            return;
        }

        try
        {
            var list = new List<RM_UNIQUE_PROCESS>();
            foreach (var process in processes)
            {
                var p = new RM_UNIQUE_PROCESS()
                {
                    dwProcessId = process.Id
                };

                long l = process.StartTime.ToFileTime();
                p.ProcessStartTime.dwHighDateTime = (int)(l >> 32);
                p.ProcessStartTime.dwLowDateTime = (int)(l & 0xFFFFFFFF);
                list.Add(p);
            }

            hr = RmRegisterResources(session, 0, null, list.Count, list.ToArray(), 0, null);
            if (hr != 0)
            {
                error = new Win32Exception(hr);
                if (throwOnError)
                    throw error;

                return;
            }

            int procInfo = 0;
            int rebootReasons = RmRebootReasonNone;
            hr = RmGetList(session, out int procInfoNeeded, ref procInfo, null, ref rebootReasons);
            if (hr == 0)
            {
                error = null;
                return;
            }

            if (hr != ERROR_MORE_DATA)
            {
                error = new Win32Exception(hr);
                if (throwOnError)
                    throw error;

                return;
            }

            var processInfo = new RM_PROCESS_INFO[procInfoNeeded];
            procInfo = processInfo.Length;

            hr = RmGetList(session, out procInfoNeeded, ref procInfo, processInfo, ref rebootReasons);
            if (hr != 0)
            {
                error = new Win32Exception(hr);
                if (throwOnError)
                    throw error;

                return;
            }

            if (procInfo == 0)
            {
                error = null;
                return;
            }

            bool hasError = false;
            int wtk = GetWaitToKillTimeout();
            var sw = new Stopwatch();
            sw.Start();
            bool finished = false;
            var timer = new Timer((state) =>
            {
                if (!finished)
                {
                    HardKill(processes);
                }
            }, null, wtk + 2000, Timeout.Infinite);

            hr = RmShutdown(session, RmForceShutdown, percent =>
            {
                // add progress info code if needed
            });
            sw.Stop();

            if (hr != 0)
            {
                if (!IsNonFatalError(hr))
                {
                    error = new Win32Exception(hr);
                    if (throwOnError)
                        throw error;

                    return;
                }
                hasError = true;
            }

            if (hasError)
            {
                HardKill(processes);
            }

            if (stoppedAction != null)
            {
                int retry = RetryCount;
                while (retry > 0)
                {
                    try
                    {
                        stoppedAction(session);
                        break;
                    }
                    catch
                    {
                        retry--;
                        Thread.Sleep(RetryTimeout);
                    }
                }
            }

            hr = RmRestart(session, 0, percent2 =>
            {
                // add progress info code if needed
            });

            if (hr != 0)
            {
                error = new Win32Exception(hr);
                if (throwOnError)
                    throw error;

                return;
            }
        }
        finally
        {
            RmEndSession(session);
        }
        error = null;
    }

    private void HardKill(IEnumerable<Process> processes)
    {
        // need a hard restart
        foreach (var process in processes)
        {
            try
            {
                process.Refresh();
                if (!process.HasExited)
                {
                    process.Kill();
                }
            }
            catch
            {
                // do nothing
            }
        }
        Thread.Sleep(KillTimeout);
    }

    private static bool IsNonFatalError(int hr) => hr == ERROR_FAIL_NOACTION_REBOOT || hr == ERROR_FAIL_SHUTDOWN || hr == ERROR_SEM_TIMEOUT || hr == ERROR_CANCELLED;

    /// <summary>
    /// Gets the root Windows Explorer process.
    /// </summary>
    /// <returns>An instance of the Process type or null.</returns>
    public static Process GetRootExplorerProcess()
    {
        Process oldest = null;
        foreach (var process in EnumExplorerProcesses())
        {
            if (oldest == null || process.StartTime < oldest.StartTime)
            {
                oldest = process;
            }
        }
        return oldest;
    }

    /// <summary>
    /// Enumerates Windows Explorer processes.
    /// </summary>
    /// <returns>A list of Windows Explorer processes.</returns>
    public static IEnumerable<Process> EnumExplorerProcesses()
    {
        string explorerPath = Path.Combine(Environment.GetEnvironmentVariable("windir"), "explorer.exe");
        foreach (var process in Process.GetProcessesByName("explorer"))
        {
            if (string.Compare(process.MainModule.FileName, explorerPath, StringComparison.OrdinalIgnoreCase) == 0)
                yield return process;
        }
    }

    /// <summary>
    /// Enumerates a specific process' windows.
    /// </summary>
    /// <param name="process">The process.</param>
    /// <returns>A list of windows handles.</returns>
    /// <exception cref="ArgumentNullException">process is null.</exception>
    public static IReadOnlyList<IntPtr> EnumProcessWindows(Process process)
    {
        if (process == null)
            throw new ArgumentNullException(nameof(process));

        var handles = new List<IntPtr>();
        EnumWindows((h, p) =>
        {
            GetWindowThreadProcessId(h, out int processId);
            if (processId == process.Id)
            {
                handles.Add(h);
            }
            return true;
        }, IntPtr.Zero);
        return handles;
    }

    // https://technet.microsoft.com/en-us/library/cc976045.aspx
    /// <summary>
    /// Gets the wait to kill timeout, that is, how long the system waits for services to stop after notifying the service that the system is shutting down
    /// </summary>
    /// <returns>A number of milliseconds.</returns>
    public static int GetWaitToKillTimeout()
    {
        using (var key = Registry.LocalMachine.OpenSubKey(@"SYSTEM\CurrentControlSet\Control", false))
        {
            if (key != null)
            {
                var v = key.GetValue("WaitToKillServiceTimeout", 0);
                if (v != null && int.TryParse(v.ToString(), out int i))
                    return i;
            }
            return 0;
        }
    }

    [DllImport("user32.dll")]
    private static extern int GetWindowThreadProcessId(IntPtr handle, out int processId);

    private delegate bool EnumWindowsProc(IntPtr hWnd, IntPtr lParam);

    [DllImport("user32.dll")]
    private static extern bool EnumWindows(EnumWindowsProc callback, IntPtr extraData);

    [DllImport("rstrtmgr.dll", CharSet = CharSet.Unicode)]
    private static extern int RmStartSession(out int pSessionHandle, int dwSessionFlags, StringBuilder strSessionKey);

    [DllImport("rstrtmgr.dll")]
    private static extern int RmEndSession(int pSessionHandle);

    [DllImport("rstrtmgr.dll", CharSet = CharSet.Unicode)]
    private static extern int RmRegisterResources(int pSessionHandle, int nFiles, string[] rgsFilenames, int nApplications, RM_UNIQUE_PROCESS[] rgApplications, int nServices, string[] rgsServiceNames);

    [DllImport("rstrtmgr.dll")]
    private static extern int RmGetList(int dwSessionHandle, out int pnProcInfoNeeded, ref int pnProcInfo, [In, Out] RM_PROCESS_INFO[] rgAffectedApps, ref int lpdwRebootReasons);

    [DllImport("rstrtmgr.dll")]
    private static extern int RmShutdown(int dwSessionHandle, int lActionFlags, StatusHandler fnStatus);

    [DllImport("rstrtmgr.dll")]
    private static extern int RmRestart(int dwSessionHandle, int dwRestartFlags, StatusHandler fnStatus);

    /// <summary>
    /// Represents the method that handles status updates.
    /// </summary>
    /// <param name="percentComplete">The percentage completed.</param>
    public delegate void StatusHandler(int percentComplete);

    private const int RmRebootReasonNone = 0;
    private const int RmForceShutdown = 1;
    private const int RmShutdownOnlyRegistered = 0x10;
    private const int ERROR_MORE_DATA = 234;
    private const int ERROR_FAIL_NOACTION_REBOOT = 350;
    private const int ERROR_FAIL_SHUTDOWN = 351;
    private const int ERROR_SEM_TIMEOUT = 121;
    private const int ERROR_CANCELLED = 1223;
    private const int CCH_RM_MAX_APP_NAME = 255;
    private const int CCH_RM_MAX_SVC_NAME = 63;

    [StructLayout(LayoutKind.Sequential)]
    private struct RM_UNIQUE_PROCESS
    {
        public int dwProcessId;
        public System.Runtime.InteropServices.ComTypes.FILETIME ProcessStartTime;
    }

    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
    private struct RM_PROCESS_INFO
    {
        public RM_UNIQUE_PROCESS Process;

        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = CCH_RM_MAX_APP_NAME + 1)]
        public string strAppName;

        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = CCH_RM_MAX_SVC_NAME + 1)]
        public string strServiceShortName;

        public RM_APP_TYPE ApplicationType;
        public RM_APP_STATUS AppStatus;
        public int TSSessionId;

        [MarshalAs(UnmanagedType.Bool)]
        public bool bRestartable;
    }

    [Flags]
    private enum RM_APP_STATUS
    {
        RmStatusUnknown = 0x0,
        RmStatusRunning = 0x1,
        RmStatusStopped = 0x2,
        RmStatusStoppedOther = 0x4,
        RmStatusRestarted = 0x8,
        RmStatusErrorOnStop = 0x10,
        RmStatusErrorOnRestart = 0x20,
        RmStatusShutdownMasked = 0x40,
        RmStatusRestartMasked = 0x80
    }

    private enum RM_APP_TYPE
    {
        RmUnknownApp = 0,
        RmMainWindow = 1,
        RmOtherWindow = 2,
        RmService = 3,
        RmExplorer = 4,
        RmConsole = 5,
        RmCritical = 1000
    }
}

Answer 2

I have honestly never had the chance to test this out, but here is a quick read: Registering Shell Extension Handlers. Essentially: you are to call SHChangeNotify specifying the SHCNE_ASSOCCHANGED event. If you do not call SHChangeNotify, the change might not be recognized until the system is rebooted.

Found this on github.com: RgssDecrypter - ShellExtension.cs. And another sample.

Answer 3

Change your command to

cmd /c taskkill /f /im explorer.exe && start explorer.exe || start explorer.exe

and you will find that explorer.exe stops and restarts