The browser in Windows Phone 7 does not currently support HTML5, though it is expected to arrive later this year. That said, I would extremely surprised if there was ever a public API that enabled a web site to open an application on the host device and I'd be surprised if this was allowed on Android or iPhone, too.
The only application that I'm aware of that exhibits this behavior is the YouTube application. If you visit YouTube.com on the WP7 browser, then it either launches the YouTube app if you have it installed, or prompts yout to download and install it. So, the capability is there, but I doubt very strongly if Microsoft would ever open it up beyond a close partnership for specific applications.
In Windows Phone 7, you can use the WebBrowser
control within an application to host/browse web content, and from the application you can call javascript methods exposed by that page as described by Shawn Wildermuth in his Navigating with the WebBrowser Control on WP7 post.