Some of what's here may repeat what's on the above links - putting everything in one place to make a complete checklist for anyone with this issue.
Add the following ports/protocols to inbound exceptions on both machines (i.e. your desktop & the sql server). Alternatively turn off any firewalls on desktop & server to skip a few of these steps.
On your desktop also add devenv.exe to your firewall's process/program exceptions list.
On the server add firewall exceptions for the sqlsrvr.exe and msvsmon.exe processes.
On the server also add a firewall exception for File and Printer Sharing.
Is the VS remote debugging service installed on your server?
Is SQL/CLR debugging enabled on the database / server?
Are you running VS Pro or Team edition?
Does your account have admin rights on the server (OS)?
Does your account have admin rights on the database instance (SQL)?
Here's more info on why sysadmin's required on SQL Server in case there are arguments from DBAs / managers about restricting access further: http://blogs.msdn.com/b/sqlclr/archive/2006/07/07/659332.aspx
Are you connecting using windows authentication?
Have you tried editing sqlserver.config as per: http://www.sqlskills.com/BLOGS/BOBB/post/SQLCLR-debugging-and-VS-2010-revisited.aspx
(please note the author's cautions; i.e. this is a hack which can cause issues with the SQL debugger's stability so only do this if you're the only person attaching to this server).
<configuration>
...
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedRuntime version="v2.0.50727" />
</startup>
</configuration>
If you're on a company network check that there are no proxies which may affect you (if your machine doesn't realise the server is on the same intranet sometimes you'll be going via a proxy). Try using the server's FQDN when connecting as having the full path explicitly stated sometimes helps (at least I've seen this get around issues with ISA Server).
If you have any virus software ensure that exceptions are setup for file extensions .mdf, .ndf, .ldf, .bak, .trn & .$$$.
Are you running the latest service packs for VS, SQL and client & server OSes?
If you still have issues after all that, have you tried running the SQL and VS installers in repair mode, or removing and reinstalling the software?