"Auth cancel" is a SSH error (and SFTP works over SSH), and it means that an authenticator failed critically (or all of them failed).
If at all possible, set debug to max (-vvv) on the server's SSHd and see what error it reports exactly. This way you'll be sure of what's really happenig.
Failing that, these are some random attempts:
Usually this is caused by a key mismatch, or by an authenticator that is not implemented correctly by the client. For example if PHPStorm's SSH auth using public-key or keyboard-interactive malfunctioned, and the server allowed it, and no other authenticators were available before that, you would get "Auth cancel", and to solve it you would need to:
- either disable the failing auth module on the server (unless it is the last, in which case it would change nothing)
- or change the order of authenticators in the client and/or server to ensure a working authenticator gets a chance to play first
One possibility is to try all authenticators one at a time, both on server and client, until you find one that works. Then put it first in the authenticator list on the server.
Update
I'm trying to set up a SFTP connection to a server in PHPStorm 2017.1:
Settings > Building, Execution, Deployment > Deployment > Options
I create a new server ("Remote Server")
Connection tab:
Type: [ SFTP ]
SFTP host: [ my host ]
Port : [ 22 ]
Root path: [ /var/www/default/htdocs ]
User name: [ lserni ]
Auth type: [ Password ]
Password : [ *********** ] [x] Save password
and it worked (I needed to supply a password). You can set Auth type to "Key pair" and in that case PHPStorm asks for the private key (either in PuTTY or OpenSSH format - that's nice, as I use PuTTY on Windows; here you'll find how to generate both and convert between them. OpenSSH public keys are stored in /home/your-remote-user/.ssh/authorized_keys in the server) with its passphrase.
Possible causes of failure
Identity mismatch between local known_hosts (in Windows 7, it is C:\Users\YourName.ssh\known_hosts) and server identification. Solution: remove the known_hosts file on Windows. It will be correctly re-created at the next connection. I successfully recreated this by simply regenerating the server key.
Cipher mismatch between local PHPStorm and remote OpenSSH server. I had to pretty wreck the OpenSSH server to do this, but it can be done. Check out on the SSH server the line specifying ciphers (if there is none, then it is the default). Apparently my own PHPStorm uses AES-CBC-256, but it actually depends on the Java version that PHPStorm uses. See this answer for more info. Again, adding logging to the server SSHd should tell something useful:
LogLevel DEBUG3
Subsystem internal-sftp -l DEBUG3
(comment them with # after you're done - syslog debug file is going to be appreciably massive).
If this is the problem you will get
no matching cipher found: client aes256-cbc server [everything but].
Solution: set a shared Ciphers that work in sshd_config. In the case above, "aes256-cbc" must appear in the ciphers. Unfortunately what the ciphers are depends on client Java package, which may vary. I believe I have a vanilla install and it uses AES-256-CBC, so removing aes256-cbc made it fail, and putting it back made it work again. Some servers might prefer CTR by default, and not have CBC at all, and this might explain the failure. See this for more.
