-1

I'm creating a server and client all-in-one chat application and I'm trying to switch to an SSL connection. I created a keystore.jks and a certificate file (.cer) but now when the program tries to make a connection the acting client throws:

Caused by: java.io.IOException: Invalid keystore format

Here is the code:

System.setProperty("javax.net.ssl.keyStore", "certificates/keystore.jks");
System.setProperty("javax.net.ssl.trustStore", "certificates/certificate.cer");
System.setProperty("javax.net.ssl.keyStorePassword", "password");

if (this.role == ConnectionRole.SERVER) {
    connectingAlert.getJFrame().setVisible(true);
    setupServer();
    do {
        Thread.sleep(10);
    } while (socket == null);
}

if (this.role == ConnectionRole.CLIENT) {
    connectingAlert.getJFrame().setVisible(true);
    setupClient(targetIP);
}

private void setupServer() throws IOException {
    SSLServerSocketFactory sslSrvFact = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
    serverSocket = (SSLServerSocket) sslSrvFact.createServerSocket(8080, 1);
    socket = (SSLSocket) serverSocket.accept();
    setupStreams();
}

private void setupClient(String IPAddress) throws IOException {
    SSLSocketFactory sslFact = (SSLSocketFactory) SSLSocketFactory.getDefault();
    socket = (SSLSocket) sslFact.createSocket("localhost", 8080);
    setupStreams();
}

private void setupStreams() throws IOException {
    dataOut = new ObjectOutputStream(socket.getOutputStream());
    dataIn = new ObjectInputStream(socket.getInputStream());
    chatInterface = ChatInterface.getInstance();
}
user207421
  • 289,834
  • 37
  • 266
  • 440
DK_
  • 137
  • 4
  • 11
  • I'm completely new to this, just trying to follow tutorials. Adding the javax.net.ssl.trustStore meant I no longer got the error: `javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` – DK_ May 08 '16 at 21:02
  • Since the version of the code used in the post (with the invalid format error) I removed the line `System.setProperty("javax.net.ssl.trustStore", "certificates/certificate.cer");` to show you what error I got. After I added the line back. With the code the same as it was in the post I went to the linked "possible duplicate" questiion and used the exact line of code that was in the top comment. I then posted the comment about the keytool error I got. Later today I will re-attempt to create the keystore and certificate and I'll comment the commands I used to do this. – DK_ May 09 '16 at 13:31
  • I did not. That sounds like it might be my problem. So I should create one for the server and one for the client. So I think I have to use a self signed certificate which will be in both the server and client key stores? – DK_ May 09 '16 at 17:56
  • Ok. I managed to get it done. I went from the start using [To Use keytool to Create a Server Certificate] (http://docs.oracle.com/cd/E19798-01/821-1841/gjrgy/index.html). Thank you so much for your patience and answers @dur. – DK_ May 10 '16 at 11:51

1 Answers1

0
System.setProperty("javax.net.ssl.trustStore", "certificates/certificate.cer");

The problem is here. A .cer file is not a truststore. You need to import it into a real Java truststore via the keytool with the -trustcacerts option.

BUT it isn't clear why you're using a truststore at all. Are you expecting peers with self-signed certificates to send them to you? Most of the time you should just use the truststore that comes with Java, and don't set javax.net.ssl.trustStore at all.

dur
  • 13,039
  • 20
  • 66
  • 96
user207421
  • 289,834
  • 37
  • 266
  • 440
  • Could you recommend what I should use? Looking online for solutions it seemed like this is what I should be using. This is a server/client in one program (depending on user input), what should I use to establish the SSL connection? – DK_ May 09 '16 at 13:20
  • `BUT it isn't clear why you're using a truststore at all.`, `you should just use the truststore that comes with Java`. I was asking how to do this. Only setting `javax.net.ssl.keyStore` and `javax.net.ssl.keyStorePasword` gives me a `PKIX path building failed` error on the client side and `Received fatal alert: certificate_unknown` on the server side. – DK_ May 10 '16 at 09:21
  • Well suppose you answer the question I asked. 'Are you expecting peers with self-signed certificates to send them to you?' And if so, why? You should use CA-signed certificates. But if that's what you're doing, that's why you would need a custom truststore. The 'unknown_certificate` problem is a separate issue altogether: the server doesn't trust *your* certificate. – user207421 May 11 '16 at 00:41