So I got this error too, and I was able to find a way around it, albeit, not a scalable way. However, if your concern is that it works on just your machine, then this solution will work for you.
To be clear to anyone else who reads this answer, the reason we are all here is that IBM Notes (IBM Lotus Notes) actually runs atop Java, specifically Java 1.6 (and actually on an IBM J9 VM). Though much of the customization code for Notes can be written in LotusScript (a VisualBasic-like variant), Notes allows you to do development in Java itself. This allows for wonderful flexibility, but if your code needs to do any communication over HTTPS and the server you are communicating with implements TLS 1.2 (a modern standard as of this answer in 2018), Java 1.6 does not natively support TLS 1.2. So what's a good way around that? A 3rd party library such as Bouncy Castle. That brings us here.
The Error
The error/stack trace is the exact same error I got. It does not give a lot of meaningful information. So what I did is a played around and modified the Bouncy Castle source code a bit (specifically BouncyCastleProvider.java). Where it uses reflection to instantiate each digest/cipher/etc., I just manually instantiated each class. You'll see that it is failing in the loadAlgorithms method. The loadAlgorithms method instantiates a list of classes, and then calls the .configure() method on the class, passing in this (an instance of the BouncyCastleProvider class. When I did it manually, I reproduced the same error but with a bit more details:
java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.initialize(J9VMInternals.java:222)
at JavaAgent.NotesMain(Unknown Source)
at lotus.domino.AgentBase.runNotes(Unknown Source)
at lotus.domino.NotesThread.run(Unknown Source)
Caused by: java.lang.SecurityException
at java.lang.SecurityManager.checkPermission(SecurityManager.java:582)
at COM.ibm.JEmpower.applet.AppletSecurity.checkSecurityPermission(AppletSecurity.java:1332)
at COM.ibm.JEmpower.applet.AppletSecurity.checkPermission(AppletSecurity.java:1613)
at COM.ibm.JEmpower.applet.AppletSecurity.checkPermission(AppletSecurity.java:1464)
at java.lang.SecurityManager.checkSecurityAccess(SecurityManager.java:1725)
at java.security.Provider.put(Provider.java:243)
at ExplicitBouncyCastleProvider.addAlgorithm(Unknown Source)
at org.bouncycastle.jcajce.provider.digest.GOST3411$Mappings.configure(Unknown Source)
at ExplicitBouncyCastleProvider.setup(Unknown Source)
at ExplicitBouncyCastleProvider.access$0(Unknown Source)
at ExplicitBouncyCastleProvider$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(AccessController.java:202)
at ExplicitBouncyCastleProvider.<init>(Unknown Source)
at TLSSocketConnectionFactory.<clinit>(Unknown Source)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
... 6 more
Now we can go into each provider, in this case, the GOST3411.Mappings class (since it is the first one loaded by Bouncy Castle) and see that it is failing on the call to java.security.Provider.put() because that method has a SecurityManager that prevents this. In other words, Notes has security in place that does not just allow a Java Agent to modify java's security settings.
How to modify Notes' Java Security Settings
Your answer already had a hint to it on how to do it, but I had to do a few more things.
On the machine where my client is running, modify the <notes installation>\jvm\lib\security\java.security file to add to the list:
security.provider.8=org.bouncycastle.jce.provider.BouncyCastleProvider
(I chose 8 because I believe security.provider.n only supports n=1 to 9, and I wasn't using Apache Harmony, which was 8th on my list).
Put the BouncyCastle JAR in the <notes installation>\jvm\lib\ext directory. In my case, I was using Bouncy Castle 1.59 for Java 1.5 onward, so I had put a copy of bcprov-jdk15on-159.jar in this directory.
In the <notes installation>\jvm\lib\security\trusted.libraries file, add a base-64 encoded SHA-1 hash of the JAR you just placed in the ext directory. I used FCIV to generate a SHA-1 hash (which is hexadecimal) of the file, then converted that hex value to base64 using any old online hex to base64 converter tool. It will be helpful if you put a comment above it too so it is like all other entries in this file.
# bcprov-jdk15on-159.jar
SHA1-Digest-Manifest: JQcgQkGrRQRWvbjowKj5huQYvZk=
Now, when you call HttpsURLConnection con = (HttpsURLConnection) urlObj.openConnection(); in your Java code, it will automatically find this Bouncy Castle provider for TLS 1.2, realize that it trusts it, and open the connection over HTTPS using TLS 1.2. This solved the same problem for me.
Disclaimer 1: Again, this must be done for the client, i.e. all clients that need to do this will have to go through this procedure.
Disclaimer 2: I did this on the same machine that I had IBM Domino Designer installed. (I believe one of the steps I went through was to add this JAR in the ext directory to the build path of the database's Java Project in Domino designer, but I don't think that was actually necessary).
Now all this being said, the only realistic way to do this for all Notes clients that must use this code is to modify your IBM Notes installer and bundle it in with the client installation for all future installations.
Instead, to avoid literally all of this, within your company's secure VPN, use either plain HTTP or TLS 1.0 (which Java 1.6 does support) over an internal Proxy. In other words, set up a web relay on your company intranet that listens to plain HTTP requests (or you can secure it with TLS 1.0 if you really want to use HTTPS) and forwards them to the destination using TLS 1.2. This is known as a proxy. This would live entirely outside of Notes and can be done using Apache HTTPD or any programming language you prefer, like Java or C#.
