Software Secret Weapons: Magic keystore: connecting to PayPal HTTPS endpoints from Java

The problem

I have been playing with various PayPal Java SDK's this weekend. Well, I wanted to play with them, but was not able to. The SSL certificates got in my way...

All connections to the live or testing (sandbox) PayPal sites require HTTPS to keep the communication private. In order to open these SSL connections in Java or any other language one must jump through some hoops and do some tricks. Unexpectedly, it took me a whole day to get everything working properly, while it should have taken no time at all. This post walks you through the issues and the solutions so you don't have to waste your time.

NOTE: This article applies only to versions of J2SE 1.4.2 prior to update 6.

Java SSL/HTTPS Background

If you ever used Java to connect to the HTTP server, you are likely to write the same kind of code to connect to the HTTPS server:

	public static void main(String [] args) throws Exception {
		URL url = new URL("https://api.sandbox.paypal.com/nvp");
		URLConnection conn = url.openConnection();

		BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
		String line;
	        while ((line = rd.readLine()) != null) {
	            System.out.println(line);
	        }
		rd.close();		 
	}

Don't be surprised when it does not work for HTTPS and the following exception is thrown:

	javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found

I don't know the deep reasons behind it, but the HTTPS server is not implicitly trusted by the Java runtime. One has to explicitly add specific HTTPS server to the list of trusted entitles. After you have done so the code above will work unchanged. But I did not want to screw around with the keystores and the certificates just to connect to a web sever over HTTPS. There got to be a better way I thought...

Disabling server certificate validation

The first time I faced the same HTTPS issue was while writing the automation and testing framework for Moola with HttpUnit. To simplify our life instead of dealing with the certificates and the server trust we just turned the certificate validation off. You can find an example of how this is done in Java Almanac. Looks likes this below more or less:

	TrustManager[] trustManagers = new TrustManager[] {
        new X509TrustManager() {
            public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                return null;
            }
            
            public void checkClientTrusted(
                    X509Certificate[] certs,String authType) {}
            
            public void checkServerTrusted(
                    X509Certificate[] certs, String authType) {}
            
	        }
	};
		
	SSLContext sc = SSLContext.getInstance("SSL");
	sc.init(null, trustManagers, new SecureRandom());
	HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());

You have to call this code just once before first HTTPS connection is made and any HTTPS server will be silently trusted. Works!!! I can now connect to PayPal and get the default server response:

	ACK=Failure
		&L_ERRORCODE0=81002
		&L_SHORTMESSAGE0=Unspecified%20Method
		&L_LONGMESSAGE0=Method%20Specified%20is%20not%20Supported
		&L_SEVERITYCODE0=Error

Well, I am quite satisfied with my hacking... At the same time, hacking does not go very far when money is at stake. Any payments API must be properly secured to protect all parties: the buyer, the seller and the intermediary (PayPal in this case). I am concerned about properly configuring the production servers to use PayPal endpoints securely. One needs to find all these public certificates and add them to the proper client side Java keystore. How hard can this be...

Locating PayPal public certificates

Inside numerous folders of PayPal (Name-Value Pairs) NVP SDK you will find paypal-endpoints-nvp.xml file that describes all PayPal connection endpoints. From this file you can determine that following twelve distinct HTTPS servers are in play for PayPal NVP API support:

PayPal Live:

https://api.paypal.com/nvp
https://api-aa.paypal.com/nvp
https://api-3t.paypal.com/nvp
https://api-aa-3t.paypal.com/nvp

PayPal Sandbox (test):

https://api.sandbox.paypal.com/nvp
https://api-aa.sandbox.paypal.com/nvp
https://api-3t.sandbox.paypal.com/nvp
https://api-aa-3t.sandbox.paypal.com/nvp

PayPal Beta Sandbox (upcoming):

https://api.beta-sandbox.paypal.com/nvp
https://api-aa.beta-sandbox.paypal.com/nvp
https://api-3t.beta-sandbox.paypal.com/nvp
https://api-aa-3t.beta-sandbox.paypal.com/nvp

Consequently, I expected twelve distinct server public certificates in the SDK distributions, but I did not find them. I found two:

paypal_cert_pem.txt
sandbox_cert_pem.txt

Not finding exact direction on what specific certificates to use I have decided to extract the certificates from live servers. This can be done by using OpenSSL. OpenSSL allows one to connect to the SSL server and "debug" the connection, capturing the public certificate along the way.

Here is how to get it to work. Download Windows binary distribution of OpenSSL 0.9.8g and install it. Use this Windows batch file to get all twelve PayPal server certificates into the corresponding text files. The export.bat connects to the specific HTTPS server using OpenSSL and gets all the debug information from it including the certificate:

	SETLOCAL
	SET OPENSSL_HOME=C:\tmp\openssl
	  
	:export_all 
		CALL :export api.paypal.com
		CALL :export api-aa.paypal.com
		CALL :export api-3t.paypal.com
		CALL :export api-aa-3t.paypal.com
		 
		CALL :export api.sandbox.paypal.com
		CALL :export api-aa.sandbox.paypal.com
		CALL :export api-3t.sandbox.paypal.com
		CALL :export api-aa-3t.sandbox.paypal.com 
		 
		CALL :export api.beta-sandbox.paypal.com
		CALL :export api-aa.beta-sandbox.paypal.com
		CALL :export api-3t.beta-sandbox.paypal.com
		CALL :export api-aa-3t.beta-sandbox.paypal.com
	GOTO :eof
	
	:export
		%OPENSSL_HOME%\bin\openssl.exe s_client -connect %1:443 > c:\tmp\%1-cert.txt
	GOTO :eof

Running export.all.bat above will produce twelve text files that contain HTTPS server debug information like this:

	CONNECTED(0000076C)
	---
	Certificate chain
	0 s:/C=US/ST=California/L=San Jose/O=Paypal, Inc./OU=Information 
	Systems/OU=Terms of use at www.verisign.com/rpa (c)00/CN=api.sandbox.paypal.com
	i:/O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International 
	Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
	1 s:/O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International 
	Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
	i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
	2 s:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
	i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
	---
	Server certificate
	-----BEGIN CERTIFICATE-----
	MIIEfDCCA+WgAwIBAgIQBVwmNOUQ1+ULF6IsBSgYhTANBgkqhkiG9w0BAQUFADCB
	ujEfMB0GA1UEChMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazEXMBUGA1UECxMOVmVy
	aVNpZ24sIEluYy4xMzAxBgNVBAsTKlZlcmlTaWduIEludGVybmF0aW9uYWwgU2Vy
	dmVyIENBIC0gQ2xhc3MgMzFJMEcGA1UECxNAd3d3LnZlcmlzaWduLmNvbS9DUFMg
	SW5jb3JwLmJ5IFJlZi4gTElBQklMSVRZIExURC4oYyk5NyBWZXJpU2lnbjAeFw0w
	NjAzMDcwMDAwMDBaFw0wODAzMDYyMzU5NTlaMIHAMQswCQYDVQQGEwJVUzETMBEG
	A1UECBMKQ2FsaWZvcm5pYTERMA8GA1UEBxQIU2FuIEpvc2UxFTATBgNVBAoUDFBh
	eXBhbCwgSW5jLjEcMBoGA1UECxQTSW5mb3JtYXRpb24gU3lzdGVtczEzMDEGA1UE
	CxQqVGVybXMgb2YgdXNlIGF0IHd3dy52ZXJpc2lnbi5jb20vcnBhIChjKTAwMR8w
	HQYDVQQDFBZhcGkuc2FuZGJveC5wYXlwYWwuY29tMIGfMA0GCSqGSIb3DQEBAQUA
	A4GNADCBiQKBgQC94WY02PsSXQiwCD8Q+cmgId9xszUt8Li6WKYyEMQP7nX5PiHZ
	ziTmblv/Gh2bYzBvMIL8YczYcSgNXQV0SlUorh7+OCG32UZw09fxiIwHxC25/y/9
	N1RL2oL1/S45jnW4AOiqg9z/spCgX58auPyTfvDoZRenM6omwvg6EePkWwIDAQAB
	o4IBeTCCAXUwCQYDVR0TBAIwADALBgNVHQ8EBAMCBaAwRgYDVR0fBD8wPTA7oDmg
	N4Y1aHR0cDovL2NybC52ZXJpc2lnbi5jb20vQ2xhc3MzSW50ZXJuYXRpb25hbFNl
	cnZlci5jcmwwRAYDVR0gBD0wOzA5BgtghkgBhvhFAQcXAzAqMCgGCCsGAQUFBwIB
	FhxodHRwczovL3d3dy52ZXJpc2lnbi5jb20vcnBhMCgGA1UdJQQhMB8GCWCGSAGG
	+EIEAQYIKwYBBQUHAwEGCCsGAQUFBwMCMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEF
	BQcwAYYYaHR0cDovL29jc3AudmVyaXNpZ24uY29tMG0GCCsGAQUFBwEMBGEwX6Fd
	oFswWTBXMFUWCWltYWdlL2dpZjAhMB8wBwYFKw4DAhoEFI/l0xqGrI2Oa8PPgGrU
	SBgsexkuMCUWI2h0dHA6Ly9sb2dvLnZlcmlzaWduLmNvbS92c2xvZ28uZ2lmMA0G
	CSqGSIb3DQEBBQUAA4GBAJqvib+dgYjKReAr7aPcZg9W3RKHqyaJIrnFS3URP4sW
	lKIKBnhXsdOXu5EPg7/arMxz/jAh6dj7jUXVYmm+llhNUcKpQ6OZHZo5TEKX/mhH
	LiMFjmH/ngo1xePdk93XSfO9mt1NDGhWi7h9jZaYwHCOsgAqytx91t4BglOfrvHh
	-----END CERTIFICATE-----
	subject=/C=US/ST=California/L=San Jose/O=Paypal, Inc./OU=Information 
	Systems/OU=Terms of use at www.verisign.com/rpa (c)00/CN=api.sandbox.paypal.com
	issuer=/O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International 
	Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. 
	LIABILITY LTD.(c)97 VeriSign
	---
	Acceptable client certificate CA names
	/C=US/ST=California/L=San Jose/O=PayPal, Inc./OU=sandbox_certs
	/CN=sandbox_camerchapi/emailAddress=re@paypal.com
	---
	SSL handshake has read 3376 bytes and written 334 bytes
	---
	New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
	Server public key is 1024 bit
	Compression: NONE
	Expansion: NONE
	SSL-Session:
	Protocol : TLSv1
	Cipher : DHE-RSA-AES256-SHA
	Session-ID: 74F5D893EDD409726F7EC977A50B6C6BE72E51BA970EA2BD63EEB8E3669C740E
	Session-ID-ctx: 
	Master-Key: 04ADE9A44CF68A68B10354F345639FE872728458893D3FF842
	08413E29D5A1B57570EF8001319FD6BAD4ADF166AEEDFC
	Key-Arg : None
	Start Time: 1199000184
	Timeout : 300 (sec)
	Verify return code: 19 (self signed certificate in certificate chain)
	---
	<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
	<HTML><HEAD>
	<TITLE>500 Internal Server Error</TITLE>
	</HEAD><BODY>
	<H1>Internal Server Error</H1>
	The server encountered an internal error or
	misconfiguration and was unable to complete
	your request.<P>
	Please contact the server administrator,
	ppmts@paypal.com and inform them of the time the error occurred,
	and anything you might have done that may have
	caused the error.<P>
	More information about this error may be available
	in the server error log.<P>
	<P>Additionally, a 404 Not Found
	error was encountered while trying to use an ErrorDocument to handle the request.
	<HR>
	<ADDRESS>Apache/1.3.27 Server at <A HREF="mailto:ppmts@paypal.com">
	api.sandbox.paypal.com</A> Port 20001</ADDRESS>
	</BODY></HTML>
	closed

All debug info for all PayPal NVP servers above can be downloaded as one zip file here.

OpenSSL reported errors trying to connect to two production PayPal servers namely api-aa.paypal.com and api.paypal.com. Not sure what that is wrong with "self signed certificate in certificate chain", but here is the command output anyways:

	C:\>C:\tmp\openssl\bin\openssl.exe s_client -connect api.paypal.com:443  1>c:\tmp\api.paypal.com-cert.txt
	Loading 'screen' into random state - done
	depth=2 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
	verify error:num=19:self signed certificate in certificate chain
	verify return:0
	4812:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:.\ssl\s3_pkt.c:1053:SSL alert number 40
	4812:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:.\ssl\s23_lib.c:188:
	
	C:>C:\tmp\openssl\bin\openssl.exe s_client -connect api-aa.paypal.com:443  1>c:\tmp\api-aa.paypal.com-cert.txt
	Loading 'screen' into random state - done
	depth=2 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
	verify error:num=19:self signed certificate in certificate chain
	verify return:0
	5048:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:.\ssl\s3_pkt.c:1053:SSL alert number 40
	5048:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:.\ssl\s23_lib.c:188:

The last step is to extract the certificate data for each server into a separate file, like I did here for api-3t.paypal.com:

All public server certificates for all PayPal NVP servers above can be downloaded as one zip file here.

Now as we have all the certificates downloaded right from live servers we have to add them to Java keystore. How hard can this be...

Adding PayPal public certificates to Java keystore

I always start with the well known initial state. This means that I recommend you to start with new keystore. Nothing is stopping you from adding the certificates right into the JRE default shared keystore located at %JAVA_HOME%\jre\lib\security\cacerts (password 'changeit'), but I advise against it. It is better to have a separate keystore for each specific purpose - in this case - a keystore to work with PayPal.

Here is the import.bat batch file that I used to generate the new keystore and add all PayPal server certificates into it.

	@ECHO OFF
	
	REM VARIABLES
		SETLOCAL
		SET JAVA_HOME=D:\MOO_NODE\bin\j2sdk1.4.2
		SET CERT_HOME=c:\tmp
		SET KEYSTORE_COMMON=-keystore %CD%/keystore.dat -storepass store12345 
	  
	REM Delete old keystore if exists
		DEL %CD%\keystore.dat /Q
		
	REM Create new keystore
		%JAVA_HOME%\bin\keytool.exe %KEYSTORE_COMMON% -genkey -keypass 
			key12345 -dname "cn=Name, ou=Department, o=Company, c=US" -alias alias -validity 3600
		
	REM Add trusted certificate
		REM CALL :import api.paypal.com
		REM CALL :import api-aa.paypal.com
		CALL :import api-3t.paypal.com
		CALL :import api-aa-3t.paypal.com
		
		CALL :import api.sandbox.paypal.com
		CALL :import api-aa.sandbox.paypal.com
		CALL :import api-3t.sandbox.paypal.com
		CALL :import api-aa-3t.sandbox.paypal.com 
		 
		CALL :import api.beta-sandbox.paypal.com
		CALL :import api-aa.beta-sandbox.paypal.com
		CALL :import api-3t.beta-sandbox.paypal.com
		CALL :import api-aa-3t.beta-sandbox.paypal.com	
	  	  
	REM List content
		%JAVA_HOME%\bin\keytool.exe %KEYSTORE_COMMON% -list 		
	GOTO :eof
	 
	:import 
		%JAVA_HOME%\bin\keytool.exe %KEYSTORE_COMMON% -import -noprompt -alias pp_%1 -file %CERT_HOME%\%1-cert.txt
	GOTO :eof

The import.bat creates new keystore keystore.dat in the current directory and adds to it all PayPal server certificates created above. Here is the command output:

	Keystore type: jks
	Keystore provider: SUN
	
	Your keystore contains 11 entries
	
	pp_api-aa.beta-sandbox.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): 82:06:CF:8F:47:06:6C:17:6A:38:1F:1B:F8:6A:E4:D2
	pp_api.sandbox.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): 89:15:7B:95:AE:BD:B1:A6:5F:06:55:B3:1A:DC:C1:39
	pp_api-3t.beta-sandbox.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): 12:85:A0:A1:95:57:49:7B:5C:2A:C8:D3:E0:7B:3D:34
	pp_api-3t.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): 23:2D:D3:0C:07:31:49:12:23:9A:B0:A2:60:7D:25:C8
	pp_api-aa-3t.beta-sandbox.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): 34:5F:1E:82:12:6F:7B:1D:F0:B9:8B:20:49:7B:6A:F1
	pp_api-aa-3t.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): F4:D7:37:42:1D:82:97:FE:AC:78:95:70:01:97:AB:C4
	pp_api.beta-sandbox.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): B3:2D:58:4F:C3:02:8B:D2:C4:06:31:24:1B:72:76:79
	alias, Dec 30, 2007, keyEntry,
	Certificate fingerprint (MD5): 32:65:4E:B1:96:66:2F:0A:B0:CD:AE:68:F1:73:AE:72
	pp_api-3t.sandbox.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): 4C:08:7F:91:86:0E:89:8B:EB:82:80:3D:93:48:93:15
	pp_api-aa.sandbox.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): E3:23:86:24:A0:A1:C1:B9:A0:05:7A:55:0D:94:5E:AA
	pp_api-aa-3t.sandbox.paypal.com, Dec 30, 2007, trustedCertEntry,
	Certificate fingerprint (MD5): D5:3B:B8:E1:7B:F6:18:40:A8:79:8C:F7:E6:B0:13:EC

The resulting keystore.dat can be downloaded here. As far as I can see all the certificates are different and have different fingerprints!!!

Configuring Java to use proper keystore

The Java runtime environment must be told what keystore to use. This can be done from the command line:

	java.exe -Djavax.net.ssl.trustStore=c:/tmp/keystore.dat -Djavax.net.ssl.trustStorePassword=store12345

or at runtime via properties:

 
	System.getProperties().setProperty("javax.net.ssl.trustStore", "c:/tmp/keystore.dat");
	System.getProperties().setProperty("javax.net.ssl.trustStorePassword", "store12345");

Now we can try connecting to any PayPal server as was described in the very beginning using URLConnection and everything will work fine.

In order to configure Tomcat to use our custom keystore edit server.xml file. Quite nicely, each connector can have independent keystore. The process is described in details on Apache Website.

Final word

There is nothing especially difficult here, but somehow it took enormous amount of time to figure out. Yes, there are complex issues, but why can't we package them properly? We have to package SDK's, libraries, tools and other software in a way that requires absolutely no manual work to get it working in the basic mode. The only way to do that is to document and package everything, even obvious. Always document the exact product and the exact context in which it is supposed to work.

The reason I go into the kind of depth as in this post is to never again to return to unnecessary issues with SSL certificates. I focus my research around how things fundamentally work (aka trust manager is pluggable, aka server must provide public certificate). Then I start with the null state where I assume no preconditions (aka do not use other people keystore, aka don't use other people hacks you don't understand) and create fully automated process (aka batch file) to get to the desired state. Altogether this achieves reproducibility and self-documentation and teaches me the fundamentals. The proper fundamentals give me so much leverage in building lasting, stable, capable, adaptable solutions.

References

As of J2SE 1.4.2_06 the PayPal certificates do not need to be imported as described here. The VeriSign top level certificates, with which they are signed, are now included in the default Java keystore. You can see this as a work item “Update cacerts with new VeriSign ca certs” here http://java.sun.com/j2se/1.4.2/ReleaseNotes.html
http://www.advogato.org/person/johnnyb/diary.html?start=143
http://www.madboa.com/geek/openssl/

Comment (1)

Comment by Igor Katkov — January 4, 2008 @ 3:15 pm

last time I needed to extract certificate from a live server I simply opened the site in IE, logged in, double clicked on the “lock” icon in the status bar that opened up the “Certificate” window, then selected the “Details” tab and finally clicked on the “Copy to file…” button.
This file one can easily import into Java keystore. Not that robust way, I guess, but only takes 1 minute.