2009/12/27

Building a WS-Security enabled SOAP client in Maven2 to the EC2 WSDL using JAX-WS / CXF & WSS4J: tips & tricks

Generating a Java client from the Amazon EC2 WSDL that correctly used WS-Security is not completely simple. This blog post from Glen Mazza contains pretty much all the info you need, but as usual there are many things to trip up over along the way. So, without further ado, my contribution.

My setup: I was using Maven2 to construct a JAR file. Running "mvn generate-sources", then, downloads the WSDL and uses it to generate the EC2 object model in src/main/java.

Blogger doesn't like me quoting XML, so I've put my sample POM at pastebin, here. Inside the cxf-codegen-plugin plugin XML you'll see two specific options, "autoNameResolution", which is needed to prevent naming conflicts with the WSDL, and a link to the JXB binding file for JAXWS, which is needed to generate the correct method signatures

Once this is done, then the security credentials need to be configured. There are some pecularities:

As laid out in this tutorial for the Amazon product advertising API, the X.509 certificate and the private key need to be converted into a pkcs12 -format file before they're usable in Java. This is done using OpenSSL:
openssl pkcs12 -export -name amaws -out aws.pkcs12 -in cert-BLABLABLA.pem -inkey pk-BLABLABLA.pem
At this point, I should admit that I spent hours scratching my head because the generated client (see below) gave me the error "java.io.IOException: DER length more than 4 bytes" when trying to read the PKCS12 file. So I switched to the Java Keystore format by using this command (JDK6 format):
keytool -v -importkeystore -srckeystore aws.pkcs12 -srcstoretype pkcs12 -srcalias amaws -srcstorepass password -deststoretype jks -deststorepass password -destkeystore keystore.jks
...and then received the error "java.io.IOException: Invalid keystore format" instead. At this point I googled a bit, and discovered two ways to verify the integrity of keystores, via openSSL and the Java keytool:
#for pkcs12
openssl pkcs12 -in aws.pkcs12 -info

#for keystore
keytool -v -list -storetype jks -keystore keystore.jks
Both the keystore and pkcs12 file were valid. Then, I realised that I'd put the files in src/test/resources which was being put through a filter before landing in "target". The filter was doing something to the files, so of course they couldn't be read properly. Duh me. I put the key material in a dedicated folder with no filtering and this problem was fixed.

My next problem was the exception "java.io.IOException: exception decrypting data - java.security.InvalidKeyException: Illegal key size". This was solved by downloading the "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files". Simple!

At this point the request was being sent to Amazon! Which then returned a new error message, "Security Header Element is missing the timestamp element". This was because the request didn't have a timestamp. So, I changed the action to TIMESTAMP+SIGNATURE (as seen in the below code sample), at which point I got a new error message: "Timestamp must be signed". This I fixed by setting a custom SIGNATURE_PARTS property also as below.

Finally, once this was all done, and everything was signed, Amazon gave me back the message "AWS was not able to authenticate the request: access credentials are missing". This is exactly the same error that you get when nothing is signed at all, which needless to say is somewhat ambiguous.

At this point I decided that I'd really like to see what was being sent over the wire. The WSDL specifies the port address with an HTTPS URL. However, I had saved the WSDL locally, and changing the URL to HTTP made the result inspectable with the inestimable Wireshark. Despite the request being sent in HTTP, not HTTPS, it was still executed. According to the docs, this should not be!

Anyway, once I was looking at the bytes, I saw that the certificate was only being referred to, not included as specified in the AWS SOAP documents, in this case for SDB. This was fixed by setting the SIG_KEY_ID (key identifier type) property to "DirectReference", which includes the certificate in the request.

...and then it worked. Oh Frabjous Day, Callooh, Callay! The final testcase code that I used is more or less as follows:

package net.ex337.postgrec2.test;

import com.amazonaws.ec2.doc._2009_10_31.AmazonEC2;
import com.amazonaws.ec2.doc._2009_10_31.AmazonEC2PortType;
import com.amazonaws.ec2.doc._2009_10_31.DescribeInstancesType;
import junit.framework.TestCase;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.UnsupportedCallbackException;
import org.apache.cxf.endpoint.Client;
import org.apache.cxf.frontend.ClientProxy;
import org.apache.cxf.ws.security.wss4j.WSS4JOutInterceptor;
import org.apache.ws.security.WSPasswordCallback;
import org.apache.ws.security.handler.WSHandlerConstants;

/**
*
* @author Ian
*
*/
public class Testcase_CXF_EC2 extends TestCase {

public void test_01_DescribeInstances() throws Exception {

AmazonEC2PortType port = new AmazonEC2().getAmazonEC2Port();

Client client = ClientProxy.getClient(port);
org.apache.cxf.endpoint.Endpoint cxfEndpoint = client.getEndpoint();

Map outProps = new HashMap();

//the order is important, apparently. Both must be present.
outProps.put(WSHandlerConstants.ACTION, WSHandlerConstants.TIMESTAMP+" "+WSHandlerConstants.SIGNATURE);
//this is the configuration that signs both the body and the timestamp
outProps.put(WSHandlerConstants.SIGNATURE_PARTS,
"{Element}{http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd}Timestamp;"+
"{}{http://schemas.xmlsoap.org/soap/envelope/}Body");

//alias, password & properties file for actual signature.
outProps.put(WSHandlerConstants.USER, "amaws");
outProps.put(WSHandlerConstants.PW_CALLBACK_CLASS, PasswordCallBackHandler.class.getName());
outProps.put(WSHandlerConstants.SIG_PROP_FILE, "client_sign.properties");

//necessary to include the certificate in the request
outProps.put(WSHandlerConstants.SIG_KEY_ID, "DirectReference");

cxfEndpoint.getOutInterceptors().add(new WSS4JOutInterceptor(new HashMap(outProps)));

//sample request.

DescribeInstancesType r = new DescribeInstancesType();

System.out.println(port.describeInstances(r));
}

//simple callback handler with the password.
public static class PasswordCallBackHandler implements CallbackHandler {
private Map passwords = new HashMap();

public PasswordCallBackHandler() {
passwords.put("amaws", "password");
}

@Override
public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
for (int i = 0; i < pc =" (WSPasswordCallback)callbacks[i];" pass =" passwords.get(pc.getIdentifer());"

provider="org.apache.ws.security.components.crypto.Merlin" type="pkcs12" password="password" alias="amaws" file="aws.pkcs12" href="http://s3.amazonaws.com/ec2-downloads/ec2.wsdl">http://s3.amazonaws.com/ec2-downloads/ec2.wsdl.

[I think I mangled somethjing here, will fix it soon]

At this, the method signatures of the generated port abruptly changed to something other, because I forgot to change the wsdlLocation in the JXB binding file. Once I fixed this, it worked again.

Some thoughts:

1) Were I publishing a library for general use in accessing AWS, I would probably not use the direct "symlink" above that always points to the latest version of the WSDL. Instead, I would link deliberately to each version, and in that way always generate ports for each version of the WSDL, this ensuring backwards compatibility.

2) Secondly, I find it inelegant to have to specify the WSDL location in two places (the POM and the binding file), and so I'd like to try and pass the binding file through a filter, using a ${variable} in both places referring to a property in the POM.

3) I find it likewise confusing that the password for the keystore is used in two places, firstly in client_sign.properties and secondly in the CallbackHandler that is invoked from within the bowels of the WSS4JOutInterceptor. In the code above, this is obviously duplicated data, however in the final 'production' version of this code I expect to have the data centralised & the code prettified around it.

2009/12/20

Using CXF instead of Axis for Java from WSDL: better results faster.

In the footsteps of the same guy, Glen Mazza, I linked to at the bottom of the previous post, who did the same thing (SOAP client / server using WS-Security and X.509 certificates using CXF), I switched from Axis2 to CXF, and had immediately better results:
  • The documentation and maven plugin instructions is current, and accurate.
  • The plugin works.
  • All the right JAR files are in repos.
  • The code generation worked fine, with some JAX-WS binding stuff added into the mix.
Which leads me to ask, why are there two projects at Apache doing essentially identical things, right down to the usage patterns for the tools they provide? (A: CXF nee XFire is from Codehaus). Anyway, I don't have to write a HOW-TO for this stuff, the docs are there and they're useful.

I have yet to look at CXF support for WS-Security, but it seems simpler from the get-go than the equivalent stuff in Axis2, despite insisting on Java's proprietary keystore, hum, I didn't read this howto clearly enough - files are supported. We shall see!

2009/12/19

Creating Java code from a WSDL using Apache Axis2: maven2 and the command line.

Four years ago, creating Java code from WSDL was difficult and annoying. Today, I'm trying to generate a client for the EC2 WSDL, that ideally would download the latest version and rebuild the API when I type "mvn clean install".

I've given up. The Axis2 Maven2 plugin does not seem to work correctly, so I've resorted to using the command-line tool, which does work. My command was:

wsdl2java.bat -d jaxbri -o -S . -R . --noBuildXML --noMessageReceiver -uri http://s3.amazonaws.com/ec2-downloads/2009-10-31.ec2.wsdl

I used the JAXBRI output format because XMLBeans is basically dormant. Unfortunately the JAXBRI compiler generates sources in an "src" subdirectory, which can't be changed via command-line options, so some manual copy-and-pasting is required.

Secondly, the generated classes then depend on axis. So this needs to be added to the POM:

(Blogger broke my XML):

org.apache.axis2
axis2
1.5.1


Thirdly, there's an undeclared dependency in Axis2-generated code on Axiom, so this also needs to be added:


org.apache.ws.commons.axiom
axiom
1.2.5


(The latest version is 1.2.8, but this doesn't seem to be in repos yet.)

Following which, attempting to run this:

AmazonEC2Stub stub = new AmazonEC2Stub("http://eu-west-1.ec2.amazonaws.com");
DescribeRegionsType t = new DescribeRegionsType();
System.out.println(stub.describeRegions(t).getRegionInfo().getItem());

...rendered many different ClassNotFoundExceptionS which were, one after another, which I attempted to solve shotgun-style by adding each new dependency to the POM as it cropped up. This was an abject failure - I stopped at org.apache.axis2.transport.local.LocalTransportSender, which apparently is only available in Axis2 v. 1.2 (I'm using 1.5.1). So instead I deleted all the Axis2-related stuff from my POM and just added the JARs from the 1.5.1 downloded ZIP file straight to the Eclipse project.

This worked, and gave me the error message that I was looking for, to whit: "AWS was not able to authenticate the request: access credentials are missing". From here, I would just need to get the Rampart/WS-Security/WSS4J stuff working properly with the Amazon X.509 certificate, and then I should be home free. We shall see.

Further light reading can be found on this article on IBM developer works and this article on SOAP clients with Axis.

Update 2009/12/20: The work has been done (I should have googled first!), and it is herculean, as you can see by reading this impressive tutorial for creating an Axis2 SOAP Client for the Amazon Product Advertising API.

2009/12/04

On the subject of being too harsh a critic

One of the great things about OSS is its transparency. This is also a great response to critics of any particular project, which is "instead of talking so much, why don't you shut up and help?".

I'm guilty of forgetting that there are real people behind most projects. With commercial software this happens a lot more and is disguised as "I'm a paying customer and I expect good service", but there's no excuse, honestly, for criticism that isn't phrased constructively and considerately when the product itself is free.

Fnar fnar fnar.

2009/12/03

EC2 upgrades again

At lunch today, I read that EC2 can now boot off EBS images, something that simplifies the whole AMI thing and brings it up to speed with Rackspace on the ease-of-deployment front. However, two points:
  1. EC2 is still the clear loser in price-performance, and charging for I/O to the root partition won't help. More specifically, when will EBS I/O become consistent? Probably this has a lot to do with being popular and dealing with shared resources at the lower end, see this HN thread.
  2. My next question is, how does this affect the attack surface of EC2? Can the work done in the "Get off my Cloud!" paper be expanded on?
Anyway, this makes my Postgres stuff interesting, guess I'll be using this new mechanism instead. It's always nice when new stuff arrives, even if it means reworking stuff.