Saturday, May 15, 2010

Using WS Security for SOAP Requests in ColdFusion

UPDATE: I've posted the source on GitHub: https://github.com/anthony-id/cfWSAuthenticator

Late last year, I had need to use a third party API that required WS Authentication. I had the WSDLs, I could get it to work in
SOAP UI but when I tried to get it to work in my code it just wasn't happening. It turns out that WS Authentication works out of the box with Axis 2, but ColdFusion creates all its web services with Axis 1, so I had to create the client myself.

After quite a bit of research, it turns out this is quite easy using a couple of Java libraries and ColdFusion's "CreateObject" function. Here's how I got it to work:

First download the following libraries:

xmlsec-1.4.2.jar - http://www.aleksey.com/xmlsec/

An updated version is available, but this is the one I have working

wss4j-1.5.8.jar - http://ws.apache.org/wss4j/
This is the library that does most of the work - Rampart is the portion that Axis 2 uses for this, but we needed to just utilize the signing portions for our purposes.


The libraries need to go to [cf_home]/lib or be added to your classpath. A server restart may be necessary for ColdFusion to be able to see them.


Now the jars are available for use. I created two functions to get these to work - one to do the signing and one to send the request. I have a component in which these live, but you could do the whole thing inline if you wanted to.

The one downside to this is that you'll need to construct the SOAP requests manually. This isn't hard if you use something like SOAP UI - I just grab the XML from there and modify as necessary. I place the SOAP request inside a cfxml block.


<cffunction name="addWSAuthentication" access="public" output="false" hint="I sign SOAP envelope using WS Authentication">
<cfargument name="soapEnvelope" type="string" required="true">
<cfargument name="username" type="string" required="true">
<cfargument name="password" type="string" required="false">
<cfscript>
// Create Java Objects from xmlsec and wss4j
var WSConstants = CreateObject("Java","org.apache.ws.security.WSConstants");
var msg = CreateObject("Java","org.apache.ws.security.message.WSSAddUsernameToken");
// Get Soap Envlope document for Java processing
var soapEnv = arguments.soapEnvelope;
var env = soapEnv.getDocumentElement();
var e = "";
// Set Password type to TEXT (default is DIGEST)
msg.setPasswordType(WSConstants.PASSWORD_TEXT);
// Create WS-Security SOAP header using the build method from WSAddUsernameToken
e = msg.build(env.GetOwnerDocument(),arguments.username,arguments.password);
// Add the Nonce and Created elements
msg.addNonce(e);
msg.addCreated(e);
// Return the secure xml object
return soapEnv;
</cfscript>
</cffunction>


This function sends the soap request and returns the result. The endpoint and SOAP Action can be found in the WSDL or by using SOAP UI. In SOAP UI, look at the raw post, the SOAP Action is in the headers. Also note the additional headers in the cfhttpparams - these are necessary to make the WS Authentication work.

<cffunction name="sendSoapRequest" access="public" output="false" hint="I send the SOAP request off retrun the SOAP response">
<cfargument name="endpoint" type="string" required="true">
<cfargument name="soapEnvelope" type="any" required="true">
<cfargument name="soapAction" type="string" required="false" default="">
<cfset result="">
<cfset soapenv="">

<cfhttp url="#arguments.endpoint#" method="POST" result="result">
<cfhttpparam type="Header" name="Accept-Encoding" value="deflate;q=0">
<cfhttpparam type="Header" name="TE" value="deflate;q=0">
<cfhttpparam type="header" name="SOAPAction" value="">
<cfhttpparam type="xml" value="#toString(xmlParse(arguments.soapEnvelope))#">

<cfreturn>
</cfreturn>
</cffunction>


Hopefully this will save you some time if you ever have to do WS Security in ColdFusion. It took me a couple of weeks of trial and error to get this working, but I'm now using it for two different third party APIs without a hitch.

Acknowledgments
The following were critical in helping me get this solved:
Ben Nadel's Blog - Listing All Classes In A Jar File Using ColdFusion And CFZip
Steven Erat's Blog - Workaround for CFHTTP and Compressed HTTP Response from IIS


42 comments:

  1. In our project, we are not able to consume webservice using cfobject or <cfninvoke. We are using only <cfhttp. Will we able consume the webservice using WS-security (SHA-1)

    ReplyDelete
  2. Yes. In fact, you can't use WS-Security with cfobject because CF uses axis-1 and Rampart is for Axis2.

    Our use case is was the same as yours - first I manually created a SOAP envelope, then sent that in to the "addWSAuthentication" function above. You don't have to make this a function, but the encapsulation makes it much cleaner. Just wrap your SOAP in a cfsavecontent or cfxml and pass that variable in.

    Also, the sendSOAPRequest function is another convenience for reuse. It does the cfhhtp for you - just pass in the variables. This could also be done inline after the SOAP is signed.

    I would recommend creating objects to encapsulate the functionality, but you don't have to - it allows for reuse and extensibility of the code - but doing it all inline will work as well.

    ReplyDelete
  3. Wow this is AWESOME, soooo cool! I didn't even think this was possible! Definately donate!!!!
    Thank You
    ColdFusion CMS

    ReplyDelete
  4. Thank you for posting this! I was able to use your code to connect to ExactTarget's Web Service API using ColdFusion.

    ReplyDelete
  5. Sweet! Glad I could help. I figured that if I ran into this issue, others may as well. It's good to know others found it useful. Makes me want to post more code-oriented entries.

    ReplyDelete
  6. I'm trying to create a web service in ColdFusion that needs to use WS-security. Do you know if and how that is possible?

    ReplyDelete
  7. Well, I think I would need to know more information before answering. I'm happy to try and answer the question, but I'll need to understand what you are trying to do first.

    This post was mostly around signing XML to POST for WS-Security purposes, but consuming WS-Security should be similarly possible.

    By the way, I've posted the code on GitHub: https://github.com/anthony-id/cfWSAuthenticator

    ReplyDelete
  8. Do you have an example of submitting a ws-security soap request to a coldfusion based web service? I get the following when I try.

    soapenv:MustUnderstand
    Did not understand "MustUnderstand" header(s):

    ReplyDelete
  9. sorry it must have removed the xml tags. I get the following
    FaultCode: soapenv:MustUnderstand
    FaultString: "Did not understand "MustUnderstand" header(s)"

    ReplyDelete
  10. No problem on the XML. I'm digging into my memory on this one (on a Friday afternoon at that...) :-)

    but here goes...the MustUnderstand header is being added by ws.security object, I think. Either the receiver is expecting a different value there or it's not expecting it at all.

    I would do a couple of things: 1) dump the generated SOAP output to see what it looks like but also capture the full POST with something like Fiddler or Charles. There are different ways to add the Security information so you may need a different WSConstants or passwordType.

    Unfortunately, the faultstring isn't very informative, so if you have any information about what the receiver is expecting (digest vs. text, for instance) that may help in the debugging.

    I'll dig into it a bit this weekend to re-acquaint myself with it since it's been a long time since I've done a deep dive into the internals of the ws.securty apache object.

    ReplyDelete
  11. The example code at GitHub outputs an authenticated envelope, so that's worth taking a look at. A header is added with your authentication information and one of the attributes in the Security node is "mustUnderstand". This attribute is set to "1".

    From what you describe, it sounds like the receiving code isn't set up to handle WS-Security or needs additional configuration. Can you tell me anything more about the Web Service that is doing the authenticating?

    This is a pretty good explanation of the issue:

    http://wso2.org/library/tutorials/understand-famous-did-not-understand-mustunderstand-header-s-error

    One thing to note, however, is that ColdFusion uses Axis1 NOT Axis2, so the issue may be in how (or if) the ColdFusion code is implementing WS-Security. I suspect it is using a custom Java wrapper, but without seeing the code, it's impossible to troubleshoot.

    ReplyDelete
  12. By the way, Axis2 support is supposed to be coming in ColdFusion X, which is pretty exciting for exactly these kinds of reasons.

    ReplyDelete
  13. This is all about consuming a WS which uses WS-Security. But what if you want to create a WS using CF and protect it using WS-Security?

    ReplyDelete
  14. This comment has been removed by the author.

    ReplyDelete
  15. I actually need to delete my last response because it was early and I wasn't quite fully in my programming mind :-/

    Creating SOAP based web service is as easy as creating a component:

    http://help.adobe.com/en_US/ColdFusion/9.0/Developing/WSc3ff6d0ea77859461172e0811cbec22c24-78b7.html

    When it comes to WSS, you would need to do something like the "Using Coldfusion..." section here:

    http://help.adobe.com/en_US/ColdFusion/9.0/Developing/WSc3ff6d0ea77859461172e0811cbec13a13-7fe0.html

    At this point, you would need to extract the Security header, check the signature and validate the credentials (or however you chose to implement WSS). Since CF doesn't give us WS "out-of-the-box" you could either parse that header manually or perhaps leverage the WS-Sec library to do it for you.

    I haven't had any need to publish SOAP web services myself, so I haven't worked much with that side of the equation. Since it's just dealing with SOAP headers, I would expect it to be tedious, but not overly difficult.

    Another approach would be to forgo WSS and use token based security whereby a user first establishes a session with an authentication call and then passes a session id in subsequent requests. This is a common approach but does require one additional call to the service to authenticate and establish a session.

    ReplyDelete
  16. Can you provide any tips for adding a BinarySecurityToken to the SOAP header using wss4j and Coldfusion? Any help would be greatly appreciated.

    ReplyDelete
  17. @Steve - Not off the top of my head since I haven't had to work with that, but if you provide more information, I may be able to point you in the right direction. I've recently been working with a .net WSE application that required first a login and then for me to put userId and access token in the username and password fields of the header.

    By binary, do you mean attachment in the header or a base64 encoded string?

    Looking at my code, I'm guessing that you would need to add a token block to the message, but I would probably start by looking at a working SOAP request in something like SOAP UI and reverse engineer that with the wss4j library.

    It sounds like an interesting challenge, though, and I would love to help and also hear your experience working with the binary token.

    ReplyDelete
  18. @Steve - aha. It looks very much like this will help you out: http://ws.apache.org/wss4j/apidocs/org/apache/ws/security/message/package-summary.html

    The WSSecSignature has an appendBSTElementToHeader() method that seems to be what you want.

    I'm guess something like binToken = msg = CreateObject("Java","org.apache.ws.security.message.WSSecSignature");

    binToken.appendBSTElementToHeader(WSSecHeader secHeader) // WSSSecHeader is also in the message object.

    You'll need to compose your own x509 but if you have that, you should be able to create the header and add the BST to your SOAP packet.

    ReplyDelete
  19. Thanks I'll give it a try and let you know how it goes.

    ReplyDelete
  20. So I've been working on it today and I've kind of hit a wall, and I'm hoping you can point me in the right direction. First I should mention that I've upgraded to wwwj4 1.6.2 and xmlsec 1.4.5. Here is my code based on yours.
    ===============
    var holder = StructNew();
    holder.msgHead = getMsgHeader(); //org.apache.ws.security.message.WSSecHeader
    holder.msgUser = getMsgUsername(); //org.apache.ws.security.message.WSSecUsernameToken
    holder.msgSig = getMsgSignature(); //org.apache.ws.security.message.WSSecSignature
    holder.msgX509Token = getMsgX509Token(); //org.apache.ws.security.message.token.X509Security
    holder.cryptoX509 = getCryptoX509(); //org.apache.ws.security.components.crypto.CertificateStore
    holder.WSConstants = getWSConstants(); //org.apache.ws.security.WSConstants
    holder.soapEnv = arguments.soapEnvelope;
    holder.env = holder.soapEnv.getDocumentElement();
    holder.certInput = CreateObject("java","java.io.FileInputStream").init("path_to_cert_file");
    holder.certFactory = CreateObject("java","java.security.cert.CertificateFactory").getInstance("X.509");
    holder.cert = holder.certFactory.generateCertificate(holder.certInput);
    holder.certArr = [holder.cert];
    holder.cryptoX509.init(holder.certArr);
    holder.certInput.close();
    holder.msgSig.setKeyIdentifierType(holder.WSConstants.ISSUER_SERIAL);
    holder.msgSig.setX509Certificate(holder.cert);

    // Set Password type to TEXT (default is DIGEST)
    holder.msgUser.setPasswordType(holder.WSConstants.PASSWORD_TEXT);
    holder.msgUser.setUserInfo(arguments.username,arguments.password);
    // Add the Nonce and Created elements
    holder.msgUser.addNonce();
    holder.msgUser.addCreated();
    holder.msgHead.insertSecurityHeader(holder.env.GetOwnerDocument());
    ===============

    ReplyDelete
  21. So far so good. I can get your results doing the following:

    ===============
    holder.newdoc = holder.msgUser.build(holder.env.GetOwnerDocument(),holder.msgHead);
    ===============

    The problem comes when I try to include the certificate and the security token. When I try this:
    ===============
    holder.msgSig.build(holder.env.GetOwnerDocument(),holder.cryptoX509,holder.msgHead);
    ===============
    I am getting an exception thrown that traces back to "Caused by: java.lang.NullPointerException: signingKey cannot be null at javax.xml.crypto.dsig.dom.DOMSignContext"

    Any ideas on what I need to do to get that signingKey in place?

    ReplyDelete
    Replies
    1. Hi Steve, did you or anybody else managed it to integrate the certificate in the cfc? Because of my minor java knowledge i don't know where and how to integrate your source code. Any help would be great!

      Delete
  22. @Steve - It sounds like you're making pretty good progress! Without digging into the crypto library, I'm not sure what is missing, but what I usually do in situations like that is start looking at the JavaDoc.

    A quick check, looks like it may be the private key needed to sign the the certificate. Let me know if this gets you moving in the right direction:

    http://massapi.com/class/javax/xml/crypto/dsig/dom/DOMSignContext.java.html

    ReplyDelete
  23. Hi Steve,

    I have to sign a soap envelope with a binary security token as well and would love to skip some hair pulling. Did you ever get this working? If so would you be kind enough to share some of that code to sign the message.

    Thanks.....

    ReplyDelete
  24. I'm still struggling with getting it in place. If I crack it I'll be sure to post here.

    ReplyDelete
  25. Steve,

    I'm looking forward to seeing your answer (I'm sure you'll get it!) I wish I had time to dive in and help more.

    I'll want to add that code in to the helper object at GitHub once you've got it, so feel free to fork it and put in a pull request. If I am able to find some time, I'll dig in as well.

    ReplyDelete
  26. Hey so I found the following code which uses java to sign a soap message. Still digging into it but if you go by the COMMENTED OUT code in the main method you'll see all the calls required. (It's a test to send an encrpyted soap message to amazon).

    http://code.google.com/p/androidzon/source/browse/AmazonClientTest/src/it/marco/axis2/amazon/TestWSS4J.java?r=71

    ReplyDelete
  27. Thanks for the link. That looks like it could solve my problem. I've been pulled into some other work for the moment, but I'll give it a try when I can.

    ReplyDelete
  28. @Steve and Anthony

    I finally got this working after several days and figured I'd share my results. Early on, I abandoned the notion of getting this working in CF, and pushed it all to a custom Java class that just takes strings from CF as arguments.

    I used Axis2 with Rampart, following their normal instructions, I created a stub and got my web service working from a standalone file. I setup the constructor to create a static ConfigurationContext with my endpoint. I then created a call() function that creates a new Axis2 stub and actually performs the sendReceive() and returns the result to CF as a string.

    Here is where I hit issues. This would work perfectly once, then randomly after that. I kept getting an "Object not instantiated" error. I was able to track it down to a bug in an older version of xml-security which is even included in the most up to date version of wss4j.

    You will need to download the latest wss4j source and the latest xml-security libraries. Update the xml-security libraries in wss4j and compile. I had to remove an "init();" line from one of the wss4j source files because the newer xml-security no longer requires (or even has) that function.

    Now you should be able to copy in all of your Axis2 and rampart libraries into the CF classpath and be good to go.

    ReplyDelete
  29. Thanks K.J.! This is really excellent stuff. I appreciate your contribution - I'm sure it will be invaluable to others doing integration work.

    ReplyDelete
  30. Anthony,

    Thanks for your tips and providing this forum for discussion. We've used your (and others) efforts here to try and get WS pulling into a CF9 Portlet. As portal native, cfusion.war doesn't spin up it's own JVM, we're getting an "Object Instantiation Exception.
    Class not found: org.apache.ws.security.WSConstants." it's as if CF, deployed as a .war, is not finding the xmlsec and wss4j jars (webapps\cfusion\WEB-INF\cfusion\lib\wss4j-1.5.8.jar and xmlsec-1.4.2jar) in the class path. The CF admin DOES indicate that they are there in server settings. We're stumped. We've put them in every lib dir we can find, tried adding them to the java class path outside cfusion.war. The statements that are calling the classes are in our WSAuthenticator.cfc



    // Create Java Objects from xmlsec and wss4j
    variables.WSConstantsObj = CreateObject("Java","org.apache.ws.security.WSConstants");
    variables.messageClass = CreateObject("Java","org.apache.ws.security.message.WSSecUsernameToken");
    variables.secHeaderClass = CreateObject("Java","org.apache.ws.security.message.WSSecHeader");
    variables.TSBuilder = CreateObject("Java","org.apache.ws.security.message.WSSecTimestamp");
    return this;



    Am I read above correctly, there's a bug in xmlsec or wss4j?

    ReplyDelete
  31. Chris,

    I this looks an awful lot like a classpath issue to me. Perhaps the jar needs to be packaged with the WAR or there is something in the WAR config that needs a pointer? I haven't worked with portlets, so I don't have any experience in that regard, so I'm just guessing.

    I would probably try to see if the ws.security object can be found - my guess is no - but if it can dump that out and see what you get.

    I don't think this a bug issue with the jars, although it's quite likely newer versions are out (pretty sure XMLSec is updated), so you could pull those to see if those work better.

    You may also want to check out Mark Mandel's Java Loader (http://javaloader.riaforge.org/) that may help as well.

    This is probably all moot with CF10 as it uses Axis2 and likely has better integration with these libraries (and Java Loader built in IIRC). I haven't had a chance to play with it yet and it's not really a solution for you, but it's worth tucking away for later.

    If you do find a solution, let us know, I'm curious to know what the issue is and I'm sure you'll help someone else out there with the same problem!

    ReplyDelete
  32. Hey Anthony, love your work.
    I'm getting an error - "Object Instantiation Exception.
    Class not found: org.apache.ws.security.WSConstants". Any idea what the problem could be?

    ReplyDelete
  33. Paul,

    Glad you are finding it useful! It sounds like the Jar file isn't in your classpath - if it is in your classpath, you may need to restart coldfusion for it to be recognized.

    If you are able to instantiate org.apache.ws.security, then you can dump that out and see if something is missing. Also check capitalization and syntax in case a space or some other mischievous character got in there.

    Hope that helps!

    ReplyDelete
    Replies
    1. Hi Anthony,

      Yeah, I got that one sorted in the end.
      A new issue though. Not sure I can explain it briefly. I've posted it on StackOverflow though at http://stackoverflow.com/questions/11822301/coldfusion-java-the-build-method-was-not-found

      You may have hit this issue before or have some insight.

      Delete
  34. Hey again Anthony. If you are available for hourly work, please let me know. We've consumed too much time and need this issue sorted ASAP. paulbaylis1@yahoo.com. Many thanks!

    ReplyDelete
  35. Paul,

    I've got my hands quite full now, so I am not available for hourly work. It is possible to construct WS-Security headers manually, so that could be an option. It's a not as elegant, but it can be done. I'm pegged out at work right now, but there are other options out there. Also, I *think* CF10 may actually have the necessary libraries built in since it uses AXIS 2, but I haven't had a chance to download and play with it (mostly force.com and middleware development at the moment)

    ReplyDelete
  36. This comment has been removed by the author.

    ReplyDelete
  37. This comment has been removed by the author.

    ReplyDelete
  38. This comment has been removed by the author.

    ReplyDelete
  39. uhm, I am confused. Can someone help me through the whole coding proccess? how to consume web service in SSL, SOAP that required certificate authentication as binary token. As I see Steve Smith is heading in that direction but I am so sure how to fit his codes in <cfhttp . . . Please help.

    ReplyDelete
    Replies
    1. not sure where the getMsgHeader(), get MsgUsername() . . .comming from?

      Delete