CFHTTP + FILE + POST (Bug Byte ?)

A few years ago, I saw a post on the adobe forums about a strange problem with cfhttp. Posting a simple .zip file with <cfhttp> somehow ended up corrupting the file. So the resulting archive file could not be read by some tools, in particular <cfzip>.

<!---
   With CF8 this code fails with the error:
   Unsupported File Format
   Ensure that the file is a valid zip file and it is accessible.
--->
<cfif structKeyExists(FORM, "myFile")>
    <cffile action="upload" destination="#ExpandPath('.')#" nameconflict="overwrite" />
    <cfzip action="list" file="#cffile.serverDirectory#/#cffile.serverFile#" name="result" />
    <cfdump var="#result#" label="Zip File Contents" />
<cfelse>
    <cfhttp url="http://127.0.0.1/cfhttpTest.cfm" method="post">
        <cfhttpparam name="myFile" type="file" file="c:/test/testFile.zip" mimetype="application/zip" />
    </cfhttp>
    <cfoutput>#cfhttp.fileContent#</cfoutput>
</cfif>

The Byte
So I decided to have a look and noticed something strange about the received file. It always seemed to be two (2) bytes bigger than the original. After a bit more experimentation, I discovered the problem disappears if you add a formfield (any formfield) directly after the file.  Placing it before the file does not work.

<!---
   This DOES work
--->
<cfif structKeyExists(FORM, "myFile")>
    <cffile action="upload" destination="#ExpandPath('.')#" nameconflict="overwrite" />
    <cfzip action="list" file="#cffile.serverDirectory#/#cffile.serverFile#" name="result" />
    <cfdump var="#result#" label="Zip File Contents" />
<cfelse>
    <cfhttp url="http://127.0.0.1/cfhttpTest.cfm" method="post">
        <cfhttpparam name="myFile" type="file" file="c:/test/testFile.zip" mimetype="application/zip" />
        <cfhttpparam name="BecauseTheFileIsCorruptedWithoutThisField" type="formfield" value="" />
    </cfhttp>
    <cfoutput>#cfhttp.fileContent#</cfoutput>
</cfif>

Now I just assumed it was an issue with cfhttp. As there was a work-around, I did not really explore it further. But a recent question on stackoverflow.com prompted me to revisit the issue and take a closer look. A poster named Kip mentioned something important I had missed the first time around. That extra two (2) bytes was a newline! So armed with that key piece of information, I decided to use Fiddler to see what was happening within the cfhttp request.

The Pest
If you are unfamiliar with multipart submissions, the w3c describes them as a ".. message contain(ing) a series of parts, each representing a successful control." In loose terms, a successful control is just a form field. (There is a little more to it than that.  But the details are not relevant to this particular case.) When a multipart form is submitted, the names and values of the various fields are submitted as parts, delineated by boundary markers.

Now in this case, there is only one (1) form field. But as you can see from the image below, the information is nested within boundary markers. Obviously the request contents are separated by a newline, at various points. But notice there is an extra newline just before the closing boundary marker? That is the extra two (2) bytes. Given that <cfhttp> is responsible for generating the request content, and the separating newlines, it certainly seemed like a problem <cfhttp>

Next I submitted the same file manually with a regular <form> post.  The results from Fiddler showed there was no extra newline in the content.

So I went back and tested the extra formfield hack and surprise, surprise.. that pesky newline was gone.

Another poster, Sergii, mentioned this issue does not occur with Railo 3.1.2. So I tested the same code under Railo. Sure enough the resulting archive was fine, no corruption. Fiddler confirmed there were no pesky newline problems when using Railo's <cfhttp>. That clinched it for me. I would say this is an ACF bug with <cfhttp> in CF8 and CF9. So if you ever experience something similar, now you know why.

(Note: I do not know if this issue applies to 9.0.1)

...Read More

My First Look at Diffie-Hellman (Merkle) Key Exchange - Part 2

After working through the basic formulas in Part 1, I felt I had a decent grasp of the overall process. So I decided to dive into a java example.  From what I have read, there are different implementations. So I looked over a few java examples before choosing what I felt was the simplest: the one from the (Sun) Java Cryptography Extension (JCE) Reference Guide. Make no mistake, there is definitely a lot more involved in establishing secure exchanges than is covered in the example. But as with most things, understanding the concepts and what the code is doing is a good start.


So when I first looked at the java code, I was a bit .. flummoxed. I honestly did not recognize much of anything, other than "DH". I suppose I naively expected I might see objects initialized with secret values, calculations of prime numbers and primitive roots. Ye-ah. Needless to say, I did not find anything like that. At least not in the basic "intro" example.

The more I thought about it, I realized how silly that would be. Remember I mentioned typical exchanges involve really, really big numbers? Well imagine having to calculate really large prime numbers or primitive roots (on your own) in order to use the library? Not fun. After reviewing the code more closely, I realized the SunJCE does indeed provide a class that does the grunt work for you.

First an AlgorithmParameterGenerator is used to generate the initial parameters. In other words the prime number and primitive root agreed upon by Alice and Bob. In this example it is initialized with a key size 512 bits. A real exchange would probably use a larger key like 1024 bits.

As you can imagine, generating the values is an expensive operation and may take a few seconds. But when finished, the generator will return a DHParameterSpec object. It is simply an object containing the settings used for the exchange: prime number (p), primitive root (g) and key size (l). To view the values, use the methods getG(), getP() and getL().

<cfset generator = createObject("java", "java.security.AlgorithmParameterGenerator").getInstance("DH") />
<!--- Intialize the generator to create a 512 bit key pair (Testing only) --->
<!--- This is an expensive operation and may take a while --->
<cfset generator.init( javacast("int", 512) ) />
<cfset params = generator.generateParameters() />
<!--- Convert the parameters to the right type of object --->
<cfset DHParameterSpec = createObject("java", "javax.crypto.spec.DHParameterSpec")/>
<cfset parameterSpec   = params.getParameterSpec(DHParameterSpec.getClass()) />

Next the code passes those parameters into a KeyPairGenerator . The generator returns a KeyPairobject which contains Alice's private and public keys. While so far things may not seem very familiar, the generator is actually doing the same thing we did in Part 1. Except it automatically selects Alice's secret number, and calculates her public number internally using the given parameters (ie prime number and primitive root).

<cfset KeyPairGenerator = createObject("java", "java.security.KeyPairGenerator") />
<cfset alicesKeyPairGenerator = KeyPairGenerator.getInstance("DH") />
<cfset alicesKeyPairGenerator.initialize( parameterSpec ) />
<cfset alicesKeyPair     = alicesKeyPairGenerator.generateKeyPair() /> 

How do you know this? Well if you display Alice's keys, you will see several parameter names followed by a long string of hexadecimal. Those values are just very large numbers, encoded as hex. The "x" represents Alice's private number and the "y" her public number. The "p" and "g" are our prime number and primitive root values. Again, encoded as hex.

Starting to seem familiar now? 

<cfoutput>
<strong>Alice's Values</strong>
<pre>
    Format    = #alicesKeyPair.getPublic().getFormat()#
    Base (g)  = #alicesKeyPair.getPublic().getParams().getG()#
    Prime (p) = #alicesKeyPair.getPublic().getParams().getP()#
</pre>    

<pre>    
#alicesKeyPair.getPrivate()#
</pre>    
<pre>    
#alicesKeyPair.getPublic()#
</pre>
</cfoutput>

Before Alice sends Bob her public number, she creates a KeyAgreement. Basically this object will be used to complete the final step: calculating the shared key. So it is first initialized with Alice's private key. Then Alice sends her public key off to Bob, and waits to receive his in return.

<!--- Alice creates and initializes her DH KeyAgreement object with her Private Key --->
<cfset KeyAgreement = createObject("java", "javax.crypto.KeyAgreement").getInstance("DH") />
<cfset alicesKeyAgreement = KeyAgreement.getInstance("DH") />
<cfset alicesKeyAgreement.init( alicesKeyPair.getPrivate() ) />
<!--- Alice encodes her public key, and sends it over to Bob --->
<cfset alicesPublicKeyBytes = alicesKeyPair.getPublic().getEncoded() />

When Bob receives Alice's public key, it is encoded in x509 format. So it has to be decoded it first. Bob then uses that object to create his own keys. Finally, he encodes his public key and sends it back to Alice.

<!---
    Bob gets the DH parameters associated with Alice's public key. 
    He must use the same parameters when he generates his own key pair.
--->
<cfset dhParamSpec = publicKeyFromAlice.getParams() />

<!--- Bob creates his own DH key pair --->
<cfset KeyPairGenerator = createObject("java", "java.security.KeyPairGenerator") />
<cfset bobsKeyPairGenerator = KeyPairGenerator.getInstance("DH") />
<cfset bobsKeyPairGenerator.initialize( dhParamSpec ) />
<cfset bobsKeyPair = bobsKeyPairGenerator.generateKeyPair() />
<!--- Bob encodes his public key, and sends it over to Alice --->
<cfset bobsPublicKeyBytes = bobsKeyPair.getPublic().getEncoded() />

Just like Alice's values, Bob's keys will be encoded as hex.

<cfoutput>
<strong>Bob's Values</strong>
<pre>
    Base (g)  = #dhParamSpec.getG()#
    Prime (p) = #dhParamSpec.getP()#

</pre>
<pre>
#bobsKeyPair.getPrivate()#
</pre>

<pre>
#bobsKeyPair.getPublic()#
</pre>
</cfoutput>

Bob is now ready to calculate the shared key. So he too creates a KeyAgreement, and initializes it with his private key. Finally he plugs in Alice's public key and calculates the shared value. Now internally this class is using the same formulas as we did in Part 1. But it is all done for you auto-magically.

<!--- Bob creates his DH KeyAgreement and initializes it with his private key --->
<cfset bobsKeyAgreement = createObject("java", "javax.crypto.KeyAgreement").getInstance("DH") />
<cfset bobsKeyAgreement.init( bobsKeyPair.getPrivate() ) />
<cfset bobsKeyAgreement.doPhase(publicKeyFromAlice, true) />
<cfset bobsSharedSecret = bobsKeyAgreement.generateSecret() />
<cfset bobsSharedSecretAsHex = binaryEncode(bobsSharedSecret, "hex") />

Once Alice receives Bob's public key, she decodes it and plugs the value into her KeyAgreement object. Then calculates the shared value.

<cfset alicesKeyFactory = createObject("java", "java.security.KeyFactory").getInstance("DH") />
<cfset X509EncodedKeySpec = createObject("java", "java.security.spec.X509EncodedKeySpec") />
<cfset x509KeySpec = X509EncodedKeySpec.init( bobsPublicKeyBytes ) />
<cfset publicKeyFromBob = alicesKeyFactory.generatePublic(x509KeySpec) />
<cfset alicesKeyAgreement.doPhase( publicKeyFromBob, true ) />
<cfset alicesSharedSecret = alicesKeyAgreement.generateSecret() />
<cfset alicesSharedSecretAsHex = binaryEncode(alicesSharedSecret, "hex") />

Thanks to the wonders of mathematics, Alice and Bob both arrive at the same shared value.

<cfset areSecretsTheSame = alicesSharedSecretAsHex eq bobsSharedSecretAsHex />
<cfoutput>
<strong>Are Alice and Bob's secrets the same?</strong> #areSecretsTheSame#

<p class="result">alicesSharedSecretAsHex</p>
<p>#alicesSharedSecretAsHex#</p>

<p class="result">bobsSharedSecretAsHex</p>
<p>#bobsSharedSecretAsHex#</p>
</cfoutput>

By itself, the shared value cannot be used for encryption. However Alice and Bob can use the shared value to generate a secret key which can be used for encryption.

The java documentation includes a simple example using DES. Obviously outdated, but it does demonstrate that Alice and Bob can successfully can encrypt/decrypt each other's values, starting only with the shared key.

Bob Encrypts

<!--- 
     The previous call to bobsKeyAgreement.generateSecret resets the key
     agreement object, so we call doPhase again prior to another generateSecret call
--->
<cfset bobsKeyAgreement.doPhase(publicKeyFromAlice, true) />
<cfset bobsDESKey         = bobsKeyAgreement.generateSecret("DES") />

<!--- Encrypt the text with a simple encryption --->
<cfset Cipher               = createObject("java", "javax.crypto.Cipher") />
<cfset bobsCipher          = Cipher.getInstance("DES/ECB/NoPadding") />
<cfset bobsCipher.init( Cipher.ENCRYPT_MODE, bobsDESKey ) />
<cfset textToEncrypt      = "StandAndDeliver!" />
<cfset bytesToEncrypt       = CharsetDecode(textToEncrypt, "utf8") />
<cfset encryptedBytes       = bobsCipher.doFinal(bytesToEncrypt) />
<cfset encryptedText     = BinaryEncode(encryptedBytes, "hex")  />

<!--- results --->
<strong>Bob's Values:</strong>
<cfoutput>
<p><strong>DES Key</strong>  : #BinaryEncode(bobsDESKey.getEncoded(), "hex")# </p>
<p><strong>Encrypted Text</strong>  : #encryptedText# </p>
</cfoutput>

Alice Decrypts

<!--- 
     The previous call to bobsKeyAgreement.generateSecret resets the key
     agreement object, so we call doPhase again prior to another generateSecret call
--->
<cfset alicesKeyAgreement.doPhase(publicKeyFromBob, true) />
<cfset alicesDESKey = alicesKeyAgreement.generateSecret("DES") />
<cfset Cipher               = createObject("java", "javax.crypto.Cipher") />
<cfset alicesCipher         = Cipher.getInstance("DES/ECB/NoPadding") />
<cfset alicesCipher.init( Cipher.DECRYPT_MODE, alicesDESKey ) />
<cfset bytesToDecrypt     = BinaryDecode(encryptedText, "hex") />
<cfset unencryptedBytes  = alicesCipher.doFinal( bytesToDecrypt ) />

<!--- results --->
<strong>Alice's Values:</strong>
<cfoutput>
<p><strong>DES Key</strong>  : #BinaryEncode(alicesDESKey.getEncoded(), "hex")# </p>
<p><strong>Unencrypted text</strong>  : #CharsetEncode(unencryptedBytes, "utf8")# </p>
</cfoutput>            

Obviously there is a lot more to consider before entering into this type of exchange. Guarding against man-in-the-middle attacks is definitely one issue. So you will want to do a lot more reading first. But hopefully this entry helped de-mystify Diffie-Hellman key exchanges ... a little.

As always, any comments or corrections are welcome!

...Read More

My First Look at Diffie-Hellman (Merkle) Key Exchange - Part 1

While I have seen references to Diffie-Hellman before, I honestly knew very little about it until this week. After seeing it mentioned on stackoverflow.com, I decided to do some research. Now I am pretty sure this protocol is not available in the standard edition of Adobe ColdFusion, and contrary to popular opinion, I have my doubts about its availability in Enterprise version as well. Though admittedly, I could not find any solid references one way or the other. So I could be wrong. Anyway, since there is a plethora of implementations in the java world, I decided to explore that route.


Disclaimers
First let me say this entry is not intended to be a "how to guide". In the world of encryption, I am most definitely a novice. But I am a curious novice. So in an effort to better understand the Diffie-Hellman (Merkle) exchange, I decided to take one of the simpler java implementations, deconstruct it, and put the results in a CF context.  If you are already familiar with it, this beginner level entry will probably be one big yawn. But any corrections or clarifications are definitely welcome.  Just go easy lest my brain implode.

What is it?
Wikipedia describes Diffie-Hellman as a key exchange protocol

"...that allows two parties that have no prior knowledge of each other to jointly establish a shared secret key over an insecure communications channel. This key can then be used to encrypt subsequent communications using a symmetric key cipher."

My novice interpretation would be that instead of exchanging a key, two parties exchange "other", transitory, pieces of information instead. Then use those "other" pieces of information to derive the key (independently) without actually sending the key itself over an open channel.

What are those "other" pieces of information? In short, they are really, really large numbers. The protocol uses several numbers in a series of simple mathematical formulas to eventually calculate the secret key. I will not go into details about how and why those numbers are selected. The wikipedia entry describes it far better than I ever could. But suffice it to say they are not arbitrary. Each has specific characteristics that directly affect both the results and the security of the exchange. So I would highly recommend you read the article.  But to paraphrase the salient points (without formulas):

1. Two parties (Alice and Bob) agree upon two (2) numbers to be used in their calculations (a prime number and a primitive root) Note: These are public values, known by both Alice and Bob
2. Then Alice and Bob each pick a private number. Note: Alice should not know Bob's value, and Bob should not know Alice's value.
3. Using their private numbers (plus the prime and primitive root) Alice and Bob each calculate a public number. They then exchange public numbers with each other. Note: These are public values, known to both Alice and Bob.
4. After exchanging public values, Alice and Bob now have enough information to calculate the shared secret key (using another formula) Note: They both arrive at the same secret key value, independently, without ever sending that value over an open channel.

Math 101
Now jumping straight into a java example from here felt a bit like sending a student driver onto a five lane highway after receiving only five minutes of instruction. So I decided to test the basic formulas from the wikipedia example first.  It gave me a better understanding of the overall process, and also provided a good basis of comparison for the java results.

Now before someone corrects me, the overview below is conceptual only.  When I finally did run the simple java example, the actual steps were slightly different. But overall the process was the same.

(On a side note, this whole thing felt a bit like something you would read about in a spy novel. But I suppose that cannot be helped...)

Step 1) Preparing for the Meeting
First, a prime number and primitive root are selected. These are considered public values, known to both Alice and Bob.   Now you may notice I am using java objects. That was necessary because the calculations involved produce very large numbers. Even using small test values like 23 and 5 the results exceeded the capacity of a basic CF integer.

<cfset prime = createObject("java", "java.math.BigInteger").init( 23 ) />
<cfset base = createObject("java", "java.math.BigInteger").init( 5 ) />
<strong>Values known to both Alice and Bob</strong>
<cfoutput>
    <p> ie prime  = #prime# AND  base   = #base# </p>
</cfoutput>

Step 2) Secret Code Words
Next, Alice and Bob each select a private number, which they do not share with each other.

<cfset alicesPrivateValue    = createObject("java", "java.math.BigInteger").init( 6 ) />
<cfset bobsPrivateValue        = createObject("java", "java.math.BigInteger").init( 15 ) />

Step 3) The Public Exchange
Alice and Bob then use their private numbers to calculate a public number using the formula: base ^ private MOD prime. They then exchange public numbers. Again, their private numbers are never exchanged.

<cfset alicesPublicValue     = base.modPow( alicesPrivateValue, prime) />
<strong>Alice's values: </strong>
<span class="code">alicesPublicValue = base ^ alicesPrivateValue MOD prime </span>
<cfoutput>
    <p>ie #alicesPublicValue# = #base# ^ #alicesPrivateValue# MOD #prime# </p>
</cfoutput>

<cfset bobsPublicValue     = base.modPow( bobsPrivateValue, prime) />
<strong>Bob's values: </strong>
<div class="code">bobsPublicValue = base ^ bobsPrivateValue MOD prime </div>
<cfoutput>
    <p>ie #bobsPublicValue# = #base# ^ #bobsPrivateValue# MOD #prime#</p>
</cfoutput>

Step 4) Finding the Key
Alice an Bob now have enough information to derive the shared secret value, independently. Alice uses the formula: alicesSharedKey = bobsPublicValue ^ alicesPrivateValue MOD prime .

<strong>Alice uses Bob's value to compute the shared key (B <sup>a</sup> MOD p)</strong>
<div class="code">alicesSharedKey = bobsPublicValue ^ alicesPrivateValue MOD prime </div>
<cfset alicesSharedKey    = bobsPublicValue.modPow(alicesPrivateValue, prime ) />
<cfoutput>
    <p class="result">Alice's shared key is <strong>#alicesSharedKey#</strong></p>
    <p>ie   #alicesSharedKey# = #bobsPublicValue# ^ #alicesPrivateValue# MOD #prime#</p>
</cfoutput>

Whereas Bob uses the formula: bobsSharedKey = alicesPublicValue ^ bobsPrivateValue MOD prime

<strong>Bob uses Alice's value to compute the shared key (A<sup>b</sup> MOD p)</strong><br/>
<div class="code">bobsSharedKey = alicesPublicValue ^ bobsPrivateValue MOD prime </div>
<cfset bobsSharedKey    = alicesPublicValue.modPow( bobsPrivateValue, prime ) />
<cfoutput>
    <p class="result">Bob's shared key: #bobsSharedKey#</p>
    <p>ie   #bobsSharedKey# = #alicesPublicValue# ^ #bobsPrivateValue# MOD #prime#</p>
</cfoutput>

If everything was done correctly, they will both calculate the same value (ie 2). This value can then be used to create a secret key (DES, etcetera) with which to encrypt and decrypt data. 

...Read More