ANNOUNCEMENT: Live Wireshark University & Allegro Packets online APAC Wireshark Training Session
April 17th, 2024 | 14:30-16:00 SGT (UTC+8) | Online

Wireshark-users: [Wireshark-users] esp decryption problem

From: Reinoud Koornstra <reinoudkoornstra@xxxxxxxxx>
Date: Sat, 21 Jan 2017 02:31:38 -0800
Hi Everyone,

I have an nice tunnel setup between two networks.
I did list the state with ip xfrm state.
It showed me this:

service@jpar1:~# ip xfrm state
1.1.1.1 2.2.2.2
        esp spi 3326951867(0xc64d41bb) reqid 1 tunnel
        A:hmac(sha1) fa6b31ae be38e9e7 798962a6 b845ea7d 73dfa8e1
        auth-trunc-len: 96
        E:cbc(aes) 7cf1de2c 794f4fd 1c682d9c 1bb3cb
        fpid 0x00000004
        fp_output_blade 1

2.2.2.2 1.1.1.1
        esp spi 3463449761(0xce700ca1) reqid 1 tunnel
        A:hmac(sha1) 54b978 bb5ff5e 3173b5ac 95cadb3 4bf1c6de
        auth-trunc-len: 96
        E:cbc(aes) 1ba5e1e4 fdf6c76 3bc18ef1 48e3db4e
        fpid 0x00000003
        fp_output_blade 1

Wireshark claims that the key from 2.2.2.2 to 1.1.1.1 is 248 bits.
However, it's length is 32, which times 8 is 256.
I wrote a small program to determine how many characters are in each key:

#include <stdio.h>

int main(void)
{
       char    a[]="7cf1de2c794f4fd1c682d9c1bb3cb";
       char    b[]="1ba5e1e4fdf6c763bc18ef148e3db4e";

       printf("amount of characters in a is %lu\n", sizeof(a));
       printf("amount of characters in b is %lu\n", sizeof(b));
       return(0);
}

The output tells me:
amount of characters in a is 30
amount of characters in b is 32

So why does wireshark say that the keylength of b is 248 bits?
<ESP Preferences> Error in Encryption Algorithm AES-CBC : Bad Keylen (232 Bits)
<ESP Preferences> Error in Encryption Algorithm AES-CBC : Bad Keylen (248 Bits)

Weirdly enough the 1.1.1.1 to 2.2.2.2 aes-cbc key is 2 characters
shorter than the 2.2.2.2 to 1.1.1.1 key according to xfrm. However,
wireshark claims the keylength is 232 bits. When I add a 0x to the key
from 2.2.2.2 to 1.1.1.1 (b) It doesn't complain about the keylength,
but then it still won't decrypt:
True: ICM matches packet content, False: doesn't match content or not
checked. The authentication data shows me incorrect.

However, the first surprise is that the two keys aren't the same length.
Secondly, I don't understand how wireshark calculates the length of the keys.
Which leads to the questions:

1) do I need to add 0x to the keys displayed in ip xfrm state?
2) I didn't see anything about salting added or nonce added in the
case of aes-cbc
3) even though the ping works fine through the tunnel, they keys of
each direction aren't the same length according ip xfrm.

Any idea what I am missing and why it's not decrypted properly?
Thanks,

Reinoud.