Decrypt F5 SSL traffic for troubleshooting

Recently, I had to troubleshoot an issue where there was some improper API use and it was being blamed on the application. The traffic is SNAT'd, so essentially backend servers are being proxied. What I mean by this, is in the "backend servers" perspective, the requests appear to be coming from the F5, not the original source IP. Since the SSL is offloaded on the F5, we can only trace the unencrypted traffic on the backend and due to how noisy it is and all request appear to be coming from the F5 self IP, this becomes very difficult to troubleshoot. So we need to troubleshoot on the frontend, where the public source IP is preserved. This traffic is encrypted, so we need means of viewing the traffic unencrypted. This can be done with tcpdump and ssldump, which both are installed on the F5 by default.

So now we are ready to start capturing the traffic. You can do so with tcpdump, but you should be very strict with your parameters, as you could cause performance issues sending to much garbage to stdout. You will want to save to file, so we can later decrypt with ssldump. Here is an example, listening on the frontend VLAN:

[root@LB01:Active] ~ # tcpdump -vvv -s 0 -nni vlan2025 -w test.pcap host
tcpdump: listening on vlan2025

30 packets received by filter
0 packets dropped by kernel
[root@LB01:Active] ~ #

Now you should have the PCAP file in your present directory and you can view it via tcpdump, wireshark, or any other packet analysis tool that you have available, as PCAP is the industry standard packet capture format. If you were to open the file up in Wireshark, you would notice that the SSL/TLS payload displays 'Record Layer: Handshake Protocol: Encrypted Handshake Message', so you aren't able to view the unencrypted data natively, which is expected. This is where ssldump comes in, which can utilized your F5 private keys to decrypt the trace. You need to identify what SSL cert/key pair are used on the VIP you are troubleshooting. You can find this out by looking over the VIP configuration, which will use a specific SSL profile. In looking into that profile, it will mention what cert/key pair it uses. You will be able to find these certs/keys in /config/ssl/ssl.crt & ssl.key on 9.x/10.x or it could be stored in /config/filestore/files_d/Common_d/certificate_d & certificate_key_d in 11.x. Once you find the key, you can decrypt the PCAP, using the following example:

[root@LB01:Active] ~ # ssldump -Aed -nr test.pcap -k /config/ssl/ssl.key/
#1: <->
1 1  1421093568.0163 (0.1353)  C>SV3.1(125)  Handshake

(.. Omitted for brevity ..)

1 10 1421093568.3602 (0.1746)  C>SV3.1(351)  application_data
    POST /example/wrong/uri/app HTTP/1.1
    Content-Type: text/xml; charset=utf-8
    Content-Length: 472
    Expect: 100-continue
    Accept-Encoding: gzip, deflate
    Connection: Keep-Alive

(.. Omitted for brevity ..)

1 1421093568.8656 (0.0000) S>C TCP FIN
1 1421093569.0030 (0.1374) C>S TCP FIN
[root@LB01:Active] ~ #

So there you have it, you can decrypt SSL traffic if you have the private key with only tcpdump and ssldump. You can perform the same task in using tcpdump to output to PCAP and then using the private key in Wireshark to decrypt the traffic, although I find it easier to troubleshoot using tools on the F5 if I can.

Let me know if you have any questions on the subject or any suggestions to improve this method.

Feel free to comment!

Capture packets on a Cisco ASA

There are many situations where you need to be able to capture traffic at ANY point in the network. Whether it testing from the client side, server side, or any transit component, such as your ASA. Since the traditional tools such as tcpdump or Wireshark aren't available, you will have to use the proprietary tools on the ASA itself.

You will start off by creating an ACL to match the traffic you are wanting to capture. This should be a strict, such as a source and dest that are experiencing the issues you are troubleshooting:

access-list test.capacl extended permit ip host host
access-list test.capacl extended permit ip host  host

Now that you have configured the access list to match your traffic, you can start the capture. I am capturing traffic before and after it goes over the VPN, so I will monitor the interface closest to the source, which is the 'inside' interface. There aren't any "required"; options besides the name but if you don't specify an ACL and interface, you won't match any traffic. There are a few options you may want to use other than those two:

  • type (default is raw-data but there are other options such as ISAKMP)
  • headers-only (no data, just headers; similar to tcpdump -q)
  • buffer (default is 512KB. Increase with caution, you don't want to use up all your memory)
  • real-time (have output display in your terminal)
  • *more* - use 'capture <cap_name> ?' to see all options

We will go with the basics; access-list and interface:

fw01# capture test-cap access-list test.capacl interface inside

After this, you can check your console to see if the traffic is matching your ACL and being captured with 'show capture'; if the bytes are a non-zero value and increasing, it's working!

fw01# show capture
capture test-cap type raw-data access-list test.capacl interface inside [Capturing - 1560 bytes]

If you want to see the content, you have to specify the capture name:

fw01# show capture test-cap

12 packets captured

   1: 01:47:27.675914       802.1Q vlan#100 P0 > icmp: echo request 
   2: 01:47:27.761236       802.1Q vlan#100 P0 > icmp: echo reply 
   3: 01:47:28.678431       802.1Q vlan#100 P0 > icmp: echo request 
   4: 01:47:28.763708       802.1Q vlan#100 P0 > icmp: echo reply 
   5: 01:47:29.688425       802.1Q vlan#100 P0 > icmp: echo request 
   6: 01:47:29.773687       802.1Q vlan#100 P0 > icmp: echo reply 
   7: 01:47:30.693262       802.1Q vlan#100 P0 > icmp: echo request 
   8: 01:47:30.778493       802.1Q vlan#100 P0 > icmp: echo reply 
   9: 01:47:31.695703       802.1Q vlan#100 P0 > icmp: echo request 
  10: 01:47:31.781011       802.1Q vlan#100 P0 > icmp: echo reply 
  11: 01:47:32.698999       802.1Q vlan#100 P0 > icmp: echo request 
  12: 01:47:32.784184       802.1Q vlan#100 P0 > icmp: echo reply 
12 packets shown

So if you were troubleshooting connectivity and you see this successful ICMP traffic, the loss you are experiencing must be somewhere else. Now you can assure that packets are hitting the firewall and this should help you verify your high level packet flow, to hopefully pinpoint where the issue is. If you want to see more information, there are a few more useful options when displaying the capture:

  • access-list (yes, you can use an additional access-list to filter the output of 'show capture')
  • count (# of packets to display)
  • detail (display more information such as TTL, DF bit, etc)
  • dump (display the hex dump for each packet)
  • *more* - use 'show capture <cap_name> ?' to view all options

So if you would like to view the hex dump of 1 packet, you can with:

fw01# show capture cap-test count 1 dump

60 packets captured

   1: 01:47:27.675914       802.1Q vlan#100 P0 > icmp: echo request
0x0000	 c08c 608c 2990 0050 569c 4211 8100 000a	..`.)..PV.B.....
0x0010	 0800 4500 0060 0000 4000 4001 a9a6 c0a8	..E..`..@.@.....
0x0020	 0a87 c0a8 051f 0800 74cd 5122 0000 0000	........t.Q"....
0x0030	 0000 0000 0000 af71 2c55 0000 0000 4c49	.......q,U....LI
0x0040	 0a00 0000 0000 0000 0000 0000 0000 0000	................
0x0050	 0000 0000 0000 0000 0000 0000 0000 0000	................
0x0060	 0000 0000 0000 0000 0000 0000 0000 0000	................
0x0070	 0000                                   	.. 
1 packet shown

Often, you may have a large capture and you don't want to sift through it on the ASA. You might want to look at the capture file in wireshark or tcpdump. You can do so by copying the file via tftp to a workstation or server. But how will the output be formatted... with just headers, hex dumps or .. ? Well, when you transfer it via copy, you specify to use pcap format, which you will be able to read in just about any packet analysis program as it is the most commonly used. You can do this via:

fw01# copy /pcap capture:cap-test tftp:TFTP-SERVER-IP

Awesome! So although the ASA capture utility isn't as versatile as tcpdump or wireshark for example, it is still great to start with and verify packet flow, etc. And if you want to save a larger capture for future analysis, it can be copied and opened with another packet analysis tool. From there you can use your advanced filters, etc.

I hope you enjoyed the basic tutorial on how to use the ASA capture utility... perhaps soon I will write one up on how to effectively use packet-tracer, which is yet another useful ASA command line tool for verifying packet flow and it will also give you a better understanding of how the ASA processes packets. (order of NAT/ACL processing, route lookup, etc.)