Recently, I had to troubleshoot an issue where there was some improper API use and it was being blamed on the application. The traffic is SNAT'd, so essentially backend servers are being proxied. What I mean by this, is in the "backend servers" perspective, the requests appear to be coming from the F5, not the original source IP. Since the SSL is offloaded on the F5, we can only trace the unencrypted traffic on the backend and due to how noisy it is and all request appear to be coming from the F5 self IP, this becomes very difficult to troubleshoot. So we need to troubleshoot on the frontend, where the public source IP is preserved. This traffic is encrypted, so we need means of viewing the traffic unencrypted. This can be done with tcpdump and ssldump, which both are installed on the F5 by default.
So now we are ready to start capturing the traffic. You can do so with tcpdump, but you should be very strict with your parameters, as you could cause performance issues sending to much garbage to stdout. You will want to save to file, so we can later decrypt with ssldump. Here is an example, listening on the frontend VLAN:
[root@LB01:Active] ~ # tcpdump -vvv -s 0 -nni vlan2025 -w test.pcap host 256.10.20.30
tcpdump: listening on vlan2025
30 packets received by filter
0 packets dropped by kernel
[root@LB01:Active] ~ #
Now you should have the PCAP file in your present directory and you can view it via tcpdump, wireshark, or any other packet analysis tool that you have available, as PCAP is the industry standard packet capture format. If you were to open the file up in Wireshark, you would notice that the SSL/TLS payload displays 'Record Layer: Handshake Protocol: Encrypted Handshake Message', so you aren't able to view the unencrypted data natively, which is expected. This is where ssldump comes in, which can utilized your F5 private keys to decrypt the trace. You need to identify what SSL cert/key pair are used on the VIP you are troubleshooting. You can find this out by looking over the VIP configuration, which will use a specific SSL profile. In looking into that profile, it will mention what cert/key pair it uses. You will be able to find these certs/keys in /config/ssl/ssl.crt & ssl.key on 9.x/10.x or it could be stored in /config/filestore/files_d/Common_d/certificate_d & certificate_key_d in 11.x. Once you find the key, you can decrypt the PCAP, using the following example:
[root@LB01:Active] ~ # ssldump -Aed -nr test.pcap -k /config/ssl/ssl.key/example.com.key
#1: 256.10.20.30(50123) <-> 192.168.255.10(443)
1 1 1421093568.0163 (0.1353) C>SV3.1(125) Handshake
(.. Omitted for brevity ..)
1 10 1421093568.3602 (0.1746) C>SV3.1(351) application_data
---------------------------------------------------------------
POST /example/wrong/uri/app HTTP/1.1
Content-Type: text/xml; charset=utf-8
Host: example.com
Content-Length: 472
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
(.. Omitted for brevity ..)
1 1421093568.8656 (0.0000) S>C TCP FIN
1 1421093569.0030 (0.1374) C>S TCP FIN
[root@LB01:Active] ~ #
So there you have it, you can decrypt SSL traffic if you have the private key with only tcpdump and ssldump. You can perform the same task in using tcpdump to output to PCAP and then using the private key in Wireshark to decrypt the traffic, although I find it easier to troubleshoot using tools on the F5 if I can.
Let me know if you have any questions on the subject or any suggestions to improve this method.
Feel free to comment!