Traffic Missing or Inconsistency Issue

Issue Description:

Customers reported problems about bank’s online services and banks unable to find user sessions in IBM Tealeaf.

Issue Summary:

The obstacle here was to get to the root cause as bank’s end customer’s sessions were missing so we had to look at all probable causes like network packets dropped, SSL certificate issue and internal traffic forwarding configuration. Capture server never drops any traffic unless there are rules in PCA defined to do so, and we checked all the rules and we don’t have any rules which drops any traffic packets. We examined SSL certificates as well and its validity and found that some certificates were expired and not updated which fixed some issues related to traffic inconsistency. None of the PCA flags on the Summary Tab showed any other concerns. We then compared the traffic stats of the network F5 switch but it did not show accurate pictures as it was forwarding lot of traffic which PCA was discarding, as per customers’ license and business requirements. Whenever we created sessions or someone from the bank’s internal network created sessions, those sessions always appeared. We were also comparing tcpdump of PCA with IBM Tealeaf traffic reports and weren’t shown any probable root cause, so we decided to use different browsers and various versions to create sessions. This idea worked because we found that sessions from the latest version of firefox were missing. So, we created a specific tcpdump of that IP and analyzed that tcpdump which showed the presence of Diffie Hellman (DH) ciphers.

Issue Impact:

DH aren’t intended for deciphering, thus, when Capture server notices the existence of DH ciphers it just relinquished those packets because IBM Tealeaf cannot decipher that traffic and therefore no further processing can be done. DH is often used by new age browsers and therefore webservers are compelled to use them as the preferred cipher.

Issue Resolution:

We first reduced the priority of DH ciphers from the list of ciphers used by Web Server for the internal traffic and tracked if there was any impact from security or any other site issues. Upon successful validation, we implemented the same strategy for all the traffic.  This lead banks to see all the traffic and also banks were able to find reported issues from customers and fixed them with much lesser turnaround time.

Please check out the video for more on IBM Tealeaf and How Royal Cyber can help.