Wireshark trace - timeout problem
I have following setup (as visible in the attached image):
A (java process) -> B (kubernetes ambassador proxy) -> C (java service in kubernetes pod)
Communication is done using HTTPS between A and B, and then ambassador strips HTTPS and continues talking HTTP with C.
The problem I'm experiencing is that sometimes, it happens that HTTP BODY message that is being sent is not transmitted 100% between A and B, but only on B side trace I can see that it stopped for some reason (in trace on A side it is shown as all is sent OK). Then, java process in C (which is waiting for B-proxy to forward all the data) is just waiting and time-outs after 30 seconds.
You can see in the image attached, in A trace it is written that whole BODY is sent, but in trace on the B side, only half of BODY is visible (delivered). I suspect on these TCP Previous segment not captured.
You can also see that after this it just waits for 30 seconds, and times-out.
It happens pretty often in my setup. Does anyone knows what could be a problem?
Comments
Were A->B and B->A anonymized/modified or is there PAT and NAT in the process?
B is in kubernetes. A is accessing it using this ambassador mapping service that is then forwarding it to the ambassador pods (B). Do you think that B should see all the packages in trace (since A says everything sent and confirmed) ? Maybe then the problem is even before B (before ambassador pod in k8s) ?
The timestamps on some of the packets in B to A are out of order.
reordercapwill sort them by time which sometimes helps Wireshark with analysis.Tried that just now. Seems more-less the same. Here is updated image with all packets: https://prnt.sc/qap1zd
Is it possible that wireshark dropped packages since it was recorder inside the pod? Maybe there is some other way to capture packages outside of the pod. I guess since on A side it looks like everything was delivered that something like this is possible? Or maybe this is due to ambassador limiting bandwidth or timeouts?
Does the proxy have statistics available to indicate its performance? Queue size? Drops? Buffer usage?