Network TCP specialist to solve XML Timeout problem

=====The issue====== We connect to a third party XML service over http using a Java application. Unfortunately we suffer intermittent timeouts that cause issues at the application level and requests need to be resent. e.g. "2013-09-23 11:19:42,511 ERROR [login to view URL]: Read timed out connecting to ..." We can go hours without a timeout and then suffer periods of very poor service quality. ==== Additional Info ======== We have confirmed that the XML provider does see and reply to each of our requests, but we do not see the response at the application level. TCP capture did show some odd behaviour, however, we are not skilled enough to interpret it. We strongly suspect that there is some network issue at play here. We thought it might be due to window scaling as described by [login to view URL] but adjusting my tcp_default_win_scale to 0 didn't seem to help. This occurs on more than one server so probably not a local HW issue, and I think wget also observes same issue which rules out the application. We only have an issue with this one service provider. The env is Linux [login to view URL] #1 SMP i686 i686 i386 we use IP tables & csf but see nothing odd at fw level. ==== What we need ======== We cannot allow root access to client machines for security reasons, however, we can supply environment, settings, and results of tests, a connection to the third party service etc. Someone that has experience of this or similar issues should be able to assist here to: 1. Come up with some possible causes. 2. Tell us what tests need to be run e.g. tcp capture 3. Interpret results. 4. Identify issue. 5. Suggest solution. Please quote for the investigation (Steps 1-3). I'll offer a 50% bonus if the cause is found (4), and another 50% if a workable solution is found (5). In your proposal please list a few possible causes to demonstrate your experience/capability in the area.