Symptoms

During a data synchronization with push or pull over HTTP/S, the node initiating the request receives the following error in the log:

java.net.SocketTimeoutException: Read timed out

Small batches may synchronize without error under a specific byte threshold, typically under 2 KB, while batches larger than the threshold get the error.

Explanation

There is problem with Maximum Transmission Unit (MTU) and fragmentation that causes packet loss along the route. The outgoing interface has a MTU that is higher than one or more of the hops along the route to reach the destination. A packet is fragmented into smaller packets to accommodate the smaller MTU. Normally, the IP stack will hold each packet until all fragments arrive, to reassemble them and pass them up the stack to TCP. But a firewall can drop all packets except the first fragment, which causes packet loss.

The first fragment has TCP port numbers, so it is easily matched by firewall rules. Subsequent fragments can't be matched on TCP port, which makes it more difficult for firewalls and network address translation (NAT) rules to match. Because of the increased complexity, state requirement, and possibility of an attack against the packet filter, many firewall and NAT devices simply drop the packet fragments.

Resolution

For each network interface along the route, set the MTU to the same value. Ethernet is typically set to a MTU of 1500 bytes. Use a utility program to find the MTU along each hop in the route. On Windows, use the following command (from the http://www.elifulkerson.com/projects/mturoute.php website):

mturoute myserver

On Linux, use the following command (usually available in the distribution's package repository):

tracepath myserver