Symptoms:

java.io.IOException: Server returned HTTP response code: 670 for URL: ...

Calling node receives a network service code 670 and is unable to push or pull data.

Explanation:

Calling node receives a network service code 670 and is unable to push or pull data. This can happen when the receiving node is overloaded servicing concurrent workers. There could be several reasons that the servicing node is unable to complete requests faster than they are coming in. Some areas to consider.

• Database connections are limited
• Database connections are slow
• Worker threads are not set high enough
• Push and/or pull job frequencies are set to aggressively
• Unexpected increase in the data to transmit

Solutions:

• Have a DBA look for any long running queries
• Verify the database connection pool provided to SymmetricDS is inline with the number of clients and the frequency they communicate. This would need to be at least double the number of clients expecting to communicate at any given moment and should even be higher so SymmetricDS jobs can also do their work (routing, purge etc).
• Verify the worker threads are aligned with the database connection pool. Remember to account for one thread for pull and one for push for each client.
• Scale down the push and pull frequencies to align better with the resources available.
• Wait for any large increase in change capture to process.

Additional Read:

Scaling SymmetricDS For Many Clients