I had a frustrating start on my first day back at work following my nice and relaxing vacation – all of a sudden it was reported to me that SVN commit and SVN update actions were now failing for the development team. What followed was a lot of frustrating troubleshooting, though thankfully I did eventually find the solution.
My first port of call was simply to restart the relevant SVN server as well as our office network IPCop machine. Following that, I confirmed that the SVN server was behaving correctly when being used by external resources, meaning that the problem had to be between our network and the SVN server. However, many fruitless hours of trawling through logs and packet capturing with Wireshark turned up nothing.
Then I stumbled across something interesting. By first tinkering with and then finally disabling Apache’s new-ish anti-denial of service module “mod_reqtimeout”, the SVN commit and SVN update actions were able to run again.
Now the Apache changelog describes mod_reqtimeout as: “a module which limits the time Apache waits for a client to send a complete request. This helps to mitigate against certain denial of service attacks. In case of problems with slow clients, the timeout values can be adjusted in /etc/apache2/mods-available/reqtimeout.conf , or the module can be disabled with “a2dismod reqtimeout”.
So one of two things must have happened: either apache was updated on the SVN ubuntu server, or our internal network has somehow slowed down. In any event, at least we can work again – just a pity it took 4 or so hours to get there!
*sob*