How do I increase the timespan of the
TimeoutPersisterReceiver? it currently retries for ~1min after arming before firing. I have looked in the documentation but cannot see a config setting for it.
We have been having some network issues that prevented a service from talking to the database causing the
RepeatedFailuresOverTimeCircuitBreaker to arm itself.
INFO NServiceBus.CircuitBreakers.RepeatedFailuresOverTimeCircuitBreaker [(null)] - The circuit breaker for TimeoutStorageConnectivity is now in the armed state
The database will eventually come back online but by the time it does the circuit breaker has been fired and the Bus has been disposed. I need to increase the time span of the
NServiceBus.CircuitBreakers.RepeatedFailuresOverTimeCircuitBreaker [(null)] - The circuit breaker for TimeoutStorageConnectivity will now be triggered FATAL NServiceBus [(null)] - Repeated failures when fetching timeouts from storage, endpoint will be terminated.Answer1:
You can try to configure it in a class that implements IWantToRunWhenBusStartsAndStops and use the SecondsToSleepBetweenPolls property of the TimeoutPersisterReceiver.
However, I would add that public access to this configuration option was removed in later versions, and I'm not 100% sure why. There may be dragons. See <a href="http://bit.ly/1ToyOHb" rel="nofollow">http://bit.ly/1ToyOHb</a> (GitHub).Answer2:
Circuit breaker behavior
After researching the code it seems that you would be able to configure the timeout poll interval via
TimeoutPersisterReceiver.SecondsToSleepBetweenPolls but that does not really matter.
The embedded circuit breaker configuration cannot be adjusted. If an error occurs it is <em>armed</em> and if no successful poll succeeds within 2 minutes the circuit breaker action will be triggered which is calling raising a critical error.
Increasing the poll interval will not help here. If you would make the poll interval more than 2 minutes the circuit breaker will <em>always</em> be armed thus trigger the action.<ul><li>
<a href="https://github.com/Particular/NServiceBus/blob/4.7.12/src/NServiceBus.Core/Timeout/Hosting/Windows/TimeoutPersisterReceiver.cs#L150" rel="nofollow">https://github.com/Particular/NServiceBus/blob/4.7.12/src/NServiceBus.Core/Timeout/Hosting/Windows/TimeoutPersisterReceiver.cs#L150</a></li> <li>
<a href="https://github.com/Particular/NServiceBus/blob/4.7.12/src/NServiceBus.Core/CircuitBreakers/RepeatedFailuresOverTimeCircuitBreaker.cs" rel="nofollow">https://github.com/Particular/NServiceBus/blob/4.7.12/src/NServiceBus.Core/CircuitBreakers/RepeatedFailuresOverTimeCircuitBreaker.cs</a></li> </ul>
There is nothing that you can do unfortunately except building your own version that removes the circuit breaker completely.
This behavior can result in issues when your database server does a fail-over.Critical error handling
The only possible solution to tackle this condition is by hooking into the critical error handler. Then you know such issue occurs and can do a application shutdown as I suspect that you are currently self-hosting.
You can also try to create a whole new bus instance and 'swap' the broken one with the fresh one but that is a pretty daunting task and I would recommend to <em>just exit your application</em>.