(update: since it was being downvoted anyway, removed the explanations part and just left the pissed off longer knee jerk reaction response to another shorter knee jerk reaction)
Coming back to your pathetic quip: what kind of piece of work one has to be to reply in this somewhat apple-fanboi-esque way (where someone says "how to fix X problem in Y" and pat comes the reply "why not just buy Y-latest, it's just $349") and what kind of unidimensional world you live in where you believe someone wanting some free and private git hosting is the problem? Ffs.
So again and with all due respect, please shut the fuck up! Because you don't know me, you don't know where I am coming from, you don't know the why, and you don't come across as someone who would ask or even care about knowing.
Shipyard has been one of the main maintainers because they have previously been an aggregator of funding — think consulting firm, that gets grants and hires people.
Protocol Labs is moving and offering to pay (as far as I understand) the same maintainers and developers directly; just not via Shipyard.
Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.
My personal take-away is this: whatever you choose, make sure it's consistent across services (not serviceA behaves like X and serviceB like Y) and make sure eng teams know _how_ these are configured and what can go wrong. They'll figure out the rest.
You'd be surprised how many engineering leaders don't understand the CAP theorem and will fail engineers on interviews for picking the one they don't agree with instead of communicating their expectations clearly (dodged a bullet on that one ...)
That's a problem for circuitbreakers on these kinds of actions, not lying on health checks.
Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed
backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain
Yeah that's why I have a manual intervention breaker that goes across all the pods/nodes etc. CrashLoopBackoff is great for selfhealing but when things go really pear shaped you want something that catches the global state. Saw it activate during an AWS outage one time where new nodes were unhealthy on start, for example.
you are the problem