We went the direction of DNS based redirects using a set of TXT records to get portability and delegation of subpaths with txtdirect.org (FOSS based on Caddy)
Currently working on a new iteration and feedback is always appreciated.
Explaining why an upgrade didn't work doesn't compromise security. I'd guess moving the data didn't work or they corrupted a database and don't know how to repair it.. Explaining that you don't have proper data backups in place can be embarrassing but with a post mortem you can at least get some trust back (like Gitlab's incident).
Not explaining why you're offline for 24h doesn't help people to trust you
If they corrupted a live database and are not able to recover it they are in a world of hurt. While it is bad form, many people keep their coins on the exchanges and even if the bulk of an individuals coins are offline, they still likely have at least a small amount on their for trading.
If a table that connects user accounts to kraken owned wallets is corrupted and not recoverable people will be out millions. For some that would be the equivalent of your 401k issuing a post mortem for losing all of your retirement.
If this worst case scenario happened they are likely in severe damage control.
Most likely explanation though is that things are just taking longer than expected to upgrade what is by all measures likely a very technical and convoluted system.
Their downtime and non usable site was so bad already that downtime is almost as inconvenient as the working site was before.
Now at least they are fixing it. They held back the update for months due to testing. They have to take the jump and after x hours of downtime the damage is done, so fixing it once and for all instead of rolling back might actually be the better solution.
Considering some of the comments I wrote at three in the morning, I guess that the actual engineers had a quick break after the social media person went home.
There are some companies using k8s not just for the ability to move, but as a hybrid starting ground.
One bandwith intense workload is running with GKE for all state/databases, but uses a bare-metal k8s cluster for compute and bandwidth intensive stuff. They actually use federation and the bare-metal cluster basically looks for a local database doesn't find one and gets routed to the next available global cluster, which is GKE.
Due to the cache and read dependend nature of the database queries the latency impact is worth it.
Gitlab reversed their decision to go with bare-metal. They still plan to ditch Azure and the most likely candidate (already testing to move their CI infra) for the move is GCP/GCE and GKE from Google.
Sidenote:
We are running coredns.io in production as authoritative internal DNS and as hidden master with NOTIFY to a secondary DNS provider (currently DNSmadeEasy).
The DNS records for the internal records are done using the kubernetes middleware (basically serving the service records).
The external records are pulled in from a git repository hosting our zones as bind files. If need be zones are split into subzones per team/project. Same permission system as our code via MRs using Gitlab.
Our recommendation is build on open standards (BIND, AXFR) and use services on top of these.
I agree that using an external mail provider is usually a good idea. It mostly is your fallback communication channel and is usually easy to switch (doing replication to an offsite mail storage needs to be done to make switching easy/possible/fast). MX records \o/
