We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages.

There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns downthread), but there is serious packet loss at the least. You'll have to wait for someone actually involved in the incident to say more.

There's some irony in that.

I’m not in SRE so I don’t bother with all the backup modes (direct IRC channel, phone lines, “pagers” with backup numbers). I don’t think the networking SRE folks are as impacted in their direct communication, but they are (obviously) not able to get the word out as easily.

Still, it seems reasonable to me to use tooling for most outages that relies on “the network is fine overall”, to optimize for the common case.

Note: the status dashboard now correctly highlights (Edit: with a banner at the top) that multiple things are impacted because Networking. The Networking outage is the root cause.

this column of green checkmarks begs to differ: https://i.imgur.com/2TPD9e9.png

except time

You want velocity for your dev team? You get that. You want better uptime? Your expectations are gonna have a bad time.

It's always interesting to see these outages at large cloud providers spider out across the rest of the internet, a lot of the world depends on Google to stay up.

Yup, I'm trying to check the Associated Press News right now and it's having trouble connecting to "storage.googleapis.com".

Incident #19008 began at 2019-06-02 12:48. https://status.cloud.google.com/incident/cloud-networking/19…

Incident #19009 began at 2019-06-02 12:53. https://status.cloud.google.com/incident/cloud-networking/19…

Times are US/Pacific

I hope they come back. This is still pretty scary

Of course, this is 2 weeks after switching everything over from AWS.

So I wander over to my Firebase console, and there's no database loading. Thank god for twitter, and people also saying that they have the same issue or I would have for sure though we've been hacked.

I hope this is a good wake up call for everyone. I know that I'm going to think more about how we do backups and fail-safes

I don’t miss being on pager duty one bit. I see it looming in my headlights, sadly.

… but not for everybody now.

  [21:55:19] POP< +OK send PASS   [21:55:19] POP> PASS ********   [21:55:21] POP< +OK Welcome.   [21:55:21] POP> STAT   [21:55:21] POP< -ERR [SYS/TEMP] Temporary system problem.   Please try again later.

I speak from experience: at Monzo we have had some pretty horrific and very public outages, yet because we communicate openly and proactively with our customers, people are very understanding and trust that we’re doing our best to fix things. The amount of love we’ve had from our customers during times when we’ve put them in a really bad spot – sometimes without access to their money – has been astonishing.

GCP has quarterly-ish global blackouts, and generally on the data plane at that which makes them significantly more severe.

AWS had the S3 incident affecting all of us-east-1: “Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.”

https://aws.amazon.com/message/41926/

I'm overall happy with it, but if I needed to run a service with a 99.95% uptime SLA or higher, I wouldn't rely solely on GCP.

AWS refunded me in the first reply on the same day!

GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me.

So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. I have called my bank to do a chargeback and put a hold on all future billing with GCP.

I'm now back to AWS and still on a Free Tier. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice….

I've yet to receive a reply from Google and its been a week now.

I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.

https://www.google.com/appsstatus#hl=en&v=status

https://i.imgur.com/pcqwwA4.png

There appears to be some irregularities on consumer services as well that are of course certainly related, youtube was behaving a bit oddly for me.

The impact seems to be cascading down from just GCE to other services as well – that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted.

It's amazing how far-reaching outages can be these days.

is what I’ve heard so far. east seems to be OK, and Europe too

Pretty much every service is down

GCE, GKE, BQ, Pub/Sub, GAE

asia-south1 us-west1 us-central1 us-west2

https://news.ycombinator.com/item?id=20077275

https://news.ycombinator.com/item?id=20077293

I'm not seeing anything at 12:47.

Next update is in about 25 minutes.

404 – Impressive

edit: wording

WARNING: The following zones did not respond: us-west2, us-west2-a, southamerica-east1-c, us-west2-b, southamerica-east1, us-east4-b, us-east4, us-east4-a, northamerica-northeast1-c, northamerica-northeast1-b, us-west2-c, southamerica-east1-b, northamerica-northeast1, southamerica-east1-a, northamerica-northeast1-a, us-east4-c. List results may be incomplete.

Luckily for us eu-west1 seems to be working normally.

The networking incident looks like the one to follow for updates now.

The interesting thing is that a couple of minutes before everything went wrong, kubectl returned a "error: You must be logged in to the server (Unauthorized)" error

especially if you use your bussiness for B2B services. Stuff like this could make you loose your bussiness, especially if some entity like google doesn't communicate and as a result, you do not have a answer for your own customers.

Medium sized private cloud providers are a lot better at this, considering the communication lines are a lot shorter.

EDIT: It's been like that since at least 12h ago though. Not sure if it's connected to Google Cloud?