Verizon outage & outage site musings

[HCHTech]

We are a victim of today's widespread Verizon outage - all phones have been down since about 9am this morning for us. We have some Comcast lines we use for backup, so we're not dead in the water, but it has been an.....interesting day.

Verizon did make a twitter post about 11:30am, but no updates at all since then - I'm sure you are shocked at that revelation.

This got me thinking about sites like downdetector.com. Here is their current graph for Verizon:

Here's what Downdetector's own site says about the reporting:

A small number of users reporting a problem does not constitute a large-scale incident. To make sure that incidents are correctly represented, Downdetector calculates a baseline volume of typical problem reports for each service monitored, based on the average number of reports for that given time of day, measured over the previous year. Downdetector only reports an incident when the number of problem reports is significantly higher than the baseline.

So the graph shows number of reports of outages greater than the baseline average number of outages for any given time of day. For example the peak of this outage was around 11:30 today, and the alt-text tells you that the baseline (i.e. "normal") number of reports of Verizon outages at 11:30am is 76. Today's graph shows the total of 104,440 reports in excess of the baseline. If you follow the graph forwards to 4:00 pm, the baseline number is 86, and the total reported outages at 4:00pm are 47,067 more than the baseline.

I guess what struck me was this language from their help pages:

Downdetector only accepts the first report for a specific company from a single user. Subsequent information about the issue is collected, but is not counted as a new problem report.

In other words, there is no tracking at all (because how could there be) of when service comes back to the folks that report an outage. So the downward slope of the graph doesn't mean or even imply that service is being restored. It just means that over time, fewer NEW people are reporting an outage. Their graph is only updated every 15 minutes, so on its face, it doesn't provide any information about the true extent of an outage or it's resolution. It's only measuring new complaints.

If you add up all of the data points on their graph, you can get the total number of people that reported an outage, but that's it. No measure of any mitigation at all. And because to total number of possible reports is not provided (or known, I expect), you can't know anything like "40% of their customers are without service".

No question here at all, but just an observation that these graphs are a lot less useful than they appear....plus, I've seen the downdetector graph in at least 3 news stories today about the outage.

 

[ThatPlace928]

The Verizon outage is nationwide and it's not affecting just a small group of people. I have Visible.com and it's a small subsidiary of Verizon. It's working fine but so many people here in Mohave County are complaining of being on SOS, emergency calls only. If Verizon says it's not widespread, who the heck are they fooling?

 

[Markverhyden]

I stopped really looking hard at DD and similar sites. The only time I'll report it to a customer is if they ask how "big" it is. Namely is it wide spread or not. Remember that large swaths of users are on towers not controlled by their provider which significantly muddies the waters.

 

[HCHTech]

Remember that large swaths of users are on towers not controlled by their provider which significantly muddies the waters.

Right as well. Plus only a small number of folks would even bother to report their troubles to a site like DD. I'd say the real outage affect has to be in the millions. My phone still isn't working for calls. I'm getting sporadic text messages, but I'm not receiving any new texts sent as a test by my wife, for example.

I expect we cling to sites like DD because it's really the only source of information beyond the pre-attorney-screened words-without-any-real-information messages we get from Verizon itself.

 

[ThatPlace928]

I suspect internet providers are using Verizon towers, as well. I started updates on 8 or 9 laptops in my shop and some of them are advancing maybe 1% an hour. Bad day for updates, I guess.

 

[Porthos]

Verizon outage: iPhones, Android devices stuck in SOS mode

A widespread Verizon outage is causing iPhones and Android devices to enter SOS mode, preventing them from making mobile calls unless they use WiFi calling.

www.bleepingcomputer.com

 

[callthatgirl]

I have Verizon/Spectrum and my phone was good all day.

 

[phaZed]

Verizon Cell and Home Internet here, all good, no outages.

 

[britechguy]

As an aside, just moments ago I got 2 separate auto-generated email messages saying that I had received my Content Contributor I and Content Contributor II badges on the Verizon Community. I cannot even recall how long ago it was that I last posted their site, so this has to be some sort of bizarre spin-off of yesterday's events.

 

[Markverhyden]

As an aside, just moments ago I got 2 separate auto-generated email messages saying that I had received my Content Contributor I and Content Contributor II badges on the Verizon Community. I cannot even recall how long ago it was that I last posted their site, so this has to be some sort of bizarre spin-off of yesterday's events.

Sounds like they may have powered on a server that they didn't know was powered off?

 

[britechguy]

Sounds like they may have powered on a server that they didn't know was powered off?

Well, it's been off an awfully lot longer than one day ago if it thinks the activity it's "awarding" me for is new. It's just very strange.

 

[Markverhyden]

I suspect internet providers are using Verizon towers, as well. I started updates on 8 or 9 laptops in my shop and some of them are advancing maybe 1% an hour. Bad day for updates, I guess.

With the advances in technologies, as in 802.11 speeds, I wouldn't be surprised if they weren't occasionally routing traffic via antennas vs copper/fiber. A few weeks ago I was installing a Cradlepoint router using SIMM's. The store was still under construction but support told me they usually bring the store live on 4g, 5g, LTE, whatever. ISP's have become very unreliable when it comes to dropping in a new service even if they are already on other neighboring businesses. When I did a speed test in the MPOE/DMARC my results were something like 300md/200mu. I have to test at two locations for this customer so I took the router up to the front of the store at the customer service counter. Around 550md/350mu!!!!

 

[Blues]

I am just now finding this out and I have service that uses Verizon as the true carrier for the servicer provider I pay and have had no disruptions in service but I have manually shut off any data on cellular devices due to suddenly exceeding my monthly limit a few days ago. Yes I don't have unlimited cellular as I have rarely even approached the 10GB limit we have on 3 lines.

 

[HCHTech]

Yes I don't have unlimited cellular as I have rarely even approached the 10GB limit we have on 3 lines.

I had their 15GB plan for years without exceeding the cap (5 phones total). Then we started getting close each month, and finally started going over - I'd get $10 or $20 in overage charges pretty regularly, but the unlimited plans cost more than that, so I just kept paying. Finally, I had enough and called to complain since their usage summaries pointed to a different phone each time as the reason for the overage. They said there are no other plans available besides the unilmited, and gave me a quote $65 LESS than I was already paying not counting the regular overages. I don't know or care why, I guess but now I'm mad that I put up with it for a year before calling. I was telling this story to a customer of mine with 11 phones on their plan and they relayed the very same experience.

 

[britechguy]

@HCHTech

Another classic example of where long term customers are treated worse than newbies or "potential customers."

Sirius XM is another example of this, where if I don't call every year threatening to cancel, they jack up the rates astronomically. I also keep my NY Times subscription cost down by doing the same. It's interesting how it's no problem for them to give me some other deal that keeps costs the same or (on rare occasion) lower.

Companies know that inertia is a powerful force.

 

[Blues]

I am looking at options as unlimited options are coming down plus my current provider is Comcast/Xfinity who uses Verizon for their cellular carrier offerings. I mostly do this as they are also my current ISP I am waiting for a good alternative which my power company is rolling out a fiber option that looks good but my area is the final roll out so I got a while to wait.

 

[ThatPlace928]

I am looking at options as unlimited options are coming down plus my current provider is Comcast/Xfinity who uses Verizon for their cellular carrier offerings. I mostly do this as they are also my current ISP I am waiting for a good alternative which my power company is rolling out a fiber option that looks good but my area is the final roll out so I got a while to wait.

I'm with Visible.com. $25/month for unlimited hot spot, calling, texts, etc. It's the whole package and I've had it for close to 7 years now. It's a subsidiary of Verizon so it uses Verizon towers. Hardly ever any interruptions or down times here. I'm happy with it.