Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Rogers network outage across Canada hits banks, businesses and consumers (reuters.com)
655 points by cupofpython on July 8, 2022 | hide | past | favorite | 387 comments


The utterly inefficient and overpriced Canadian telecoms exist only because we allow one specific behavior:

1. A new local player comes to town, builds their own infrastructure (e.g. connects one building to fiber) and starts offering competitive service.

2. Rogers/Shaw/Bell/Telus immediately offer better terms for the residents of that building.

3. The competitor runs out of money and leaves.

4. The telecoms revert to their usual pricing.

This hurts competition, this hurts customers, this hurts long-term infrastructure resilience, but not a single politician ever comes close to even admitting the problem. I understand you cannot do it on the federal level where both major parties are on the telecoms' payroll, but it could be a very low-hanging fruit for someone running for a municipal position to address. But nope, identity politics, words, feelings and fighting climate change with paper straws seem to be what the electorate wants instead. Sad.


> 2. Rogers/Shaw/Bell/Telus immediately offer better terms for the residents of that building.

I was about to write a whole comment about how I've never seen this myself and to disagree with you I was going to compare the price of Bell Fibre in my building to Beanfield. The latter of which I have at $50/mo at 1Gbps.

But then I loaded up Bell's availability page and entered my address... it was almost like a sick joke. The spinner was saying "Checking availability" or something like that. However, I could see the page with prices behind it darkened out while it was running. It was $115/mo for 2 years and $125/mo after that for 1Gbps fibre. But when the spinner was done and they "checked my availability"... well what do ya know! It's now magically $50/mo. Pretty disgusting that they can get away with that.


Man those prices, Ill use other currencies to be more precise, but in Paris, the 1Gbps was 30 euros and now that I live in Hong Kong it's 250 HKD. Canadian prices could be explained by the increasing cost with lower density, but damn, high speed dedicated fiber is still a luxury in some place, we forget, us dense cities' people.

Competition is not the only thing, in France we have 4 highly regulated fiber providers who must compete or pay fines, but in HK it's way more shady and we choose building and flats to get the best ones (cable is shit, former public telco is expensive, new kid is fast and cheap). It's still nowhere near your prices: a shit cable is 98HKD and an expensive HK Telecom is 500...


The density argument is utter BS. A vast majority of the population is concentrated in 5 very dense cities, and no one is asking the incumbents to provide FTTH to every house in Nunavut.

It’s cartel behavior at its best. The Canadian telecom landscape is a sick joke.


The rural argument also never made sense because why not just start a nationalized ISP or subsidy that purely serves the rural north? Why sarcrifice the competitiveness of whole countries ISP market for a very small part of the country?

The same argument is used for keeping the postal service public when you could just subsidize the parts of the market that wouldn’t be self-sustaining rather than propping up the entire money-losing crown business. Burning tax dollars at the parcel delivery business in the age of e-commerce is a sad joke.

The whole “rural people are being protected” seems like a cover story for powerful friends of politicians to keep the money train rolling, and regular people eat it up on social media.

Plus Starlink et al has rendered the rural argument moot.


Well in downtown Budapest 1Gbps is like 5 euros a month. There's no wonder why so many people become developers in Eastern Europe. Cheap and fast connectivity is key. The sooner western leadership realizes this the better.


>the 1Gbps was 30 euros

That's still pretty good. In Austria that's 80 euros since telecoms here have the same government protected cartel as in Canada. And it's not even available in every building of the city, but you need to check before you move whether you have fiber or not in the building.

Similar in Germany.

One of the things I miss from Eastern Europe is the fast and cheap internet that's available in any building in the city.


You should check what an actual competitive market looks like, Bucharest for instance.


I use 1Gbps symmetric Bell fiber small business package which includes static IP. Pay $129/month and generally happy. Static IP is important as I host some stuff right in my basement. You got me curious with this Beanfield. It is indeed $50 for 1Gbps symmetric (no static IP of course) but when I looked at their business package it is $300 for the same speed. Static IP is extra. I wonder what does such a big difference include.


You can use a Dynamic DNS service and a dynamic (residential) IP, unless downgrading to a dynamic IP incurs severe tradeoffs like CGNAT and/or blocking port 25.


If i had to guess maybe it is a guaranteed 1Gps slot? It does seem like an excessive markup.


"markup" is one way to price things, but not necessarily the best way.

A better way, and likely the thing in play here, is to price according to a mix of value provided, and the ability of the market to pay.

If you are spread enough this will result in some markets paying effectively a large markup, and done a much smaller markup, and some, potentially, less than cost.

The reason the package costs $300 is because that is what the market will bear, and the utility to the customer exceeds the value of that cash.

Incidentally that value may also be in support etc.

In summary, markup is one of the least effective pricing methodologies, and invariably leaves a lot of money on the table.


Is this not just competition working the way it's supposed to?

ie, another player shows up in the market, offers a more attractive price and forces the other players to reduce their prices or shed customers.

It's not like Bell, in this case, were offering 1Gbps for something absurdly below cost, like $1/mo or something just to drive the new comer out of business.

Matching and lowering prices is good for everyone. The new player running out of money and going out of business is really just a lack of foresight and planning on their part. It's pretty absurd to not expect the incumbents to reduce pricing when they are directly completed against.


I think the current answers to your questions are fairly inadequate - so I'm going to give you a more nuanced answer.

> Is this not just competition working the way it's supposed to?

Yes. This is competition working exactly as you would expect it to in a free market: One company is a dominant supplier, and they are able to match rates for a new entrant. They are able to out compete this new company, and they will do so - more marketing, lobbying, cost matching (or undercutting), etc. They have the ability to beat the competition, and they will.

So everything is going exactly as you'd expect - except the points from the parents comment still apply: the consumers in this market are actually getting fucked - the price drop will be temporary, the competitor will be forced out of business and leave, and the total infrastructure investment in the area will go down.

This is what's termed "Market failure". The free market here operates in a way that doesn't increase the well being of all (or even most) participants.

So yes - this is just competition playing out as we'd expect in a free market, but instead of doing what it normally does in an area (force infrastructure updates, service improvements, cost reductions, and higher efficiency as companies compete - all good things) it's doing bad things. Why?

Well - this case is the literal textbook definition of a "Natural monopoly". It turns out that when competition appears, Bell is able to outcompete them not by actually improving, but by leveraging existing infrastructure and scale in a way that the startup company cannot.

The free market isn't making Bell better - they're not having to work any harder or improve. That's great for Bell, but pretty bad for everybody else.

So (at least in theory) we regulate this case of the free market, because we've seen that "normal competition" doesn't actually work here.


> forces the other players to reduce their prices or shed customers.

Excpet they revert back to old prices once the competition is dead. This is textbook predatory pricing.

> It's not like Bell, in this case, were offering 1Gbps for something absurdly below cost, like $1/mo or something just to drive the new comer out of business.

What's the significance of "absurdly below cost" here? They are doing exactly what you describe except the exact number here is not 1 but 50.


No, its just basic supply and demand...


When you price good below your production costs and have most of the market, then that is considered anticompetitive, monopolist behavior and is illegal in many places.

This is because if a large enough company does this, they can lower their prices locally to below cost whenever a new company enters a local market and subsidize this with their other markets. This creates a stranglehold on the market that can allow a company to charge artificially high prices for sub-par services.

This isn't just theory there is a well established pattern here and that is why laws prohibit it in many places.


> Is this not just competition working the way it's supposed to? > > ie, another player shows up in the market, offers a more attractive price and forces the other players to reduce their prices or shed customers.

No, because any time a company shows up that could pose an actual challenge to their stranglehold, all 3 companies gang up and lobby against them.

E.g.

https://globalnews.ca/news/753408/new-ad-against-verizon-exp...

https://www.iphoneincanada.ca/carriers/rogers-telus-bell-job...


> t's pretty absurd to not expect the incumbents to reduce pricing when they are directly completed against.

But they should be forced to reduce it everywhere where they're offering similar service, not just a limited local area.


> The utterly inefficient and overpriced Canadian telecoms exist only because we allow one specific behavior:

> 1. A new local player comes to town, builds their own infrastructure (e.g. connects one building to fiber) and starts offering competitive service.

You forgot to mention that whoever is the ILEC in an area (Telus, Bell, etc) and/or the incumbent 35+ year early advantage local cable TV operator (Shaw, Rogers) has an immense advantage in owning and controlling existing right-of-way to reach whatever is the last mile service delivery location, whether it's aerial pole to pole or duct routes.


They legally have the right of way to build infrastructure to connect to newly developed, last mile locations? Interesting.


whether in usa or canada, local governments treat the incumbent local phone company and cable tv company as a default essential utility, so if it's not a legally enshrined right, it would certainly be shocking and weird if they didn't build to new places

of course in places where the local phone company or cable tv company shares aerial utility pole based infrastructure with the local power company, they're very close buddies as well


In most of the USA the local telcos are obligated to extend service to new construction as part of government grants they've received in the past. The obligation is usually for something like ~10 years, but the grant cycle is shorter than that.

The recent "RDOF" mega-grant has one of these clauses (at the census block level):

"All support recipients must serve locations newly built after the revised location total but before the end of year eight upon reasonable request"

https://www.fcc.gov/auction/904/factsheet

These mega-grant things happen every 6-8 years. The last was "CAP II", preceded by "CAP I", etc. The telcos are basically treated like municipalities when it comes to federal grantmaking. This is basically the system that was lobbied into place after AT&T was broken up; the FCC just took over AT&T's local-loop capital allocation.


These observations are very true. A friend was living in a Toronto condo building, and I saw them choose to switch from Beanfield to Bell/Rogers for the same monthly price but higher Internet speeds. They took the bait and now everyone pays the price in the long run. And yes, Bell/Rogers gouges with high prices for houses/apartments that don't have an independent ISP like Beanfield, etc.


I recently moved to a newly built condo in Toronto and there was a special resident onboarding email, which happened to include a promotional offer from Rogers, advertising a rock bottom deal on internet that I couldn't refuse. I later asked the Rogers representative on the phone how they managed to score the deal with the building developer, and he basically told me that they have an ongoing relationship. They basically have the deal made as soon as the project gets off the ground.


That's a shame. I recently moved to Toronto and got Beanfield mostly by chance. I'm really pleased with the service, hope to stick with them for a long time.


After seeing how the telecom infrastructure was targeted in Ukraine, No sane government in the world can afford to maintain an oligopoly telecom infrastructure anymore; Ironically often in the name of national security.

I think the best way to address it is to open the industry for disruption by encouraging new players(without Billion dollars in market cap) by removing barriers for entry, With stringent monitoring/punishments for anti-competitive behavior including backroom deals among carriers and mobile ecosystem duopoly.


Last mile connectivity is a bit of a natural monopoly, in Australia they converted that to a government owned monopoly that leases their infrastructure to ISPs. I wonder if Canada should do the same.


Does Canada not have strong antitrust laws? As far as I know, this is explicitly illegal in the USA.


On paper, yes. But over the last few decades our regulatory bodies have been subverted by influence from our telcoms.

Recently our telecom regulator (the CRTC) was considering whether MVNOs should be allowed in Canada. Our Competition Beaureau, which is supposed to be enforcing antitrust laws, provided guidance to the CRTC[1]. Here's an excerpt:

"MVNOs can drive lower prices and greater choice, but they also could threaten the demonstrated progress in enhancing competition in this industry to date."

That's right. "Increased competition and lower prices is bad for competition." "Demonstrated progress" in a country with some of the highest telecom prices in the world.

Let's not even get into how the chairperson of the CRTC is literally a former telecom lobbyist.

Ultimately, MVNOs were "approved", but with practically insurmountable requirements for new entrants, effectively blocking them.

[1] https://www.competitionbureau.gc.ca/eic/site/cb-bc.nsf/eng/0...



There really should be a big investment in an anti-regulatory capture organization the way there are anti-trust lawyers and privacy czars.

Which would have the dual purpose of employing researchers and public outreach and have powers to stop hires, limit grants/subsidies, and collaborates with with anti-trust.

But who are we kidding, politicians never bite the hands that feed them.


I'm interested in any and all ideas to mitigate regulatory capture, no matter how crazy.

My only notions so far:

- Add explicit carve outs for customers (of the regulated entity). Where the representatives are chosen by jury duty or sortition or other suitably random method, so they too are resistant to capture.

- The customer's have veto power. Maybe thru an ombudsman structure. Maybe thru majority of the regulatory body.

- Like citizen's assemblies, customers have investigatory powers. They can ask questions, they cannot be denied. They benefit from expert testimony.

- No artificial deadlines for deliberations. Some decisions just take longer.

- All decisions (laws, rules, procedures) have built-in expiration (TTL), and must therefore be regularly reauthorized.

- All artifacts and hearings made public by default.

- Pay board members and their staff. So that normal people can afford to serve.

> a big investment in an anti-regulatory capture organization

I think you're on to something.

I keep thinking of the law practice that makes terrific money suing pharmacy benefit managers (or maybe just the pharmas themselves) for violating pricing rules. Sorry, no cite.


> this is explicitly illegal in the USA

Oh my sweet summer child.

The Canadians learned this scam from us. Centurylink has been doing this for decades. They ran several WISPs out of business in Bellingham, WA.


regardless of the law, canada chooses not to enforce it when it comes to the big telcos. they do what they want, and the politicians obey.


Canada is rife with oligopolies and the government does nothing to make this right. Everything is much more expensive in Canada vs the US because there's simply no competition.


Isn't this a simplified version of what Rockefeller did with the oil market more than a hundred years ago? Drive out competition by financing any extra costs with profits from the markets where there's no competition.

Such behavior spawned lots of laws designed to prevent it from happening again. And, just like the telecoms extort huge subscriptions from their clients using extreme product bundling, politicians engaged in ideology and law bundling to get voters to flock to them attracted by some laws, enabling them to enact (or in this case, repel) the laws preventing them from going back to 19th century capitalist practices.


Sounds like how the free market was designed to work.

Not that I find it a particularly good design, though.


Utilities like telecoms can't be free market due to the natural monopoly effect.

We can't have a hundred different companies burying infrastructure or stringing up lines on poles.


Many places have solved this long ago, both for electricity and for telecom: the network is a separate, neutral entity, co-owned by all the players.

One cable/fiber, rented by all providers. The network only has one goal: distribution. The providers are the ones competing.


Is that a free market solution or has that been mandated by government regulations?


It’s good for the consumers, so government mandated. Free market yields the US or Canadian telco landscape with bought out regulators, no real competition, price gouging, and poor service.



You can, but profitability is limited. You can have 5/6 maybe max in a dense country. With 4 in France, they each slowly fibered the whole country reaching an approximate 25% coverage each and compete in wave: one day they are at 20MBps and suddenly one explodes at 100 and the rest run after to restabilize market share. If they dont they all get fined by the regulator looking at european average speeds.


Yes and: Profitability is predetermined. Converting public goods and services into securities. Ideally. So a public utility issues bonds, instead of stocks.


Not sure what you mean, bonds and stocks are different liabilities and a state utility by definition cannot issue equity or the state would lose ownership. It means they must pay their bond coupons and therefore are very risk adverse and change resistant.

A security-emitting entity doesnt need to pay back the debt too much (it should but doesnt have to, via dividend) and therefore can afford to try things to raise returns for both shareholders and management and absorb failure.

The worst situation is when a state utility tries to innovate and fails: this leads to privatization which actually transforms it in a public companies (the words are weird: state companies are not directly publicly owned by the citizens, private companies on the public market can be). If a public market private company fails to innovate and just produces riskless cashflow forever, it becomes a good candidate for state private ownership.


Municipal bonds.


An extra 2 or 3 might be ok.


Seriously, the world will not end because more than two companies are laying fibre. Especially as the country grows in size.

This weak apologia for monopolies is what keeps them around. Even when the cost created via gov-backed pseudo-private monopolies are obviously worse than their hypotheticals.


> free market

> was designed


Freemium Markets™

aka Pay to play.


It definitely won't happen in Toronto, seeing as John Tory was, and still is, in bed with Rogers.

https://www.theglobeandmail.com/news/national/john-a-tory-a-...


Curious though what the government could do about this pricing scheme here? Seems like the solution is to setup a bureaucracy to watch pricing and then micro-manage the situation whenever its tried?

The ride-hailing services pulled this in various places around the world. Airlines in Canada did it as well. Feels like whack-a-mole.


The most straightforward solution would be to require the telecoms submit pricing tables to the regulator and force them to stick to them. If they want to make a change, they have to do it for the whole region, not just the area where small-time competitors have installed new service.

Price transparency in general is a huge issue with Canadian telecoms, it's well known that it's a "complaint based system" with the big three. They'll hook you with an introductory rate and then start jacking up the prices after a year or two. The usual dance is you go to one of the other big telecoms and get a quote for their introductory rate, and then go back to yours and threaten to leave unless it's matched. It's a pretty awful system that requires people to know how to navigate it. Most don't, so after their intro period expires they just get gouged. My father in law is paying almost $200 a month for internet and TV.


Promote fair competition.

Verizon tried to get in Canadian market in 2019ish but top three telecom formed a mafia and didn't let that happen. Same thing with Delta, would be a much better alternative to Air Canada.


Verizon was already in the Canadian market once before, holding a sizeable stake in Telus. I would say that they, along with AT&T owning a stake in Rogers during the same period, and of course AT&T also establishing Bell Canada earlier, are responsible for the present state of telecom in Canada. There were a number of other players who were trying to compete, but is wasn't easy going up against the offspring of the American behemoths.


Air Canada isn't perfect but Delta is junk. I have flown on both many times.

Many counties have their own airline. Canada shouldn't automatically give up our airline in favor of a US one, because the US one wants in.


the government is the reason there is such a triopoly - Bell/Telus/Rogers is a triopoly set up and supported by the government itself. The CRTC last mile service rule (forcing the triopoly to let other vendors use their physical cabling) is the only thing holding back a complete closed market.


In Canada, the entire public apparatus (federal and provincial at least) is geared toward keeping big businesses alive and happy, no matter the costs, because that means keeping jobs, because people with jobs paying taxes is the only meaningful source of revenue for governments since corporations can evade most taxation.

So, the various Canadian governments will do whatever they can to maintain corporate status quo and keep the money flowing. That's how it works. Just please big players and the rest follows.

Then, don't expect too much. Canada has never been a real country, its people never decided to take over and govern themselves, it's just an ex-British colonial body from a bygone era (one of many), and has always been acting as such.

Because of that, it's also one of the few places in the world where true personal and community freedom can be achieved, probably because it's so harsh and so vast and so highly dysfunctional and so challenging and so unimportant. Whatever it is, it's nice to be a Canadian for the most part. I'm not proud of being one, but I don't have to be, and that works pretty well for me.


One thing some other countries have done is break up the old monopolies into a last mile company that owns all the copper/fiber vs a services company.


Acknowledge that internet / phone service works much better as a utility and nationalize or highly regulate (i.e. fixed pricing schemes) the telecom companies.


The CRTC does dictate a fixed pricing scheme for mobile phones:

https://crtc.gc.ca/eng/phone/mobile/occa.htm

Near the bottom it states that those companies are supposed to promote those plans. They didn't the last time I tried looking them up. Regulation is meaningless if there is a lack of enforcement.

(Note: I'm not saying that the pricing scheme is particularly good since it leaves the needs of many people unaddressed. On the other hand, it does address the needs of those who would have the most trouble affording phone service.)

Also consider the continual battle of third-party ISPs to provide affordable Internet access. The CRTC says the major providers have to lease out their lines and stipulate what those rates are. On the other hand, those third-party ISPs are constantly fighting to keep the rates low and are pretty much tied to providing service levels that match the major providers.


I thought I'd dive in and look for these plans, and as expected they're not at all easy to find.

For Telus, I had to go to four different plan/pricing pages before I found a link mentioning that cheaper plans are only available on Koodo (their lower priced brand). And then, the $35 plan is as lousy as they can make it and still comply with the CRTC - 3GB of data at 3G (only 2 generations ago, awesome) speeds, with overage costs of $13/100MB. Just absurdly bad. Or maybe you want something more reasonable, let's cut out that data access, you just want talk and text? That'll save you a whopping $3, at $32/month.

So that's the telecom situation in Canada; the first 3GB cost you $3, the next 3GB cost you $390.


Maybe re-examine the foreign ownership rules for telecommunications companies...

"As it stands now, foreign ownership of a telecommunications company is limited to no more than 20 per cent of a company’s voting shares and no more than 33.3 per cent of the voting shares of a carrier’s holding company, and an effective total limit of 46.7 per cent (as long as the foreign entity doesn’t have control). On top of that, at least 80 per cent of the board members must be Canadian citizens." [0]

(Although seems not to apply to small actors under 10 percent of the market share anymore)

[0] https://financialpost.com/telecom/tight-reins-leaves-our-tel...


The idea around the foreign ownership laws is to prevent jobs and knowledge from leaving the country hopefully creating entities that can compete at the global level. It has worked with banking but failed with internet providers. It costs regular citizens but not strategically doing this is costly.

The industry is large/stable enough to allow 100% foreign ownership.


Its already illegal if its foreign physical goods that are being imported (https://www.cbsa-asfc.gc.ca/sima-lmsi/brochure-eng.html ). I guess they could extend that.


I also remember Air Canada or WestJet doing the same to some small air company offering cheap flights from Kelowna (?), but I cannot find the news piece. Does anybody here have a better memory than myself?


The airport in Kitchener has a deal where an airline is offered exclusive rights to a new route out of the airport for, I think, two years. Westjet/Swoop was recently throwing up a storm in the media over this when Flair took advantage of the deal, calling it anti-competitive. Never mind that they had, and still have, access to the same deal.

The reality is that they were only interested in running those routes as long as another airline was. YKF has been down this road before.


Are you thinking of Flair or Swoop? Never flown them out of Kelowna when I was living there, but I remember both being a big deal when they finally arrived.


Swoop is owned by WestJet, it's just their discount brand. Flair's actually independent and used to be based in Kelowna but is now based in Edmonton – they're probably who GP is thinking of.


I'm incredibly thankful to Flair as my flight to Montreal now costs 1/3rd what it cost last year.

It's unfortunate that I need to visit a province that discriminates against my mother with its regressive laws, but at least I can pay my respects and get out without contributing to the oligopoly that runs the country.


> a province that discriminates against my mother with its regressive laws

Can you elaborate? I only very vaguely follow Quebec news so I'm curious what you're referring to.


She was a teacher and she wore a hijab (thankfully she had retired long before they decided that was a problem)


They're still around


While I believe that this is a tactic that gets used, it shouldn't work. And therefore it is points to probable regulatory failures. Note that the theory here is that the majors will offer cheap prices if the infra is duplicated.

That means "I" (my building) could build my (our) own fibre connection and then use that as leverage to get cheap internet without having to actually run it. It'd be a bit silly, but sounds long-term profitable.

The focus should be on enabling this sort of competition.


I was really hoping Musk would bring Starlink for cheaper then Rogers et all. I think for now they will mainly focus on rural areas but as time goes on hoping they lower the prices enough that even city folks with clear line of site to satellites can hook up and ditch Rogers.


Canada is rife with monopolies because of this. It's a very unhealthy market.


Why do new local players invest in new offerings if it is apparently so obvious that the big telecoms will come in and undercut them long enough to exhaust their capital?


Why does anyone try to sell books when Amazon exists? It's the nature of the optimist.


But people actually do sell books. It’s not the case that Amazon always comes to new book store locations and undercuts them on price just long enough for them to go out of business.


Amazon undercuts pretty much every bookseller on price. A few local/specialty shops can survive but Barnes & Noble is pretty much the only national competitor left in the US and they are on the ropes. My local B&N is now a pile of rubble, soon to be a fast food place.


What exactly does prevent almost any company in the free market to protect their market share with this way (if they can)?


Deeper pockets because of income from different markets.


why not fix the bigger problem instead. that politician can be « on telecom payroll ».

we can all agree politicians are supposed to represent the population of the country not the company and their board of directors.


People in other countries probably don't understand just how powerful - and depended upon - the big telecoms are here in Canada.

They have no presence outside the country - they are so bloated and inefficient they couldn't compete - but they are omnipresent here.

There are lots of people who depend on Rogers for home Internet, home phone, mobile phone, TV, and home security, not to mention business services that consumers also depend on, like the payment networks that are down.

Plus their media properties, which probably come in useful when the government starts to think about allowing more competition or decreasing the taxpayer money being funneled to them.


I work in telecom and tell this to nearly everyone who asks me about Canada, and its ISP/telecom market and competition...

"Canada is an endangered species protection wildlife reserve park for dinosaur telecoms"

It's a travesty that the government is actually going to let Shaw and Rogers merge to even further concentrate power in the hands of a few families and reduce market competition.


In the article, it mentions that the Canadian government blocked this merger. I believe it said that was in April though. Has that changed, or did I misread what the reporting said? Just want to clarify in case I missed something.


The Canadian government have put up a mild roadblock that will be addressed with non-substantive changes to the structure of the merger, enforced by a little known, toothless regulatory agency who will be instructed to look the other way.

The regulatory environment Canadian telcos operate in is completely captured.


Ah okay, this helps a bit. Not in Canada so I wasn't aware. Thanks for clarifying.


I would bet $100 they'll ultimately let it go through anyways due to the absurdly outsized political influence and lobbying of the equity shareholders of Shaw. One of the wealthiest families in the country.


Practical experience working with Telus has been astounding. I have never met a set of system admins so incredibly ignorant of and incapable of using any of the systems they ostensibly run.


I think people in the USA understand bloated telecoms with monopoly power that means they don't need to compete.

In my neighborhood, I have one effective ISP, that is the cable company (who offers phone+internet+cable), the other is an old-school phone company who offers DSL (with bonded DSL, up to 12mbit). I thought that I could get a slow DSL connection to back up my cable, but no, there's some neighborhood line concentrator that means I can't get DSL at any speed.

So I really have only one ISP to "choose" from. (well, in theory I could get Starlink, but my area is still waitlisted, as is Verizon 5G home internet)


I have a lot of experience in both places. Americans may think think they do, but I don’t think so. As bad as Comcast and others are in the USA, I have found them to be far more efficient and competitive than Rogers or Bell in Canada.

Same for banks in Canada. It’s unbelievable what they do and get away with.

And airlines. Rather, Airline.

I believe there’s a cultural element here: Canadians are more risk adverse and place a higher value on “established” companies and brands. This influences consumer behavior and legislation (Canadians don’t seem to care about laws that make it hard or impossible for market entrants).


If you asked 100 Canadians if they’d prefer more competition to the telcos, 97 would say yes (1 works for Bell, 1 works for Rogers, 1 now works for the CRTC). We’re not risk adverse. It’s just that our government / regulator is captured.


I wonder how many of them would actually buy them though. There's been attempts at mobile providers, sometimes offering prices that the big telcos won't try to match even to flood them out, and they tend to wither away.


Very few of them have offered decent service at good prices. Usually outside of ~5 major metros you have effectively no coverage and speeds are often subpar compared to the incumbents.


Mobile providers have to make huge investments in infrastructure to provide reasonable coverage in Canada. It happens on a small scale but it's like an entirely different market from the main players.

Now I have little doubt that if foreign providers were allowed in Canada that Verizon or AT&T would be able to setup reasonable competition in short order.


This happened: Wind Mobile. They couldn't immediately build infrastructure across the entire country, so IIRC you were "roaming" outside a few major urban centers. However, I believe they also had unlimited US roaming (much better than the big three).

I expected more uptake since it would have been a better deal for most Canadians in those urban centers, but I think there's a subtle bias against taking these kinds of risks.


The Egyptian billionaire CEO had some choice words about the Canadian business climate as he was in the process of divesting of wind mobile: https://www.cbc.ca/amp/1.1013522


They never really expanded close to enough. People didn't want to be roaming when they went to the cottage or camping or whatever and even then the urban areas served were still too concentrated and small.

I think had they focused on southern Ontario or BC and expanded from there rather than aiming to hit the all the big metros first they might have done better.


Decisions are made at the margins. It’s always possible to look back and say “they should have done X, and they would have succeed.” It possible that there were many factors, but in my mind no minor change to the strategy would have helped. Had they done what you suggest, I’m sure the respond would have been that they targeted too small a market, or they didn’t cater to folks traveling between major cities frequently.

IIRC Wind was offering unlimited plans at 50% the price of a mediocre plan on major providers. You could have hours of roaming calls and still come out ahead (and this is in the edge case, where you happen to go camping that month). Most things being equal, Canadians broadly prefer to pay more to have a sure thing. It’s not a bad instinct, but it has consequences to how industry is shaped in the long term.


Most of the smaller providers have either been acquired (e.g. Public Mobile) or downright always been run by the larger telcos (e.g. Virgin Mobile). For both mobile and ISPs, very few companies actually own the infrastructure, most are just leasing bandwidth.


And since you can't count on them to stick around, it's not worth it for many of us to go through the hassle of switching only to have to switch again a year or three later when the company goes under. Plus the risk of poor service compared to the known mediocre service you get from the major telcos.


The poor service often has to do with the fact they don't own the infrastructure, and rely on the parent's technicians. I'd be genuinely surprised if those service calls were put on the same level of priority as their own customers.


New Zealand often seems like a toy country with one of everything. There's the One Bank, the One Telecom, the One Airline.

Not ... exactly. But effectively, yes.

A combination of a tiny population (5.1 millions) and a long way from nowhere (roughly 3,200 km / 2,000 mi from Sydney).

Australia and Canada already have issues from their own small scale (and vast land areas). NZ takes it up a notch.


Mobile data in NZ is pretty good nowadays, for NZD $60/mo we had unlimited/unthrottled 4G broadband good enough to play FPS games on. In Canada the best we can get is CAD $169/mo for average to middling speeds with pings over 180.


At least of late NZ seem to have pretty good governance. That may help ameliorate the more agregious harms.


Interesting you'd say that in a thread about ISPs, considering we have regulation-enforced separation between the owners of our government-funded nation-wide FTTH network and the actual service providers. This means I have over a dozen ISPs to choose from, and they all over gigabit fibre and compete on price, customer service, and things like IPv6 or CG-NAT.


On the other hand it made it quite an enviable place for much of a global pandemic.


> And airlines. Rather, Airline.

I would dispute this one, only because in addition to Air Canada (fleet size 312), Westjet does exist (fleet size 162). And the other regional airlines offer (-ed, prepandemic) some price/service competition. Comparisons to the US aren't as useful with airlines as with some other industries due to population differences.


Yeah, I thought about Westjet and decided that I wouldn’t quite call them a national competitor. I hope they continue to grow, both because I like Westjet and competition helps everyone. There’s a reason Air Canada’s unofficial slogan “we’re not happy until you’re not happy” is widely known.


I mean WestJet hits destinations in every province and two of the three territories (though Whitehorse is apparently going to be cut come September). I think that's a pretty solid national competitor. What reason is there to exclude them?


Porter is really the best way to get in and out of Toronto...


While I used to love Porter both for location and better service at comparable price, it's not much good if you're flying to Vancouver or Toronto, and the one time I took it to St John's was miserable

Edit: and the union Pearson Express made the main airport a lot more accessible


> Same for banks in Canada. It’s unbelievable what they do and get away with.

Canadian banks are so bad that they actually make the service and fees with a USD checking account with all the normal features at Wells Fargo look good.


I'm a US citizen, Canadian permanent resident.

I think it has to do with the fact that historically much of Canada's lands/provinces were literally just land owned by very massive British crown corporations. Massive corporations are baked into the history of Canada as a political entity.


No. "Crown Land" in Canada is owned by the provinces.

There is no such thing as a "British Crown Corporation."

Crown Corporations are a Canadian thing.

I lived in Canada for 30 years and now live in the Scotland. CalMac and Scottish Water are not "Crown Corporations," but are nevertheless owned and controlled by the Scottish Ministers. This is contrast to BC Ferries, ICBC, and BC Hydro which are Crown Corporations.

In the UK there is something called the Crown Estate, which is somethint again different.


>> No. "Crown Land" in Canada is owned by the provinces.

Crown land is owned by the Canadian Crown; the monarchy owns all crown land officially. It's administered by a split across federal and provincial jurisdictions.


>> Crown land is owned by the Canadian Crown; the monarchy owns all crown land officially. It's administered by a split across federal and provincial jurisdictions.

No, Crown Land is controlled by the provinces. No, the Canadian monarchy does not own crown land. The Crown owns crown land. Essentially The Crown in Canada is its own legal entity and instrument for the purposes of administration of public lands. The Crown Estates of England and Scotland are something different again, and the Crown Estate owns things like tidal lands and actual real property.

There is Federal Crown Land, but that is almost completely in the northern territories. This is down to Canada's constitutional makeup because provinces control resources.

The Federal government can expropriate land from the provinces when it has a reason to do so like for military bases.

Here is a link that shows Canada Lands. The white area are Provincial Lands.

https://clss.nrcan-rncan.gc.ca/data-donnees/sgb-maps-dag-car...


Yeah but there is a difference between crown land owned by the crown in right of a province vs in right of the federal gov.

(From what i understand. IANAL)


I suspect that was a reference to the Hudson's Bay Company. I would have to brush up on my history, but they effectively controlled a huge tract of land pre-Confederation and continued to serve many smaller towns until well into the 20th century. I don't think the Hudson's Bay Company was considered a crown corporation, but they were granted a royal charter.


Yes, HBC has a Royal Charter from King Charles II which granted HBC exclusive economic activity in the lands that drained into Hudson Bay.

HBC lands were surrendered to the British government in 1868 ahead of confederation.


HBC (which is now a department store) was still required to give the crown 2 elks and 2 beavers whenever they visited Canada up until 1970.


I think they're referring to the fact that the Hudson's Bay Company used to be the legal owner of vast swaths of what is now Canada before selling/surrendering the land to the Canadian government.


In Canadila about half of the crown land is federal and the other half provincial. In 2013 11% was private, 41% federal crown and 46 provincial crown


Federal Crown Land is almost exclusively in the northern territories.

Resource administration is a constitutional power that provinces administrator.

The area in white in the below map are not Canada Lands which makes them Provincial Lands, ie Provincial Crown Lands.

https://clss.nrcan-rncan.gc.ca/data-donnees/sgb-maps-dag-car...


I expect gp was referring to The Hudson's Bay Company / Rupert's land


You're at a National Park. In Canada, it's crown land. In America, it's your park. The Canadian mentality is different.


National Parks are not Crown Land. National Parks are reserves of land owned and managed by the Federal Parks Department and development is not allowed on that land whatsoever.

Crown Land is land owned and administered by the provinces. Crown Land can be licenced for many uses and sometimes it can be purchased.

Also, Provincial Parks are not Crown Land.


We also have national and provincial parks, Crown land is different. A national park is maintained for visitors and for nature, crown land is closer to unused or unclaimed land, generally nothing is done with it unless it's sold off. You are free to camp on crown land, but it's nothing like going to a park, it's full on wilderness camping with no amenities, sites, or anything else.


My point is, Canadians think of government land philosophically as something that is not theirs, but rather, owned by the crown -- the Queen. While Americans see public land as their shared land. This mentality penetrates into many of the differences between the two countries in how they govern.


"Crown" (or "Royal", "Regina", etc.) is used pretty extensively in the Canadian governmental or legal system, and any association in Canadian's minds with the Monarch are pretty much nil.

If you talked about "crown corporations", or "Regina vs." (for criminal cases) no one would associate that with Elizabeth II. "Crown" and anything like that basically just means government.


No, Canadians do not see Crown Land as owned by the Canadian Monarchy, they see Crown Land and owned and controlled by the government as a public asset.

Canadians see the Canadian Royal Family as nothing more than a figurehead of the state.

I am a Canadian and I have studied British and North American History extensively.


I somewhat agree with your main point, but I don't think most Canadians associate the "Crown" as in Crown Land or Crown Corporations with the literal monarchy. Rather it's just a synonym for the (Canadian Federal) government. Most Canadians don't tend to think about the monarchy much at all. (Certainly the Canadian Governor General, theoretically the Queen's representative as head of state, would never be expected to actually take instruction from the Queen.)


At least the Canadian banks and airlines have international operations.

TD has more US than Canadian branches.

But Rogers, Bell, Telus? Nobody has heard of them ex-Canada unless they’re an ex-resident with bad memories.


TD Bank (and Scotiabank in Latin America) operate as independent entities outside of Canada -- those American TD Bank branches are branches for a different bank (TD Bank USA). In fact, I believe that TD Bank (Canada) simply acquired a US bank rebranded it. I believe they have basically nothing in common operationally.


Half right. Yes, TD has a lot of divisions, each separate: https://www.td.com/about-tdbfg/corporate-information/corpora...

But it’s also true that over the years, TD has had to merge systems and make changes. They’ve been operating in the US for about two decades now. So they’ve had time to merge and continue growing by merging companies in the US.

It’s true though. They aren’t entirely integrated across borders the way other companies might be. The plans and cards they offer in the US are a much better value than those in Canada, too.


They are becoming more integrated than they used to be. I just opened a TD Bank account linked to my TD Canada Trust account that I can (theoretically, still getting it set up) manage through my Canadian login.

At least a few years ago that wasn't possible.


Oddly enough, Telus is a medium-sized but well regarded player in the fiercely competitive market for outsourced call centres in the Philippines.


It's kind of funny that Telus, which at it's core is a merger of British Columbia Telephone Company and Alberta Government Telephones (tough to get more entrenched and bureaucratic than that) is actually a pretty innovative and diversified company... outside of their core telecom business where they're just as bad as the others.


Or a hockey fan. It seems like every Canadian NHL team has one of those companies (or ScotiaBank) as their main sponsor.


In Jersey City I can get xfinity (cable) Verizon 5g, or Fios 1gbit unmetered for $65 a month

It's probably the density and income levels that make it worth putting out the infrastructure


Not really. Most Canadians live near the US border and concentrated in high density. There's no correlation to what you are proposing. Income levels are good. Looking at developing countries, they have cheaper and better plans than Canada.



Having lived in Canada and the US, with telecom plans with providers in both countries, I think Canada’s telecoms take it to a new level. To give you a sense of how distorted the market is, it was cheaper for me to keep my US Verizon wireless plan than to use Rogers, Fido, etc. when I lived in Canada!


I do this.

Unlimited (international!) roaming and everything isn't that expensive (Only about 100$) compared to whatever insane amount I'd be paying if I had some ripoff, under 5 gigabyte or whatever pittance data capped Canadian phone plan for no reason. I also maintain a Central American phone plan that doesn't cost me very much either (2$ every 3 months just to keep the number outside of the country and receive SMS/use chat apps basically, but I don't do roaming when out of that region - When in the region, I spend maybe 20$ a month).

Since the Rogers website is down (Great sales strategy!) I can refer anyone curious to a portion of their menu from some image I found; [1] 95$CAD (500MB until you hit overage) + 60$ for 'roam like home' (no mention of data, canadians still use SMS way too widely), and then probably more to up that data cap with no guarantee that applies to roaming? Yeah, no thanks! Oh yeah, the cops just install malware on your shit with reckless abandon, too. [2] I see no reason to consider Canadian nationality as anything but a nice passport at this point.

[1] https://cdn.mobilesyrup.com/wp-content/uploads/2018/05/roger...

[2] https://www.theguardian.com/world/2022/jul/07/canada-police-...


I've heard they stop your service if your roaming is half of your data plan.


Wow, that's incredibly rude. I've never experienced something like that outside of Canada before.

Like what are you even paying roaming for if they pull stuff like that? Is there any acknowledgement that some of us are actually abroad most of the time?


Telecoms routinely sell people services based on the assumption people won't use them. This gigabit link? The user isn't ever going to saturate it, right? They are pretty much immediately proven wrong and get angry because their infrastructure is over capacity.

That they can get away with this false advertising astounds me to this day.


I've heard something similar but that they'll stop service if you're out of the US for over 6 months.


My Rogers plan is $90 for 65 GB of data, it is unnecessary (average use for me is 4 GB/month), but better than going over or throttled. When I was supporting an MPLS Rogers fibre connection, we had zero downtime in 2.5 years.


I bet you could twist their arm into a better deal if you spend a while complaining on the phone about today.

I do a lot of my work off of a mobile hotspot so I'm not tied to an office or home. I try to make the most of what's paid for and it goes very far.

I think that if you're getting a decent price it's probably just customer retention at that price point. If they're ripping anyone off widely at the prices I saw, it's new customers... Immigrants etc, pretty nasty.


I (america) also have only one internet provider to choose from (Spectrum TV, formerly Time Warner Cable). It's really annoying as just barely a mile away, my neighbor has a choice to use Verizon DSL (and does). -.-

I dropped TV and a landline a long time ago, and use cell (google-fi/t-mobile).

The annoying thing is all the spam mail I get from spectrum to bundle all the above for much lower than I'm paying now for an introductory rate. I don't want to play that game but it kills me what I could be saving for at least 6 months to a year.


Spectrum is so incompetent that even if you were a Spectrum subscriber, they'd still spam you with those offers.


Most people also don't realize telcos rake in more revenue than the entire tech sector. And that it is no where close to enough to maintain or upgrade the pipes. Cloud has been built on lot of assumptions that are close to breaking point.


The entire tech sector would not exist without the telcos. So there's that


Deploying infrastructure is not easy or cheap. But is clearly profitable if you can find investors to finance the capital spend.

Google have tried to enter the FTTH market thinking there must be opportunities to innovate and reduce the cost to consumers. But even they have struggled due to the complexities of deploying fiber in the existing built environment...


Google was trying to get started with the flushable fiber, right?


Isn't that true in most countries?

Is there any place in the world that can claim to have a diverse, competitive Telco market?


The model of separating the fiber network from the telcos that offer service on it can work of the incentives are well aligned. Many Asian countries, eg Singapore, do this.

Or you could put the fox in charge of the hen house and end up in Australia's situation, where local monopolist Telstra owns the phone network and is supposed to play nicely with its competitors.


for last mile ISP options, there are some very fortunate US counties in WA state that have

a) last mile dark fiber network operated by local public utility district which is also the electrical grid operator, and rents access to the fiber to 3rd party ISPs

b) local cable tv operator, legacy coax operator, often docsis3 cablemodem

c) local ILEC/POTS phone company that may or may not have overbuilt its last mile copper/DSL service with its own singlemode fiber and 1Gbps GPON service.

for primarily mobile phone carriers like rogers, there's an effective RF planning limit of around four major LTE/3GPP technology based operators in any given geographical area.

USA used to have 4 with sprint until the tmobile/sprint acquisition.


no but i think theres similar zombie giants that is essentially a hybrid state run enterprise but Canada has explicitly designated industries where you simply cannot execute because they are anti-competitive.

and its not clear how individuals are being handed out rights to say run a casino in canada.

if you took a look at canada's corruption, you will be shocked. we don't quite live in a first world even though we like to tell ourselves all of the government officials/servants are honest.

after all the canadian embassy staff in Hong Kong happily handed out PR residency to hardcore organized criminal groups in exchange for various luxuries and perks. when a staff tried to expose it, he was quickly removed and media began to attack him.

its not only that but you see NGO's championing for racist white supremacy linked groups who vandalize Chinatown in Vancouver and with a large chunk of locals who feel that they are "being overrun by a certain ethnic group" eat that up and the same populist individuals get elected again and again. then my tax goes towards those interest while none for me because i'm 'privileged'

in the long run, I see this system breaking down, if not already. Canada no longer feels like a country but some feudalistic interest group driven, poorly run corportation.

so glad i dont have to pay taxes here. the savings and currency difference allowed me to create jobs in another part of the world. there was a time where I hoped things would get better and I could be creating jobs locally.

i just regret wasting my youth in canada and west coast. canada is a bubble and lot of us are moving capital/jobs out of it.

why contribute to a country that just sees you as an ATM to transfer payments to others who blame everyone but themselves and constantly wanting hand outs?

im done with canada and im warning anybody who still attach romantic outlooks, especially in heavily marketed cities like vancouver.


Singapore does a decent job. I was paying $10/month for a 10GB mobile plan.

There are 5 different mobile providers in Singapore: Singtel, Starhub, M1, MyRepublic, ViewQwest and WhizComms.

Many of them also do fiber broadband. Some government body owns the infrastructure and the providers provide the connection to the internet. There are 5 providers as well.

I was paying $50/month for 1GB fiber and I paid more since I did a shorter term plan. If you buy in for 2 years, it's can be $35/month.

And this is in a country of 5.5M people.


Mountain West state, middle of nowhere town. I have the option of choosing between a 1Gb+ fiber provider, a 1Gb fiber provider, a 100Mb fiber provider, multiple wireless line of site providers in the 30Mb range(more targeted to the people outside town but have line of site to the mountain), or T-Mobile home internet in the 30-180Mb. Small towns are super friendly and responsive to the 'work remote' crowd.


Just wait until you hear about the UK.

BT/Openreach has control of about 80% of the backhaul/trunk connectivity.


Completely not comparable. Openreach has significant marketshare but is also extremely tightly regulated. It has to allow other providers access to basically every element of its offering, now including its physical ducts and poles.

Hence you have a very competitive market landscape in the UK. There are at least 5 major national retail players using various wholesale products (BT, VM, TalkTalk, Sky, Vodafone), plus dozens of altnets offering FTTH on totally seperate fibre infrastructure have started (Hyperoptic, Cityfibre, GNetworks, Community Fibre).

In my flat in London I have access to 4 seperate FTTH networks (with completely different infrastructure) - Openreach FTTH, VM DOCSIS, Hyperoptic FTTB and Community Fibre FTTH. The market works here, prices are low and there are very few data caps.


>> London

Yip, things in London are quite different than the rest of the UK.

While Openreach has to allow access to other providers to its infrastructure, most infrastructure is owned and controlled by BT.

If you want FTTP, the underlying service is still BT Openreach. Openreach is very profitable.

At my previous employer it cost us £30K to have BT put in 3000m of fibre that took them 1 afternoon, then it cost £1K a month for a 30Mbs service.

BT owns provides the vast majority of broadband service in the UK and it is somewhat disingenuous to claim BT is on an equal footing as Hyperoptic, VM, or Community Fibre.

TT and Sky use BT OR infra, and EE, Plusnet, and BT Retail are all owned by the BT Group.

The telecom market in the UK is very consolidated and controlled by BT.


You don't even have to leave London to see a night and day difference. I currently live in a new-build flat and have multiple choices for fibre (Hyperoptic are great). I am hopefully soon to be moving less than a mile down the road into a house where my choice is either an offensively slow BT-backed service or an offensively unreliable (so I have heard) Virgin Media service.

If you have a new-build flat in a major UK city you probably have a similar decent choice but the rest of the housing stock (the majority by far) is stuck with BT/Virgin.


You'll almost certainly be getting BT FTTH in the next year or two if you have access to VM. The program is running quickly, something like 6 million premises a year and ramping.

VM can be ok or can be congested depending on the area. BT FTTH doesn't suffer from congestion issues (nor does the FTTC).

Check https://bidb.uk/ for more information. It collates all the altnets and planned roadworks into one dashboard.


>> BT FTTH doesn't suffer from congestion issues (nor does the FTTC).

That is just not true. BT consumer FTTC is contended up to 40 to 1--that is you are sharing your backhaul with upto 40 of your neighbours. I am assuming that their FTTH services are the same.


> If you want FTTP, the underlying service is still BT Openreach. Openreach is very profitable.

Not always. There are many altnets now (cityfibre etc).

> TT and Sky use BT OR infra, and EE, Plusnet, and BT Retail are all owned by the BT Group.

TalkTalk also use cityfibre.

> The telecom market in the UK is very consolidated and controlled by BT.

If you mean physical infrastrure, yes it is (though changing rapidly). Consumer level pricing is competitive though and the services are reliable in the large. It is a different planet compared to the US and especially Canada.


I see a lot of people noting the lack of communication and updates from Rogers.

To me it would be shocking if they were providing updates. Canada’s telecommunication monopolies are not known for their customer service and, even worse, they seem to have internal corporate cultures of entitlement and arrogance that drive an “F U” attitude in general when it comes to external communications and accountability. So you won’t see a post mortem. You won’t see timely updates. You certainly will never see some kind of status page!

I’m sure Rogers will be forced to do a post-mortem for large commercial clients where contractually required, but I highly doubt we will hear a word from them about the cause of this or what was/is being done to rectify it. They probably have an ETA but feel no obligation to share it.

Rogers isn’t alone. I would expect the same behaviour from Bell or Telus. Canada has serious issues when it comes to its big telco carriers and it represents major risk to the Canadian economy.


It's not possible to pay with debit card in Canada because Interac is down, because Rogers is down. Many atm across the country aren't working.

It's not just Rogers' client who are impacted, the whole banking ecosystem is impacted.

Even rumors that Costco can only accept cash, the Capital One Mastercard network might be down.

I don't see how Rogers could keep quiet on this, unless it's from some malicious actor.


> I don't see how Rogers could keep quiet on this, unless it's from some malicious actor.

Canada will get the same action on this as Americans do about mass shootings. “Thoughts and Prayers”.


Have you ever seen any similar company give meaningful updates in a situation like this?

It is odd they don't have PR people trying to spin it. But I am not surprised in the slightest that there is no technically useful updates happening.


If you are already the most hated company in Canada, why do you need to bother spinning anything or trying?


Indeed, I can easily see them get dragged before a parliamentary inquiry.


Do those happen before or after the beer drinking with the regulators?


These seem to occur rarely. Unlike US Senate inquiries, Parliament seems to be less aggressive about these, reserving them for things like indigenous issues. Quite disappointing IMO.


> I don't see how Rogers could keep quiet on this

My sweet summer child, you have no seen how meek and docile Canadians are.

They'll take a page from the Federal Liberals, spout a bunch of unrelated platitudes and nothingness, and pretend nothing happened.

The oligarchs control Canada and the Federal Liberals are just a finger in their hand. There will be no inquiry from regulators - what's more - they'll rubber stamp the Rogers - Shaw merger that's been in the works.

If Canadians say anything, Bill C-11 will be ready to squelch them online. If they say anything in person, a quick speech from Trudeau calling them foreign-funded MAGA terrorists will cause the government-funded CBC to produce an article. That article will then be used by the Trudeau Liberals to invoke the Emergency Act once more, then de-bank and trample them.


I would say if you want to engage in the process of dialogue, don't bring a large cache of firearms with you while blocking an international border crossing.

https://www.cbc.ca/news/canada/calgary/coutts-protest-blocka...


Which has nothing to do with the ottawa protests? Are we pretending that they suspended the charter because of that single event in manitoba now?


> I’m sure Rogers will be forced to do a post-mortem for large commercial clients where contractually required, but I highly doubt we will hear a word from them about the cause of this or what was/is being done to rectify it. They probably have an ETA but feel no obligation to share it.

I'm sure there'll be industry wide participation in a post-mortem. Translation for non-Canadians... The head of the CRTC will book a tee time for him plus the CEOs of Rogers, Bell, and Telus.


I see so many people get these two conflated, across the board, any time something like this happens. Some big leak or outage or whatever.

The shocking part is not that they are giving no updates. Everyone is to expect that. The shocking part is that we (collectively) don't care that they give no updates, and let them get away with it.


What actionable steps can consumers perform?


The French People walk for less reason. (J/K they only walk for serious issues)

This outage is way too disruptive for polite exchanges of platitudes.

The public have to make their point of view high-key known. The public should respond so strongly that the other companies start saying things in public. Then the rich will get queasy and ask themselves, what do we have to do to buy back social peace?


This is probably the best question that can be asked.

Vote with our wallet? C'mon, what else?

>What actionable steps can consumers perform?


> it represents major risk to the Canadian economy.

I actually think this outage will show that there's no major risk to the economy and every Canadian can just take every Friday off with no ill effects.


Agreed, although, Rogers has had notably more vaguely explained outages. With this, if the CRTC does not revise their decision to let Rogers acquire Shaw (CRTC 2022-76), then, as Canadians, we must expect to continue to overpay for shotty telecoms. But hey, there's maple sirup and affordable health care.


https://blog.cloudflare.com/cloudflares-view-of-the-rogers-c...

Trust an American company to come up with a clearer picture of what's going on, highlighting the mediocrity of Canadian businesses.


It feels like there’s been a rash of companies messing up BGP stuff lately - Facebook, cloudflare, and now this.


Having formerly lived in Canada for 30 years, I can concur.


Trust an American company that did the same boneheaded thing not long ago to be able to explain it well.


The lack of communication from Rogers for an event of this magnitude is quite something. Three tweets[0] with zero useful information so far today. The first one not coming until ~4 hours after the interruption started.

[0] https://twitter.com/rogers/


The timing of that update tweet is what gives pause for concern. If it were the same or similar thing as to what took down their network nationwide in April, it seems reasonable that they'd just say something like "Hey guys, sorry but the network is down due to <insert similar cause here>."

A lot of us are in IT and for a network of national importance in a "developed nation" to be taken offline so easily is worrisome.


They probably can't get online to tweet. /s


Seriously though, someone working there had to find a friend with a Telus hotspot to borrow, I imagine.


Wait until you find out that the cable that Rogers uses to peer to the Telus network got unplugged!


Their 2FA passcodes are sent via SMS to number(s) on the dead network. :_(


Your mistake is assuming that their twitter account is for providing useful information to customers, rather than signaling their allegiance to state-sanctioned social and political causes.


In Toronto, here's a small and non-exhaustive list of affected services:

- Point of sale machines. Supermarkets are accepting only cash.

- ATMs. Several banks are incapable of dispensing cash.

- Public transit ticketing systems. The TTC is effectively running for free, today.

- Public bicycle rental stations.

- Public parking locations.

How long before the public realizes that we are not Rogers' customers, but its hostages?


>How long before the public realizes that we are not Rogers' customers, but its hostages?

Won't happen. The average Canadians are one of the most sheep like docile people out there.


Remember when we all banded together to prevent Verizon from introducing competition into our market because it would “take Canadian jobs” and “give our hard earned dollars to the states”?

That happened because of a hostile attack ad campaign by the big 3.


I'm not sure an aggressive astroturf campaign is the same as "us banding together". There was even an anti robellus site[1] set up.

[1] https://realfairforcanada.ca/


What actionable steps can Canadians take?


Well, elect politicians who encourage free markets, and have the least govt in our personal life as much as possible. Easier said than done. Canadians (like most other people) are too stupid to gauge what policies/politicians are good for them.


Good intention. How do you keep markets free?


Like I said, Canadians (like most other people) are too stupid to gauge what policies/politicians are good for them. So true free markets will never happen in practice.

Yes, I'm being pessimistic, but it's the way things are.


Has any of the big cities ever considered its own municipal internet? I would vote for any politician that has that as a platform immediately.


City of Toronto has done some work on it but the mayor of Toronto is also on some Rogers board with a pay for $100k a year


My jaw literally dropped. I cannot believe this level of corruption. Goodness gracious.


You must know that it'd run on Rogers infrastructure though.


Not those kind, I'm wondering if any city has considered making their own infrastructure.

The Ebox and Teksavvy are great, but like you say it's hard to survive on other people's infra, that's why big cities need to make their own.


I have a feeling that to do it right, it'd have to be a collaboration with an enthusiastic private partner, if for no other reason than available expertise. One city, Hamilton, Ontario, years ago tried out free public wifi. Great idea, but this was way back before smartphones were as big, so basically everyone used computers to access the internet. Not a huge deal, but the city rolled it out so poorly that the end result was signal strength that was only usable outside. Since few, again because of the time in which this happened, use an internet connection outside, few people were able to use the service, and it eventually just went away. A week of testing would have found this issue, but nobody in the city thought of doing so, likely because nobody technical enough with enough time was involved in the entire project -- it was simply a great idea, but nothing to back it up.


> Point of sale machines. Supermarkets are accepting only cash

> ATMs. Several banks are incapable of dispensing cash

Back to bartering.


At my local supermarket debit was down but credit card (Visa) was working.


That would make sense, since only Interac relies on Rogers, and it processes debit cards, not credit cards.


Could be one of two things:

1) the site has connectivity so only Interac is down

2) the site has no connectivity and can process credit transactions offline and queue them (don’t tell anyone their expired card will work), but not for debit.


> don’t tell anyone their expired card will work

The expiration date is stored on the card, so expired cards wouldn't work. Revoked cards (eg. ones replaced by the bank) would, though.

Keep in mind that the merchant sets terminal limits for what transactions are allowed to be processed offline, so one might be able to get a burger without connectivity, but not a new TV.


Also, the CRTC helpline and Service Canada and CRA account logins. We were unable to access government services.


Alright, when do we start carrying chickens to buy our gas?


I was in Whistler today -- a lot of multinational retailers running cash only businesses today.


I don't understand how this is not on CNN/BBC news. This isn't the top news at the moment but significant .. I don't know if people in other countries know how this feels. People are saying this is pre-1990 .. but it is not really. People don't have landlines any more .. all our media is streamed via Internet. Stores can't take payments or orders. People can't call Ubers/taxis Millions of people don't have cellular. 911 issues have been reported. It has been over 12 hours and issue is not fixed.

This video shows a senior person at Rogers giving an update on the situation. It feels like I am in the twilight zone: https://www.youtube.com/watch?v=MYfRZZaPrC0



Reuters, the news organization, is headquartered in London. Thomson Reuters, the media conglomerate that owns it, is headquartered in Toronto.


Yes but.. we have municipal wifi, public libraries, etc, most of which are fortunately not running on Rogers (although admittedly they probably don't have much redundancy in place, here in Montreal, buses use Telus and city wifi is often Bell).

And Rogers broadband is mostly just Ontario?

re:landline, I'd say go a step further, and don't use cell services at all. All my phone calls use Signal or VoIP, sms is routed through VoIP.ms. My cell is data only and costs 15$/month for 3GB (fido/rogers).

I know, I know, some conditions may apply, but I really hate telcos and enjoy finding workarounds. Hopefully this will encourage a discussion about monopolies/redundancy.


> voip.ms

> redundancy

You do have a point in general. I wouldn’t put VoIP.ms and redundancy in the same sentence though. Unless they’ve drastically changed their infra, they had a massive 15 days (yes) outage not that long ago because of no redundancy and a poorly architected infra.


wasn't that primarily because of ransom-DDoS? I mean sure, not a good amount of DDoS protection but still


Your cell is Data only? I have never seen that as an option. How do you take calls on 3GB/mo? What VoIP service do you use for receiving calls?


If you look at the front page of the BBC News website right now, it is a fairly high ranking item directly below the Japanese assassination.


Right? I saw that and I was like... who gave this guy the right to talk to the media? If I were corporate communications at Rogers I'd be freaking that this person made it onto camera with this crap

(Luckily my phone is Freedom and my Internet is an independent small Hamilton local ISP.)


Aren’t a lot of smaller ISPs going out of business right now because of the CRTC’s decision to allow the big 3 to charge more for wholesale rates? I recall TelSavvy being going outspoken about this.


I'm not sure of their business state, but yeah, it's not good.

For me, I'm rural and I use a 900mhz point to point wireless connection that caps out around 15-20mbps on a good day. The upside is it's provided by a local old school indie ISP ("NetAccess"). It's $130 CAD a month, which is... expensive... but unlimited which is far cry from the theft that Rogers was performing on me prior with the 3g (then LTE) connection I had that at first capped me at 25G per month @ $100 a month and eventually went up to a "generous" 100G and then charged insane overage fees. My teen daughter ran us up a $400 bill one month.

Bell and Rogers are now offering more competitive rural options (at the behest of the gov't) but I refuse to give them any more money.


The guy in the YouTube video is terrible. It's like he's trying to be a politician and not answer the question, but he's incredibly bad at it


The Canadian telecom space is dominated by 3 primary companies -- Telus, Bell, and Rogers. They form an effective oligopoly that is quite detrimental to the Canadian consumer.

In the ISP space, there is a bit more competition. Namely, Shaw provides additional coverage in some regions of the country. However, Rogers wants to buy Shaw. You can imagine how bad that will be for Canadians.

I do wonder what the Rogers outage is about. Ransomeware? State attack? Something stupid? If anything, it shows how we should not have critical infrastructure centralized. Competition between ISPs is important.


In the ISP world, what will happen with Rogers buying Shaw...

In the western provinces, AB and BC, Rogers runs a very widespread and strong LTE network. What they do not have is a DOCSIS3/coax and GPON cable TV plant. Nor do they have much terrestrial right of way for aerial plant or underground in conduits, which is where Telus (the historical copper POTS ILEC) and Shaw (the historical cable TV operator going back to the mid 1970s) are by far the strongest.

Rogers is a facilities based last mile cable operator/terrestrial operator in Ontario.

Shaw runs the landline cable tv networks in most of the metro Vancouver area. And many other small to mid sized cities in the west.

Letting Rogers control both one of the largest/strongest mobile phone networks and the only viable land line terrestrial broadband competitor to Telus by acquiring Shaw's cable tv plant is an absolute outrage.

To use a USA analogy, it's like if T-Mobile already owned RCN (Astound) in some big part of the country and then proceeded to buy Comcast.

Or if Verizon bought Spectrum (Charter/historical TWTC cable).


There's also Vidéotron and Cogeco but these are not Canada wide so .. yea.


> Something stupid?

Whatever it is… it's definitely something stupid.


-- Started to fall to bits around 4:00am EST - got woken up by my phone non-stop glowing as news alerts about abe flooded in - got out of bed to check the news on my laptop - wasn't working - tethered to my phone & checked BGP - AS812 was announcing bogons & dropping peers like it was going out of style - thought maybe they were doing a config update - went back to bed - was incredibly surprised to see it dropped all the peers & was sitting dead when I woke up --


Can you please show bogons learned from AS812? Thanks!


https://bgp.he.net/AS812#_bogons

They announced 216.176.216.0/21.


Completely unrelated. Network in questions was advertised from Rogers for last 3+ years.


Interac is down because Interac uses Rogers' network.

Interac's backup network also happens to be with ... Rogers.

Canadians across the country can't use debit cards, regardless of their financial institution. Credit and cash only. E-transfers are unavailable. Fun for people trying to pay their bills online.

And that's just the financial side of the situation. Millions of people have lost Internet and phone service.

Someone on Reddit wrote: "Maybe the worst part is that they have literally no open lines of communication. No twitter posts. All chat options (twitter, IG, FB, web based) are all unresponsive. Can’t call tech support, as the call will fail. Can’t even call billing or general inquires as it’s the same thing. Shit happens, I get it. But to leave your customers completely in the dark? Brutal. Just brutal."


It's hard to understate the impact of this across all of Canada. As someone who uses Apple Pay for almost every transaction, this underlines the fragility of a "cashless society."

Also, around half my team are unable to effectively work today. We all work remote, and half the team's home internet is on Rogers, which is down. Some people are tethering, but people who are also on Rogers for their wireless are out of luck.

I'm not a Rogers customer anymore, but my boss showed me how you can't even login to their account anymore. Their website is down too, just timing out.

Funny, I left Rogers last year and just last night a telemarketer phoned me asking me to return to Rogers. What a sign that I made the right call by politely refusing.


Tethering with Canadian data prices is..spicy. A Zoom call uses about a gig per hour with HD cameras, and you typically get about 10 gigs of data per month. Overages charged at one firstborn per gig


> this underlines the fragility of a "cashless society."

And a society that refuses to invest in robust infrastructure.


You don't need an internet connection to use Apple Pay. Unless you mean you didn't use credit cards before, the situation has been the same for the last 20-30 years when we stopped using carbon paper imprints.


The payment terminal still needs internet access to process your Apple Pay transaction


It does not for credit. Credit transactions can be batched offline and transmitted when things come back online. Debit must be online, though.


What happens if the transaction is over credit limit or if someone uses a fake (once valid but now cancelled, generated, etc) card?


Imagine Bell was down too. What then?

I really hope this results in the CRTC starting to allow more competition. I hope, but I'm not optimistic.


I'd suggest giving the CRTC a call with your thoughts... but their phone system is down :)

https://twitter.com/CRTCeng/status/1545421218534359041


I think we're most likely going to see them using this as justification to let Rogers get their way even more. Maybe the CRTC will try to convince the rest of the government that it's a reason to give away more cash and preferential treatment to Rogers and Bell as well.


applepay works fine. not on the interac network.


Does Apple Pay work even with debit transactions?


It likely only works with credit cards.


> It's hard to understate the impact of this across all of Canada. As someone who uses Apple Pay for almost every transaction, this underlines the fragility of a "cashless society."

It seems like Bitcoin and crypto fared well during this outage.


If you could get on the internet to transact your crypto, that is.


You can if you use cellular data to transact with Bitcoin, not everyone is using the Rogers network, but the banks in Canada were.

So no online banking, withdrawals, payments in affected parts Canada.


Its crazy that interac does not have backup independent connectivity.


Truly, and hopefully this leads to change. But this is also a consequence of Canada's lack of competition with ISPs.


To a certain extent yes. But its not like rogers literally has a monopoly. The isp i use is doing fine.


When there is so little choice, it's essentially an Oligopoly.


As long as there are at least 2 choices, it should be possible to have an independent backup.


Apparently it does - with another Rogers connection.


They probably bought into SLA clauses claiming Rogers won't have maintenance downtime on both circuits at the same time. That shouldn't be an excuse not to have a backup with another provider.


It's hard to believe these jamokes don't have redundant providers. It's something I roll out to even my smallest consulting client...


They have a system-wide outage. How can tech support help in any way?


The VP of Rogers admits they have yet to identify the root cause:

https://twitter.com/PnPCBC/status/1545512971878662145


This is absolutely astonishing.


> This is absolutely astonishing.

It's quite astonishing by any reasonable standard, but this is Rogers we're talking about, so pretty much par for the course.


I was hoping there would be a discussion of this here with some inside baseball, but then HN was down.


The Cloudflare blog suggests it's BGP problems: https://blog.cloudflare.com/cloudflares-view-of-the-rogers-c...


this is pretty far outside my area of expertise so maybe i'm missing something, but isn't the cloudflare blog just describing what they're seeing from the perspective of somebody who monitors BGP traffic, rather than assigning the blame to BGP?

it seems like the BGP traffic signature of any isp-level outage, regardless of the cause, would look like this. they went down, so they stopped advertising that they could accept traffic. then they tried to accept a little bit of traffic, but that didn't work so they turned it off again.


Yes, exactly. BGP withdrawal is likely a symptom, not the cause.


How does that relate to the Rogers cell service outage, though? Surely SMS doesn't require BGP?

It feels like there's a big, fundamental problem affecting a variety of Rogers infra, where BGP and cell problems are just symptoms we can observe. CloudFlare says it looks more like a failure than an attack; I'm not entirely convinced.


I don’t think it’s an attack.

I’m in Canada at the moment with my UK SIM and using Rogers internet just fine. Completely unaware of any problems until my friends warned me about needing cash and I then double checked and still, phone internet via Rogers on a roaming SIM worked fine.

My guess is their voice, sms etc all are IP these days which was impacted by their BGP issues. Whereas my UK carrier roaming gives me a UK IP still, unrelated to Rogers advertised blocks.


> My guess is their voice, sms etc all are IP these days which was impacted by their BGP issues.

Impacted, yes. But "no service"? That's dubious.

I hear you on foreign SIM working. I got online at a cafe just long enough to buy a Yessim eSIM. When I activated it and turned on data roaming, I had LTE working on... the Rogers network! That I didn't expect.


> Impacted, yes. But "no service"? That's dubious.

Nope, in contrary I will fully believe it. Circuit-switched networks are dead, period. Most "circuit-switched" networks are actually IP networks just with custom FEC'd and prioritised protocols so that there's backwards compatibility. Most Subscriber Authentication systems since UMTS and 3G CDMA (except for super-early deployments, and since they have LTE they would have migrated it if it were the case) have moved into an IP-based system because it's cheaper, and if that's down then everything is down.


It's always BGP......


Sometimes it's DNS...


Should just merge this all into systemd-bgp-dns; kill all the birds with one stone.


Well, Pottering quit RedHat to work for MS, maybe that's exactly the plan


Unfortunately BGP problems make DNS problems look trivial :(


I was so confused when I couldn't access HN - I thought to myself maybe HN was hosted on Rogers network or something.


There's a Wikipedia article, though it's still sparse. It should have post mortem information once that's official.

https://en.wikipedia.org/wiki/July_2022_Rogers_Communication...


Ditto, I expected to this be front-page on HN with many upvotes.


Staying with the inlaws that use Rogers' reseller for wired internet.

Using Fido for mobile internet access (also by Rogers).

Zero internet. It's wild to discover how reliant I am on the internet:

- Can't get around this city (no waze, gmaps).

- Can't figure out what's going on (AM radio is non stop commercials)

- Can't figure out where we might get some internet (maybe WeWork somewhere? How do you find one without internet)

This is just wild. Almost as critical as no water, or no power in the winter.


> Can't get around this city (no waze, gmaps).

offline gmaps!

Thankfully Canada’s horrendous roaming and overage charges has taught me how to use my cell phone effectively without service.

(Also have the entirety of Wikipedia on my phone, thanks kiwix!)


How much storage does Wikipedia take up?


88gb for English


For more information:

https://en.wikipedia.org/wiki/Kiwix

~88 GB w/ images, English. 47 GB w/o.

There are also Wikipedia apps which opportunistically download pages. You'll have access to what you've previously read offline, but will require connectivity to read more.


Still no internet at all? I have a suspicion that you found some…


My reseller connection (Start.ca) is down, and Telus’ cell phone service becomes near unusable because of the extra load.

I expect the device I send this from will be offline again in the next few minutes, but I’m willing to be pleasantly surprised.


Have been getting all circuits busy alerts periodically throughout the day.

Fun stuff!


The HN downtime was to add a post-by-postal feature


Still no sadly. My phone shows 2G network. The last time I use 2G network was probably...decade ago.


> - Can't figure out what's going on (AM radio is non stop commercials)

Don't you have public service? Isn't that what CBC Radio One is, with no commercials?


The outage took down some CBC stations https://www.cbc.ca/amp/1.6514376


I'm genuinely curious now how you access HN.


Drove around for 2 hours to see if another cell might be up, nope. Stopped a test stop that had internet. Looks like Fido is back up, but the isp is still down.


680 news is AM radio version of CP24 at least.

680 news is owned by rogers and cp24 is owned by bell lol!


https://www.cbc.ca/news/business/rogers-outage-cell-mobile-w...

"we're getting very close to understanding the root cause of the of the failure. And we're taking actions along with our network vendors to recover the situation."

"We don't understand how the different levels of redundancy that we build across the network coast to coast have not worked," he said.


Translate: "We have no idea what's wrong or how to fix it. But we've almost decided who to blame."


Here in BC Rogers came back about 4pm, but went down again around 7pm.


Hopefully this will contribute to a regulatory denial of Rogers' upcoming purchase of Shaw, if only to keep a competitor with its own separate infrastructure to minimize outages through consolidation.


omg I didn't even know about that. How could that be allowed to go through


This is funnier if you look at it from the English slang point of view when something gets thoroughly Rogered.

This network has definitely rogered itself.


If this doesn't show Canadians that telecommunications are a public utility, I don't know what will. The CRTC's (Canadian FCC) own phones are down[1] today because they're a Rogers customer. If that doesn't scream regulatory capture...

1: https://twitter.com/CRTCeng/status/1545421218534359041


I mean... it isn't as if the CRTC is going to set up their own telco just to operate their own office.


Yeah, but ultimately they are pretty much owned by the big Telco so we are fucked


Late this afternoon a Rogers Executive in charge of the response gave an interview to the CBC. He said they hadn't even identified the root cause of the issue at that point (~12 hours into outage) https://twitter.com/PnPCBC/status/1545512971878662145


That seems reasonable if service is still not back to 100% -- they need to focus on fixing the problem. They can investigate what caused the problem later.


It seems like some sort of hardware/software rollout going wrong. We had similar outage last weekend in Japan with #3 service provider KDDI/AU, a 86 hour outage of both cell and data service. Lot of services like weather stations went offline, people were unable to reach/use emergency, banking, barcode/app based payment services. Some ATMs and Vending Machines went offline.

Mobile and data service has become quite embedded in daily life.


I'm using a Roger's number in India currently.

Unfortunately, I've been using that number for most of my 2FA so essentially, I've been unable to access a lot of services because of this (ones where I didn't use Authy/Google Authenticator).

Lesson learnt: use a separate VoIP business phone number instead.


Isn't that susceptible to the same problem though? I mean, couldn't any service provider be down? I avoid SMS 2FA whenever possible, but sadly that's all some services offer.


Are you using Wifi calling to receive 2FA? What would you change to?


Same here in the US. Which service will you port to?


I'm just using OpenPhone now for services tied to my startup.

At least, I'd be getting verification codes without having to worry about the service being down.


Looks very similar to Facebook's outage. A combination of BGP and DNS problems that started with a configuration change.

https://blog.cloudflare.com/october-2021-facebook-outage/


What evidence are you seeing that makes you think this is happening in the Rogers' case? I haven't seen any diagnostics or analysis by anyone.


Rogers did just abruptly withdraw all their routes, then reannounced some, then withdrew them again. It's not dissimilar at least.


You probably saw it up thread but since rogers isn't talking this is all we really have: https://blog.cloudflare.com/cloudflares-view-of-the-rogers-c...


https://radar.cloudflare.com/asn/812?date_filter=last_24_hou... look at the IPV4 vs IPV6 traffic, when there is IPV4, there is no IPV6 and vice versa. Issues announcing IPV6 BGP prefixes and IPV4 at the same time?


I’m surprised there’s no chiming in from Bell customers. I’m on their Fibe or whatever plan, no issues all day whatsoever. My phone lines, however, are all down, as is merchant processing etc etc.

Really strange day but not unexpected. We “know” the telcos are screwing us but we don’t have a way to gang up against them, as people easily buy into their feelgood propaganda.


From what I've heard, some Bell lines are really just Rogers lines with a Bell label on top


I would imagine it’s the other way around, but maybe the infra has changed. I always thought it was Bell at the very top.


Just wait until Rogers and Shaw finish their merger so that the entire country can be plunged into the stone age every time a technician trips over a wire in Toronto.


Just got back online a day later. Progress on side projects aside, the day was a write off.

In terms of causes, I think it was one of a few possibilities in order of percieved plausibility and likelihood:

1. newb engineer updated router configs using a CI pipeline that caused them to stop talking to each other.

2. standard mistake of redistributing iBGP with a bunch of static routes for load balancing into eBGP causing their internet facing ASN to announce the internet, and it took down their internal MPLS routes for backhauling their cellular traffic over IP.

3. geopolitical situation where someone was making a deniable example of their capability against a target who can't fight back. That was just a friday in Canada, it's just as easy to do D.C., Chicago, or New York.

4. a political stunt and pretext for "a safer internet" with "more oversight," orchestrated by party hacks, or an exercise to see how effectively they could do this. Least likely, but that's the level of trust there is in this govt right now. If you don't want conspiracy theories, try being credible enough people don't have to invent explanations because nobody believes you are in control.

I'd bet on 2 or 3. Items 1 and 4 are weak ideas, but 4 will likely exacerbate confirmation bias. What will likely not come out of this is liberalization of the telecoms market to add competition. However, regardless of teleological fallacies, I don't see this as a crisis that will be left to go to waste.

It has been a good fire drill for business continuity and preparedness anyway.


I see it seems to be a day for shitting on Canadian telecoms. May I say a hearty "Up yours CRTC".

Seems as good as any time to share a Rogers story with HN:

My mum passed away in 2009, she owed Rogers $200 and change. We took over her contacts/mail forwarding/etc to tie everything up. A month or two afterwards; we got a call from Rogers looking for their money, explained she passed away, they said how sorry they were and figured that was the end of it.

3/4 months after that we got a call from a collections agency, mentioned her passing, got apologies and hang up. This then continued every few months bouncing between Rogers and a collections agency. For some reason they kept passing the debt back and forth? The frequency was low enough, that we never bothered going any further with it, after a couple years it was less annoying and more a nice reminder of mum :)

It took about 6-7 years before they finally clued in, and that's why I'll never have anything to do with Rogers.


I used to work in a debit processing command center where we would monitor all of the acquirer and issuer links in real-time. The amount of paperwork and phone calling required just for a <60 second MPLS VPN drop-out was pretty incredible. I can't imagine a day like this in that room. Would have been an absolute circus.


Not a good week for internet infrastructure in Canada. Yukon Territory had essentially no internet for most of this Wednesday after the only fibre was broken.


that, at least, was a very canadian outage caused by a beaver chew

it's hard to build resilient rings of fiber between towns/cities where there is only one linear path (road) for right-of-way that you could economically build the fiber along...


I think my weirdest experience was a server that was jammed on "apt update". I looked up the IP using whois.. and it belonged to Rogers.


Where was the server?


The server was setup to use ftp.ca.debian.org, which currently resolves to 207.210.46.249, and the whois for that mentions Rogers.


Canadians like to complain about the lack of competition.

For wireless Freedom Mobile was independent from 2008 to 2016. Mobilicity was independent from 2010 to 2015.

They were always cheaper than the incumbents. However, they struggled to get customers because their coverage and speed lagged and they were eventually bought by incumbents for their wireless spectrum license.

The government tried to foster competition by letting them bid on spectrum set aside for new players. And removing strict foreign ownership rules.

Canada is a small market. Geographically larger than the US with the same population of California.

It's just not an attractive telecom investment.


Sounds like those are good reasons to nationalize telecom infra.


Clearly a correlation between 2 hard drives failing at HN [1] and the Canadian Rogers network outage [2].

[1] https://twitter.com/HNStatus

[2] https://www.reuters.com/business/media-telecom/rogers-commun...


this is quite a significant outage and has been ongoing for over six hours now.

would love to see a tech write up or blog post or even a Twitter hint but there seems to be absolutely zero transparency.


> ongoing for over six hours now

Closer to 12 hours now I think. Certainly it was out at 7am EST this morning when I woke up.


Problems started to appear around 4am EST and services became completely unavailable by 4:40am EST.


It's quite surprising. I wonder if the technical side and the PR side of the company just aren't sufficiently connected.

It would be beneficial to have unvarnished tweets from someone in a technical role, written for technical peers rather than the general public. The press and tech journalists can filter the jargon into something useful for customers. "Dashboards are showing ABC, we're trying XYZ, so far it's looking like LMN", even if the public don't understand what is being said, is more reassuring than a generic statement. It suggests competence and hints that yes, it really is a difficult problem.


The first withdraw was at around 7:45 UTC, a little over 13 hours at the time of writing this comment.


At what point does the anti-competitive nature and monopolistic qualities of Canadian telecoms become a national security threat?

Maybe when it destroys the whole Canadian economy for a day


>Maybe when it destroys the whole Canadian economy for a day

Won't happen. The average Canadians are one of the most sheep like, docile people out there.


Harsh, but so true lol


Nah, that's not enough, but maybe when it does that twice in 15 months.


never. It's a very deeply corrupt country.


“In April 2021, Rogers customers reported interruptions to wireless voice and data services for several hours. Rogers blamed that outage on a glitch tied to an Ericsson software upgrade”. Rogers has rolled out 5G with Ericsson equipment and I hope this outage is not related to Ericsson activities. Network upgrades are usually performed late at night during low traffic, so it shouldn’t be the case.


Thanks for mentioning the last major outage. That one was memorable for me as my son had just been born a few days prior. If that had happened the day of, I honestly don’t know how we would have managed it logistically.


For the record, I noticed at 1:50 PST or so - meaning a network upgrade would actually line up exactly.


Goodbye Rogers, Hello Telus!

Glad that Best Buy has a large stock of Telus SIMs.

Never again, seriously.


Interac.

This outage was eye-opening. You’d think that Interac (a prolific payment method for using your bank account to pay for products that hit the scene early enough to get entrenched in Canadian habits) would be infrastructure-agnostic, but it’s clear that they are extremely dependent on Rogers’ network. All Interac was down. Places still processed credits cards as normal, the machines still worked, and it didn’t matter what ISP was used by any particular store.

I do not understand how they were comfortable floating billions in transactions over a network with a single choke point.


Given that private business won’t magically reduce their profits to have (more) backups and hire more competent it staff, I guess you all want the internet to return to being a regulated utility? The only really serious aspects were letting debit transactions be run by a single provider along with some aspects of border control. These should have mandated fall over to other vendors to keep running. The banks and credit cards did fine as long as the store was on line.


I am in Toronto and I don’t use Rogers. Nevertheless Rogers slows down 4g traffic for most providers. It is fun watching how tones of services and sites simply stop functioning when internet slows down


Canada's main issue is that we've allowed duopoly on ALL telecom (i.e. we essentially have 2 major national communications networks encompassing all forms of telecom, including phone, TV, cell phone, internet etc.)

Rogers is one of those networks, and it basically covers 30-40% of the country's communications.

There are a few smaller regional players, such as Shaw, Cogeco, Videotron etc. I think we might be the only developed country that's allowed this. I know in the US, there are 5 major mobile networks, some of who offer internet etc., but there are major internet providers separate from the mobile network providers.

The reason this has happened, is because building networks is really expensive making it so that being a regional provider is a tough business, and the 3 major companies have played the laws to their advantage. They've done acquisitions over time, and used the rules around foreign investing to block a lot of major development in 3rd and 4th networks from happening.

This is why the Rogers/Shaw deal should be blocked and the government needs to allow foreign investment to be able to come in and build out networks.

This problem can only be solved in 1 of 2 ways, and only through regulation unfortunately: - A complete overhaul of our rules which will allow foreign investment in our communications networks in Canada - A break up of the 2 networks

The second is not likely to happen. Rogers/Bell/Telus are some of the wealthiest companies in the world, and are at the top in terms of deep pockets in Canada. They will essentially fight the government on any sort of attempt to break them up and handily win.

The government has a much better shot at changing the rules to allow foreign investment, but they don't want to do it. The reason that's the case is because the leadership of the telecom companies has many shared members with prominent politicians, and also outright owns all forms of media in the country except for CBC, Post Media, and Corus.

Post Media is a known right-leaning org and Corus is effectively owned by Shaw, so their sway in the country's politics is not as big as Bell Media and Rogers Media who both own much bigger networks.

That means that they can run a very effective smear campaign against any politician or part who dares to allow foreign competition to enter Canada.


Sounds like a BGP problem where people needs to get onsite to fix the issue but since they never did the drill they can't connect etc ...


FTA - "Rogers has not identified a cause."

Seems like a rather large impact. Maybe Russia really wants its gas turbine back? [0]

[0] - https://www.reuters.com/world/exclusive-ukraine-urges-canada...


After around 20 hours, it just started working for me. I am with Fido for mobile and use Tecksavvy for home internet but both of them rely on Rogers at the end, so both were down. Walmart was only accepting cash or credit card. Debit cards were not working as Interac was down.


Would Starlink work in Canada during an outage like this if the base stations don’t have internet?


Starlink worked throughout the Rogers network issues for those I know using it in Canada.

From what I understand of their current network setup, if their base stations in a region had no connectivity then the starlink customers in that region would lose their connectivity too.

This is likely going to be able to be mitigated when Starlink inter-satellite communications is up and running and they can route packets in space.


I don’t know but this makes me want to get out the Starlink and try. It is my disaster backup.


I don’t understand have 911 service is impacted, doesn’t essential services have redundancy?

Society is fragile


The actual dispatch appears to be fine. It’s just the network isn’t allowing mobile connections. Everyone has “No Service”.


https://www.cbc.ca/amp/1.6514453

And some hospitals appear to have similar issues.


My spouse is a division chief in a surgical subspecialty - hospital is mandating anyone on-call to not leave the premises. People are sleeping in their offices.


Well here's one reason not to retire cash just yet.


Looks like it is still going on. I only have 2G network. I also realized that I cannot use Google Map for driving without network.


That's why I use Bell or Vidéotron



[flagged]


I know many countries have problems but Canada is in a okay spot. Specific cities have looming crisis on their hands but this is largely the fault of provincial governance.

Little of that I attribute to Canada as a hole, I just find BC in particular to be quite mediocre in terms of value compared to what you pay.


with the massive property price increases that have taken place Canada has become even more mediocre than ever before. With the increases we've seen some movement out of Toronto and Vancouver areas into other areas of the country, driving up prices even more. Young people are basically out of luck in terms of hopes of ever owning property. I know multiple immigrants who are talking about returning to their previous country, or trying to figure out getting legal in the US.

There isn't enough housing nor enough being built. Discussing immigration is not allowed, socially speaking.


the thing is they blame immigrants but won't do anything against the real driver of housing unaffordability, the collusion of regulators and corporations offering large liquidity to housing market.

its breaking apart. the city of vancouver is broke and basic services are being scaled back.

theres also a large brain drain here. only idiots work for vancouver startups/companies that tries to gaslight you that sunshine tax is real.


It's not about vancouver. Everyone who wants to live in vancouver knows what they're getting into. It's the fact that at the current rate of immigration they're adding a small city worth of people every single year into the middle of a healthcare service crisis, a housing crisis, an infrastructure and services crisis, etc, and there are no plans to deal with any of those issues.

Of course there's a brain drain going on, but in terms of the average canadian that's not the issue. the issue is that Canada's immigration policy far outpaces its ability to provide for canadians and immigrants, and the government doesn't care that their immigration policy is seriously injuring the quality of life for people already living in canada.

It doesn't affect the people in government because their politics makes questioning immigration targets and policies verboten and you could be ejected from polite society if you do. If you have a problem with their policies and the massive crisis its introduced to the country its obviously because of racism, right?

It literally has nothing to do with vancouver on its own.


that's an awful long winded way say you blame immigrants for problems that have long existed before their arrival


thanks for proving my point.


Why do you say that? I'm quite happy paying what I do renting in Vancouver. I usually find that people who actively want out of BC either desperately want to own property and can't, don't have any hobbies that suit the west coast, or can't make friends


It's a great weigh-station for immigrants who want to eventually move to the US. Canada is easier to immigrate to as a high-skilled person.


> Canada is easier to immigrate to as a high-skilled person.

False.

> It's a great weigh-station for immigrants who want to eventually move to the US.

Many do not, they move back home.


Many people who don't win the H-b lottery, or who do win the H1-b lottery but get turned down for an H1-b for other reasons, end up heading to Canada. Canada has a simple points-based residency system and it is very easy to get enough points if you have a bachelor degree and speak english fluently (already gets you a TON of points).


you realize there are many many affluent immigrants who do not participate in the labor market but pay taxes, invest, and otherwise provide liquidity for the locals holding real estate portfolios?


I believe this is a Russian hack. Some group took credit for it.

It's an act of war.

I'm wary we may not find out for a long time.

Debit cards were out, huge swaths of wireless and internet, this was a huge failure.


I always assume incompetence before malice, for Rogers, that goes triple. It is a shit network build on layers of shit.

It is owned by a crazy family in a perpetual feud that could have inspired Succession. https://www.ctvnews.ca/entertainment/succession-actor-brian-...

https://www.wsj.com/articles/rogers-family-feud-creates-boar...

Rogers and Bell executives are more interested in their ego than their networks and use their oligopoly money to buy sports teams: https://www.theglobeandmail.com/sports/no-love-for-blue-jays...

Last time this happened, they blamed Ericsson, but let’s face facts, they aren’t competent: https://mobilesyrup.com/2021/04/20/nationwide-rogers-outage-...

Their cellular business used to be called Cantel 30 years ago and we called it Can’t Talk.

Their CIO is from Vodaphone. https://about.rogers.com/team_bios/jorge-fernandes/

They have no oversight because the CRTC executive are best friends with the big 3 telcos: https://www.iphoneincanada.ca/carriers/bell/crtc-chair-on-dr...

Interac (Debit) apparently has their primary and backup network services with Rogers. They need a new executive team ASAP.


I too think this is more likely to be a mistake, but incompetence also makes you easier to hack.


Do you have a source you could use to convince readers why you believe this? As written, your source is "trust me, bro"


It's my bad, I found the quote - it was about the Canadian outage, but at the bottom they wrote:

"Hours before the Rogers outage began, an apparent cyberattack took down the U.S. Congress website. According to Fox News, the pro-Russia hacking group Killnet took credit for crashing the website around 10 p.m. Thursday evening."

I didn't see the first part, i.e. hackers taking credit for the US Congress attack.

So I don't see anyone actually taking credit for it.

That said - this major attack is happening 1) at the very same moment US Congress is attacked, and, 2) Canada is at this very moment deciding on whether or not to return gas turbines to Russia as part of a maintenance agreement. The Turbines are part of the 'banned goods' but Germany wants the gear to go back to Russia for a key pipeline. Canada announced today they will send the gear to Russia.

So it's hugely speculative, but I'm still inclined to believe this was a hack - it's just too hugely suspicious.

[1] https://theparadise.ng/canada-suffering-massive-nationwide-i...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: