Garmin server issues

[OtayBW]

Originally Posted by trener1

Just plugged my Vivosport into my laptop and I am still getting the "server down" message on the Garmin Express app.

It also still says "down for maintenance" on the Junior app

The check is waiting to clear....

[adrien]

I get that for us, it's mostly and inconvenience. The more worrying thing is those who rely on these, notably for aviation.

I do hope that this will be a bit of a wake up to the broader industry on security protocols.

[peanutgallery]

Usually, it takes an incident like this coupled with an airplane falling out of the sky or a cruise ship getting stuck like Shakleton. Due to the lack of travel because of the pandemic...it's going to be a while

Originally Posted by adrien

I get that for us, it's mostly and inconvenience. The more worrying thing is those who rely on these, notably for aviation.

I do hope that this will be a bit of a wake up to the broader industry on security protocols.

[benb]

Originally Posted by ColnagoC59

Leave the world of banking and insurance and i think you'll find that even in this day and age many companies under-invest in disaster recovery and data security. it's always something they'll address next year. it's not an expense that directly benefits the bottom line "today". Bad actors such as the ones purportedly behind this attack take advantage of this short termism

My experience working at a vendor is banking & insurance are the most likely industries to have nigh incompetent IT... they have a lot of process but they seem to always have the bottom of the barrel when it comes to IT and engineers. They mostly get to where they're going to by raining money down on consultants/contractors/vendors and demanding the sky. But it's pure pain the entire way. Either they can't attract good employees or they're too cheap to hire the right people, but then they blow money like crazy to external companies to make up for it.

It's a constant facepalm of "what! you are a senior principal grand dragon wizard IT engineer and you need us to hand hold you for that???" Stuff like wanting a remote session for us to type in the serial # into the software for them, or we're trying to debug something and we ask them to list the files in a directory on a server and they tell us they don't know how to do that and we'll need to take control and show them how.

Now I have also gotten the impression that Garmin is stuck with the same level of people for some reason.. pay, their HQ location, who knows.

People (not you, in general) also need to keep in mind this is not about "what software garmin buys". They develop(ed) a ton of their own public facing software. Running connect is nothing like buying a message board software or blog package and following the instructions on whatever else you need to buy/lease to get the website up and running... They had to build their own stuff and you have to secure that yourself. I work for a tool vendor and that is specifically what we help with, but we don't fix anything for you.. we just tell you about it. Whoever is developing the software is 100% responsible for actually acting on what the security software reveals.

Garmin is also less likely to have been "pwned" by random ransomware... if hacker was involved it would have been more likely to be a specific custom developed hack.. you don't hack custom software with off the shelf stuff, but Connect in particular is huge.. tons of public APIs, tons of different pages in the webapp. Anyone in the world can open up an account and/or buy a Garmin device to get an account and then start probing the software for weaknesses.

It's just as likely this was a data center fire or an Air conditioning failure or something and they didn't have a proper backup data center and now they've having to replace servers and reload from backup. Nobody knows.. some of that stuff is why so many companies elect to put stuff like Connect in the cloud.

[stev0]

looks like some syncing is starting to happen. and...Strava seems to be adding activities I already uploaded manually for some reason.

[Latestart]

Finance and insurance are regulated industries. They have a hard time getting good people because the work is *hard*, there are lots of external compliance issues (Sarbox), you are a high-priority hacking target so lots of testing is needed to do a decent job and the rate of change is slow. If you F*ck Up, it will actually kill of revenue - Imagine this outage at BofA?

In addition, most of the F&I companies are big and public, so you won't get rich on the equity. Asymmetric risk - a good day is when nothing bad happens...

The typical 'rock star coder' from a top 20 school has no interest, and with good reason. Same reason why the best mechanics don't work at a chain muffler shop...

On the other hand, Garmin *should* be able to spend for decent crew. They are not in a regulated industry (yet) and people buy the hardware not the software, so they can take some risk to innovate. But I would guess the CISO and the CIO will both be 'on the street' after this debacle...

Originally Posted by benb

My experience working at a vendor is banking & insurance are the most likely industries to have nigh incompetent IT... they have a lot of process but they seem to always have the bottom of the barrel when it comes to IT and engineers. They mostly get to where they're going to by raining money down on consultants/contractors/vendors and demanding the sky. But it's pure pain the entire way. Either they can't attract good employees or they're too cheap to hire the right people, but then they blow money like crazy to external companies to make up for it.

It's a constant facepalm of "what! you are a senior principal grand dragon wizard IT engineer and you need us to hand hold you for that???" Stuff like wanting a remote session for us to type in the serial # into the software for them, or we're trying to debug something and we ask them to list the files in a directory on a server and they tell us they don't know how to do that and we'll need to take control and show them how.

Now I have also gotten the impression that Garmin is stuck with the same level of people for some reason.. pay, their HQ location, who knows.

People (not you, in general) also need to keep in mind this is not about "what software garmin buys". They develop(ed) a ton of their own public facing software. Running connect is nothing like buying a message board software or blog package and following the instructions on whatever else you need to buy/lease to get the website up and running... They had to build their own stuff and you have to secure that yourself. I work for a tool vendor and that is specifically what we help with, but we don't fix anything for you.. we just tell you about it. Whoever is developing the software is 100% responsible for actually acting on what the security software reveals.

Garmin is also less likely to have been "pwned" by random ransomware... if hacker was involved it would have been more likely to be a specific custom developed hack.. you don't hack custom software with off the shelf stuff, but Connect in particular is huge.. tons of public APIs, tons of different pages in the webapp. Anyone in the world can open up an account and/or buy a Garmin device to get an account and then start probing the software for weaknesses.

It's just as likely this was a data center fire or an Air conditioning failure or something and they didn't have a proper backup data center and now they've having to replace servers and reload from backup. Nobody knows.. some of that stuff is why so many companies elect to put stuff like Connect in the cloud.

[peanutgallery]

What did the MAMILS do to raise the ire of Putin?

He put a crypto henchman on the case

https://nypost.com/2020/07/27/cyberc...garmin-outage/

[zmudshark]

Looks like Garmin paid the ransom.

https://news.sky.com/story/garmin-ob...ttack-12036761

[benb]

Originally Posted by zmudshark

Looks like Garmin paid the ransom.

https://news.sky.com/story/garmin-ob...ttack-12036761

Article could be complete BS if it turns out Garmin doesn't run its infrastructure on Microsoft Windows. It's complete speculation as they have no comments/confirmation from Garmin or the hackers they think did it.

They're talking about a Ransomware package for Windows..

I just looked through the engineering job openings Garmin has... you can't tell for sure but the tech stack they are running Connect on does not have any telltale signs they run stuff on Windows. It wouldn't make a lot of sense.
They're running the stereotypical stuff you run on Unix, at least for connect.

The way they are slowly bringing stuff back online would not be consistent with paying the ransom and suddenly getting everything back.

[sg8357]

Originally Posted by benb

Now I have also gotten the impression that Garmin is stuck with the same level of people for some reason.. pay, their HQ location, who knows.

S/W engineering at Garmin,

B+ or better, Aviation
B, Maritime
Interns, bike stuff
IT staff, "lowly contractors"

[thegunner]

it's hard for them to attract good talent, the comp is significantly lower than any of the FAANG companies and the lack of remote working is also a bummer.

most SWEs i know aren't jumping at the opportunity to move to KS (this is not a knock on Kansas, just a misalignment between the industry and the location)

[benb]

I'd bet not that many cyclists/runners/triathlon/hiking obsessed engineers or otherwise want to move there either.

Although I looked up where their HQ is.. it looks flat but otherwise looked like there are probably decent areas to ride nearby.

I do get a huge kick out of the tech news sites being so sure it's ransomware. They all say they have sources.

Usually one of them makes it up, and the others all make the story and the "source" is the first one that made it up.

Garmin did confirm it was a cyber attack but with the magnitude of what they had to take down it's hard to believe it's as simple as a windows ransomware attack.. maybe more like a multi-prong attack that was pretty well researched to be able to attack them multiple different ways.

[doomridesout]

I grew up near their HQ and worked at a shop which was a feeder for their CS team. I would not be surprised if many of their hires are strictly regional convenience. I’m not knocking my buddies who went to work for them, but they’re not drawing from the most talented tech labor pool.

[C40_guy]

Originally Posted by benb

Although I looked up where their HQ is.. it looks flat but otherwise looked like there are probably decent areas to ride nearby.

The riding is surprisingly good in the area. I've done some riding south and west of Kansas City (not too far from Olathe), and more north and west.

Lots of rolling hills. Nicely laid out road grid. Friendly local drivers who wait 50 yards behind you if you're coming up on a hill, and won't even take a wave-by if *they* don't think it's safe to pass.

Friendly local people who would probably ask you if you'd prefer crushed or cubed ice for your waterbottle if you knocked on the door searching for water...

Quite a change from New England riding.

But it's damn hot there this time of year. I have a photo of my $armin bike computer showing 106 degrees. Yea, it was hot.

Course it's in the mid-90s here, right now...

[fkelly]

Originally Posted by unterhausen

are there other apps that will upload activities via wireless? RWGPS, for example?

The Android and IOS apps on rwgps upload data directly to the rwgps site. For Garmin units rwgps uses Garmin connect (or at least used to)so those uploads could be affected.

When I used a Garmin 800 my rides would get stored on both the Garmin site and on rwgps. Garmin served as a useful "backup". (I also downloaded a CSV file from rwgps to my PC as a third level of back lest any rides should get "lost"). With the smartphone app my data sits "naked" on rwgps (backed up to CSV).

The Android upload is instantaneous. By the time I can get up a flight of stairs my ride is waiting for me on the computer.

But no site is immune to ransomware ... which usually involves an element of social engineering.