PDA

View Full Version : Garmin server issues


adrien
07-23-2020, 11:00 AM
Ride this morning didn't sync, so I checked online.

Servers apparently are all down and there's speculation that it's a ransom attack.

https://news.ycombinator.com/item?id=23926289

Makes me realize I'd love the option to just update to the phone directly, rather than relying on the servers. I'm guessing when they fix this there'll be a brownout on Strava, too.

kppolich
07-23-2020, 11:08 AM
First World Problem.

Bostic
07-23-2020, 11:34 AM
Saw it on my Sysadmin forums I frequent.

unterhausen
07-23-2020, 12:16 PM
can't ride, because if it doesn't upload, the ride didn't happen.


I was going to zwift today anyway, that is unaffected.

retropean
07-23-2020, 03:12 PM
I think the ransomware attack has been confirmed.

https://www.zdnet.com/article/garmin-services-and-production-go-down-after-ransomware-attack/

If you sync via a USB, open your garmin's folder that pops up when you connect it, go to the folder /data/, find your activity's .fit file (sort by modified date), and upload it manually to Strava (upload->file). :banana:

trener1
07-23-2020, 03:23 PM
Yeah you can upload to Strava no problem.
I'll also say that dismissing this as a first world problem is kind of missing the point.
If they can hold a tech company hostage, they can just as easily shut down your bank, or your cities water supply, or the Hospital down the road.
The way I see it it's actually a pretty serious thing, even though didn't affect my ride.

kppolich
07-23-2020, 04:03 PM
Politely, any public website can be scraped and attacked which can bring them down. The servers going down and being unreachable is a better scenario than them staying up while under attack and being compromised while people snoop around and grab data. This literally happens every day, and if a company doesn't have the correct security in place to mitigate and deflect these attacks these things happen.

Garmin in particular, with various API's in play is probably more exposed than others due to the sheer amount of Integrations they have with 3rd parties. Strava, for example is cutting off some of those 3rd parties and getting rid of their open API will be protected against at least some of this.

The security of anything is only as strong as the person with the key. More doors in and out mean more keys floating around.

The issue at hand is that a $200 cycling computer that not syncing via bluetooth to your smart phone , via wifi to your home network is the definition of a first word problem.

You can plug your garmin into the computer, and grab the .fit file and manually upload it to strava in 60 seconds to see your results if it is indeed that high of a priority.

unterhausen
07-23-2020, 04:26 PM
I figure garmin is still running a pirated version of winnt Server 2001, and that's why they were vulnerable.

C40_guy
07-23-2020, 05:08 PM
The servers going down and being unreachable is a better scenario than them staying up while under attack and being compromised while people snoop around and grab data. This literally happens every day, and if a company doesn't have the correct security in place to mitigate and deflect these attacks these things happen.


Shoot, I get reports on a weekly basis of an attack on my home router.

Frankly, I can only imagine how many home networks are left wide open...

phoenix
07-23-2020, 05:17 PM
Ah, this is why my ride and walk didn’t upload this morning. Technology :banana:

FlashUNC
07-23-2020, 05:17 PM
Totally sounds like ransomware. Awkward.

Elefantino
07-23-2020, 06:01 PM
Definitely ransomware and they're going to be down for a while.

DfCas
07-23-2020, 06:44 PM
That didn't answer my Tacx support email anyway. Good riddance.

gbcoupe
07-23-2020, 07:13 PM
I think it's been 4 years since I've uploaded my Garmin 200. Does what I need it to do. Don't need print-outs to tell me I'm not nearly as fast as I used to be.

unterhausen
07-23-2020, 07:32 PM
I never used to upload my rides, but now that the 830 does it automagically, I find it useful. Going to be annoying until they fix this.

tctyres
07-24-2020, 07:19 AM
US tech company.
Happens the day after US forces Chinese embassy closed in Houston.

Probably not coincidence.

AngryScientist
07-24-2020, 07:27 AM
And just when i sprang for a new solar fenix!

AngryScientist
07-24-2020, 07:28 AM
The issue at hand is that a $200 cycling computer that not syncing via bluetooth to your smart phone , via wifi to your home network is the definition of a first word problem.

You can plug your garmin into the computer, and grab the .fit file and manually upload it to strava in 60 seconds to see your results if it is indeed that high of a priority.

why so surly? this is a cycling forum after all, and talking about cycling stuff is why we're here.

trener1
07-24-2020, 07:43 AM
Interesting point about the Chinese connection.

BTW It's more then just cycling, though yes I rode Zwift yesterday, so that uploads to Strava automatically.
However I also track my sleep, my sons daily activity (via a Vivofit Jr, which is a great watch BTW if any of you have young children) and other metrics.
Yes granted these are not life and death situations but it's not just that rides can't be posted on Strava (as you pointed out, they still can).

Ti_on_Steel
07-24-2020, 12:17 PM
I think the cyber security practices of a company that holds a huge amount of our location, travel patterns, and personal biometric and health data is something we should all be aware of. Not sure what was lost here, but I hope we find out.

azrider
07-24-2020, 12:19 PM
Hilarious [emoji1787]

https://uploads.tapatalk-cdn.com/20200724/370fd1e855081d74c9e97e5c9924da97.jpg


Sent from my iPhone using Tapatalk

unterhausen
07-24-2020, 12:24 PM
Garmin was jealous of all the attention Wahoo was getting about their devices not uploading and decided to go big.

Funny thing about shutting down a Chinese consulate is that they mostly rely on NOC agents (people not at consulates), so some of them might actually work for Garmin.

many_styles
07-24-2020, 02:00 PM
The issue at hand is that a $200 cycling computer that not syncing via bluetooth to your smart phone , via wifi to your home network is the definition of a first word problem.

You can plug your garmin into the computer, and grab the .fit file and manually upload it to strava in 60 seconds to see your results if it is indeed that high of a priority.

You can also resort back to friction shifters too. It certainly works, but the user experience is much better with index shifting.


Sent from my iPad using Tapatalk

benb
07-24-2020, 02:11 PM
They’re having the biggest outage of the time I’ve had one of their products!

Every online service is down and the status page says they have no phones, email, or chat capability!

https://connect.garmin.com/status/

AngryScientist
07-24-2020, 02:13 PM
They’re having the biggest outage of the time I’ve had one of their products!

Every online service is down and the status page says they have no phones, email, or chat capability!

https://connect.garmin.com/status/

you got merged there big ben.

yea, it's a giant mess. this will be interesting to watch.

benb
07-24-2020, 02:16 PM
I hope their aviation assets are segregated out to a different facility.

If this is just a random hacker taking them down it’s pretty pathetic their network was that insecure.

The cycling related stuff might be frivolous but they do have some products that are a lot more serious and could have safety or national security implications.

AngryScientist
07-24-2020, 02:18 PM
I hope their aviation assets are segregated out to a different facility.

If this is just a random hacker taking them down it’s pretty pathetic their network was that insecure.

The cycling related stuff might be frivolous but they do have some products that are a lot more serious and could have safety or national security implications.

no such luck...

https://www.digitaltrends.com/news/garmin-ransomware-service-outage/

Users with Garmin wearables have been unable to sync their data, and those using the aviation navigational equipment are no longer able to meet Federal Aviation Administration requirements for flights.

benb
07-24-2020, 02:23 PM
Absolutely ridiculous that the aviation stuff wasn’t locked up tight in a separate setup.

There have been a lot of signs they had a lot of software issues for a long time but I always expected this stuff was just on the consumer side.

There are a lot of Garmin Nautical units too but I think they’re generally less dependent on network access.

Still if this blocks getting aviation or nautical chart updates or anything else like that their are going to be some really angry folks.

That’s the kind of thing I might expect SLAs and such covering downtime.

kppolich
07-24-2020, 02:31 PM
-On average 30,000 new websites are hacked every day.
-Hackers create 300,000 new pieces of malware daily.

From:
https://www.webarxsecurity.com/website-hacking-statistics-2018-february/#:~:text=On%20average%2030%2C000%20new%20websites% 20are%20hacked%20every%20day.&text=These%2030%20000%20sites%20are,that%20are%20u nwittingly%20distributing%20malware.

thegunner
07-24-2020, 02:32 PM
Interesting point about the Chinese connection.

why? Garmin's chairman is taiwanese and the majority of their production is in Taiwan.

I'm pretty sure there's no connection to be drawn here.

benb
07-24-2020, 02:42 PM
-On average 30,000 new websites are hacked every day.
-Hackers create 300,000 new pieces of malware daily.

From:
https://www.webarxsecurity.com/website-hacking-statistics-2018-february/#:~:text=On%20average%2030%2C000%20new%20websites% 20are%20hacked%20every%20day.&text=These%2030%20000%20sites%20are,that%20are%20u nwittingly%20distributing%20malware.

A lot of that is amateur websites & home users getting hacked though.

Garmin is a tech company full of engineers, this is kind of a big deal. It's not even that they run a website, they build a lot of software.

They most likely have a full time security team, if they don't it's a little ridiculous.

It'd be interesting to know what products they use.. I work in Cybersecurity.. it's not like I've ever had the customer lists memorized but no product I've ever worked on has ever had them as a customer IIRC and at least 2 of the last 3 would have been highly relevant to Garmin.

pooneej
07-24-2020, 03:24 PM
Polar fitness FTW !!!!!!!!!!! lol /jk

ColnagoC59
07-24-2020, 07:24 PM
cable upload works. its not so bad. i just hope account data isn't compromised.

CMiller
07-24-2020, 07:46 PM
First World Problem.

99% of problems posted on this forum are first world problems including probably any thread you or I started.

ColnagoC59
07-24-2020, 07:51 PM
A lot of that is amateur websites & home users getting hacked though.

Garmin is a tech company full of engineers, this is kind of a big deal. It's not even that they run a website, they build a lot of software.

They most likely have a full time security team, if they don't it's a little ridiculous.

It'd be interesting to know what products they use.. I work in Cybersecurity.. it's not like I've ever had the customer lists memorized but no product I've ever worked on has ever had them as a customer IIRC and at least 2 of the last 3 would have been highly relevant to Garmin.

Leave the world of banking and insurance and i think you'll find that even in this day and age many companies under-invest in disaster recovery and data security. it's always something they'll address next year. it's not an expense that directly benefits the bottom line "today". Bad actors such as the ones purportedly behind this attack take advantage of this short termism

If I was a stock or bond holder I'd immediately want to know what security and regular audits were conducted and what they found. Once this is over I suspect heads will roll and whether they have a reporting structure that takes data security seriously. My guess based on the duration of this outage and reading the new and my own intuition is that Garmin does not.

trener1
07-25-2020, 07:16 AM
Considering that we are going on day 3 now, I would say that this is pretty huge.
Usually if a company like this is down for 30 minutes it's a problem.

unterhausen
07-25-2020, 08:18 AM
I give even odds that there are people at polar and wahoo that have been meeting all weekend about this, even if their online presence isn't quite so ambitious as garmin's

AngryScientist
07-25-2020, 10:00 AM
Considering that we are going on day 3 now, I would say that this is pretty huge.
Usually if a company like this is down for 30 minutes it's a problem.

Yea this is massive and there will be many hours of post crisis analysis on this.

I work in the heavy industry/power/petrochemical business and we had a client hacked with ransomware; and though I don’t begin to understand the particulars, they were essentially told by an expert firm: under no circumstances pay the ransom; they basically canned all their hardware and started over. Took about a day and a bazzilion man hours of work.

This stuff is serious

joosttx
07-25-2020, 10:15 AM
Funny I haven’t noticed the quality of their service being worse than usually.

unterhausen
07-25-2020, 10:28 AM
Took about a day and a bazzilion man hours of work.

This stuff is seriousThat's pretty impressive they could do that. In my lab, it seems like recovering one machine from a mobo failure takes forever.

A single point failure taking out an entire company seems mind boggling to me. I am waiting to find out what bad software they were running. Surprised that no ex-employees are blabbing about it already.

73Bronco
07-25-2020, 10:30 AM
Funny I haven’t noticed the quality of their service being worse than usually.That's funny, thanks for the laugh!

Sent from my SM-G960U using Tapatalk

Elefantino
07-25-2020, 05:37 PM
It's the Russians. (https://cyclingtips.com/2020/07/hackers-demand-10-million-from-garmin-to-release-data/)

This is not a hoax.

unterhausen
07-25-2020, 06:33 PM
are there other apps that will upload activities via wireless? RWGPS, for example?

zmudshark
07-25-2020, 08:00 PM
It's the Russians. (https://cyclingtips.com/2020/07/hackers-demand-10-million-from-garmin-to-release-data/)

This is not a hoax.
The last time the Russians were on our side was the Civil War, and then, only if you were on the side of the Union.

Mr. Pink
07-26-2020, 10:50 AM
Wait. Was that when the Germans bombed Pearl Harbor?

zmudshark
07-26-2020, 10:58 AM
https://www.rbth.com/politics_and_society/2017/08/16/what-role-did-russia-play-in-the-us-civil-war_823252

NoMoreParagon
07-26-2020, 11:56 AM
99% of problems posted on this forum are first world problems including probably any thread you or I started.

not sure about that...with Garmin Connect down, if you have to do a hard reboot, you can't load any route. Say you are in the middle of a bike packing trip, unit freezes, and by rebooting you lose all your routes. Only way to load routes wirelessly is strava or komoot apps. and w/o GC, you can't load apps.
So its kinda of a big deal. This is the high season of bike packing and many, including me, rely on Garmin Navigation to find the right route, often on solo trips and potentially on remote areas.

Keith A
07-26-2020, 12:07 PM
Not sure if anyone knows the answer to this, but I wonder if I will end up with duplicate entries in Strava once Garmin is back online. I went ahead and updated Strava directly with the files from my Garmin, and am guessing that the Garmin Connect might update Strava too once things are working again.

54ny77
07-26-2020, 12:14 PM
Anyone know if this Garmin issue impacts marine systems? I can't find news on that. Saw aviation down also though.

(Got a buddy on a big sailing trip, but from what I recall he has Raymarine gear as primary nav tools.)

thegunner
07-26-2020, 12:34 PM
not sure about that...with Garmin Connect down, if you have to do a hard reboot, you can't load any route. Say you are in the middle of a bike packing trip, unit freezes, and by rebooting you lose all your routes. Only way to load routes wirelessly is strava or komoot apps. and w/o GC, you can't load apps.
So its kinda of a big deal. This is the high season of bike packing and many, including me, rely on Garmin Navigation to find the right route, often on solo trips and potentially on remote areas.

bike packing is a 1st world thing mostly.

phoenix
07-26-2020, 12:38 PM
I usually upload to my phone via BT. I finally had to go dig up a data cable! The horror!

NoMoreParagon
07-26-2020, 12:40 PM
bike packing is a 1st world thing mostly.


And? What does it have to do with safety when riding?
I don’t care where you from but if you tell me you lost in a remote place and you rely on GPS and it doesn’t work, well that’s a problem for me.

thegunner
07-26-2020, 12:42 PM
you gave two other ways to load routes though, I was mostly commenting on how these are all 1st world problems.

weisan
07-26-2020, 12:57 PM
Common theme:
The awesomeness of shared networks, global travel, open interconnectedness - all come with associated risks and vulnerabilities

Like Corona, this is just a prelude.

gdw
07-26-2020, 01:42 PM
not sure about that...with Garmin Connect down, if you have to do a hard reboot, you can't load any route. Say you are in the middle of a bike packing trip, unit freezes, and by rebooting you lose all your routes. Only way to load routes wirelessly is strava or komoot apps. and w/o GC, you can't load apps.
So its kinda of a big deal. This is the high season of bike packing and many, including me, rely on Garmin Navigation to find the right route, often on solo trips and potentially on remote areas.


If people choose to travel in remote areas and don't carry backup maps and a compass they should stay closer to home and avoid unfamiliar routes. Too many folks dabbling in backcountry/bikepacking adventures aren't prepared to deal with the malfunctioning of their tech toys or equipment these days.

mcfarton
07-26-2020, 01:43 PM
Not sure if anyone knows the answer to this, but I wonder if I will end up with duplicate entries in Strava once Garmin is back online. I went ahead and updated Strava directly with the files from my Garmin, and am guessing that the Garmin Connect might update Strava too once things are working again.


Strava won’t double post the same entry


Sent from my iPhone using Tapatalk

2metalhips
07-26-2020, 01:54 PM
If people choose to travel in remote areas and don't carry backup maps and a compass they should stay closer to home and avoid unfamiliar routes. Too many folks dabbling in backcountry/bikepacking adventures aren't prepared to deal with the malfunctioning of their tech toys or equipment these days.

Word^

NoMoreParagon
07-26-2020, 02:01 PM
Sorry if you too cool for electronic navigation. But let me explain you that a Garmin is not a toy. It is used in both high end applications in aviation and boating. I am not an idiot and I have back up if a failure occurs. But the experience of exploring new areas with the aid of a turn by turn makes the whole experience more enjoyable as I can avoid stopping and check maps on trails.

jb_11
07-26-2020, 02:08 PM
If people choose to travel in remote areas and don't carry backup maps and a compass they should stay closer to home and avoid unfamiliar routes. Too many folks dabbling in backcountry/bikepacking adventures aren't prepared to deal with the malfunctioning of their tech toys or equipment these days.

Along with your point, I think I read that Garmin's InReach service is down as well. That's probably some peoples last/best form of remote communication in an emergency.

Sucks all around.

rkhatibi
07-26-2020, 02:10 PM
I am waiting to find out what bad software they were running.

It's almost never the software as everyone runs the same stuff above a certain scale or outside of specific niches. It is however almost always the process used to manage the development of the system (software and infrastructure). Teams incentivized to chase new features don't prioritize breaking organically grown systems into structured components, testing disaster recovery, etc.

kppolich
07-26-2020, 03:28 PM
I wonder what happened as Garmin hasn't been too busy developing any new features or products. It's not like the pile of Tech Debt around security could have been that high? Or could it?

They do integrate with quite a few vendors, likely an inside job there.

tsarpepe
07-26-2020, 09:23 PM
99% of problems posted on this forum are first world problems including probably any thread you or I started.

That doesn't make it right.

jimoots
07-27-2020, 05:23 AM
Back online

peanutgallery
07-27-2020, 06:13 AM
Then some Russian hacker is now $10 million richer

Back online

R3awak3n
07-27-2020, 06:42 AM
Yeah seems to be back, I got a strava message saying that a few of my things uploaded overnight.

jimoots
07-27-2020, 06:43 AM
Then some Russian hacker is now $10 million richer

I’m not saying you’re wrong. Neither of us are “in the room”.

But given that the execs will be on the hook for jail time if they pay up - plus given how much press this has gotten - i’d be seriously surprised if they paid the ransom.

peanutgallery
07-27-2020, 06:50 AM
Venmo and Rubles:)


I’m not saying you’re wrong. Neither of us are “in the room”.

But given that the execs will be on the hook for jail time if they pay up - plus given how much press this has gotten - i’d be seriously surprised if they paid the ransom.

schwa86
07-27-2020, 07:07 AM
Maybe some things working and some not, my forerunner still not syncing, phone app says "down for maintenance"

Keith A
07-27-2020, 07:25 AM
I was able to sync with my old 500 and it sort of worked. It said the sync was complete, but when it tried to check for new updates, it reported an error about not being able to communicate with their servers.

To answer my question, it doesn't appear that I have any duplicate entries on Strava for the rides I uploaded directly.

Keith A
07-27-2020, 07:26 AM
Maybe some things working and some not, my forerunner still not syncing, phone app says "down for maintenance"Here is a page of their status...
https://connect.garmin.com/status/

trener1
07-27-2020, 08:11 AM
Maybe some things working and some not, my forerunner still not syncing, phone app says "down for maintenance"

Yeah same here, when I go to garmin connect on my laptop it comes up, but when I try to connect my Vivosport to my phone app the app still says servers are down and it isn't loading any data and the same goes for my sons Vivopsort Jr.

Elefantino
07-27-2020, 08:13 AM
Here is a page of their status...
https://connect.garmin.com/status/
Maybe they only paid $5 million.

peanutgallery
07-27-2020, 08:14 AM
My Fenix synced this AM for the first time in days. Looks like all my data is available. Correct that, only some

peanutgallery
07-27-2020, 08:18 AM
Bunch of MBAs got involved, not the teck folks

Maybe they only paid $5 million.

trener1
07-27-2020, 08:19 AM
Just plugged my Vivosport into my laptop and I am still getting the "server down" message on the Garmin Express app.

It also still says "down for maintenance" on the Junior app

OtayBW
07-27-2020, 08:38 AM
Just plugged my Vivosport into my laptop and I am still getting the "server down" message on the Garmin Express app.

It also still says "down for maintenance" on the Junior app
The check is waiting to clear....:rolleyes:

adrien
07-27-2020, 09:12 AM
I get that for us, it's mostly and inconvenience. The more worrying thing is those who rely on these, notably for aviation.

I do hope that this will be a bit of a wake up to the broader industry on security protocols.

peanutgallery
07-27-2020, 09:29 AM
Usually, it takes an incident like this coupled with an airplane falling out of the sky or a cruise ship getting stuck like Shakleton. Due to the lack of travel because of the pandemic...it's going to be a while:)

I get that for us, it's mostly and inconvenience. The more worrying thing is those who rely on these, notably for aviation.

I do hope that this will be a bit of a wake up to the broader industry on security protocols.

benb
07-27-2020, 09:37 AM
Leave the world of banking and insurance and i think you'll find that even in this day and age many companies under-invest in disaster recovery and data security. it's always something they'll address next year. it's not an expense that directly benefits the bottom line "today". Bad actors such as the ones purportedly behind this attack take advantage of this short termism



My experience working at a vendor is banking & insurance are the most likely industries to have nigh incompetent IT... they have a lot of process but they seem to always have the bottom of the barrel when it comes to IT and engineers. They mostly get to where they're going to by raining money down on consultants/contractors/vendors and demanding the sky. But it's pure pain the entire way. Either they can't attract good employees or they're too cheap to hire the right people, but then they blow money like crazy to external companies to make up for it.

It's a constant facepalm of "what! you are a senior principal grand dragon wizard IT engineer and you need us to hand hold you for that???" Stuff like wanting a remote session for us to type in the serial # into the software for them, or we're trying to debug something and we ask them to list the files in a directory on a server and they tell us they don't know how to do that and we'll need to take control and show them how.

Now I have also gotten the impression that Garmin is stuck with the same level of people for some reason.. pay, their HQ location, who knows.

People (not you, in general) also need to keep in mind this is not about "what software garmin buys". They develop(ed) a ton of their own public facing software. Running connect is nothing like buying a message board software or blog package and following the instructions on whatever else you need to buy/lease to get the website up and running... They had to build their own stuff and you have to secure that yourself. I work for a tool vendor and that is specifically what we help with, but we don't fix anything for you.. we just tell you about it. Whoever is developing the software is 100% responsible for actually acting on what the security software reveals.

Garmin is also less likely to have been "pwned" by random ransomware... if hacker was involved it would have been more likely to be a specific custom developed hack.. you don't hack custom software with off the shelf stuff, but Connect in particular is huge.. tons of public APIs, tons of different pages in the webapp. Anyone in the world can open up an account and/or buy a Garmin device to get an account and then start probing the software for weaknesses.

It's just as likely this was a data center fire or an Air conditioning failure or something and they didn't have a proper backup data center and now they've having to replace servers and reload from backup. Nobody knows.. some of that stuff is why so many companies elect to put stuff like Connect in the cloud.

stev0
07-27-2020, 10:48 AM
looks like some syncing is starting to happen. and...Strava seems to be adding activities I already uploaded manually for some reason.

Latestart
07-27-2020, 11:22 AM
Finance and insurance are regulated industries. They have a hard time getting good people because the work is *hard*, there are lots of external compliance issues (Sarbox), you are a high-priority hacking target so lots of testing is needed to do a decent job and the rate of change is slow. If you F*ck Up, it will actually kill of revenue - Imagine this outage at BofA?

In addition, most of the F&I companies are big and public, so you won't get rich on the equity. Asymmetric risk - a good day is when nothing bad happens...

The typical 'rock star coder' from a top 20 school has no interest, and with good reason. Same reason why the best mechanics don't work at a chain muffler shop...

On the other hand, Garmin *should* be able to spend for decent crew. They are not in a regulated industry (yet) and people buy the hardware not the software, so they can take some risk to innovate. But I would guess the CISO and the CIO will both be 'on the street' after this debacle...




My experience working at a vendor is banking & insurance are the most likely industries to have nigh incompetent IT... they have a lot of process but they seem to always have the bottom of the barrel when it comes to IT and engineers. They mostly get to where they're going to by raining money down on consultants/contractors/vendors and demanding the sky. But it's pure pain the entire way. Either they can't attract good employees or they're too cheap to hire the right people, but then they blow money like crazy to external companies to make up for it.

It's a constant facepalm of "what! you are a senior principal grand dragon wizard IT engineer and you need us to hand hold you for that???" Stuff like wanting a remote session for us to type in the serial # into the software for them, or we're trying to debug something and we ask them to list the files in a directory on a server and they tell us they don't know how to do that and we'll need to take control and show them how.

Now I have also gotten the impression that Garmin is stuck with the same level of people for some reason.. pay, their HQ location, who knows.

People (not you, in general) also need to keep in mind this is not about "what software garmin buys". They develop(ed) a ton of their own public facing software. Running connect is nothing like buying a message board software or blog package and following the instructions on whatever else you need to buy/lease to get the website up and running... They had to build their own stuff and you have to secure that yourself. I work for a tool vendor and that is specifically what we help with, but we don't fix anything for you.. we just tell you about it. Whoever is developing the software is 100% responsible for actually acting on what the security software reveals.

Garmin is also less likely to have been "pwned" by random ransomware... if hacker was involved it would have been more likely to be a specific custom developed hack.. you don't hack custom software with off the shelf stuff, but Connect in particular is huge.. tons of public APIs, tons of different pages in the webapp. Anyone in the world can open up an account and/or buy a Garmin device to get an account and then start probing the software for weaknesses.

It's just as likely this was a data center fire or an Air conditioning failure or something and they didn't have a proper backup data center and now they've having to replace servers and reload from backup. Nobody knows.. some of that stuff is why so many companies elect to put stuff like Connect in the cloud.

peanutgallery
07-27-2020, 12:01 PM
What did the MAMILS do to raise the ire of Putin?

He put a crypto henchman on the case

https://nypost.com/2020/07/27/cybercriminal-maksim-yakubets-is-the-hacker-linked-to-garmin-outage/

zmudshark
07-27-2020, 01:33 PM
Looks like Garmin paid the ransom.

https://news.sky.com/story/garmin-obtains-decryption-key-after-ransomware-attack-12036761

benb
07-27-2020, 01:48 PM
Looks like Garmin paid the ransom.

https://news.sky.com/story/garmin-obtains-decryption-key-after-ransomware-attack-12036761

Article could be complete BS if it turns out Garmin doesn't run its infrastructure on Microsoft Windows. It's complete speculation as they have no comments/confirmation from Garmin or the hackers they think did it.

They're talking about a Ransomware package for Windows..

I just looked through the engineering job openings Garmin has... you can't tell for sure but the tech stack they are running Connect on does not have any telltale signs they run stuff on Windows. It wouldn't make a lot of sense.
They're running the stereotypical stuff you run on Unix, at least for connect.

The way they are slowly bringing stuff back online would not be consistent with paying the ransom and suddenly getting everything back.

sg8357
07-27-2020, 02:33 PM
Now I have also gotten the impression that Garmin is stuck with the same level of people for some reason.. pay, their HQ location, who knows.

S/W engineering at Garmin,

B+ or better, Aviation
B, Maritime
Interns, bike stuff
IT staff, "lowly contractors"

thegunner
07-27-2020, 02:49 PM
it's hard for them to attract good talent, the comp is significantly lower than any of the FAANG companies and the lack of remote working is also a bummer.

most SWEs i know aren't jumping at the opportunity to move to KS (this is not a knock on Kansas, just a misalignment between the industry and the location)

benb
07-27-2020, 03:03 PM
I'd bet not that many cyclists/runners/triathlon/hiking obsessed engineers or otherwise want to move there either.

Although I looked up where their HQ is.. it looks flat but otherwise looked like there are probably decent areas to ride nearby.

I do get a huge kick out of the tech news sites being so sure it's ransomware. They all say they have sources.

Usually one of them makes it up, and the others all make the story and the "source" is the first one that made it up.

Garmin did confirm it was a cyber attack but with the magnitude of what they had to take down it's hard to believe it's as simple as a windows ransomware attack.. maybe more like a multi-prong attack that was pretty well researched to be able to attack them multiple different ways.

doomridesout
07-27-2020, 03:10 PM
I grew up near their HQ and worked at a shop which was a feeder for their CS team. I would not be surprised if many of their hires are strictly regional convenience. I’m not knocking my buddies who went to work for them, but they’re not drawing from the most talented tech labor pool.

C40_guy
07-27-2020, 03:26 PM
Although I looked up where their HQ is.. it looks flat but otherwise looked like there are probably decent areas to ride nearby.


The riding is surprisingly good in the area. I've done some riding south and west of Kansas City (not too far from Olathe), and more north and west.

Lots of rolling hills. Nicely laid out road grid. Friendly local drivers who wait 50 yards behind you if you're coming up on a hill, and won't even take a wave-by if *they* don't think it's safe to pass.

Friendly local people who would probably ask you if you'd prefer crushed or cubed ice for your waterbottle if you knocked on the door searching for water...

Quite a change from New England riding.

But it's damn hot there this time of year. I have a photo of my $armin bike computer showing 106 degrees. Yea, it was hot.

Course it's in the mid-90s here, right now...

fkelly
07-27-2020, 03:46 PM
are there other apps that will upload activities via wireless? RWGPS, for example?

The Android and IOS apps on rwgps upload data directly to the rwgps site. For Garmin units rwgps uses Garmin connect (or at least used to)so those uploads could be affected.

When I used a Garmin 800 my rides would get stored on both the Garmin site and on rwgps. Garmin served as a useful "backup". (I also downloaded a CSV file from rwgps to my PC as a third level of back lest any rides should get "lost"). With the smartphone app my data sits "naked" on rwgps (backed up to CSV).

The Android upload is instantaneous. By the time I can get up a flight of stairs my ride is waiting for me on the computer.

But no site is immune to ransomware ... which usually involves an element of social engineering.

unterhausen
07-27-2020, 05:21 PM
The Android upload is instantaneous. By the time I can get up a flight of stairs my ride is waiting for me on the computer.
I really got spoiled by this. I hit save on the garmin, phone still on the bike, go inside, open the garage door, and by then the ride is on the web. I have no idea how that happens. It didn't at first, but then started working.

The idea of going back to a cable just doesn't appeal.

Latestart
07-27-2020, 05:58 PM
https://www.bleepingcomputer.com/news/security/garmin-outage-caused-by-confirmed-wastedlocker-ransomware-attack/

Confirmed WastedLocker ransomware attack
A source close to the Garmin incident response and a Garmin employee confirmed to BleepingComputer that the WastedLocker ransomware attacked Garmin.

A Garmin employee told BleepingComputer that they first learned of the attack when they arrived at their office on Thursday morning.

BleepingComputer was told that the Garmin IT department had tried to remotely shut down all computers on the network as devices were being encrypted, including home computers connected via VPN.

After being unable to do so, employees were told to shut down any computer on the network that they had access to.

In a photo of a Garmin computer with encrypted files shared with BleepingComputer, you can see that the .garminwasted extension was appended to the file's name, and ransom notes were also created for each file.

djg21
07-27-2020, 07:37 PM
It had to reinstall Garmin Connect on my iPhone after latest iOS beta. Apart from my Edge not uploading workout data, I could log into Garmin Connect for the past few days. Tonight everything functioned, albeit a little more slowly perhaps.

schwa86
07-27-2020, 08:34 PM
I just looked at the status page posted further up in this thread and all services are currently listed As down. Seems like they are still working on it.

benb
07-27-2020, 08:50 PM
If it hasn't been mentioned already... good time to change your Strava, Training Peaks, Trainer Road, Peloton, Golden Cheetah, etc.. passwords.

This is not a time to trust Garmin that they didn't lose passwords.

C40_guy
07-27-2020, 09:00 PM
I just looked through the engineering job openings Garmin has... you can't tell for sure but the tech stack they are running Connect on does not have any telltale signs they run stuff on Windows. It wouldn't make a lot of sense.
They're running the stereotypical stuff you run on Unix, at least for connect.



It's amazing how much you can learn about a company just by reading the tea leaves, so to speak.

54ny77
07-27-2020, 09:08 PM
Hmm, will a Thomas Bros. map book fit in my jersey pocket?

:D

BobbyJones
07-28-2020, 07:31 AM
Hypothesize all you want, but if you can get to it, so can they.

It's that simple.

benb
07-28-2020, 02:53 PM
My last 4-5 rides synced today after my ride and ended up on Training Peaks.

That's about all I really need.... I log into Garmin Connect about once a year and barely ever make courses and such.

trener1
07-28-2020, 02:54 PM
Is it back down? I haven't synced a ride today but the app isn't connecting to my watch at all today (it was yesterday)

benb
07-28-2020, 03:14 PM
It says it's down when you go in the phone app but the portion of the software that sends the Activities up to the cloud appears to be working.. you just can't read them back in the Garmin App.

I went to plug my Edge into my computer to manually upload to Training Peaks and the rides were already there...

edit: Just now I can go into Garmin Connect on my iPhone and it still has the Maintenance message but I can see the rides there too.

unterhausen
07-28-2020, 03:44 PM
my watch didn't sync using my phone the way it usually does. Plugged it in to charge it and it synced.

Latestart
07-29-2020, 06:08 PM
https://arstechnica.com/information-technology/2020/07/garmans-four-day-service-meltdown-was-caused-by-ransomware/

Keith A
07-29-2020, 06:48 PM
This is just messed up. This is a quote from the article...

"Not only did ransomware last year cost US governments, health care providers, and educational institutions a combined $7.5 billion, the resulting disruptions can cause hospitals to turn away patients seeking emergency care, dangerous meddling of critical infrastructure, and hardships for millions of end users. "

Mark McM
07-29-2020, 07:10 PM
Citing an unnamed number of security sources, Sky News reported that Garmin obtained the decryption key. The report lined up with what the person with direct knowledge told Ars. Sky News said Garmin "did not directly make a payment to the hackers," but didn't elaborate. Garmin representatives declined to provide confirmation that the malware was WastedLocker and if the company paid any sort of ransom.

Anybody want to lay odds whether they ended up paying the ransom (perhaps through a 3rd party)?

zmudshark
07-29-2020, 07:16 PM
Anybody want to lay odds whether they ended up paying the ransom (perhaps through a 3rd party)?
Anyone want to lay odds on whether they will be charged for that crime?

ColonelJLloyd
07-29-2020, 08:01 PM
Anybody want to lay odds whether they ended up paying the ransom (perhaps through a 3rd party)?

Anyone want to lay odds on whether they will be charged for that crime?

I'm gonna go with "they did and they won't".

zmudshark
07-29-2020, 08:48 PM
I'm gonna go with "they did and they won't".

Not taking that bet, but you are correct, I think.

Bazartorie
04-13-2023, 07:26 AM
I stumbled upon this thread and saw that you were having issues with Garmin servers a while back. It's unfortunate to hear that you experienced that problem, but I'm assuming it has been resolved since this thread is quite old now.
However, in case you or anyone else reading this is still having trouble with syncing or relying on servers for their fitness tracking, I wanted to mention that there are alternative options available.
Also, have any of you played 7 Days to Die before? If you're into survival games, I highly recommend checking it out. You can find more information about 7 Days to Die server hosting at 7d2d.net[/URL]. This game really heps me relax after a long day.