The one good reason is that there is nobody skilled left in the office of the supermarket. The days of people with the skills to hit a supermarket HVAC system with a hammer are long gone. When the fridges are on the fritz, the minimum-wage shelf packer goes and packs shelves as far away from the fridge as possible.
While the app is beeping at them to pack more shelves, and they have already used up their only toilet break of the day to trying to stop the resourcing AI chatbot from putting them on triple shifts for the rest of the week.
The trained and retained supermarket employees are gone. The refrigeration service engineer has been replaced by an AI chatbot. We can't scale to qualified people running complex systems on remote air-gapped networks.
IoT platforms provided all the required functionality. Device provisioning (it is not easy to do identity on 'things'). MQTT for service-assisted communication where devices don't need to accept incoming connections. Over-the-air software updates. Over-the-air kernel security updates.
The 'fridge company' and other product manufacturers decided to skip all that and run everything on a raspberry pi with apache, or an ESP32 soldered onto the control board, and call it finished. There are millions of these devices out there.
I just read the article again. It mentions "Danfoss" HVAC controls.
I happen to have a Danfoss HVAC controller here, and its controllable via Modbus. Its an older model, but its my understanding that larger HVAC systems commonly use Modbus.
Modbus systems are multi-drop serial, and the hack likely came in from the server side of one of the companies involved, and the HVAC controllers were all sent defrost commands. Thats my guess anyway.
I have a pump thats Modbus controllable as well, made by a company called Grundfos.
Modbus is the equivalent of VGA connector in displays and projectors. It does have a TCP/IP version as well, but it has no security whatsoever. Or any features besides reading/writing a 16-bit value (or a single bit, charmingly referred to as "coil")
Yes, that is likely. Although Danfoss seems to have their own controllers, I think that you are right, that the attack was not an 'Internet connected fridge', but an HVAC or FMS (Facilities Management System) that was connected to the fridges.
The biggest issue is that many of these IoT devices are badly secured. I spent years trying to convince people to build more secure IoT systems, but they always preferred cheap and easy solutions.
> I spent years trying to convince people to build more secure IoT systems, but they always preferred cheap and easy solutions.
For me this has always been true of systems in general all across the decades that I've worked with technology. It's like pulling teeth to convince anyone in charge that backups or security are even a little bit important until after there's been a costly disaster, then it's all about "Why didn't you warn me this could happen?" - Umm... You mean like every single day for many months until you told me to shut up about it because my paranoia doesn't justify the tiny expense of ensuring you don't lose many tens of thousands of dollars or years of irreplaceable work or important records?
I went through a manual you mentioned in another comment. It listed "High level of security against undesired user
access" as one of the features but it wasn't expanded upon. So I did more searching and found cybersecurity best practices from Danfoss.
They highlight connection through a firewall, mention secure remote access methods, monitoring for data breaches, strong access controls, etc. Reasonable, but these are just recommendations, so the stores could have followed all of them or none at all.
> we searched publicly available internet-wide scanning platforms, including Shodan and Censys, for signatures associated with its management interface. It didn't take long to find what we were looking for. Numerous controllers [2765] were directly accessible from the public Internet.
I get that having networked devices is a very nice to have feature. However, putting them directly to the open internet is bewildering. Any person worth their salt in IT would know that's a bad idea. Were these people never brought into the conversation? Do they connect to the store's WiFi so that IT isn't actually needed to install once the WiFi creds are obtained?
I imagine a very conscientious vendor could easily set up a system that detects if the device is trivially exposed to the internet, causing warning messages and beeps and general pestering.
... And then the customer demands it be turned off, and the customer is always right...
IIoT (Industrial IoT) doesn't have time to deal with the load of bullcrap that cybersecurity is. Half of it is not even about security, but setting up moats and keeping control over users, which is very much not what you want to prioritize when setting up industrial infrastructure, and is rightfully rejected by engineers working there, who are concerned with safety and reliability first and foremost.
Though I'm afraid things are getting worse, or at least it looked so a few years ago, back when I worked in this space. The hot thing in IIoT / Industry 4.0 was basically taking all webshit technologies and deploying them on industrial hardware, to "leverage mature innovation and OSS ecosystem" or such nonsense. The problem (one of many) being, once the engineers were done trimming excessive security measures that negatively impacted safety, they were still left with node.js running on industrial controllers, now even less secure in general than it was originally.
Did you know that buildings have doors that cannot be locked to prevent them from being opened from inside, because otherwise in case of a fire (or in the US, active shooter), people will die if they cannot exit the building?
Did you know that many industrial plants intentionally disable certificate revocation checks, because having a certificate expire on you for whatever reason is a sure-fire way to break the plant (and in some case possibly kill people), and overall a nuisance when trying to keep the plant running, while the threat it protects against can be handled elsewhere in the stack?
Security and safety are very often at odds - not just in emergency, but during regular operations.
Yes, there is little demand for this sort of thing when the potential loss, should it even come, is already insured. Your customer might actually not be users themselves but their insurance companies. It may be a better model to say pitch to them and then have them offer discounts to their clients who are using your tech that mitigates whatever % risk. Someone pitched that car probe to Progressive after all.
There are a bunch of good reasons to connect a commercial refrigeration unit to the Internet (via IoT). Notification when failover to the secondary cooling system happens, notification of when ambient temperature is too high, optimisation of energy consumption (especially after hours).
Doing it insecurely by, for example, having a device with a web server on the fridge is problematic. Not being able to deploy firmware updates. Having too much of the control loop exposed (as opposed to monitoring only), can lead to issues.
Service-oriented industries with costly assets will look to Internet connectivity for competitiveness and efficiency. It needs to be done better.
I didn't expect to spend so much time today thinking about fridges.
I have so many questions. If it's all so connected and monitored and notifies of issues, couldn't it notify anyone that the settings are weird, like continuous 50 degree Celsius defrosting? How much access did the hackers have, apart from temperature controls? Could nobody intervene quickly before the damage was done? Was there a way to "unplug" the fridges or they would just keep frying, so they had to get people across all the stores doing manual resets?
It can do all that IF YOU THINK OF IT and then write the program to do it. If you think of it - which is unlikely because it never happened before (or perhaps it did before but electronics are reliable and so you think it won't ever again).
These are commercial units. There is no plug. There is a circuit breaker in the back room, but general rules is only an electrician is allowed to touch them: if they are off there is always a good reason (until now) and so random staffers touching the panel to turn a breaker on is likely to kill somebody.
Look for "Danfoss AK-SM 720 user manual" and scroll through the pdf.
You'll find that these are quite complex control systems and their data may not be integrated into an operational or security control centre. So if they are not configured for, or resourced (in terms of operators), to handle anomalies, how do they notice and respond?
Without knowing anything about the attack it may be that the controllers were not connected directly to the public Internet, but an on-site VPN that became compromised some other way. Or with a compromised laptop of a technician that connected.
So while you are thinking about fridges today, think about how you would secure and operate control systems across tens of thousands of stores. Would that be easier without Internet connectivity? How do you get qualified and experienced technicians into each (air gapped) store? Another complexity is that many of these systems are part of facilities management systems (HVAC, lighting, security, fire control) and managed or owned by a completely different organisation that is not going to share their data.
reply