Some days ago I was waiting for a Windows server (which is doing nothing but hosting a dozen VMs using Hyper-V to do its quarterly updates and wondered “why are they only done four times a year and why do they have to take hours”?

Simple answer by the IT department responsible for the machine: “Updates take long and have been requiring constant attention/supervision in the past. Sometimes something was breaking or the systems did not reboot after updating or the applications/payload did not launch. Third-party software has been breaking often (e.g. Veeam). Updating servers is always considered a high risk and thus avoided as much as possible.”

Wait. What?

We’re always considering regular (and timely!) system updates a matter of mitigating security risks and now the department who is supposed to consider it their near religious duty tells me “mitigating security risks is considered a high risk itself”? There must be something wrong. Yet we all have accepted that stance since the day in the ancient past where some badly ordered system updates cause us to restore a server from backup media.

Today I needed to take a look at some public sources and stumbled over the ChromeOS documentation repository which I have never seen before (and am wondering so much about the reasons that I’ll ask Google to comment on hiding them so well. The Whitepaper “Security in ChromeOS” caught my attention (even if it was just for its clear message that was obviously not just intended for their own developers).

The section

• Users should not have to worry about “patching” their systems for security bugs. Updates (including security updates) should be seamless.
  • This means that maintaining your security posture is simple: reboot and you‘re up-to-date. ChromeOS is designed to boot quickly, which helps ensure staying up-to-date isn’t a burden for the user.

immediately caught my attention. This is not just an announcement of a nice to have; it is hitting the nail on the head. You have to (be able to) trust updates not to take your device offline longer than necessary (and certainly not for an extended period of time, to never change system behavior in undocumented and clearly stated ways and to be a simple as possible. This is even more important in OT environments where computers are just “another part of machinery”. If an office IT system fails and you did your job well most of the time nobody will even notice that something went pear shaped. The same happening to an OT system usually means physical production is halted and in the worst cas the vendor has to be called in to fix the problem as the real-time state of the system is important and cannot be restored from backups. This all comes down to “the risk to the most important goal of OT systems (availability) from security improvements is perceived as higher than the risk of malware having negative effects”.

OT vendors still explicitly exclude most responsibility of updating systems unless they have extensively tested updates, often taking months to permit application of patches while leaving the responsibility for mitigations of security risks to the clients (contacted about the known vulnerabilities in their ancient (several years out of date even in relation to the rather outdated application itself) versions of JBoss at the base of their entire application an MES vendor replied that “it is a known risk” but that their “application was meant to be kept on an isolated system and not connected to a network”).

Something has to change there…

(PS: I am running workplaces on ChromeOS and can testify that updating the systems is as easy as getting a notification about an update having become available, saving the things I was working on if necessary (many do that on their own) and booting the machine. The system will be back by the time I’m returning with a cup of coffee.)