Security for IoT – a secure thing! Logging and legal security for critical infrastructures

Hello Andre, hello Corinna and hello Michael – welcome to the IIoT Use Case Podcast. I am very happy that you are here and that you have taken the time. Today, we have taken on the topic of IoT security and want to illustrate it with concrete use cases from practice. I would start with a short round of introductions and hand it over to you, Corinna. Would you like to say a few points about yourself and also about your company – perhaps also about the constellation of CyProtect and sematicon?

Corinna

Hello Madeleine, thank you very much for the invitation and the interview. About me: My name is Corinna Weiss. I have been in sales and marketing for 20 years. Exactly four years ago, I started at CyProtect AG and took over the role for Marketing and Communication. I am responsible for setting up and expanding the IT and IoT marketing strategy, PR, in-house trade shows and much more. Due to my sales background I am also a contact person for the area of industrial security and the sales of the sematicon product range. So I have an exciting variety of tasks. Now to the introduction of CyProtect: We have been an independent IT and OT cyber security specialist for 20 years and have focused on offering custom-fit IT and industrial security from a single source to medium-sized businesses, large infrastructures, public clients, etc. Due to the fact that IT, information technology, and OT, operation technology, are coming ever closer together, we entered into a sales and development partnership with sematicon two years ago.

 

To continue the round of introductions: André, would you like to add points from your side and also introduce yourself briefly?

André

Gladly, my name is André Neumann. I work at sematicon as COO, i.e. Chief Operational Officer, and basically take care of the entire operational business, i.e. everything non-technical. This also includes marketing and sales strategies and the like. sematicon, as Corinna just said, deals primarily with the area of IT, similar to CyProtect, and crosses over into OT. OT and IT actually have to merge more and more in the course of Industry 4.0. And sematicon AG really focuses very strongly on the topic of industry. We want to provide solutions for industry, because IT solutions usually don’t quite fit. This can be seen from the fact that an industrial machine is not replaced every 5 years like a laptop, but can be used for 30 years or more. A quick word about sematicon: We are based in Munich, and research and development is also in Munich. We really have the people sitting here on site, everything is made in Germany. In addition to company and software development, this also includes corresponding hardware development. We focus on IT security, cryptography, especially for OT or industry. Ultimately, three business areas have developed from this: One is the safe management of industrial plants, which we are also talking about today with the corresponding audit and critical infrastructures. In the second part, the cryptography topic for IIoT and embedded systems. This is about really getting the cryptography right from the ground up for sensor technology, for example, so that we simply don’t see scenarios like “15,000 cameras being taken over somewhere or something like that” in the future. The third part results from the other two business areas: That is the whole subject of service, certificates, cryptography, PKI, etc. 

 

To close out the round, I would hand over to you, Michael. Would you like to say a few more points about yourself?

Michael

My name is Michael Walser. I am the CTO at sematicon and am responsible for the technical side of things – in other words, everything that André doesn’t do. About my background: I’m actually an electrical engineer and belong to the engineering faction, so to speak. In the last few years, now for more than a decade, I have been primarily concerned with the topic of IT security. That means my focus is on cryptography, smart card development, digital identities – anything to do with identification, encryption and cryptography. In sematicon AG, we have placed the focus on industry topics. In IT, there is already a whole range of nice tools for implementing the topic of security, but in industry we are still missing something. We want to make sure that the tools and skills from the IT world are perhaps adopted a little bit. But with the credo: don’t copy, but first and foremost look at what we can learn from each other’s domains in order to then bring the whole thing together in a meaningful way. 

 

Then I would just jump right into the topic. The topic of security or industrial security is always a bit intangible for me, especially in the context of new topics such as IoT. That’s why I’d just like to ask you in advance how your customers actually see it. Corinna, I’m looking in your direction. You probably face a wide variety of challenges and issues from customers every day. What are the top 3 challenges, perhaps also in the area of Industrial IoT?

Corinna

On the one hand, we have remote maintenance, which is nothing new in principle, but industrial remote maintenance is something else. You buy a PLC machine and usually you buy a maintenance contract with it. This maintenance contract means that any technician can connect to this machine. That means you have no way to control the whole thing, you have no traceability. That’s a challenge, this figuring out of “who did what, when, how, where.” Then we have wild growth, acquisitions of machines. Let’s take this issue of waterworks, CRITIS facilities. Such a waterworks was not just born yesterday, it has been around for 40, 50, maybe 60 years and is not connected to the grid. And number three is the risk of production plant downtimes. 

 

You had just mentioned CRITIS facilities. What does that mean exactly? Perhaps also asked in the round.

André

CRITIS facilities are critical infrastructures, such as oil pipelines, nuclear power plants, coal-fired power plants, water plants. This is about the basic supply. Ultimately, this also includes hospitals and similar facilities. What we also see in terms of challenges – besides data security and data integrity, of course, which is there anyway, especially in the IIoT world with sensor data, etc. – is that the worlds of IT and OT have developed somehow separately for decades. OT has gone completely to production and IT has grown differently towards networking, data, clouds – all these issues. However, due to Industry 4.0, the two must somehow grow together again. IT security has a very, very high priority here. For example, in the case of automotive suppliers: Some data goes into the cloud and conclusions can be drawn about the share price, for example. So someone produces fewer mirrors, for example, then I also know that fewer cars will be produced in the end. Such data must be protected accordingly. That’s why IT and OT must increasingly grow together there and also need other approaches. What we have still seen in terms of challenges in this is also the consolidation and safeguarding of various systems. Our customers run machines that have MS-DOS. Partly everything there runs on Windows 3. 11, XP and Windows 10 . These machines are now all to be connected to the Internet in some way and perhaps linked to predictive maintenance and many, many other things. Here I need a solution that combines all of this and, of course, also brings these things safely into Industry 4.0. 

 

Is it actually a reality today to negotiate with hackers? Data is “siphoned off” somewhere and you are virtually blackmailed with it – does something like this really happen in practice?

Corinna

Yes, that happens. Basically, the fact is that we are often like 10 to 15 years ahead. Cyberattacks are absolutely taboo subjects. What company likes to admit that it has been hacked, that perhaps it was not as secure as it should have been? The standstill of production lines can be accompanied by high contractual penalties or order cancellations. That is perhaps even the smaller problem. The bigger problem is systems that are hacked, because that really is a matter of life and limb. In the last two to three years, extortion with stolen data has caused 11 billion euros in damage. That’s really a lot! That`s why the Job Description “Ransom Negotiator” exists. Or like the water plant in Florida, where hundreds of times more chemicals were added than normal. Fortunately, an employee in the control center was able to locate this and switch it down. This is not only about money, but in the end about human lives!

Michael

These extortion or crypto Trojans or Ransom, as they are also called, have a huge disadvantage: they first encrypt data and then the information comes that you do want to put in some Bitcoins to get your decryption key. But then it is usually already too late. If some tool doesn’t already exist, if some security analyst hasn’t already taken the thing apart, then the odds are mostly against it. Because even if I pay, it is still not guaranteed that I will get the key at all. You have to imagine it like this: When I approve a product in the European Union today, there’s always this nice CE mark on there. The customer takes care of the so-called EMC tests, he makes sure that electrical equipment can be approved. And this requires special, very expensive measuring equipment, a special shielded room, in other words a huge amount of equipment. What happened to colleagues in our partner network: On many of these measuring devices, on these expensive devices, some operating system is running in the background, in this case it was Windows XP. You have to imagine that a device like this, which costs 40,000 or 50,000 euros, runs Windows XP. You can’t see it because the interface hides it, but then someone plugged a USB stick into it or accessed it remotely and suddenly there was a prompt: “Please throw in some Bitcoins, we’ve encrypted your data.” I’ll put it this way, renting a room costs quite a bit of money, since the equipment isn’t cheap either. And now you can’t work because the thing is encrypted. The manufacturers then say, for example, “You’ll just have to send it in and we’ll exchange the hard drive”. But then a few days just go by. 

 

I would make the transition to the use case now and maybe come back to it later. Today, we have taken up the topic of critical infrastructures and, in particular, the topic of waterworks. Corinna, can you introduce us to it and tell us: What kind of customer was that? What challenges did he have, and what might be the challenges or vulnerabilities in waterworks like that?

Corinna

This involved consolidation and audits for external and internal maintenance processes and a clear summary and traceability.

André

A second major issue is also: Who actually accesses my assets? Is he actually authorized to access it? And what was he actually doing there on the plant? Did he increase the caustic soda level here or did he not increase it, for example, on a pumping system – that I can trace something like that. That has been the second main focus, so a certain authentication or authorization on the device and of course a certain integrity of the data that flows there. You don’t usually have 10 pumps in one location, they are usually spread out a bit. 

 

Michael, what does the infrastructure of such a waterworks look like on site, can you give us a small visual image? And what have been the challenges from this customer?

Michael

I think the waterworks itself is actually irrelevant. Whether it’s a water company, an electric company, or another industrial customer, they all basically have the same challenges. There are a few systems on site, called SCADA systems or control systems. These are classic PCs, with a screen attached, as we know them. And there are so-called programmable logic controllers, i.e. small PLCs. There are small mini PCs, they don’t have a screen, they don’t resemble a PC. Actuators, i.e. pumps and sensors, are connected to these, a program is run on them and they are controlled. It is, so to speak, the heart of every industrial plant. Such a small PLC is really only there for this control purpose, a PC has other things to do. Such systems are primarily designed to be available, to be real-time, to be reliable. The topic of security does not really come up. And especially with these kinds of systems, I don’t have a user name, I don’t have a password as I know it, they’re not secured. If I can get on it, then I can do something with it. That’s a bit of the challenge. Most of the time, the people who maintain this equipment are not our own employees, but external collaborators. People like service technicians are companies that are on the road and have their own notebook, they have their own PC, and they are now accessing the internal system with their foreign hardware via a VPN tunnel, for example, and doing their work. 

 

What does a job of such a classic service provider at the waterworks look like? What data are you interested in?

Michael

I’m not that deep into exactly what kind of data is being sent. Most often, an update is applied to a firmware, a few configuration parameters are changed on a pump. A software is played on the PLCs, maybe some configuration is changed on a PC or something is set, as we know it. Our day to day is there quite classically for these industrial systems. Now an external party is accessing it, but there is no longer any real barrier. The problem is always the same: I have systems internally that are actually wide open. There`s not really a barrier, there`s sometimes maybe not even a password, as it is with the programmable controllers. External colleagues access it internally. It is irrelevant whether he is plugged directly into the system with its cable, whether a USB port is plugged in or whether he sits externally. VPN only extends the cable, so to speak, the system. That means, I do not know what he is doing. I don’t know what he’s really accessing. I actually have to trust that everything is going to be okay. If he now has some malware or similar on his PC that exploits a vulnerability in the network, then the problem suddenly becomes the operator’s problem. And that’s an extremely big challenge for many, to have old assets. In IT, when we do VPN, it’s relatively easy, we have patched assets. This means that security updates will be applied. You have to imagine it like this: A complex software has errors and errors are not found immediately. These errors are simply in the application. Now, for example, there was a huge outcry, mail servers were attacked, Exchange servers have somehow fallen into disrepute everywhere because of this. The BSI had issued a warning, this was quite a big fuss. This is such a classic example. This Exchange Server is a complex mail server product and there was an error that was not noticed. It hadn’t been noticed for the last 20 years. Now suddenly someone has found it. And then a so-called zero-day attack is carried out. That is, I as a hacker know the problem, the others do not know it yet. I now write malware that exploits this problem and brings down thousands of mail servers. By the time the vulnerability was discovered, the attack had already happened. And by the time most companies responded, there was a week in between. There have already been a lot of problems. 

 

And at a waterworks, do you also have complex software systems or how does that work there?

Michael

On the one hand, you also have complex software systems that have vulnerabilities. On the other hand, sometimes you don’t have to go that far, since many systems don’t have authorization. I can think of it as leaving the front door open at home. And as I said, with the other systems, the front door may be nailed shut. But I might be able to get in through the bathroom window if I throw in a pane. IT systems have their weak points. And if I just plug in network cables now, then I can’t protect myself from that. What should I do now? Should I turn off the email server? I may not be able to patch it because that’s a big issue, and the systems are usually delivered ready to go. You have to imagine it like this: I get a system, it’s delivered ready to use, and it’s set up at my place. I don’t have access to it at all. I also don’t buy a car and remove the car radio and then install another one – those days are over, I think. Here’s how to think about it. This is a completely integrated system, no updates are applied, nothing is reconfigured, the system is made available. And now there’s some bug in a program somewhere that I can exploit remotely and I can’t stop it. A basic rule is actually, you would have to isolate the networks completely. This means that I am no longer allowed to give an external party – let me use Corona terminology – access to vulnerable risk groups. 

 

Now, of course, I also talk primarily about the topic of Industrial IoT in the podcast. Now, the fact is that many machine and plant manufacturers who, for example, also have machines in such a waterworks or other critical infrastructures, naturally use dedicated data to leverage added value, to optimize processes, to save costs or to optimize services. If I would like to share this data with my manufacturer because I see added value, what do I need to do to ensure that? How do you see it at that point?

Michael

The question is: Do I want to share my data? I think that is the first point. The second point is: What do I do with my data? I always hear “increase production capacity and so on,” but I see it a bit differently in practice. When I talk to people about what they really do with their data, there’s usually no answer. That just gets pushed into the cloud and no one actually knows what to do with it. But one topic that we are actively pursuing, for example, especially with a major German partner, is predictive maintenance. So the fact that maintenance is handled automatically. This means that I use certain algorithms to enable early detection of the fact that a machine may fail in the future, so to speak, via the data that I collect – and only those that are really necessary, all of which are not the target, but selective. You can actually evaluate data like this, it really makes sense. On the basis of these evaluations, we provide a way to fully automate access to the plant, maintenance. This means that the activation takes place exactly at the time when it happens. The technician is triggered, he then has access to the system according to his authorizations and the whole thing is audited and recorded. This means that the plant operator can go to the office the next day and look at what maintenance happened yesterday. He sees: Aha, there was a temperature problem, it had to be readjusted. The technician did such and such, that was the goal, such and such software was updated, such and such things were clicked. 

 

I can see the added value: If I have a connection to my manufacturer somewhere, I may also benefit from years of research and development that may also be involved in these individual processes. André, I’ll look in your direction again: How do I make sure that access via the cloud or these accesses are secure? So now when I say, hey, I’m a waterworks operator and I’d like to share this data because my manufacturer has incredibly valuable data for me. We want to tackle the issue of predictive maintenance together, optimize service, etc. 

André

As Michael said, VPN is a bad solution to do the whole thing. I provide a kind of tunnel, a direct IP connection ultimately to the machine and thus all the hygiene that might otherwise act like a firewall for the entire system or even make an AV scanner absurd because I access the machine directly from the outside. That’s why there really should be uniform secure access here with complete isolation of the machine network. You can do that with our se.MIS™ product, which works without a VPN. With this, old and new machines can be connected accordingly. In principle, this is a system through which I make my external and perhaps also internal data uniformly accessible and the whole thing is centrally documented. Where I also have maintenance windows, for example. I have heard that it has happened before that someone quickly switched to a machine on a Saturday during the half-time break from soccer, and on the following Monday the entire production line was at a standstill. Something was done quickly and in the end no one did it. I have a loss of production and then what insurance will pay for it? This is then usually not traceable and then no one pays in the end. In the end, the customer is left with the damage. Such scenarios can be prevented and seen with se.MIS™. se.MIS™ does not use a VPN connection to the machines, but in principle creates a completely isolated machine and plant network. You can imagine it like this, I’ll take the Corona example now: When you go into the isolation ward at the front, you have appropriate controls, entry controls. You wear a mask, you might get a suit on or something, all the hygiene, so to speak, until I finally get to my relative. When I do a VPN, it basically means I tunnel directly into the ward into the patient’s room and then I’m there without all the hygiene measures. se.MIS™ here is a system that builds in a “digital pane of glass,” so to speak, through which I can talk to the patient by telephone or ultimately access my facility through this digital pane of glass. Of course, the camera is always over my shoulder, recording everything and telling me at the end what I have ultimately done with this patient or the system. 

 

Michael, another quick question for you: That means now, in practice, I connect my infrastructure, my different plants, for example in a waterworks, with the system and then have uniform access to these systems and the possibility of handling the whole thing not via VPN, but via an intermediate, secure system?

Michael

Yes, now that depends entirely on the technology used. I’ll say a system, classic PCs or similar, we hide them under a kind of digital glass pane, as I said. You can think of it as not being able to access the system directly anymore. This means that we completely terminate the insecure interfaces on our system, convert them and only provide the user with what he really needs. Such a remote desktop can do much more, there are also other data transferred that I may not even need. In the case of PLC controls, for example, we work very intensively with a company that has made a name for itself in the field of hacking and security of such systems. And that is the company Alpha Strike from Berlin. It is quite exciting to combine the hackers and the manufacturers to find out what is actually happening? What do we need to protect ourselves against? For example, we have developed a solution called PLC-Guard, which we can use to isolate such PLC systems. That was a bit of a challenge, but we made it work. This means that we can introduce security without installing anything on the PC, without changing anything on the system – that is always the challenge – but we cannot actually influence the systems themselves. That means that if it is somehow technically possible, then it is best not to install anything, no additional hardware, but simply to say: Okay, we will implement this in the data center and on the one hand have a possibility for IT, and on the other hand have a possibility for industry to integrate their systems in order to then completely isolate access. This means that we control the data traffic in the network and with PLC Guard we are also able to actually extract the software that lands on the PLC from the data traffic. You may know Stuxnet from the media, when some nuclear facilities in Iran were attacked. Viruses that hide on such systems can be identified relatively well. I have created the isolation between the external technician and the internal plant and can additionally monitor what exactly he is actually doing on the plant. So I can actually rule out a lot of these vulnerabilities that come up over years because they don’t affect me anymore because the protocol that has that vulnerability is no longer released externally. This means that I can continue to operate an XP machine for the next few years without any problems and integrate it into the IT. 

 

Is there actually a life hack or anything? So tips and tricks where you say, these are actually topics that I could implement on my PLC today that might make the whole thing safer? Or does it make sense to implement this holistically? What is your experience with this?

Michael

It is always difficult to change anything in the system. I’m very cautious about that, because it always means that I’m interfering with a functioning system. We actually want to avoid that. That means the only thing I can do is figure out how to design remote access to my facility. Do I allow remote access in principle, are they really necessary and if I allow them, what solution do I use? There may be solutions where a VPN box is just as suitable. But there can also be other solutions, where you can use a manufacturer like us, for example, to ensure isolation in this way. Every manufacturer has to think, and this is really a manufacturer’s task: What do I want in my network? Where do I want to go? I have to say in all honesty that I believe Industry 4.0 is a huge opportunity for the industrial location here in Germany. However, I also believe that you have to tread this path carefully. To say: Nah, I don’t want to. I do everything offline, I turn everything off. This is just as bad a solution as: I network everything completely and without brains. And that is perhaps one of the points where you have to find the right balance. But I don’t have many options on the system itself without changing the configuration. 

 

One more substantive request: if I imagine I now have unified access through your system and now also want to share data with a manufacturer. How exactly does this work in practice?

Michael

If it’s purely about the data from the machine, then it’s relatively simple. We are happy to support you there, because most of the time I don’t need complex intelligent sensors. Even from a 20 year old PLC, data can be extracted nicely. But I should think about when I do something like this, what do I want to share and for what purpose. Do I have to share all my data with the manufacturer? Do I really need it? What is he doing with it? All of this should perhaps be questioned a bit. We can ensure the encrypted transmission. But what the manufacturer does with it, we can not influence. Everything that leaves the house can also be misused in some way. That means I have to think about how do I deal with it? Or maybe you can still anonymize data in some way. This is always an option. We have ways of keeping the data that goes out anonymous – but the question is whether that really makes a difference. You have to think about: What is my use case and what is the lesser evil. This all sounds like a big challenge, but I have to say IT had the same problem. Even if we encrypt data in IT, then the first big challenge is always the classification of data. That’s a huge expense, too. This is going to sound a little funny, but there is this triad when you talk about security. It is also called the CIA triad. This has nothing to do with secrets, but with the availability, confidentiality and integrity of my data. integrity means they are unchanged. Availability means I can always access it. And confidential means they are encrypted. And if I have something very extremely confidential now, then the data probably also has incredible integrity, but then it’s no longer available. It would mean: encrypt and throw away the key. It’s always such a balance of power. You always have to think about: Do I always have to encrypt at all? It sounds funny, but encryption is usually not even necessary. Most of the time, it makes more sense to look at the data you’re submitting and see: can I make sure the data is coming from this point? Is it safe to transfer the data from A to B? And then, of course, important is what data. It’s very, very easy to distinguish between what I have to transmit and what I don’t have to transmit. And on the subject of the cloud: There are also manufacturers in Germany. We work together with Software AG, for example. They have a solution called Cumulocity IoT. This is a German manufacturer, with German locations and German data protection guidelines. That’s where you can try a collaboration. It doesn’t always have to be the usual suspects. So maybe just look for alternatives. 

 

André, I’ll look again in your direction: If I have now ensured uniform data access and implemented this solution. What is now summarized in the end the advantage for the operator? 

André

The advantage is that I know who is on my machine, when, how – I can answer all the W questions eventually. I increase the cybersecurity level by not accessing via VPN, but through the digital glass pane. I have a certain ability to plan. I can set maintenance windows. If I’m working on the machine on site and realize “Oh man, I’m not getting anywhere now,” then I can also do session sharing and get an expert from the manufacturer on the machine in a safe way and solve the problem together. This saves me time, because I don’t have to let the manufacturer arrive with the car for ages or have certain production downtimes, which I can naturally avoid in this way. I don’t think we need to talk about the C02 balance if I don’t travel. Often, that would be long-distance travel to remote locations. For example, we have a customer who has a factory in the Czech Republic and in Taiwan. If I always have to send someone there and have them fly over to take a look at the machine, it simply takes too much time. Of course, I can save a lot of time with a remote management solution. In short: I have control, overview, the whole thing is finally manageable and I have it consolidated and not 20 different systems in use, where I might also have to train 20 different people on 20 different systems in the end. 

 

Now your customers are not just water utilities, but generally from critical infrastructure and beyond. Corinna, I’d like to say one more time in your direction: This is now a pointed use case that we have discussed today with the topic of waterworks. You have a wide range of customers – how can this use case perhaps be transferred to other critical infrastructures or manufacturing companies?

Corinna

We work with customers of different sizes – it can be manufacturing industry, a machine builder, but it can also go in the CRITIS area, hospitals, power plants, any color.

André

At that time the banks were interesting, there you could possibly transfer money directly. But nowadays, hackers are somehow becoming more and more interested in it and find it more exciting when a centrifuge like this in a nuclear power plant or in a processing plant spins a bit faster. That could then also still bang at the end. This is perfidious, but it will go in that direction.

Michael

If I want to be real-time and available, then security is usually problematic. And that is also a challenge. I don’t want to beat up on the manufacturers, they can only do so to a limited extent. Because their primary task is actually something else, namely that the plant stays up and running. So the operator also has to invest a little more brainpower to find a solution. And that’s also an operator issue. By way of comparison, I didn’t sue Deutsche Telekom because I caught a virus via the Internet line. And I think you have to look around a little bit at that point here as well. I can’t blame the equipment manufacturer because I, as the operator, don’t do my homework properly. It is easier to attack a system that is unprotected, perhaps 10-15 years old, with known vulnerabilities or known problems than a modern managed and updated IT infrastructure. 

That was a nice closing. Thank you for the session today. It was really super exciting.