The topic of this podcast episode: Anomaly Detection in networks and an integrated solution for robust control networks and reliable data security. Companies actively lose money today when there are production disruptions due to issues like faulty devices providing incorrect or no information. In the worst-case scenario, this can lead to so-called “ransomware attacks.” In this episode, you will learn, among other things, what that is, what it could mean for a business, and how such attacks can be prevented.
Speaking about the development and implementation of a solution for anomaly detection in networks are: Bosch Rexroth AG – represented by Director Ecosystem ctrlX World Michael Krause – and IoT partner Rhebo (a Landis+Gyr company) – represented by Co-Founder & Business Developer Frank Stummer.
Episode 96 at a glance (and click):
- [5:35] Challenges, potentials and status quo – This is what the use case looks like in practice
- [16:59] Solutions, offerings and services – A look at the technologies used
- [34:29] Results, Business Models and Best Practices – How Success is Measured
Podcast episode summary
The increasing interconnectivity of industrial plants is leading to a rising number of cyber attacks and security risks. To effectively address these threats and ensure production security, Rhebo and Bosch Rexroth AG have developed an integrated solution, which is comprehensively explained in this podcast episode. The focus is on a cybersecurity solution for the connected industry and how innovations in industrial automation can contribute to safeguarding businesses and valuable resources.
Rhebo is a leading provider of solutions for monitoring industrial control systems and detecting anomalies in real-time. Cybersecurity, as well as the availability and stability of production processes, are central topics for this hardware-independent security specialist.
Bosch Rexroth AG is a global provider of drive and control technology, specializing in factory automation. They assist machine engineering customers in engineering machines faster and help machine operators minimize production risks and achieve higher availabilities. Additionally, their goal is to reduce energy consumption and environmental impact.
The partnership between Rhebo and Bosch Rexroth AG offers companies an integrated solution that combines the benefits of both technologies. The cybersecurity from Rhebo and the stability of Bosch Rexroth AG’s industrial control systems ensure optimal protection and efficient monitoring of production processes.
Congratulations for choosing this episode, because today we’re talking about a really important industry topic that everyone should have on their radar: it’s about anomaly detection in networks. That doesn’t sound very appealing at first, but it’s not at all, because with every day, with every hour and with every minute, more and more amounts of data are produced, coming from devices, machines and also sensors.
Companies actively lose money here because, for example, production downtimes occur due to devices that send incorrect information or no information at all. In the worst case, so-called “ransomware attacks” or breaches can occur, which need to be prevented!
Today you will learn what this is and what it can mean for a company in the worst case. Directly from the electronics manufacturing plant for servo drives at Bosch Rexroth, represented today by Michael Krause and their IoT partner Rhebo, a Landis+Gyr company, represented today by Frank Stummer.
Hello Frank and Michael, first of all welcome to the IoT Use Case Podcast! Glad to have you here. Frank, how are you and where are right now?
Thank you, Madeleine. I’m doing well and I’m in Berlin right now; both customer visits and working group visits are coming up. For example, we are at Bitkom, which is why we are in Berlin.
Berlin is always worth a trip, we founded our company in Berlin, I know that very well. Michael, are you also in Berlin?
I’m also in the big city, I’m in Lohr am Main, near Würzburg, at our headquarters. Beautiful little town on the edge of the Spessart, 11,000 inhabitants. I’m back in the office today; seeing colleagues, it is nice to talk to them after all the time working from home.
Is that a big location?
Yes, this is our headquarters with over one thousand employees and production, right on site.
About Rhebo briefly from my point of view: You guys from Rhebo offer OT and IoT monitoring solutions for threat and intrusion detection, everything around the topic of cybersecurity in industrial networks. It doesn’t really matter where the plants are located, whether they’re on-site or distributed somewhere in the world, you’re providing the monitoring solutions for that. What I find exciting about you guys is you do a lot with anomaly detection! You can detect and evaluate deviations within networks.
Frank, you are Co-Founder and Business Developer respectively responsible in Business Development at Rhebo. Which customers do you work for today? Who are your typical contacts?
Everything you said is right. In addition to cybersecurity, it is also about availability and the stability of production processes; on the other hand, of course, it is strongly about cybersecurity. This is the reason why we currently have a strong focus on critical infrastructure companies, known as “KRITIS” in Germany, including energy supply, water utilities, and gas supply, as well as many applications and customers in manufacturing companies or process industries, such as food and pharmaceuticals, where cybersecurity is also important, but more in terms of stability and availability of the production process itself. At the end of the day, they are two sides of the same coin.
The contacts for these matters include both the security officers responsible for cybersecurity, as well as the production managers who are responsible for the operation and smooth functioning of production processes. They represent both sides of the coin, with cybersecurity responsibilities on one hand, and production and operational responsibilities on the other.
You have been taken over by the Landis+Gyr Group, which has been know for 125 years for any solutions around electricity meters or even energy solutions. Can you tell us anything about that? Do you work with them? How do you get involved here?
Landis+Gyr are a Swiss company and the global market leader for advanced metering solutions, i.e. the electricity meters that are installed everywhere and, of course, smart meters. It is ultimately an integration partner of ours, so our “mother” is also an integration partner. This means that we integrate these functionalities of OT monitoring, surveillance and also anomaly detection directly into these smart meters or into the complete infrastructure of all smart meters. So from the front-end all the way back to the back end, where large amounts of data are processed. That’s where we integrate the solutions and the functionalities.
Challenges, potentials and status quo – This is what the use case looks like in practice [05:35]
Can you and describe what projects and use cases you have on Rhebo’s part in general? Which one are we looking at in detail today?
On the one hand, we have these integration use cases, we also have one featured on our joint platform in the network, that is with sonnen, a Shell company, it is about integration in battery controls. Here in the use case, it’s about working with an automation platform, the one from Bosch Rexroth, but also the application directly.
What does the end customer do with it, so to speak? That is, the use of such anomaly detection, those are the two big use cases and then it differentiates greatly. Then it depends on which sector I am in, do I still have to take special requirements into account? If it’s about energy distribution networks or if it’s about energy transmission networks or even food production, at the end of the day from a technical point of view, from our point of view, it’s always very similar, but sometimes the requirements from the standards are a little bit different.
sonnen is also a very exciting project that you have worked on.
Today, Michael, we will focus on your project; how did you actually meet each other?
In fact, I don’t even remember because it was such a long time ago. Rhebo is a partner of Bosch Rexroth, at that time we didn’t even have an official partner program. We are a provider for an automation platform, “ctrlX Automation”. I met Frank through a mutual colleague who said in 2017 that it would be super exciting to see what we were doing, that would become more and more important in factory automation, which is where we are today, with our platform, and who proposed working on something together.
So there’s a long-standing partnership then. Bosch Rexroth also has its own partner program, a kind of ecosystem, where you have a wide variety of solutions in the network. Can you talk about what exactly you do there?
Exactly, ctrlX Automation is our automation solution, typically for factory automation. And because we say we have an open platform, we don’t do everything ourselves, we focus on certain topics where we are doing very well, so PLC, motion control and also the whole platform. Per se, we introduced the platform in 2019 and then in 2020/21 we established a partner network, nowadays we would say ecosystem, where we have gathered 73 companies today. They help our mechanical engineering customers to engineer machines faster and our machine operators to minimize production risks and achieve higher availability. That’s exactly where Rhebo fits in beautifully.
How did that come together in terms of content? What were the potentials in the partnership with Rhebo where you said: This fits exactly what we need for machine builders and customers!
We claim that we have a very secure automation runtime system, which meets IT standards. Rhebo is active in the KRITIS area, critical infrastructure facilities, but we believe that in factory automation it is also increasingly important to monitor networks. Many incidents that have occurred in recent months and years show that there is also a need for machine operators to catch up in factory automation.
We have to distinguish between the two here. It is important on the one hand, you have your own ecosystem, on the other hand it is about your own production.
Pilz, Krauss-Maffei and others have fallen victim to so-called “ransomware attacks”. This is a big issue, the threat level is still high. There are now frequent cases where the public also reports on it. Many also do not come to light at all, and the degree of suffering of those affected is high because many victims have to pay demanded ransoms. You can read a lot about it online at the Federal Office for Information Security.
Can we briefly summarize together what’s happening right now? What are the potential damages that arise from this issue?
You’ve already given some very important examples, also in the sense that they cost a lot of money. Attacks have increased in recent years. Fortunately, state prosecution by police and others has also increased, but it is still a huge problem. That’s simply about the money, that’s about criminal organizations trying to extort someone.
It is also the case that in such attacks, it usually still goes far beyond ransom payments. That means there is a negative impact from hacker attacks, from malware that may not have been intended for the company, but ends up in there anyway. If not only IT is affected, but also OT, the production processes, then it quickly becomes very costly.
Unfortunately, the fact is that we have to talk about state actors, we have to talk about state attackers, who have a whole different set of resources. This includes production processes and also our mechanical engineering in Germany. This also includes our automobile manufacturing in Germany, and we must not forget that either. Five years ago, I had to talk my head off about it; unfortunately, you don’t have to do that anymore, but you can do something about it.
We are talking about your production today and you are also experts in the individual processes, you have seen the issue, this is to be avoided at all costs. Can you situate for us how you guys approach the issue at your end?
Everyone thinks of the use cases Frank described, but there are much more mundane use cases where network anomaly detection can help. This already starts with the fact that we see an increasing number of Ethernet subscribers in production or even in the machines. Ethernet networks are getting bigger on the shopfloor and so is network traffic.
We are talking about IT and OT convergence, about connected machines, and there is a corresponding data traffic. In our production here, we are in the headquarters, we produce servo drive technology here, so power electronics for our servo motors in Lohr am Main. Here we have many Ethernet subscriber and a congested network can lead to problems there, for example production orders could no longer get down to the machine. Cyber-attacks come to mind, but there are network devices that may have some kind of damage and are sending packets through the Ethernet uncontrollably.
i.e. incorrect packet info.
Exactly, and thus effectively closing off the connection. Within Bosch Rexroth, we have very good central IT that protects us from the outside, and we also have very good security concepts on the shop floor. But nevertheless, we found it interesting to test the Rhebo solution at our plant as well, simply out of this network utilization idea and to simply avoid possible production downtime, again the topic of higher OEE.
That’s where the stability of the network is crucial, because the data traffic is increasing all the time. You have the challenge of dealing with these potential network disruptions because there’s a lot of traffic coming in and there’s a lot of data. Can you put it that way?
The important data has to go from the ERP, from the MES down to the machine, because you have to tell the machine what to produce. And if they aren’t received, as banal as it sounds, you can’t produce.
Do you have examples of data types?
On the one hand, we have the typical fieldbus data, from the machine down to any fieldbus subscriber. But we also have a lot in production. We also have manual processes or work stations where there are laser scanners that document a process step, for example, when the worker is done, then there is a laser scanner and then the drive moves on to the next station, so to speak; these are all Ethernet devices, label printers, for example, that print out some kind of packaging labels or print out some kind of serial numbers that go on the device.
They get their data from the MES, from the ERP system with the serial number. We have a kind of digital twin for each product that is stored in the system, all of this is Ethernet communication in the plant.
Solutions, offerings and services – A look at the technologies used [16:59]
What are the technological requirements that the solution must definitely bring to the table? What are the general requirements for such a project to be successful?
The network data to be monitored here is similar to that in IT. We ourselves, the founders of Rhebo, originally come from IT. We were even on the backbone of the Internet, developing DPI technology there 20 years ago, deep packet inspection technology, so that’s what we’re using now.
What we then learned, had to learn and were allowed to learn: Yes, it is very similar to what we talk about on the shop floor, but at the end of the day it also has some peculiarities. The most important peculiarity is: the production process must always be running. I still remember when we started out with our prototypes, we were allowed to test them in Leipzig in a large automotive plant and then said: let’s see, then we can stop it right away; if you do that you’ll be sent home immediately. This means that you must not interrupt a production process like this, you should analyze what is happening so that you can then take reaction paths and react in a targeted manner.
On the other hand – and this is very positive – in production networks we communicate with very many subscribers who are communicating with each other, but they are always the same ones. This high degree of determinism is useful. That’s why our anomaly detection, which we have installed, has a significantly different approach than in IT, for example.
In IT we wouldn’t see the forest for the trees, but in OT it works very well. And you don’t just solve cybersecurity issues with that, but I see a lot of other things: stability issues and so on. Because at the end of the day, we are “only” monitoring the network. Who is in the network, who is talking to whom, what is being said? We look into the communication, also see individual values and then only see changes in the network, changes in values. In production processes, such changes are always noteworthy.
What is the common solution?
I’m describing the part that I can cover, and that is we have within ctrlX Automation an industrial controller that then has a certain function due to the software loaded onto it. When I load the PLC app on the industrial controller, the ctrlX CORE, it is a PLC. If I load a robotics app on it, then all of a sudden it’s a robotics controller, and if I load the Rhebo Sensor app on it, then it becomes an intelligent network detection device.
That’s right, it’s an appliance that’s put on top of it. And what we do is, we then get as a monitoring solution the complete data communication from the ctrlX that runs over it and for us that’s the data access point. That’s the raw data that we work with to then do the anomaly detection.
That is, Michael, you have your own controls that you use. Software is installed there, a kind of app that comes from Rhebo, and this app provides all the functionalities for this data communication to take place, or intelligent software that makes it possible to listen in on this network. Could you put it that way?
Exactly and then the client is this control and there is the central device from Rhebo, which then evaluates the data in production on-premise.
How does your controller collect data today? Do they use different protocols?
Exactly, the controllers are classically multiple Ethernet ports. This is once down to the field level EtherCAT master, with which it talks to EtherCAT devices and the machine, this is a profiNET master or slave, with which profiNET devices can be addressed and of course it also has a normal Ethernet port into the production network, which is above the controller and can forward the network traffic there.
When other manufacturing companies come in, they have a wide variety of hardware, is the system or the application that is installed there hardware-independent or is that something that always has to come prefabricated by a certain partner?
We always need a central device. What we have talked about so far is the data sensor, so to speak, to get the raw data, but the evaluation, that then is not performed on the controller or on the platform, but its performed in our device, the so-called “Rhebo Industrial Protector.”
This runs on-premise, but can also be integrated, for example in control centers. We then connect to data sensors and one possibility is then the automation platform; there you are directly at the machines, very close to the plants and get a very high visibility. But in the same way, I can also take a data sensor, for example at the top on the ERP system. This is all collected at the central device, aggregated, evaluated and displayed. This is then the complete solution.
For me, a sensor is a classic temperature or measurement value. But you are talking about a different sensor here. Could that be?
This is about the data access point, to avoid the word sensor. A point where I can passively listen in on all communication without any feedback and then route it to our device.
The analysis, how does it work, how do you do it? What kind of data do you need?
We basically monitor 100% of the data communication, really everything. This is one of the principles that can be done very well in production IT; in fact, it can be done very easily here. Then we present any change as an anomaly. What we do then is we make that more readable so that a non-security expert can still do something with it. That is, we then represent what his network looks like.
Most of the time, at that point, you realize: Oh, that one device, I don’t even want it in there anymore. We also see when a device stops communicating, for example. Then there is further grouping, for example there is normally a communication every second and now only every five seconds, which means something is no longer right or the bandwidth is too high or something along those lines.
That goes all the way to security use cases, of course, something like a port scan that is performed or I have passwords in plain text. The reaction path is then the most important thing. Our solution is nice, but it is even more important that something is then done with it, that people work with these findings and react. It depends on the people and or the connection to other systems.
You create 100% transparency and visibility for whatever anomalies. You don’t have to be a cybersecurity expert, which means that even if I’m a production manager, for example, I could learn the ropes and work with your software. Did I understand that correctly?
Yes, it is usually like that. Every morning, the production manager looks at it or gets a message when we have messages. Since it is one of the most important messages from our device: how many messages are there anyway? Normally, such a network does not change very much. There’s a lot of communication going on in a production, but it doesn’t change much. If I have more than 100 messages, within ten minutes, I don’t know what’s happening yet, but I know something is happening.
Then I bring in my security expert, my network expert, and thhe can drill down and look deeper. For me the most important functionality of our device is that I get a Pcap. I’m a data forensic scientist myself, which means I’m not interested in all that beautiful and colorful stuff. I want to have the Pcap in the back if there was an incident somewhere, if something was broken somewhere and I can look at the last bits and bytes.
With this solution, not only can I see the communications within the network and different use cases that could potentially cause disruption, but there’s also the cybersecurity aspect behind it.
Do you also detect new attacks?
Exactly, that is in the nature of things. Zero-day attacks are also detected immediately, but that’s actually a given. Because when I observe something, then I see the communication, then I also see changes in the communication. We can’t then, of course, say this is attack X, Y, Z, because it’s brand new, but we at least see the effects and we give indications of that.
This means that you have the possibility to see very quickly that something is not right, in order to then analyze more deeply and also to react. Even if forensic experts as cybersecurity experts wish they could, it is not so much a matter of actually going down to the attack and analyzing it, but rather the most important thing is to see that something is happening at all so that we can then react and close off something on the firewall so that we can restore something with a backup and so on.
Michael, are any other partners relevant in this conglomerate, also regarding the shop floor? How does the cooperation within the partner network work on this topic?
We address many different categories with the partner ecosystem and security is one of twelve. These are simple IoT applications, OEE dashboarding to, for the machine builder, rather engineering tools or sensor, create compatibilities with sensor technology. It’s really quite broad and is intended to help the machine builder and the machine user.
This is helped by security applications, integration solutions, faster integration of machines, existing lines. This is also helped by simple MES applications or factory IT as a whole, and we are looking very broadly at this and are happy to welcome any partner, large or small, who is interested in working on this open ecosystem.
This topic is so insanely relevant that you don’t just bring in IT, you bring in all the OT players. There are many partners in our network who come from exactly this area, such as ifm, WAGO or Turck. All those from the OT corner.
We work with many, we are open there as well and that goes from interfaces, simply connecting two devices together to deep integration into the own controls of such devices. WAGO mentioned us in a previous podcast episode, also Welotec and others, not to mention switch or router manufacturers.
For example, we integrate the sensor technology from our side, such as from ifm, among others, the data is brought to the data layer in the control system, which is the central data interface and then we forward it to the central Rhebo system. Thus, one or the other partner plays very well with each other.
Today, you can’t do anything on your own; you have to join forces in networks. There are specialists for OT, for IT and security like Rhebo. That’s how today’s production works and in interaction you create unique and cool solutions. There is no one who can do everything anymore; your network also shows that.
Meanwhile, it’s coopetition; you always find yourself together on the shop floor because you have to provide the data at the end to solve a business case.
Results, Business Models and Best Practices – How Success is Measured [34:29]
What is your business case for Bosch Rexroth in working with Rhebo?
The controller is determined by the software that runs on it, and we want to offer our customers as wide a range as possible. Through a marketplace, the ctrlX store and customers can download the Rhebo app relatively easily. That also has a very interesting price and then bring it on the control and then turn to Rhebo with confidence for the central device and the total solution.
I would also like to see this done much more in mechanical engineering. We also have a focus still on critical infrastructure because critical infrastructure colleagues used to be forced; now they do it voluntarily, to do more for security.
Standards such as IEC 62443, for example, also lead in this direction, as does TISAX. At the end of the day, the point is that with such an automation platform, I make my life much easier as a mechanical engineer. It’s relatively easy for me to add those applications that are going to be important for my end customers, so I can build my own business case.
We have that for example in the food industry, there the functionality is first only the availability of such a data access point and then a second, the complete monitoring solution. That’s simply already a checkpoint there when I buy a system. This makes it easy for me to check off all these standards, which in turn then require my end customers.
What are things that are still to come in the future? What do you see here in the market?
In research projects that go further into the future, we see very clearly that cybersecurity, but also stability and availability, is thought of end-to-end and that is built in everywhere. You have that everywhere now, but more in set pieces. Something like this also exists in OPC UA, for example. Everything has to work together, because the automation pyramid, as it was taught some time ago, has already dissolved or is in the process of dissolving, from headquarters, from ERP to the edge and down to the shopfloor. I also need other solutions to ensure cybersecurity or data integrity.
This does not stop at OT, which means that when you use an industrial control system in your machine, you have to make sure that it is secure by design and secure by default, for example. When you log in to the controller for the first time, you must change your password and cannot use the default password, which is probably always the same in many PLC systems, depending on the manufacturer. You can achieve a lot as a mechanical engineer; think about it: Am I still on the right track with what I’m using?
I can add a note here because I just recorded a podcast with the OPC Foundation, because it’s insanely important to have topics like this on your radar not only with OPC UA, but also with Security by Design.
Thank you for this exciting podcast. Thank you Michael and Frank for joining us and for presenting your solutions on how it all works. See you then!
I enjoyed it very much and of course Michael to you. I always enjoy talking with you and look forward to seeing you again at the next fair and talking longer again!
Congratulations to all listeners for addressing the issue of IT Security; a very important topic! Thank you Madeleine, it was fun.