PASM and PSsystec: IoT fault management for 3000 sites

Hello and welcome, dear friends of IoT. Today we have another episode with IoT from real practice. As always, we look at how other companies are already implementing these topics successfully. Our guest today is PASM, a fully owned subsidiary of Deutsche Telekom AG. Some of you may know the name. The full name is Power and Air Solutions GmbH. What exactly they do, we will find out in a moment. This episode focuses on IoT use cases around fault management and energy consumption. More specifically, on IoT devices deployed across more than 3000 sites, which means large device fleets. We talk about how energy supply and availability for internal Telekom customers are monitored, what concrete savings this transformation brings and how such a solution can be scaled. We also discuss what to consider when integrating across different sites, especially when working with cellular connectivity. Joining us today are Alexander Barnstorf-Brandes, Application and System Manager at PASM, and their IoT partner PSsystec, represented today by Managing Director Michael Knöpfle. It is a particularly exciting project, and you will hear why in just a moment. All information about similar implementations can be found at www.iotusecase.com, and the contact details are listed as always in the show notes. So with that, enjoy the episode. Let us head into the studio.

Welcome to you, Alex, and to you, Michael.

Alexander

Hello, good to see you.

Michael

Hello Madeleine.

Hi Alex, I will start with you. How are you today and where are you joining from?

Alexander

I am working from home today. I had a very eventful week, a lot going on, quite intense. Monday in particular was very busy.

So many projects. That is positive to hear, right?

Alexander

Absolutely, of course. It is fun.

Very good. Just quickly regarding your region, where are you located?

Alexander

I live in Braunschweig, in the Braunschweig-Hanover region. Our sites are spread across Germany. Telekom is represented throughout the entire country. Most of my colleagues are in Munich or Münster.

Greetings to my former home region, to everyone listening from Lower Saxony and of course from all other regions as well. It is great to have someone from Braunschweig and the surrounding area again. Michael, nice to have you with us as well. How are you today and where are you joining from?

Michael

I am in our office at our headquarters in Augsburg. The week is, just like for Alex, busy but still connected with a lot of joy in the work.

Very good. I already gave a brief introduction earlier, so let us dive in directly. Alex, I will start with you. You are Application and System Manager at PASM. As I mentioned, you are a fully owned subsidiary of Deutsche Telekom AG. If I understood correctly, you are responsible for providing the products you deliver and for fault management within the group. You also oversee projects related to the system upgrades and you have been involved in this project for quite a while. Is that correct? And what fascinates you personally about these kinds of projects, especially in the IoT context?

Alexander

I have been with the group for quite a long time. I joined Telekom in 2012 and moved to PASM in 2022. I have been working with IoT since 2016. It is important for me to emphasize that this is always a team effort. We work closely together internally and also with our external partners. The projects are so large that even the project management would be challenging to handle alone. That is why I am glad I can rely on many great colleagues.
What I really like is that with IoT you can bring things into the digital world that were not connected or digitalized before. I enjoy that I can literally touch my work. I can walk into a building and see the gateway we planned and then look at its counterpart in the cloud, essentially the digital twin of that system or building. I think that is very cool because it is easy to show and explain to colleagues or friends.

That makes sense. As a Telekom subsidiary, your customers are mainly internal Telekom units, right?

Alexander

Our customers are actually exclusively the individual building blocks of Telekom, i.e., subsidiaries of Telekom, primarily Telekom Technik, but also DESUS, our colleagues in the office sector. This means we ourselves have no contact with external customers. We are the internal service provider responsible for availability. We supply the energy and provide a certain level of availability, depending on what the customer requires. For example, this can mean that the energy supply is bridged for several hours during an outage.

What kinds of devices and systems are you connecting?

Alexander

This includes classical HVAC systems. At home you would call it an air conditioning unit, but ours are much larger. We also have compressed air systems and, in smaller buildings, standard heating systems, just like the ones you know from home. These are the systems we monitor and send signals for.

I see, very interesting. Michael, let us come to you. You are the Managing Director of PSsystec and you have been working together for quite some time. Can you explain how relevant this project is for you and how long you have been working with PASM?

Michael

Yes, we have been working very successfully with PASM for several years now, together with our team of developers, technicians, project managers and many others. For us it is always a team effort. In the large projects we have implemented with PASM and continue to work on, all parts of our company are involved. What is fascinating is that they took this digitalization step and moved from the analog to the digital world by enabling analog systems to deliver digital data.

Great. It is also important to mention that you come from a traditional hardware background. You have your own SMARTbox, which serves as a hardware platform with a SIM card, and you bring strong mobile connectivity expertise. We will talk about that in a moment, especially about what needs attention and how it works in detail. Can you explain what the initial goal of this project was and why fault management is so important for you and your customers?

Alexander

Gladly. The systems we operate in our buildings need to be monitored continuously.
We have a large service management center, actually two geo-redundant sites, that monitor our buildings and evaluate the signals generated there. With the large number of Telekom sites and the wide geographic distribution, this is quite a challenge. All buildings had signaling technology installed, some of it still analog, based on ISDN or even older. That was exactly what needed to be replaced. There are different ways to solve this.
Back then, when I was still in another part of Telekom, we applied for this project together with PSsystec to develop an alternative to classical building signaling. Classical building signaling usually involves cost-intensive hardware that is routed through internal building wiring with encrypted connections or DSR lines to reach central platforms. That is exactly what we wanted to replace.
Cloud technologies and IoT were already present at the time, but projects of this scale were rare. We took the first steps in 2020 and really accelerated in 2022. At the beginning the SMARTbox was just one of several options, but it prevailed because of its flexibility, especially regarding installation. Since we rely on mobile connectivity we have significantly lower hardware costs and can monitor the systems just as reliably as with the older technology. We also have the option to connect to different cloud platforms. We use Telekom’s Cloud of Things, but in principle a SMARTbox could also be connected to another cloud.

[08:58] Challenges, potentials and status quo – This is what the use case looks like in practice

I believe many listeners have similar projects or challenges. What were the specific savings or areas where faults were actually costing you money? In which situations does the investment in technology pay off most? Where did you realize in day to day operations that processes were unnecessarily complex or too expensive, for example by sending too many technicians on site or handling fault messages inefficiently? Why is fault management so critical and what other aspects matter for you?

Alexander

That is indeed a broad topic. In general, monitoring the systems is absolutely essential. I want to know the condition of my systems across all sites. That alone creates huge value. I can signal different types of faults, depending on the equipment.
One major advantage for us is the gateways themselves. I can perform mass operations much more easily.

What does that mean?

Alexander

For example, when I want to shorten reaction times or adjust the transmission intervals for temperature data. One example: In summer it gets very warm. I measure the temperature once per hour and have a hysteresis in place. That means if the temperature rises by at least zero point four degrees, an additional measurement is created. Now imagine a heatwave and I want to monitor my sites much more closely. With the combination of cloud and IoT I can adjust these intervals quickly and easily. Targeted for specific regions. I can group them by federal state, by operational area, or I can switch the entire deployment area at once.
Then there is the topic of over-the-air updates, meaning functional extensions of our gateways. Michael mentioned this earlier. We have made tremendous progress together, especially compared to 2022. The devices PSsystec provides are relatively simple compared to classical building control systems. But we have built more and more business logic onto these devices and added many functions. The hysteresis is one example. Another is that after a fault alarm the device automatically increases the transmission cycle. This can be configured for every individual data point. We have improved this a lot over the last years.
Another example is full remote configuration of these gateways. We can configure the hardware completely remotely, input by input. We can connect up to forty data points to such a gateway and define the input type remotely. The gateway applies the configuration immediately. A technician on site does not need a laptop. He just connects the sensors and we handle the rest remotely.

Great. There are many companies that are not yet where you are today. Often these setups are still implemented with classical building control systems or traditional controllers. Do you have best practices where you would say that if you start a similar project you should watch out for specific cost traps because they really matter? How did you handle it before? And are there things where you say that you can avoid them today?

Alexander

One important cost factor is mobile connectivity. We use our own Telekom mobile tariffs of course, but these SIM cards are available on the open market as well. Optimizing the data volume is absolutely sensible and something you should keep an eye on. But the real cost trap used to be somewhere else. We have many relatively small buildings at Telekom. If you describe it in T-shirt sizes, we have everything from S to XXL. In these small buildings we would have been using a sledgehammer to crack a nut with classical building control systems. The hardware procurement alone, the operation of that hardware and the integration into the building were major cost factors. And this is exactly where the first cost savings appear when you use IoT.

Understood. And a quick note for everyone who wants to learn more about Telekom IoT: I recorded a separate episode about this, Episode 186, specifically about Telekom IoT connectivity. I will put it in the show notes. You can also easily find it on Spotify or Apple.

[13:51] Solutions, offerings and services – A look at the technologies used

This is a technical topic, but mainly driven by real business cases.
The points you mentioned fall under device management and asset management. Meaning how you increase cycles, perform over the air updates, handle remote configuration and so on. That refers to both the hardware and the device management layer. Can we summarize it like that, and what were the biggest advantages for you?

Alexander

Definitely. Fleet management, meaning the comfort factor, is enormous. And I can hand such a platform to someone who is not deeply involved in the technical details. That is a major advantage. The IoT platforms people use today are far more convenient than having to program a classical controller. That would be a huge amount of work and you would need specialists who are qualified for that.

Absolutely. Michael, you have been working with your SMARTbox as a hardware platform and with mobile connectivity for a long time. What would you highlight here? What is important when you build such a system independently from other infrastructure? You mentioned that the focus shifts away from classical IT as we know it and toward mobile connectivity. What is the special aspect of this project for you, or the technological advantage that stands out?

Michael

In general, the independence from IT is a major advantage. I do not need integration into existing networks and I do not have to build a local network on site. I use a network that is available worldwide. I use this network and still have data security because the transmission runs over secure channels and the cloud platforms are certified and secured. This fear you sometimes hear that it is not safe is unfounded. These are all secure connections.
Our devices are designed so that you plug them in and they work. As Alex said, I can connect up to forty IOs to a SMARTbox system. The wiring is done on site and I can configure everything cleanly later via the cloud. I do not have to program anything or adjust software parameters on site. This flexibility is a major advantage.
Regarding data consumption: That is always a topic, whether mobile or any other network. If I have an office building with one hundred people and run a server farm, I also need a lot of data. The key question is always what the business case requires. Do I really need a measurement every thirty seconds? Does that bring any additional value? Or is every fifteen minutes enough? That depends on the use case. This consideration is positive because it forces you to think about what you do with the data and how you actually need it. This way you can optimize the consumption and still stay flexible.

Cool. So what Alex described earlier, for example when a heatwave comes and quicker reactions are needed, would be exactly such a case. Then you do not process the data at a permanently high frequency. Instead you adjust situatively, maybe to once per minute, every fifteen minutes or even once per day in a consolidated way. That means you look closely at how the cycle must be configured so that device management works optimally?

Michael

Yes, you can configure that through device management. You have exactly that level of flexibility.

Alexander

And I would really like to pick up that point again, because this critical reflection on your own systems happens automatically when you implement such a project. If you ask someone how many data points they would like, they usually say all of them. Everyone wants to know everything about a system. But in very few cases is an almost real time resolution actually needed.
Our devices are always in standby. They only send a signal when the state of a system changes, and that is exactly the moment when we need the data. Temperature values, for example, sometimes have legal or contractual requirements. In those cases, we must capture them at least once per hour or every two hours.

Michael

Exactly. And another example comes to mind that shows very well how flexible such a solution can be. It is about mobile power generators, large trailer based units. The requirements there were: I want to know where the unit is located, so GPS. And I want to know the condition, for example the charge level of the starter battery, the tank level or whether the generator is currently running.
We implemented this with a SMARTbox as well. The box was installed and wired in the control cabinet by a regular technician. The actual parameter setup was done completely remotely. The Modbus connectivity was configured remotely. Register 1 is the tank level, and so on. You can cover several requirements very easily with one solution and, I agree with Alex, at a very low cost.

Great. I think this is relevant for many companies that have similar projects. And let me say it again: if anyone listening thinks this matches your own challenges, I will put both of your contacts in the show notes. Michael, yours and yours as well, Alex. I hope that is fine for both of you in case someone wants to exchange experiences. And of course you are warmly invited to join our community, which deals with exactly these topics.
One more question: This solution is especially interesting for companies with many decentralized assets or devices, right? Because these scaling effects appear mainly when you have many small devices or large fleets in the field.

Alexander

I would not generalize that. We are certainly an extreme example in terms of quantity and distribution. But why should it not make sense if I only have one or two large buildings and want to monitor my systems or energy consumption there? From my point of view, IoT definitely makes sense in that case as well. And when I use technologies that utilize both mobile communications and, as is very common in our case, low-energy in the mesh network, I also have a non-invasive technology. I do not need to pull cables or open fire barriers. These are extremely cost-intensive tasks that I can avoid.
Of course, this technology also has limitations. There are things that are difficult to implement. You should take a careful look at that first. But on the whole, we have had very good experiences. And there are now mobile connectivity technologies like NB IoT, which we use as a backup everywhere, even to reach building sections or shafts that are up to two meters underground. Even there we still get reception.

So that means you also have devices that have to be connected underground?

Michael

There are also completely different use cases where a single device is sufficient. At PASM this is not the case, of course, but if I think about HACCP in the food sector, there is the requirement to log the temperature every 15 minutes. We also have devices in use there. In that scenario, it might be a pharmacy or a butcher shop monitoring one to three cold rooms. Then it is just one device, and it is simple because it is noninvasive. I mount the device on the wall, use wireless sensors like BLE beacons, stick them onto the surface, power the device on, and the data starts flowing immediately. This simple implementation is ultimately the major advantage.

Yes, interesting. How does data integration work for you? Are your data streams integrated into other building-monitoring systems such as an EDM? Or do they flow classically from OT, meaning from the device into the cloud? Or do you also provide IT/OT integration for your internal customers?

Alexander

For us and for our customers, the data naturally needs to be further processed in other systems. We also pull additional information from external systems. So the integration definitely happens. In smaller companies that might not be necessary. There, a cloud platform can be used as a standalone solution. I used to be a project manager in the IoT domain, and we implemented several projects where companies did exactly that and maintained all their data directly on the platform. For us as a relatively large company, that is of course not sufficient.

Makes sense. What does the concrete solution look like for you, especially from the user’s perspective? What have you built, and what does the result look like for the end user?

Alexander

We have many different user groups. There is no single end user, but rather different stakeholders who work with the data. If we go back to the original use case, meaning fault management and capturing the condition of the equipment, then our colleagues work directly in the web interface. This interface shows, at a glance, all information about the unit or the units connected to the respective gateway. You can view the history to see whether there are recurring messages that might indicate a fault that should have been fixed earlier. You can also refresh the current status of the equipment at any time. So our colleagues work directly in the web interface of the Cloud of Things, the Telekom product, together with the devices.
We also use the platform extensively for fleet management, which is one of my core responsibilities.

So your end users actually do not notice the scalable architecture behind it. That is more the device-management layer, which is integrated into the Cloud of Things. Somehow the devices need to communicate with the cloud layer. That is probably where PSsystec comes in, providing the connectivity and device logic, and from there the cloud-based device management takes over, right?

Alexander

Exactly. In the cloud, device management is fully integrated. We can build our own dashboards and interfaces there. When we receive devices from PSsystec, they are fully preconfigured. That is also a logistical advantage, because we have different use cases. The devices are labeled accordingly and handed over with the right configuration for each purpose. From the moment the device is mounted on the wall, we can manage it through the platform.
And the scaling effect is enormous. Whether we have two hundred or three hundred devices or two thousand or three thousand makes practically no difference for the individual user in the interface.

That is the backend architecture you have built in a scalable way. Very interesting. Have you calculated a return on investment for your setup?

Alexander

Yes, of course we thought about that. One of the main factors is the hardware. I would say we can save by a factor of ten here. For us, it is an expansion based on actual demand. I can monitor individual systems or smaller parts of a building very well. The technology has limits, but if you understand those limits, you can use it very precisely.
I cannot give an exact number for the return on investment. For hardware procurement, I would say the factor of ten is realistic. Then there is the topic of connectivity. Connectivity and the offers available, and I would really say worldwide and of course from Telekom, have developed enormously in recent years. There are very attractive options on the market, which make smaller fleets viable as well. Tariffs and SIM cards that run for ten years are very affordable and work reliably.

Michael

The overall effect is that I increase operational reliability because I have more transparency. I have far fewer failures. And if you look at that over five or ten years, the impact can reach several million.
I can give another example from a different project, not from PASM. There are several tens of thousands of devices in the field. Through monitoring, they reduced service deployments by fifty percent, simply because the data allowed them to decide more logically whether they even needed to react, whether someone had to drive out or whether it was enough to restart a process. That alone saved fifty percent within a short time. And I will take a bold stance and say that if you have a clear business case, meaning you know what you want to do with the data, you will always see savings.

Very nice. That is a great example. And that is precisely what makes investments in IoT technologies and such products worthwhile. Service costs go down, hardware is cheaper and behind all of it stands a scalable architecture. Thank you for sharing that so openly.

[28:01] Transferability, scaling and next steps – Here’s how you can use this use case

Alex, a quick question for you. How is the project evolving? Are you already finished or are there future plans? What can we look forward to?

Alexander

We are probably never finished, that is how I would put it. We are constantly evolving. PSsystec introduced a new version of the SMARTbox to us this year, and I hope I am allowed to say that, Michael. This new version will be significantly more powerful. It will allow us to run small services directly on the SMARTbox and monitor larger systems as well. And all of this still within the IoT ecosystem, which brings major savings for us simply because of the scale.
Overall, this also has a certain impact within the group. Use what you sell, as they say. Telekom itself is very active in selling IoT products. Greetings to our colleagues in the IoT division. They also use the examples we have built together here.

Great. Maybe we will give another update in the future on how the project continues to evolve. Michael, one last question for you: Which new features or topics are on your roadmap? What is coming next year?

Michael

Yes, we are currently working on a very exciting new product that I can present here for the first time. We have two major product areas. One is energy data management and the other is remote maintenance. And especially in the remote maintenance projects we see that the requirements for data volumes, processing speeds, connection options and access to the end device or the machine are rising sharply. Existing mobile network based IoT gateways are reaching their limits.
That is exactly why we are taking the next step with a new, much more powerful SMARTbox. We have developed a version that offers more power, more ports, more computing performance and higher data throughput. A device that can confidently handle complex or highly connected systems.
Technologically, we are deliberately taking a smarter approach than classic VPN. Instead of opening external access, we use a fully encrypted cloud tunnel for the new SMARTbox ETH. This works without external software and without IT configuration. No open ports, no external access, just a secure tunnel. And again over mobile connectivity. In short, simpler, same level of security, but in a completely new performance class.
We believe that the new SMARTbox ETH will create real value. And we already see that some customers are very interested and are waiting for us to provide them with a model.

Yes, great.

Michael

And at this point I would like to thank Alex and PASM for going down this road with us. For saying, you have a great solution and we may need a little more here and there, but we are taking this journey together. I also have to say, it is really fun working with the team at PASM. It is a great group. Thank you for that.

Alexander

Thank you. I can only return the compliment. It is a collaboration on equal terms. And PSsystec has, in my view, shown a level of flexibility in many areas that we have not seen from other manufacturers.

Great. Then thank you both for being here today. I really enjoyed that we had a very concrete real world focus. The use cases were very clear. We talked about challenges, about learnings on the technical side and about the business case and the added value. We also talked about your collaboration and there were some technical details as well.
To wrap up, thank you again to both of you. All information about this episode, contacts and further details can be found in the show notes. Feel free to check them out. Have a great rest of the week. Take care.

Alexander

Take care.

Michael

Many thanks. Thank you.

Bye.

Michael

Bye.