Explainable AI, opportunities and risks

Thomas Sinnwell: Welcome to our new podcast episode. Today we will be talking about explainable AI. And I have been able to get a very special guest. Professor Doktor Antonio Krüger, he is the managing director of the DFKI and at the same time director of the research area "cognitive assistance systems" at DFKI. A warm welcome to you.

Antonio Krüger: Nice to be here.

Thomas Sinnwell: Yes, maybe I'll tell everyone a little bit about you. I also had to-, or I'm going to look at my cheat sheet. I can't even recite my own curriculum vitae fluently. Professor Krüger studied computer science and economics at Saarland University. Completed his doctorate in computer science in 1999. From 2004 to 2009 he was professor of computer science and geoinformatics at the University of Münster and Managing Director of the Institute for Geoinformatics. Since 2009 he has held the Globus Foundation Professorship for Computer Science at Saarland University. I would ask you to perhaps introduce the DFKI. Most of our listeners probably know what the DFKI is, but perhaps not all of them know exactly what the DFKI does. What do you do?

Antonio Krüger: Yes, the DFKI, the German Research Center for Artificial Intelligence, has set itself the goal to support the transfer from basic research to application through application-oriented research. This means that all directors at the DFKI are also professors at their respective universities. DFKI is now active in six "Bundesländer". And we have 1,400 employees, if we count the assistants, which of course we do. And at the DFKI we transfer the results with many industrial partners, in many projects. At the DFKI, we have over 300 projects running at any given time, in different areas. That is our mission, this idea of transfer, that is very important. And we have been doing it for 30 years as a non-profit limited company in a public-private partnership model, where both shareholders from industry and the public sector jointly support the DFKI.

Thomas Sinnwell: Yes, thank you very much. Today we would like to explore the question of what actually lies behind the term explainable AI. And I think it's a good way to start if we first talk about AI. The very first podcast that was produced here in this room, in which I was also allowed to be a host, I had the pleasure of talking to your Research Fellow, Doctor Jan Alexandersson. And we looked at AI, let's say, in a ramble, and also briefly looked at a few aspects, legal, ethical aspects. But not everyone will have heard this podcast, not everyone still has it in their head. I would like to ask you to explain very briefly what AI is?

Antonio Krüger: I don't know if that will work very briefly, but I'll give it a try. If you ask ten AI researchers what AI is, you will probably get ten different answers. In this respect, what Jan Alexandersson said and what I say won't be completely congruent. But artificial intelligence naturally deals with algorithms, with procedures from computer science that are closely linked to systems that do things where we, as humans, would say: "You need a certain degree of intelligence to make that work". From this you can see that this is of course also a moving explenation. So for example, back in the nineties, when Blue, from IBM, beat Kasparov in chess, the then world chess champion, then that was clearly a domain of artificial intelligence. It was even said: "Oh, now the crown of intelligence has been snatched from man". Of course, anyone can smile about that nowadays, noone would probably even really seriously argue that chess programmes that now run on smartphones, for example are AI. But you can tell from that, it's moving, the explenation is moving. But what is clear is that certain human characteristics like use of natural language, systems with which I can have dialogues, with which I can that I can have dialogues with, that I can talk to, I think everyone would agree that that has something to do with AI. Robotic applications, that is, physical behaviour, which we associate with a certain intelligence, for example a machine that assembles complex parts, that would be called AI. And of course, very importantly, and this was already at the centre of research back in the sixties, when the term AI was coined in the USA, is of course the ability of self-learning. That means learning from mistakes, learning from experience. And that means having a certain dynamic in a system, which is natural to us as humans. Of course, we would all hopefully claim that we at least learn after a certain number of mistakes. We learn and develop further. And if machines can do that, then we certainly also talk about artificial intelligence.

Thomas Sinnwell: Yes. When people ask me what AI systems are, I like to say that they are digital systems that contain algorithms developed by humans. Admittedly, that's a dry introduction, perhaps also an engineering one, but in my opinion it grounds the topic immensely. In order to dispel any question marks that may have arisen, I usually like to add that the algorithms in this digital system serve to enable the machine to learn from input data. So. And what does this learning mean? Seen from a very high altitude, there are two large classes of algorithms. And they are suitable for different areas of application. One class is used to recognise patterns in huge amounts of data. And it is precisely in this discipline that AI systems, we humans, are really vastly superior. In the second class, it's about learning specific features that can be given a meaning or an assignment. It sounds abstract now, but it's actually easy to explain. For example, when a camera sees a picture, the machine knows: "Okay, what I see here is a nut or a screw." Yes, that is certainly not the whole truth. And to rekindle the fascination after the dry introduction and to complete the topic a little more, I like to explain the structure of deep neural networks. The basic structure of these networks is already modelled on our brain and they pick up elements from it. But the great thing about these networks is that they are able to generate algorithms themselves within the framework of ring phases. In other words, the algorithms I've been talking about all along, in order to be able to learn specific features from input data. But you have now mentioned something, this self-learning and that it sounds as if an intelligent human being would also do that. What is the state of the art today and what is the DFKI working on? What are the topics that are burning under your nails now, that will determine the future of the DFKI?

Antonio Krüger: Well, at the DFKI, also due to our growth in the last 10, 15 years, we already say that we deal with all facets of artificial intelligence. That means everything that I just mentioned as exemplary, from natural languages, to robotics, to machine learning, i.e. self-learning systems, to a whole range of applications where AI plays an important role, the DFKI is on the move everywhere. And  of course, the aim is to continue to improve the processes that already exist. This means, for example, improving image recognition processes so that they can distinguish, recognise and classify things better than humans. I am thinking, for example, of systems that support radiologists in recognising tumours, certain types of tumours. In the meantime, there are procedures that actually make better decisions than doctors on average, i.e. they make the right diagnoses. But of course, since we as a transfer institute are committed to application, it is particularly interesting for us to bring AI into industrial application. That is a very, very important topic for us.

Thomas Sinnwell: What are you doing there?

Antonio Krüger: We are doing a whole range of things in the context of Industry 4.0, a term that was co-invented at the DFKI by Wolfgang Wahlster, with Wolf-Dieter Lukas and Henning Kagermann. They developed the term together and presented the first concept paper at the Hanover Fair like 15, 18 years ago, so it's a relatively long time ago. And it is indeed a very important topic for us, because industry is so important for Germany and for Europe, much more so than the service industry, for example. And because we in Europe also have a competitive advantage in the use of AI technologies and methods due to the strong SMEs that we have, due to the strong industrial companies. It is clear that in the consumer market, when we think of large internet platforms and so on, let's say there are also some players from Europe. But of course not nearly as important or as big as in the USA or China. But it also has to be said that here in Germany we have SAP, the market leader in business software. And that is not by chance. That is due to our strong industrial structure of medium-sized companies. And one of our tasks is indeed to take AI methods and bring them into industrial production. Robotics, predictive maintenance, for example, is a very, very important topic.

Thomas Sinnwell: Why don't you explain to the listeners what predicitive maintenace is? What's behind it? I don't think everyone knows.

Antonio Krüger: Predictive maintenance is about directly deducing from data collected from machines in the industrial process where maintenance work or quality defects are to be expected in the future and at what point in time. And this then leads to the fact that these machines are of course maintained in a more targeted manner, which means that they have fewer standstills. And at the same time, in principle, by adapting the maintenance intervals very individually to the processes and the machines, of course. Maintenance can then be carried out at lower cost. So that is an important aspect. The other important topic, I think, in Europe is human-robot collaboration. The big-, there are some really globally significant companies in Europe that are in the "cobot" space. "Cobot" refers to robots that can work closely and directly with humans in production. And not in a cage like the large, industrial robots that we know from these welding lines. From car manufacturing for example. And this is a topic that is much more advanced in Europe in research, but also in the companies, than for example in the USA or in Japan.

Thomas Sinnwell: I think what you say is very important. Because I often come across this view that AI is a topic that is being developed across the pond by the big players. But you've already mentioned it, it's heading in a very specific direction. And when it comes to this industrial application, or also to issues, to ethical issues, to legal issues, much more is happening here in Europe, in my opinion. Am I assuming right?

Antonio Krüger: You are right. I don't think it's all that surprising, because it's about the data. We talked about machine learning, machine learning needs data. Now, of course, the US and China have incredibly good access to consumer data because they have the corresponding platform services. And of course they learn with it and offer services for consumers accordingly. That's kind of obvious. But we have a great opportunity due to our high industrialisation, the high level of SMEs and the many hidden champions to work with high-quality industrial data. And that is something that is not really the focus of attention in either the USA or China, and where the data is not as qualitative as it is here. That is a great opportunity for us.

Thomas Sinnwell: Yes. And we are now in this industrial context. And I think that's a great transition to the core topic of our podcast, the topic of explainable AI. What is explainable AI?

Antonio Krüger: Well, I'll follow up on "what is AI". Yes, if you want to say what is explainable AI, I first have to say what AI is. We talked briefly about the concept of AI. I think one thing that perhaps came out of my from my attempt to explain it, is that it is not so clear what AI is. How AI works. There are many, many different methods and on top of that, the success of the last ten years in artificial intelligence, especially in machine learning. The great success of learning about huge amounts of data and using it and also tabulating it to improve all kinds of applications. And-.

Thomas Sinnwell: And these are areas where the machines are vastly superior to us. They are much better there.

Antonio Krüger: In some cases vastly superior. There are many areas where man and machine together are better than man alone or the machine alone. And in some areas it's still the case that humans are still ahead. So we have the whole spectrum. But the crucial thing about machine learning is that it's a statistical learning process. That means they take huge amounts of data. When I look at the great successes in the last ten years-, especially due to the scalability of the neural networks, both in terms of the architectures, so we have larger and larger neural networks that we use. Some of which also achieve better and better results. And we have the corresponding hardware on which we can train these networks, because this is very, very time-consuming.

Thomas Sinnwell: That has pushed the topic immensely.

Antonio Krüger: A big disadvantage of these statistical methods, especially neural networks, is that they are black box methods. This means that in case of doubt, we can, for example, feed an image to a neural network and a classification comes back out. That means the neural network tells me, yes, is it a tumour for example or is it not a tumour. But we don't know what's happening inside. The other observation is that statistical methods work statistically. That means that on average they are always-, they can be better than humans. But not for every decision. It's similar to what happens with humans themselves. In fact, it is neural networks are sometimes miles off the mark. Not very often, but a certain percentage of the time. And when we say that they are better than humans, that does not mean that they are 100 per cent correct. It means that they are perhaps one or two percent better than humans. That means that if humans make 90 per cent or 91 per cent of correct decisions, such a very, very good neural network-, is at 92 or 93 per cent. That means that eight percent are still wrong. And if I now imagine I'm at the doctor's and we're looking at an X-ray together. And there may be a dark spot discussed and the neural network says, "That's a tumour." And the doctor is not sure, then he cannot easily ask the system, as things stand today, how it comes to the conclusion that this is a tumour and does not simply have some other cause. And here it is really very important that we develop systems that can answer such questions. That's what explainable AI is all about. It's not only super important in medicine, but everywhere where wrong decisions cost a lot of money or affect personal fates. And there are many, many examples where I am firmly convinced that we need approaches that can explain how such a system arrives at a decision or a recommendation or a recommendation suggestion.

Thomas Sinnwell: Well, my perception is that we currently have an enormous technical gain, especially through machine learning methods, but that it also leads in part to a loss of control. Because, if I have black box approaches, is the result correct? Can I trust it or not? And if I want to accept that as a human being, then I still want to understand how the system arrives at it. And that is a very exciting topic in our domain right now. We develop monitoring systems with large IT infrastructures. Monitoring in a very positive sense. We try to find out if there are technical problems and how we can detect them early. To support the staff in analysing and solving these problems. But there are also questions about whether a cyber attack is taking place or whether data is being exfiltrated. These are sometimes very, very important issues for companies. And if I now have such a monitoring system and it takes a black box approach and says: "Oh, we are being attacked. We have this alarm. It has a high criticality. That's quite problematic." And maybe gives out another probability. You don't find that anymore, that was a few years ago. Then it is of little help to the person in front of the system. He asks himself the question: "Yes, am I being attacked now or am I not?" And if I want to accept what the machine says, then I want to know how it comes to that.

Antonio Krüger: Yes, I think that's important. Above all, that the systems also offer different degrees and levels of explanation. It also depends very much on who is being communicated with. In this case, it is an expert who already has a certain background. Maybe not an expert in machine learning, so he won't have a really good idea of how the system itself works and can draw conclusions from that. But he has domain knowledge, he knows his way around the domain, he has a certain experience in the use of certain tools. And that's where you have to start. In other areas, such as in the medical field, it makes a difference whether a system explains something to a doctor or to an end user, like the patient. That is a different level of explanation, so you need systems that can adapt a bit there. Because in the end, an explanation that I don't understand, even if it is logical, is not helpful. That is the crucial thing. That's just the difference with a mathematical proof, which is just logical and demonstrably correct. Nevertheless, if I understand the mathematical proof, then it is of no use to me, then I will have no confidence in such a system. And that is the big challenge.

Thomas Sinnwell: I heard a wonderful lecture in your house. It also dealt with ethical issues, legal issues. An example was presented and it also dealt with imaging procedures. And it was about the system learning whether this is a very serious disease or something that can still be managed. And what was actually done, the people who simply had a better prognosis, who were more mobile, were examined with a completely different system than those who were very seriously ill. And the algorithms had learned to recognise a specific feature on the photo, on the image data, and thus had this high degree of accuracy, that this person is now really at risk of death and this person is not yet. And that is, I think, a very good example of why it is so important to understand how the machine gets there. Couldn't it have told me, well, there's in the bottom right corner, there's a little signature or an element, that's the decisive factor for me, you would have known immediately that you had trained the machine wrong.

Antonio Krüger: And there are other well-known examples. There is one where a system learned to distinguish wolves from dogs. And the main criterion was the background. Because most of the image data of wolves were presented to the system against completely different backgrounds, mostly even with snow and so on. And these systems learn that from what they get. And one must never forget, they do not learn a real model. Although the computer scientist often likes to talk about the model, a statistical model. We have learned a model, but not in the sense as we as humans would say, a model from which I can derive predictions or even derive new things. And that is missing. And that is also noticeable in the modern language models, statistical language models. For example, GPT-3 from "open eye", which is one of the systems that has been trained with millions, billions of data sets and which also produces astonishing results, for example, rewrites stories itself and so on. But if you ask this system nonsensical questions, you get completely nonsensical answers. A human being would never answer like that. Because a person has an idea of what he is talking about. And above all, people usually have an idea when they don't know something, and that's why they say: "I'm sorry, I can't answer that question. It's nonsense." But modern AI systems actually answer every question somehow. You often don't get that presented. I think if more of these nonsensical system answers were made public, then people would understand much better what the limits of today's AI systems are. And we would also understand that we absolutely need this explainable AI, because you can't rely on AI systems one hundred percent, but only 92 or 93 percent. And the systems would never say, "I don't know." Or very, very rarely. Because-, it's difficult for such a system.

Thomas Sinnwell: Yes. And I think this is perhaps also a very good point at which we can talk about how it is possible to make an AI system explainable.

Antonio Krüger: Yes, that is of course partly a discussion in research. Nevertheless, there are already some very good approaches, different ways of making such AI systems explainable. And essentially you can distinguish between three phases differentiated. One is the phase before the use of such an AI system, where the AI system itself is perhaps configured or set up before it learns. For example, by using simulations to try to find out how systems react to certain things. That is a very, very important topic. Because we don't even have a lot of data yet to train systems. For example, if you think of a self-driving car that is supposed to avoid a child or a ball that rolls in front of the car, and then conclude that a child is about to run after it. Fortunately, we don't have much data on such things. And you might not want to collect real data in such situations. But what you can do, you can you can generate synthetic data. So we can basically like in a 3-D computer game, we can have virtual children running after virtual balls and we can have virtual cars driving. And you can actually learn from that. And if I then have procedures that predict what works well and what works badly, then I can already try to find out what works during the design process. That is the first phase, so to speak. The second phase is perhaps one of the most important phases, and that is during operation. If something is-, you mentioned the example of monitoring a network situation and explaining what is going on during operation. That's probably one of the most important things. And the third phase is afterwards, when something has gone wrong, to be able to analyse afterwards what actually went wrong. All three are important. All three require slightly different procedures. As a rule, we can look into this black box to a certain extent. And we can also deduce, as you said, whether, for example, a neural network always makes a decision based only on the image section. Then it is immediately clear to us, and we can also build systems that recognise this automatically, it's immediately clear to us that something can't be right and we can go deeper into this.

Thomas Sinnwell: But there are also quite beautiful mixed forms where people help again.

Antonio Krüger: Exactly.

Thomas Sinnwell: If I really make this neural networks identifiable, visualise the whole thing, then I often have a very good chance as a human being, if of course they are not too deep, to still be able to recognise quite well how the algorithm now arrives at this result.

Antonio Krüger: Exactly. Of course, you need a certain background knowledge for that. So for a certain user group, this is definitely a feasible way. For others, a natural language must be generated from this information. From a model that is built up from general knowledge or expert knowledge, which is combined with statistical models. That is also a very promising approach. We often talk about hybrid models that can generate a much higher degree of explainability for different user groups.

Thomas Sinnwell: Yes, it is precisely these hybrid approaches that are currently driving us quite a bit in-house. Of course, packet inspection gives us the chance to extract a lot of information that can be interpreted relatively well. You can also combine that with machine learning.

Antonio Krüger: This is actually something that can be observed in general in the use of AI in industry. There are hardly any companies that rely purely on a statistical method. It is a very important building block. The greatest successes in the last ten years have been achieved with this method. But now back to autonomous driving. No car manufacturer would think of checking the car's traffic rules only on the basis of a neural network. But of course, there are road traffic regulations that I can-, we computer scientists have no problem casting that into a corresponding model in rules. Why let a neural network learn that again. That's an interesting exercise, but it's also done in research. The systems achieve 99 per cent, but if I could reach 100 per cent, why should I be satisfied with 99?

Thomas Sinnwell: Yes. I think we have now succeeded in making quite a good breakthrough in this topic of "explainable AI". And yes, so in the old tradition, I would like to maybe summarise our findings. And yes, I think the goal of AI or explainable AI, can be explained quite well. Ultimately, it's about the human being being able to understand how the algorithm arrives at the result. And now I would like to ask you to perhaps explain again why this is so important from your point of view.

Antonio Krüger: Well, I think it's so important because it's important to remember why we're researching AI systems in the first place, at least at the DFKI. And that's because we want to build systems that support humans in doing the things that humans want to do and in solving the problems that humans have to deal with. That it's clearly a human-centred approach. It is also clear that for the foreseeable future, many of these AI systems will only be able to realise their full potential in close collaboration with humans. Because the errors that the AI systems make are still so high that without human supervision. I don't think in all areas, there are even some that can run fully automatically, but in many areas humans will still play an important role. And that's why it's clear that we have to work on this interface. Where man and machine work closely together and the machine's ability to explain this is a very important building block. On the other hand, it is of course also important that humans can specify exactly what they want from the machine. But that's another topic, we can do another podcast about that. But this direction is very, very important. And I believe that if we don't succeed in building systems that gain the trust of people in important, critical applications, then these systems will not have a chance to establish themselves.

Thomas Sinnwell: I think that's a wonderful conclusion. It's been a lot of fun talking to you about this topic. Thank you very much.

Antonio Krüger: Yes, thank you very much for the invitation and for the exciting discussion.

Thomas Sinnwell: Yes. It was a great pleasure. I would now like to give you a short preview of the next podcast, which I will be hosting. We are starting a three-part series on the topic of "cyber security". Also quite exciting. Of course, it has something to do with machine learning as well. Until then, I wish you all a good time. Bye!

Antonio Krüger: Bye.

So, that's it from us again. We hope you enjoyed today's episode and that we have that we were able to bring you a little closer to the topic of AI. You can find more links to the episode can be found in the show notes. And if you are interested in the wonderful world of technology and software development, we would be delighted if you would subscribe to us. In our next episode, we will start with a three-part series on the topic of Cyber-Security. Host Dr. Thomas Sinnwell is here for you again and has invited a special guest. See you next time. We look forward to seeing you!