Mishcon
I'm delighted today to be joined by Dr Aurel Sauri, who is a Professor of Public International Law at Exeter University, um, to talk to me about the laws of armed conflict and how it might apply to defence technology in a modern context. Um, before we get into our conversation about your work, if you could just give me a very brief introduction, um, as to who you are and the work you've been doing most recently.
Dr Aurel Sauri
Uh, great to be here, thank you very much for having me. So as you’ve just mentioned, I'm a Professor of International Law at the University of Exeter, and my main area of specialisation is the International Law as it relates to conflict and security. So I do a lot of work on the law of armed conflict and various other areas of operational law. I spent the last decade perhaps looking at the legal dimension, legal aspects of grey zone warfare, hybrid threats, that sort of stuff, but then more recently I've pivoted to looking at AI in particular, various technologies about AI and how the law of armed conflict applies to the military use of AI.
One of the projects I'm involved in is an expert manual that is being prepared under the auspices of the US Military Academy at West Point, and I'm one of the core drafters looking at how the law of armed conflict applies to AI.
Mishcon
So the reason I thought this conversation would be so interesting in the context of our series on the future of defence and defence technology is that we see, and even actually during my time at BA Systems a long time ago, we were made aware or shown concepts or future capabilities that were quite some way beyond what our forces had available to deploy with at that time. And that was a long time ago, and, and that probably has accelerated hugely even since then. And so now companies, as we're well aware, that are venture backed or privately funded are developing capabilities that go way beyond what conventional weaponry might be able to do now. And so in the context of that focus on the future, I think what is forgotten about is what we're actually not just able to do, but what is permissible in, in, in a conflict. And so this conversation hopefully will shed some light on that. And you've written a paper recently, which I'm not sure if it's been published yet.
Dr Aurel Sauri
Not yet, no.
Mishcon
So I won't give too much away, but soon to be published on precisely how, um, the laws of armed conflict might apply to AI systems, for example. So can you give us a very high level, and I'm sure you could talk for hours, but a very high level on, on how the laws of armed conflict generally apply to both states, state actors and agents, so members of the armed forces, and then we can go in from there.
Dr Aurel Sauri
So the law of armed conflict is, is a branch of international law. It's one of the older branches of international law and essentially what it does is, as it says on the tin, it regulates the conduct of hostilities. So it applies to States, but also to other parties, uh, in armed conflict. One of the main reasons we have these rules is, is essentially not to prohibit warfare, but to limit it, to impose certain restrictions and to channel violence and warfare within certain areas.
So to channel violence against lawful targets, but then at the same time impose obligations on States not to use violence, not to conduct attacks against, for example, civilians or civilian objects. And, again, the main reason for doing that is humanitarian, um the idea is to limit warfare and to expose those to hostilities who are members of the enemy's armed forces, to focus on them, but then those who are innocent civilians, those who are other people requiring protection, um, objects like cultural objects, to protect those from, from, uh, hostilities.
Mishcon
And recently the UK, as have many other, um, governments and MODs within those governments, have been focussing heavily on AI and the adoption of AI in, in modern warfare. And we in the UK have published a series of documents in the last, I would say, 10 months. So in each of those documents or sets of guidelines, there's very little reference made to compliance with laws, right? So we see lots of references to ethics. I think you pointed out in your article, an awful lot of reference to ethics, which is one thing, but that is distinct from the law. So why do you think that is, firstly? Because I would have thought, I mean, obviously I'm a lawyer, but that sounds quite important, right? Complying with the law or building capabilities that comply with the law, kind of important and then, you know, what your take on, on the lack of reference to the law in this context is?
Dr Aurel Sauri
Yeah. So when you look at what has been published, uh, you know, in terms of doctrine, official statements by, by the UK, um, there seems to be, you know, quite an obvious imbalance between, on the one hand, the ethical considerations, which get a lot of, a lot of attention, there's a lot of detail, the UK has adopted five ethical principles, it has adopted guidance on how these should be implemented. And then when you look at the law in particular, the law of armed conflict, there's this very clear recognition that this is important, that the UK has to comply, that the UK will comply, but it kind of stops there, so there's very, very little detailed guidance. Now, why is that the case? I think, first of all, this is hard. So, uh, one of the difficult, difficulties in this whole field, and this is exactly why we're writing a manual on this and why guidance is needed, is there's a lot of uncertainty as to how the rules actually apply to AI, partly because the rules themselves are uncertain, but AI as a new technology raises genuinely interesting novel questions, which haven't really been decided, haven't been settled. Um, probably high-level doctrine is not the place where you want to, you know, do a deep dive in some of the legal intricacies, and also it's not necesSaurily the kind of audience that you want to, want to, uh, communicate these sort of legal details to. Um, so, I think it's understandable why there is that sort of very general level treatment of, of the law, certainly compared to the ethical dimension, but I think it does have two important consequences or implications.
One is, when you talk to, um, commanders, soldiers, you know, rank and file in the armed forces, there's a pretty widespread conflation between the ethical principles and the law, um, and, and there's this sort of general feeling that the law isn't that particularly difficult, I mean, you just apply the law, right, so, you know, what's the big deal? Um, and you don't really have to worry too much about the legal implications, because we've, we’ve dealt with it under the heading of ethics. Now, that's not the case, there are some, as I mentioned, some very difficult legal questions, uh, that need to be resolved, and I think the focus on the ethics, that very prominent focus on the ethics, at least in this republic communications because there's a lot of work on the law going on in the background. But in terms of the public focus on the ethics, that breeds a little bit of complacency in the mind of operators, the non-lawyers.
The second major implication I think is for industry and for, you know, the wider public, if you will, for very similar reasons. I mean, when you look at how AI is being developed under the innovation, the drive is very much bottom-up, so unlike in, in previous decades where, you know, uh, the MOD would commission a tank or, or an aircraft carrier, what we see now is a lot of that acquisition process is driven and the innovation is driven from the bottom-up. Many of the, of the companies that are involved in this are new to defence, I mean, there's a lot of dual-use technology, and as a result, many of them also, you know, there are a lot of start-ups, so in terms of size, they don't necesSaurily have, uh, uh, the legal capacity in-house, nor do they have the expertise, because they're new to defence.
So, there is a bit of a gap, I think, understanding gap in industry that the law of armed conflict is incredibly important to their customer, and unless the customer, in this case the MOD, gets better in communicating that and explaining why law compliance matters, what that involves, how that translates into say, capability requirements or performance metrics and communicates this to industry, I think there, there's a gap that needs to be filled.
Mishcon
So to contextualise it for people listening who are not lawyers, which is probably everyone, um, could you give me an example, maybe taking the proportionality rule, which I know is very complicated in itself, maybe take that rule, um, and could you just maybe apply it in a scenario where an AI system is being deployed, so maybe an autonomous targeting system, and obviously, um, uh, targeting web is very topical at the minute, and the UK are focussing on it as our, sort of partners and allies, so, and then just say how that maybe works together.
Dr Aurel Sauri
Yeah, proportionality is a rule that many people have heard about, but it's also one of the more challenging rules to, to apply, uh, even if you, you know, before you even get to AI and automation.
Mishcon
Yeah.
Dr Aurel Sauri
But, um, I mean, just so briefly to remind us what, what proportionality is and why it matters. So the law of armed conflict prohibits parties from directly targeting civilians, in other words, I cannot shoot civilians, I cannot attack civilian objects. However, the law recognises that even if you direct your operations very clearly, uh, against military objectives, lawful targets, it's almost inevitable that there will be collateral damage, that civilians might get caught up in the attack, that there will be damage to, to civilian objects. So the law of armed conflict accepts collateral damage, it is permissible, or incidental civilian harm, but it imposes a maximum ceiling. So it says it's fine, but you cannot go further than this, and that's really the role of the proportionality rule. So proportionality says that attacks which cause disproportionate collateral damage in relation to the concrete and direct military advantage expected from the attack, that these are prohibited.
So, how can AI come into the picture? And I guess, um, so one, one of the difficulties with AI, it's that there's so many use cases, there's so many different technologies involved, but, um, we can probably think about AI coming in, and sort of from modest to quite extensive ways. So I think at the bottom, uh, you might think of AI, uh, being used for the purpose of intelligence analysis. So a military command saying, right, you know, here's a system and it helps me to identify collateral concerns, you know, are there any civilians in my area of operations, what is the kind of military advantage? You might then use that same system to quantify that, so how many civilians, um, look also at military advantage and try to put a number on that. I think the next step would be to say, right, we've quantified this now, so now I'm asking the system to actually make the assessment to compare these two.
Mishcon
Yeah.
Dr Aurel Sauri
You could then ask the system to come perhaps up with a recommendation, you've done the numbers, so is the civilian harm, incidental civilian harm, excessive in relation to the military advantage? And then I guess this is where we come to like a full automation, you might, um, also end up with a system where you put everything onto the same platform, so you have the platform sensing how many civilians, deciding, calculating the civilian harm versus the military advantage, and then perhaps also if it has that capability and, and it decides that the attack would not be disproportionate to actually conduct the attack. So there's a whole host of things that AI and automation could do.
Mishcon
So on that point, um, who is responsible? So let's say the machine does its thing, you know, there's a target that's selected, it is wrong, let's just put it in really simple terms.
Dr Aurel Sauri
Yeah, yeah.
Mishcon
Is it the UK government, the MOD, is it the individual operator, could it be the company, right?
Dr Aurel Sauri
Yeah.
Mishcon
Is there sort of corporate liability there?
Dr Aurel Sauri
The, so the main, I guess, liability would be if, if you have industry actually and as I say, um, participating in the conduct of hostilities. The second mode of liability would be, uh, for instance, under the UK's Geneva Conventions Act 1957, which imposes liability for aiding and abetting grave breaches of the Geneva Conventions and their additional protocols. So for instance, again, if a company were to supply, uh, an AI system, a weaponised system or something like that to, uh, to a State, it could be the UK it could be someone else, um, and they know that this will be used to commit war crimes, that may well then trigger the, uh, the aiding and abetting provisions of the Geneva Conventions Act.
Mishcon
Right, so, so broadly, there are potentially significant implications.
Dr Aurel Sauri
Yeah.
Mishcon
Perhaps not directly, but certainly indirectly and as I said, more practically from design process, from why, you know, it makes life very difficult to sell a system that could potentially do serious harm or put, you know, um, war fighters and users in, in sort of, in harm's way of liability, I suppose.
Dr Aurel Sauri
Yeah.
Mishcon
So then my next question is, do you think, going back to the example we talked about, where, you know, in that autonomous system, somewhere in the process, a human will have to intervene and start to make some calculations of their own, and the judgement call piece. Do you think we're putting, potentially, in that scenario, human users under an undue burden, right, cognitive burden, for one thing, right? In the middle of a conflict, in the heat of a decision-making process, they're having to process not only real information coming in, tactile information, all the rest of it, then they're having to look at a machine, trust that machine to come up with some answers, review that machine's outputs to some degree or other, you know, depending on what the machine is doing, and then make a decision off the back of that?
Dr Aurel Sauri
So from a legal point of view, you know, the kind of scenario that you described where the human is, is reviewing and has to, you know, shoulder some of that cognitive burden, um, it may be a bad thing, and it may raise legal concerns, equally, it may not necesSaurily, because this may actually be the price that you have to pay to the human being involved…
Mishcon
Yeah exactly. Yeah.
Dr Aurel Sauri
…for making sure that there's compliance, because the machine can do some of these functions, but not all of them. So therefore, I mean, we talk a lot about human-machine teaming, so that also can happen, certainly, in, in terms of how you comply with the law. I mean, you get the machine to perform some cognitive functions that you are reassured it can do consistently with your obligations, but then the other, the hard questions, that's where the human has to come in. So actually slowing down and, and, you know, taking on some of that cognitive burden, which is the human's to begin with, anyway, right? So this continuing to discharge some of that cognitive burden may not necesSaurily, uh, be a, be a bad thing. Um, of course, also, you know, the reason why, why we want to use AI is to speed things up, is to have better situational awareness, is to make use of the data.
And, I mean, some of that will be directed towards, you know, finding, you know, enemy formations and, and speeding up the targeting cycle, but some of it equally must, because it's a legal requirement, be focused on preventing harm to civilians. Um, so again, AI can play a very significant role, and I think for now, in terms of where the technology is, a lot of that will have to be very much human-machine teaming, where AI does what it can do in a way that, where the human is reassured it's consistent with the law, and then the harder, more difficult bits, the human will continue to have to carry out.
Mishcon
Yeah. This might be an unfair question to you, but if you were given all of your experience and everything you've just sort of described, and I know you go into much more detail in your paper and other published works, but in your experience, how early on in the design process, if you are a privately backed, you know, defence technology company doing your own R&D and your own product ideation. At what stage in that process would you start to consider compliance with public law, laws of armed conflict?
Dr Aurel Sauri
Ideally, as early as possible, which is, you know, how long is a piece of string, that kind of answer.
Mishcon
Yeah, sure.
Dr Aurel Sauri
Um, but the reason I say, ideally, as early as possible, because, um, what you want to avoid, what industry, I'm sure, wants to avoid, but also what MOD from a, you know, capability development perspective wants to avoid, is going down a road where you put a lot of effort, you put a lot of money, you put a lot of, you know, resource into developing a product or a service, fail to look at what legal concerns, law of armed conflict implications arise, you do that at a later stage, and then it turns out, well, actually, you know, there are lots of issues that need to be addressed. At that point, you know, when you're down that road, um, you know, by quite some time, it might be very costly to, to fix this. There might be delays. You might say, well, actually, there's only so much I can do, so I need to constrain the capability. You might not, in the end, even be able to use that capability. So that, that is obviously something that both industry, I, I suppose, and government wants to avoid. So from that point of view, the earlier you can look at this, the more sense it makes, the more you can avoid this outcome. I think what that means for industry is, uh, first of all, industry needs to be aware of its own LOAC obligations, but perhaps even more importantly of the law of armed conflict obligations of the client, which is the State, which is the MOD. So first of all, understand what is it that the UK can and cannot do, and how your product relates to that.
Mishcon
Yeah.
Dr Aurel Sauri
I think, secondly, it's important to do, essentially, a legal risk assessment. Like, you know, we're thinking about developing this product, this particular service, so what concerns, what questions does it raise? What do we need to, to look out for? I would also say, engage early with the MOD. Uh, I mean, they might not be able to answer all your questions, but certainly, there's, there’s nothing wrong in asking those questions and, and to seek clarification as to what the MOD thinks the legal requirements are.
Mishcon
Fantastic. Well, thank you very much. Um, I found that particularly interesting. Um, I'm sure people watching did, and I really appreciate your time.