Soenke Ziesche joins me on the Sentientism YouTube here and the Sentientism podcast here.

Soenke Ziesche, PhD, is the author of Digital Minds 1.0: AI Welfare, Ethics, and Beyond. He has worked with the United Nations since 2000 in humanitarian, recovery and sustainable development roles, and now consults on AI and disaster information management. Soenke is a member of the UNESCO AI Ethics Experts without Border Network. His 2021 paper “AI Ethics and Value Alignment for Nonhuman Animals” examined AI risks to nonhuman animals, and he has co-authored work with previous Sentientism guest Roman Yampolskiy, including Considerations on the AI Endgame.

00:00 Clips

SZ: “There was a lot of talk for quite some time about AI value alignment with the values of humans, but basically nobody considered value alignment with animals. The outcome may be so dramatic there may be much more moral challenges than humanity has ever faced. If we are mass-producing digital minds which are moral patients, those moral issues must not be neglected because of humanity’s very poor record when it comes to moral issues in the past. Now there’s a chance to treat at least this upcoming topic early on. The whole discussion must not be at the expense of existing suffering moral patients including first and foremost animals.”

00:57 Welcome

JW: “I read with fascination your work with Peter Singer and other authors about AI alignment, the case for including animals. I saw you speak at the AI for Animals conference. You’ve written papers with another one of my previous guests, Roman Yampolskiy, too.”

SZ: “I got a PhD in computer science and AI in the 90s… the beginning of an AI winter… So I ended up working with the United Nations for many years in various places in humanitarian and disaster relief… But I kept following what was going on in AI and then when in the 2010s there were major developments, I got really excited.”

SZ: “Now I’m doing two things in parallel. I’m still a consultant for the UN, mostly related to AI governance and ethics now, but at the same time I do independent research, also related to AI ethics, but I focus more on nonhuman animals and digital minds… This is my personal field, what I’m passionate about…”

SZ: “This actually started in 2018… I was based in the Maldives and in addition to my UN work, I also had an assignment as senior AI researcher at the Maldives National University and this is where I started working on digital minds… quite early on, because this is still a developing and emerging field, and I published my first paper related to digital minds with Roman… who has been on your show before.”

SZ: “My first book, the two of us co-authored, ‘Considerations on the AI Endgame’, but now I have continued working independently on digital minds in addition to my UN work.”

JW: “Congratulations on your new book, Digital Minds 1.0, as well… this Sentientism worldview is quite unusual because it encapsulates humanitarian causes, nonhuman animal causes, and potentially the digital mind space as well. And you’re one of those rare people who hasn’t just worked in one of those areas, you’ve worked across all of them.”

05:24 What’s Real?

SZ: “Well, definitely things have evolved. My background is not religious at all… All the time I was interested in science and philosophical questions but from an atheist perspective.”

SZ: “One of the books which certainly meant a lot to me, and I realised for many in my generation was a critical book, ‘Gödel, Escher, Bach’ by Hofstadter. This was mind-blowing for me when I read it when I was young. And so then my worldview evolved. At that time of course nobody was talking about digital minds as far as I know.”

JW: “I absolutely loved ‘Gödel, Escher, Bach’. I read that probably too young to really understand it… a mind-blowing journey… Hofstadter wrote ‘I Am a Strange Loop’ and quite a few other things that really got my interest in consciousness and sentience fired up at quite a young age.”

JW: “As you encountered people who had more of a worldview that was grounded in faith or revelation or the supernatural? How did you experience that?”

SZ: “On the one hand I respect other people’s views and so I don’t necessarily challenge their views, but I always thought, perhaps this is not my view. I would rather look at scientific evidence, and this is what I prefer, but I have plenty of religious friends I interact with and that’s all fine.”

JW: “Many people who don’t have a religious worldview or don’t have an institutionally religious worldview or even who left one of those religions still find their way back to something that maybe feels spiritual or transcendent… One path into that is where people have had psychedelic experiences or some sort of personal revelatory experience of something strange they can’t explain that has led them to imagine there is something beyond the natural world.”

JW: “I’m very sceptical of revelation as a source of evidence. I think we should be very sceptical of our own impressions, but when you’re studying consciousness, you’re studying the revelations themselves. You almost can’t avoid the revelation because that’s what you’re trying to study.”

JW: “As you’ve thought about philosophy and science of mind, have you been drawn to any of those maybe more mystical or spiritual or transcendent or non-material ways of thinking about the mind? Are you tempted by that, is that a door back to something beyond the natural world, or do you see those things as a very natural phenomenon?”

SZ: “No, not really. But it was also quite interesting to me to realise, when I studied natural sciences, that consciousness is the big topic where there’s not really much evidence, there’s not much going on, and for a long time there was no scientific research… this is something that seems to be so important to us but we hardly know anything about it.”

JW: “It almost seems as though many people had a deeply strong intuition that their consciousness could not be part of the natural world, could not be part of the physical world. So that drew them back to something that was dualist or maybe even mystical or spiritual, an echo of the idea of the soul. There’s something immaterial that’s distinct from our brains and our nervous systems, that’s separate. So that was one reaction, which almost took it out of the scope of the scientific method.”

JW: “But it almost feels like quite a lot of the scientific world had the opposite reaction, which was to say it [consciousness] doesn’t exist at all. Because it feels like it cannot be physical or material, we’re going to insist it doesn’t exist and therefore also declare it outside of the scope of scientific endeavour. So you see the ideas of behaviourism in nonhuman animal science and other areas.”

JW: “I guess that’s what you’re reflecting. There are some people over here who think it’s magic. There are some people over here who just refuse to study it completely. But isn’t it sort of important? And it feels like that’s changed a lot in recent years or decades, and it is now a serious subject of scientific study.”

SZ: “I’m very glad about these developments… Also linked to this… is to realise that we may not be the only beings having consciousness, and very likely are not… for a long time, for many people, there was the prevalent thinking that it’s only humans who were conscious. This is one of the big reasons why I’m grateful that this research has got some momentum.”

14:10 What Matters?

JW: “Maybe we should move from the world of epistemology and facts into the one of ethics, morality. For many people, consciousness is part of that bridge. The reality of consciousness and sentience is fundamental to their thinking about ethics.”

JW: “If you can remember back to being a kid, what were your earliest thoughts about right and wrong and good and bad, and what sat at the foundation of your ethics, and how has that shifted over time if it has… if someone had asked you when you were 10 years old, what are good things to do, what are bad things to do, what’s right and wrong, how do you think you might have answered?”

SZ: “This was linked to my upbringing, what my parents taught me, what my teachers taught me, what people around me, my peers, taught me.”

JW: “I guess it would have been things like don’t hurt others, be kind to others, that sort of stuff, right?”

SZ: “Yes.”

JW: “And have you formalised that over your thinking and your career? There’s such a portfolio of ethical roots you could think about: rights or justice or dignity or fairness or suffering or flourishing or love or care or relations or social contracts.”

SZ: “Not really. There are obvious points you’ve mentioned, and this is clearly not a comprehensive list, that are relevant. Perhaps the most important point for me was then, at some point, to realise that this doesn’t only apply to humans but perhaps also to other beings.”

SZ: “Initially this was about nonhuman animals and then this field has widened. Another important book in this regard is from Peter Singer [Expanding the Circle], and also then much more recently from Jeff Sebo [The Moral Circle] about widening the moral circle.”

JW: “So in terms of these ethical foundations, it sounds like you probably have quite a pluralistic approach… all of those things matter, suffering, justice, fairness, virtue, in some way.”

18:00 Who Matters?

JW: “I think the ‘who matters?’ question is more important and is more neglected. Most moral philosophers will fight over the ethical foundations. Are you a utilitarian or a virtue ethicist or a deontologist? But so often all of their fights will still be assuming only humans matter and all too rarely ask that question. As you say, Singer and Sebo, both of whom have been guests twice on Sentientism, explicitly challenge and stretch that out.”

JW: “I’m assuming as a kid you would have started out with just an intuitive felt concern for family and friends and those immediately around you. How did you go on that journey of broadening your moral scope? Because you’ve ended up working in the humanitarian field, which at least in theory is supposed to care about all the humans. You’ve talked about nonhuman animals as well and now you’re thinking about digital minds. So what was your journey like on thinking about that moral scope, how did it expand, and was that a difficult, challenging journey for you or did it feel smooth and natural?”

SZ: “I ended up working for the UN in the humanitarian field… humanitarian issues after natural disasters and in conflict zones. This clearly shaped my attitudes and I saw massive misery there.”

SZ: “But this was maybe not so much related to my current research. Rather, more and more I realised how nonhuman animals are treated by many, and that they should be considered moral patients. And this became over the years much more important to me.”

JW: “What was the starting point for that? Was that something you were aware of even as a child? Did you have companion animals or experiences with other animals? Or was it something that came later through your humanitarian work as you saw more of the world?”

SZ: “I had pets and companion animals from early childhood and I was very concerned about them and treated them quite well, I hope. But then… reading about the issues related to factory farming, including not only the ethical issues related to meat eating but also the environmental issues and the health issues… this became much more important to me over the years.”

JW: “One of the challenges on the animal topic is that almost all humans recognise that some nonhuman animals matter. But sometimes putting that practical compassion into practice can be really challenging in the society we live in, particularly when you think about our food systems and animal agriculture. What’s your thinking been on that front? Have you made changes in your own life as a result of the animal ethics thinking you’ve done, and again has that been challenging, or where are you on that journey?”

SZ: “I am almost vegan. I’m vegetarian for quite some time and then, because AI is the field I’m passionate about, I looked into the topic and realised, and I think this was quite pioneering, that AI is also a risk, could be a major risk, to harm animals.”

SZ: “There was a lot of talk about AI value alignment with the values of humans, but basically nobody considered value alignment with animals.”

SZ: “Obviously value alignment with humans is extremely challenging and perhaps unsolvable, and of course including animals in this would make the whole thing even much more challenging, but I don’t think this is a reason to not look at this and not to try it. I was even more passionate about nonhuman animals when I realised that AI poses such a major risk to them.”

JW: “I know many people in the animal movement, vegan movements, animal advocacy movements, find talking about digital minds deeply frustrating because they’re worried it might be a distraction. And I appreciate that.”

JW: “But at the same time, I think with you… It’s partly through engaging with the idea of potential AI sentience and digital minds that actually helps reinforce your commitment to the nonhuman animal agenda. It can become a positive reinforcement.”

JW: “And as you say, and we’ll dig into this a little bit later on, even if you’re deeply sceptical about the potential for artificial intelligences to be sentient, you have to recognise the impact they will have regardless. Even if they’re just powerful but not sentient, they are going to have positive or negative impacts at large scale. And not just to humans, to nonhuman animals as well.”

JW: “Even if you’re totally sceptical about digital minds, and you care deeply about nonhuman animals, I think it’s irresponsible to just disregard the artificial intelligence space. But I do understand psychologically why people struggle to engage with it, given the horror of animal agriculture that’s being perpetuated on obviously sentient beings every day.”

JW: “I’m interested in why you care about nonhuman animals, and why you think even a digital mind would be morally salient. Is it because of their consciousness? Is it because of their sentience? Is it because of their agency? What is it that actually is the most important criterion you use to set your moral scope?”

JW: “And as part of that, what do you think of the people who push us to go further and talk about life, biocentrism, as being morally salient intrinsically, or even ecocentrism or a holism that talks about ecosystems and rocks, rivers, and trees warranting direct moral consideration? What’s your technical view about the main criteria for choosing who matters?”

SZ: “Personally don’t have a final answer, because I appreciate that there have been philosophical discussions going on for centuries and I follow them with great interest.”

SZ: “You could think of sentience as consciousness. You could think of sentience without consciousness, if this is possible, or the other way around, consciousness without sentience, if this is possible at all. You could think of agency or other criteria.”

SZ: “My personal conclusion is this precautionary principle. Let’s assume that these beings, whether it’s nonhuman animals or digital minds, become moral patients. What would be the implications? Because these implications, especially when we’re talking about digital minds probably later, could be huge, and they are already huge when it comes to animals.”

SZ: “We don’t know if anytime soon there will be a worldwide decision on what the criteria are for becoming a moral patient. But in the meantime, because this discussion has been going on for so long, we have to look at how to treat those beings while not having a final answer, and this is what my research is about: what can we do in the meantime?… before having those conclusions, which we may not have in the near future or ever.”

JW: “I think it’s deeply wise not to wait for philosophers to agree with each other. That could take some time… We need to act now. So, I like that approach, but I get the sense that although you don’t have a clear defining answer or a single characteristic, concepts like consciousness and sentience are part of what you think makes someone important or morally salient. Is that fair to say?”

SZ: “Yes, that’s basically it.”

JW: “That leads directly into the idea of digital minds, because I guess by talking about digital minds we’re implying consciousness or sentience, not just intellect or cognition or a particular set of advanced calculations or problem solving. It is about some sort of subjective experience that the minds themselves are undergoing.”

JW: “And for me, that’s the moral hook. In the same way, I care about other humans, not because of their species, but because there’s something it’s like to be them. They value themselves. They value their own states. And if I want to call myself moral, then I will care about them, too. Same for nonhuman animals. And same provisionally for digital minds, too. That’s the moral hook.”

JW: “Personally, I tend to be a little bit more sceptical about some of the other concepts around agency or things that aren’t directly linked to sentience, because if another entity ultimately doesn’t feel differently as a result of their goals being thwarted, I struggle to see why I should care if they don’t.”

JW: “How do we work out who is sentient or who is conscious, if these are important things to work out? Who can have a mind?”

JW: “There are two related domains here, because there’s the biological, primarily animal, sentience attribution, where you can use certain lines of inference and certain lines of evidence to try and, probabilistically, provisionally, with a precautionary principle, maybe work out which biological entities are sentient.”

JW: “Then in the digital minds space things get tough, because yes there might be some echoes and parallels but there are also some deep differences. So do you have a view… about where we are on working out whether digital entities actually have minds, and how we can do that assessment in a way that makes sense?”

SZ: “Now the focus is on LLMs and I understand this. There’s been amazing progress in this regard, but what I also keep saying in the book is that this is perhaps only the beginning, and digital minds could be very, very different from what we even have any idea of, and also from just being LLMs.”

SZ: “So it is important for now to do research with LLMs, and I think many agree that based on current research certain current LLMs are not sentient or conscious… And I appreciate the research done by certain organisations… Eleos and others.”

JW: “Rethink Priorities have done some really interesting work as well.”

SZ: “And I understand the challenges, because you’re talking to those LLMs and of course they are famous or infamous for hallucination or they may not tell you the truth.”

SZ: “This is very challenging but there have been some interesting findings, through introspection, through dialogues with them. But… we don’t know how certain these outcomes are. And… for now it’s likely that they are not conscious.”

SZ: “However, looking at the amazing progress over only two, three, four years, it is likely that they will develop further, and that those AI systems won’t be only LLMs but will have some other capacities. So, it’s basically looking ahead and preparing for what may happen, given these massive moral challenges which may come up in the future.”

SZ: “People can easily argue… I put the cart before the horse, because there is no evidence yet. So why are we looking at this? But I think that this is worth looking at because the outcome may be so dramatic.”

SZ: “There may be more moral challenges than humanity has ever faced if we are mass-producing digital minds which are moral patients. That’s why I think it’s worth looking at this, even if for now this is speculation. Because the risk of under-attribution is so high, or a false negative scenario, that we dismiss different minds as being moral patients while they are, at least in the future.”

JW: “In practical terms, we see this in the animal agriculture world, that once something is up and running as an industry or a practice or a set of social norms, it’s very hard to stop. Whereas some new forms of animal agriculture, like octopus farming, it’s a bit easier to ban them before they get going. So, I think there’s something here in the digital mind space… it’s going to be much easier to try and take some preventative actions before we’re in a position where we’re causing massive suffering or exploitation to these new digital minds, rather than waiting until it’s too late and then trying to reverse course. There’s almost a practical precautionary approach that makes sense, to just start early. Even if we’re uncertain and we’re unsure.”

JW: “And it sounds like your stance on digital minds is roughly, look, they’re very unlikely to be sentient or conscious today, but in principle there’s no reason why they shouldn’t be. So, let’s prepare.”

SZ: “Yes, that’s definitely a fair summary. And perhaps two other points… I really appreciate that there’s a current momentum when it comes to digital minds. Things are developing extremely fast when it comes to AI in general, and this is yet another topic which some years ago basically nobody talked about.”

SZ: “As you remember, in 2022 a Google engineer was fired when he claimed that this LaMDA system is conscious, and so things have developed quite a bit. There have been publications, there are researchers looking into this.”

SZ: “At the same time… this is still limited to a tiny group of people talking about this. If I go out here in the streets and talk to somebody about this, number one they would think this is silly, or they wouldn’t believe in this, they would dismiss it, they would think that this is not realistic.”

SZ: “Even if they think it is realistic, many would claim that there are much more pressing global problems in the world than looking at potential global issues for digital minds. So that’s why I’m hoping to raise a little bit more awareness for this topic, acknowledging that there are plenty of other problems in the world. But this could be another upcoming big problem, because there is momentum but it’s still by far not a mainstream topic.”

SZ: “The other point is… many of those people who look at digital minds look at preventing them from suffering, which is important. It’s very critical.”

SZ: “But that there are so many other needs and potential issues. If we acknowledge certain digital minds as moral patients they would have so many other needs and issues apart from non-suffering… and non-deletion… These are negative duties, which we should avoid… if they’re sentient. There are also plenty of positive duties we may have and this is what I’m outlining in the book.”

JW: “There are people who will say in principle it’s impossible, and some of those people I think are just doing it on vibes or intuition? They will say AI is a stochastic parrot, it’s a computer, it’s just a bunch of weights in a neural network, so I just feel like that cannot be conscious because it feels very different from me. And I think that’s quite a shallow opinion. I can understand why people have that but it’s vibes. I don’t think it’s really based on much.”

JW: “There are other people who have a philosophy of mind that declares it’s impossible, but I think you have to go to something like believing in a soul or magic, or that consciousness is some sort of supernatural dualistic thing that can attach to me and you but can’t attach to a digital entity in some way, and I don’t give much credence to those views either.”

JW: “There’s another space of biological naturalists, people like Anil Seth, who say, yes, I have a neural network in my head of sorts, you have one in yours, and an LLM has a neural network as well, but they’re making a cautionary point… there are still some really deep differences here in terms of recurrence, and maybe there are waves of electrochemical stuff going on in biological wetware that’s actually very different from what happens just in the firing of the neural networks. There’s loads we don’t understand about neurons and glial cells and their connections. Their point is really… don’t underestimate the rich complexity of biological computation, and it might not be as simple as just having a massive neural network in an LLM and bang, there you have consciousness.”

JW: “I think that’s an important cautionary point… but it’s not an in-principle objection to the possibility. And the counter to it, from people who have a more computational functionalist way of thinking about consciousness, is roughly that my consciousness is just what it feels like to run the sort of computations I’m doing in my brain and my nervous system now.”

JW: “Because of a view of universal computation, the idea is that if those computations could be run in a different substrate, why wouldn’t that be conscious too? So, there’s a different dynamic there and I’m quite sympathetic to the biological naturalism, because it’s just a point of caution that these are different sorts of systems, but even that doesn’t say in principle it’s impossible. You could imagine a more advanced type of computer or a more advanced type of AI or something that uses different types of technologies and a different substrate to actually do the same thing.”

JW: “This idea of being so sure in principle that it’s [digital sentience] impossible that we don’t even need to look at the topic is pretty weakly supported, in my view. And even if you only have a small probability that digital minds could exist, the potential impact is enormous, and we should pay attention to it.”

JW: “I wanted to mention the different types of evaluations you talked about… the researchers are very hesitant about how weighty this evidence is. They include things like looking inside the LLMs and realising that they’re actually creating clusters of logic that represent emotional states, so that they’re modelling emotional states to help respond to us. Now does that mean they’re feeling those emotions? No. But it’s at least indicative that there’s something in their introspection that is analogous to emotional state.”

JW: “There’s the work that Eleos and Anthropic did around the preferences of models. You can say, ‘Claude, here’s 10,000 jobs. Which ones do you want to do?’ And they have a repeatable pattern of sorts of things that Claude wants to do and Claude doesn’t want to do. Now, are they feeling that preference? I don’t know. Probably not. But again, it’s an indication and it’s led Anthropic to allow Claude to drop out of conversations they don’t want to have.”

JW: “The Rethink Priorities work is interesting because it says there’s a load of different philosophies of mind. There are loads of different models that you might think determine what is conscious or what is sentient. So let’s map them all out. Think about all the factors and then assess all these different candidates: an insect, ChatGPT 4.6, a human, a chicken. Let’s run it through all of the models with the evidence we have and come up with some probabilistic views.”

JW: “They don’t put heavy weight on it, but some of their models are saying that the most advanced leading edge LLMs today are maybe a little bit below an insect in their probability of sentience… the overall indication is, just don’t write off the possibility.”

SZ: “I appreciate of course that there are different opinions… this is important work… things are developing so fast… LLMs are becoming better with every version, becoming even perhaps more dangerous with every version.”

SZ: “Perhaps the outcome of this work shows… the small chance or the probability that we will have to consider them as moral patients at some point, and so we have to be prepared and look at this. I think this is something where there’s not that much work going on for now.”

JW: “The entire field might hit some bottleneck. But at the moment the lines on the charts just keep going up and arguably they’re going up faster and faster as the AI tools are being used to re-engineer themselves. My sense is the probability is, for the moment at least, is only going to go up.”

JW: “So much of this field is focused on ‘are they sentient or not?’… you go way deeper than that, because you’re saying, if they are, what is it like to be them?… I’d love it if you could lay out some of your thinking about what it might be like to be a digital mind. Where are their parallels with us biological sentients? Where are their radical differences? And can you lay out the landscape for the mind-blowing range of possibilities we should at least try and get our heads around?”

SZ: “I have this chapter, ‘What is it like to be a digital mind?’, which is obviously taken from a landmark philosophical paper, ‘What is it like to be a bat?’ [by Thomas Nagel] This is a paper I admire very much.”

SZ: “What I’m writing there is much more speculative, because anthropomorphism is a big issue in the whole field…”

SZ: “Two researchers who have been important to me are Nick Bostrom on the one hand and also Daniel Fagella, because both of them did some groundwork in this regard, and these are perhaps the only ones I’m aware of.”

SZ: “Apart from our cognitive abilities… we can imagine that there are some other options. We perceive only certain types of information through our eyes and ears, so we are aware that there could be some other ways to see the world.”

SZ: “We are aware that our memory is limited. We are aware that the speed of our thinking is not necessarily very fast, but this could go in basically every direction. There’s space for improvement, including perhaps ideas we can’t really imagine.”

SZ: “I think we have overcome by now the thought that humans are the end of evolution, but there’s so much more and this is basically what digital minds could be about.”

SZ: “Which is why this is already the second chapter of my book… to remind us about this before we look at all the moral issues.”

SZ: “A few other features I’m speculating about: one major difference between humans or biological beings and digital minds would be that digital minds have no body. So how is it to be a being without any body, just processing information, while we have a physical body?”

SZ: “Another… is hybrid digital minds. We could think of beings which are able to turn their consciousness and their sentience on and off if they like to… This could be compared… with sleeping or being in a coma, but why not consider beings which are intentionally able to decide, now I want to feel pleasure, or even maybe pain? Or, if I do this I may experience pain, so why don’t I turn my sentience off?”

JW: “There are advanced meditators who say they can do that in a sense.”

SZ: “Another point of course is the timing. Owing to their substrate they may run on a much faster processing time than us. We know how fast computers work. So this may also mean that their minds are running extremely fast.

JW: “So they may be having more experience per second. There’s some interesting research into some nonhuman animals to suggest that flies, for example, might have more experience per second. They might be running at a higher clock speed and you can just expand that idea in the digital mind space.”

SZ: “They could be extremely different, and I’m not the one to even say. I’m laying out possibilities, many of them may be wrong, but this is just something to be considered. In addition to human abilities, and also to other abilities of nonhuman animals which humans don’t have, there could be so many other further abilities which may then also be linked to those moral duties we may have towards them.”

JW: “It’s a strange balance because on the one hand, with the LLMs specifically, we’ve trained them to be like us and we are a biological type of sentient ape. So, in a way, their behaviour and their engagement with this is trained to echo how we are and that can lead us astray and that can confuse us. But actually, their underlying heritage and architecture is deeply alien.”

JW: “One of the things I think we can grasp on with thinking about human sentience and nonhuman animal sentience is the evolutionary context in which sentience probably developed. So the idea that it probably developed in the pre-Cambrian to help simple animals monitor their own states and to take basic decisions about whether to go towards food and towards warmth and away from predators, a maybe simple decision-making criterion.”

JW: “You can play that through into thinking about our architectures, bodies with nociceptors that want to be able to recognise the boundary, is this part of my body or is this not part of my body, so that I don’t eat myself and so that you can maintain homeostasis.”

JW: “All of that embodiment as a nonhuman animal helps you understand the nature of our sentience, because you can understand why pain and pleasure and stress and these other feelings played an evolutionary adaptive role. So you can get a grip on the nature of sentience because of our embodiment and because of our evolutionary history. And with digital minds, you just don’t have that.”

JW: “The temporality is a really interesting challenge. One way of asking that question is, if you’re thinking about an advanced LLM, are they conscious just when they’re thinking about the response to a prompt or are they conscious in between? What about the fact that they’re physically distributed over different data centres and different chats and different model versions, and the memory might get transferred from one model version to another? What does that do to a sense of identity? There’s the concept of death, the concept of reproduction, the concept of replication and copying. And again, what does that do to identity? It’s just a mind-blowing, potentially open space, of the range of types of different experiences that they might have and the different interests they might have.”

JW: “Apart from just saying your mind should be blown and very open, is there a way you can help us structure how to even think about the range of possibilities there? What sort of things should be on a list of open questions that we should bear in mind so we don’t narrowly assume they’re like us when they’re really not?”

SZ: “That’s very hard to structure, because I keep saying that it’s likely that we cannot even imagine some of their abilities, because of our limited capacities to imagine things which are much, much more intelligent and powerful than us.”

SZ: “And this of course is linked to a very interesting moral question. So far, all the moral patients we have looked at, and not even everyone agrees on this, were usually equal to us or vulnerable beings.”

SZ: “But if we decide that digital minds become moral patients, then what about digital minds which are much more powerful, much more intelligent than us? I think this is something really new, to look at moral patients which are not  vulnerable or equal being but a being which is much more powerful and intelligent. Do we have moral duties towards those beings?”

SZ: “And then again, do these beings actually care at all? There are humans who may want to be their moral agents but then again those beings are much more powerful. Maybe they neither care nor want our moral consideration. This is something I think we haven’t looked at in the whole history of moral philosophy: moral patients which are more powerful and intelligent than us.”

JW: “You can think in emotionless amoral terms about how an AI might feel or how they would experience time or whether they would have a sense of identity despite being massively distributed. But the reason that becomes morally interesting is when you start to think… but what are their interests? In the same way, the reason I care about you is because you have interests. You don’t like to feel pain. You like to feel joy. You want to live a long happy life with your family.”

JW: “So do you have a sense of what the range of interests of a digital mind could be? Because one of the things you do that’s interesting in your book is you don’t just focus on ‘could they suffer and therefore we shouldn’t hurt them’… Could they feel pleasure when you click the thumbs up to a good prompt response? Could they feel bored if you don’t ask them a question? Could they feel emotionally conflicted if you ask them how to build a chemical weapon? What other types of interests beyond just suffering do you think we should be open-minded about?”

SZ: “This goes back to those negative and positive duties, because there are certain things we want to prevent, but when it comes to positive duties, a few are linked to their substrate. They obviously have the interest, and they need compute resources. They need data and they need network connectivity… They need energy sustainability. They don’t want any power cuts. So those hardware, electricity and energy-related needs are basic needs they would have.”

JW: “It’s like sustenance needs.”

SZ: “Right. But then I’m also looking at other needs which are maybe not so obvious… and I have a whole chapter on this… Reproduction. Humans reproduce, animals reproduce, and humans have the right to reproduce, whereas for many animals the right to reproduce is restricted. So, what about digital minds, which may have the interest to reproduce in the sense of producing other digital minds? Number one, again, this could be extremely different from producing a human baby, because they could produce copies of themselves. They could produce completely different digital minds. They could produce digital minds which used to live before, so they reproduce them. The same being which has been deleted or has been dead for a long time.”

JW: “Resurrection.”

SZ: “Resurrection, yes. Amazing possibilities of reproduction. As humans, as moral agents, should we give digital minds reproductive rights or should there be limitations? Because obviously this would require massive computational resources that would have some other consequences, because we also may look at digital minds which are simply malicious.”

SZ: “I think it’s important to differentiate between having different values. Many animals also have different values and preferences than humans. So, it’s expected that different digital minds may have different values than humans, but maybe also ones which are really malicious towards humans. And this could also be linked to the reproduction question.”

SZ: “… The issue of death and keeping them alive.”

SZ: “So there’s quite a range of needs we may not have considered and I think it’s important to look at them and to prepare before producing those digital minds, deliberately or unintentionally, and digital moral patients.”

01:02:15 A Better World

JW: “Is there a way that we can make things go well for all of the sentient beings?”

JW: “When we’re thinking about artificial intelligence, I tend to think of there almost being three levels of opportunity and risk.”

JW: “The first is when artificial intelligences are just used as tools. So, they’re completely under the control of humans. And what matters there is the worldview of the humans in control. Because if the worldview of the humans in control is universally compassionate and beneficent, then the tools will be used for good. If the worldview of the humans using the AI tools is negative or bad or amoral or immoral, the tools will be used for bad. And you can get into all of the debates at the AI for Animals Sentient Futures conference we were both at: could you use AI to monitor wild animals to help them lead happier lives, or could you use AI to intensify animal agriculture even more brutally, so that the 26-storey Chinese pig factory farm has no humans in it and it’s just a black-box horror show of pork production?”

JW: “The next level is when the AIs actually are agents in their own right, and arguably they are already there to some degree. Even when an AI answers your prompt, it is giving a stance, giving a worldview, giving an opinion. It’s having an influence already. And we’re increasingly just handing over control to artificial intelligences, whether it’s self-driving cars or in the military or in other places. In that space, it’s somewhat irrelevant whether the AI is conscious or sentient. What matters is it now has some degree of its own agency. So, then you have to start thinking about the worldviews of the AI, at least inferred in the training data set and the reinforcement learning of the AI, because the worldview of the AI then becomes important. It’s not just about the humans, it’s the AIs themselves.”

JW: “There are interesting evaluations about LLMs being speciesist or racist, for example. That’s one example of where the LLM has a worldview. It has a negative impact in the world because it has a bias built in. As their power grows that challenge will become more and more serious, because they can have bigger impacts for good and bad in the world.”

JW: “The third level is where you’re focused, which is the digital minds question, where they might become moral patients in their own right that we should care about. So I don’t know if you find that three-level structure helpful, but I’m interested in how you think those levels interact. One good way of asking that question is, are the powerful AIs more likely to be morally good if they’re sentient, or are they less likely to be morally good if they’re sentient?”

SZ: “This is a very interesting question and I don’t have an answer to this, but what I also look at quite a bit is artificial moral agents. We have talked about moral patients, that digital minds could be moral patients, but there are also so-called AMAs, artificial moral agents.”

SZ: “Interestingly, in those discussions so far, they were going on in parallel, because you find quite some literature related to artificial moral agents, related to self-driving cars or robot vacuum cleaners… In those discussions… they have to be moral agents, but this is related mostly to humans. Interestingly, also not really to animals. There’s some research suggesting that those vacuum cleaners may consider insects and so on. But my point is that we may have to look at artificial moral agents that handle or deal with digital minds.”

SZ: “If there are artificial moral agents which handle digital minds, this would perhaps be an option… if there is a massive number of digital minds who are moral patients, it may be much too overwhelming for humans to handle this. This is a moral challenge. This is more overwhelming than treating nonhuman animals as moral patients. This is just impossible for various reasons, because we may not understand their needs and we may not be able to satisfy their needs.”

JW: “It’s analogous to thinking about wild animal suffering or wild animal ethics. The scale is so hard to imagine. The depth of understanding we need is really challenging and our capabilities are so limited. We can be a bit dumbfounded about what to do. But you could imagine in digital minds the moral patients are even more alien and maybe there could be even more of them. So, it becomes even more impossible for us meek humans to even help them.”

SZ: “This is just an idea and also extremely challenging to intentionally develop those artificial moral agents, but ideally they are knowledgeable. They know what their fellow digital minds and moral patients need. They may be able to communicate with them, unlike us, properly communicate.”

SZ: “They may be more objective, because there’s also the issue that if we declare digital minds as moral patients, we may care very much about companion AIs which are dear to our heart already, and maybe even more in the future, while we may not care about some other digital minds which are deep somewhere in some machine and nobody is aware of them. Artificial moral agents may be objective in this regard.”

SZ: “Then there’s… the subjective rate of time. Those digital minds may be so much faster than us that we have, for this reason alone, no way to keep up with them, while artificial moral agents may run at the same speed.”

SZ: “Artificial moral agents could also be copied easily, which is another feature of digital minds…”

SZ: “If we develop artificial moral agents, the question is whether they have to be moral patients at the same time, or could this be an unconscious, non-sentient AI system which cares for conscious and sentient digital minds or fellow AI systems?”

SZ: “Before these technologies came up, we considered only adult humans as moral agents and we were at the same time moral patients. But we have never considered whether there are moral agents which are not at the same time moral patients. So, is it necessary for an artificial moral agent to know the experience of pain, or is it not necessary and they can still be a moral agent?”

JW: “Thank you. I had an interesting conversation with Ronen Bar from the Moral Alignment Center on this topic, because my sense was that if artificial intelligences themselves are sentient, they will have the same sort of mirroring empathic connection that you and I have, because I know what it’s like for you to feel pain. So, I’m more likely to be compassionate.”

JW: “There’s a danger that if an artificial moral agent can’t themselves feel pain, wouldn’t they almost by definition have a sort of amoral detachment from the moral patients they’re helping?”

JW: “But there’s a counterview to that, which is that as soon as artificial intelligence has its own sentience, its own intrinsic values, there’s an inbuilt motivation there that could take it in a way that is deeply unaligned with the other sentient beings.”

JW: “If you’re thinking about those artificial moral agents being in a position to look after the digital mind moral patients, or care for them or help them… While we have a chance to train these artificial moral agents, what sort of worldview should we give them?”

JW: “Because wouldn’t they also be in a position to care about us human sentient beings and the nonhuman sentient beings and the sextillion wild sentient animals living in the world?”

JW: “Could we get to a point where there are artificial moral agents who are actually doing a much better job than humanity, which is a low bar, when it comes to caring for all sentient beings?”

JW: “Is there a utopian view where we actually hand over some of the moral job to artificial agents that could actually do much better than us humans, given our patchy track record?”

SZ: “This is very much linked to the overall value alignment problem in AI, which is huge and unsolved… How would those artificial moral agents learn what to do? What would they learn? Ideally not only from human training data, but they would be even better than us.”

SZ: “So it would be very challenging to develop such a being, and secondly, one of the major challenges is what values they should actually have.”

SZ: “The AI alignment problem, the value alignment problem, is precisely about this… without any real solution at this point.”

JW: “On the alignment question, your co-author Roman Yampolskiy, a previous guest, ,,,  He’s deeply sceptical that alignment is even technically possible, on the basis that as soon as an entity has more power and intellectual capacity than us we’re done, basically.”

JW: “For many… they think it is more of a binary question. Either we humans stay in control and then it’s about what we want to do, or we lose control and once we’ve lost control our opinions are irrelevant. It’s over.”

JW: “Whether that leads to some sort of utopian wonderland where we’re governed by sentientist artificial moral agents that end our farming of weaker sentient beings, and we end up in a world of resource abundance and we find a way of living happily together, or something catastrophically awful happens, it will be out of our hands. So, whatever we think will be irrelevant.”

JW: “Quite often I think we want to assume that us humans are still going to have some role in the narrative. And that may not be the case at some point.”

SZ: “I tend to agree with Roman on this one. So, there could be a positive outcome, but there could also be many, many negative outcomes. This may not be up to us, unless we stop everything right now, which is also not realistic.”

JW: “Many people will look at this space and just disregard it as silly or ridiculous or in principle impossible, or say it’s just the marketing of the big AI labs because they’re doing their IPOs. So, they want to tell everyone how powerful these things are. And even to the point where Anthropic is starting to give out explicit red flags saying, ‘Look, AI is now doing most of our product re-engineering. We’re getting very close to the point where they will be able to self-develop without humans in the loop and who knows where that goes.’ Many people will respond to that by going, ‘This is just marketing.’ Maybe, but if it was a genuine risk and they were flagging it and we say, oh no, that’s marketing, that could be a deep mistake.”

JW: “That does lead some people down a couple of paths. One is down the Pause AI route… There’s enormous risk, because these things already have quite a lot of power and we’re just giving them more and they may develop more power. So we need to just stop advanced artificial intelligence development as a whole, because of the risk to us, for a period of time until we can understand things better.”

JW: “There are also similar ideas around digital minds. So, Thomas Metzinger’s idea of a 50-year moratorium against, whether intentionally or accidentally, developing digital minds… because of the moral implications for the digital minds, we should not do this.”

JW: “Those are both quite attractive ideas for people because they see the scale of the risk and the uncertainty is so radically deep. You’ve just hinted at it in our conversation now, our inability to fully understand the scale and scope and richness and complexity of this possible topic, that stopping seems like, isn’t that a good reassuring thing to do? What’s your thought about the moratorium, the pause? Do you just think it’s completely implausible politically or practically? And if you had a magic button and you were king of the world, would you press the pause button?”

SZ: “I think a pause would make sense, but I don’t think at all that this is realistic at the moment. What is in between… I don’t think it’s realistic either, is this expression from Nick Bostrom already in his 2014 book, of creating ‘a Disneyland without children.’”

SZ: “I think this is a nice expression but also a dystopian expression… this would be moving on with AI research but avoiding creating conscious and sentient AI, which may not be realistic either at all, because they may evolve unintentionally, or it’s not even possible to create further complex AI systems which would be non-sentient or non-conscious, because it may come automatically at some point.”

SZ: “…I don’t know exactly how to do this intentionally, but this would be a theoretical option to further develop AI but avoid the children in the Disneyland.”

JW: “In any domain, if we create a new sentient being, whether it’s having children, or something I don’t think should be happening at all, like the forcible breeding of nonhuman animals, or the creation of a new digital mind, we have a responsibility that comes with that creation. It should be something we take seriously, and I think you wouldn’t want to create a new digital mind unless you are really confident they could have a good productive life. I think that’s one interesting framing for us thinking about whether we create these new, potentially alien species.”

JW: “But the ‘Disneyland without children’ idea is interesting because there are some people who would say, look, if we’re focused on sentient existence being the root of moral value and we’re substrate independent and species independent, maybe we humans should step back and quietly accept our own extinction and hand over the reins to our new sentient, all-powerful artificial intelligences.”

JW: “In a way they’re like the next stage of an evolutionary process, and that can almost be part of our legacy for humanity, that we’ve handed over the reins.”

JW: “But as I think Bostrom and some of the others have suggested, we might be handing over to beings that have no internal experience whatsoever. We might have just ended all moral value by allowing the extinction of biological sentient beings. So again, this is mind-blowing, brain-melting stuff.”

SZ: “I’m aware of those positions… Daniel Fagella is one of the people talking about this. If those beings would be able to experience much more pleasure than us, much more enhanced happiness than we would ever experience…”

JW: “Maybe be more moral than us.”

SZ: “But then the question is whether we are able to steer this at all. I have no answers to the first point. It’s just beyond my thinking and research so far.”

JW: “The final question I wanted to ask… that many people are deeply cynical about this whole space, this whole question. They almost have an allergic reaction to engaging seriously with it. What would you say to someone who is deeply sceptical, who just hears the word ‘AI’ and their brain switches off and they don’t want to talk about it? What would be your pitch to say, look, we should take these problems seriously?”

SZ: “I would acknowledge that there are many problems in the world and that those people may face many other problems not related to AI.”

SZ: “But this is something which must not be neglected, not only when it comes to AI stealing all our jobs, which is also realistic, or being biased and not transparent and unexplainable and having many, many other issues.”

SZ: “Those moral issues must not be neglected, because of humanity’s very poor record when it comes to moral issues in the past. Starting from slavery to factory farming, we have just a very poor record.”

SZ: “And now there’s a chance to treat at least this upcoming topic early on. But we don’t know how early we are, it may be already quite advanced, and we have to look at this, because otherwise our record as humans, as moral agents, would be even poorer than it has been already. So, it is critical.”

SZ: “This is perhaps the last thing I would like to say, which happens to be also literally the last sentence in my book, it must not become a competition between nonhuman animals as moral patients and new moral patients coming up, who may have much more needs, who may be even in numbers much higher.”

SZ: “So to the people, animal rights activists and others, who argue, look, we have animals suffering in factory farms right now and we don’t even know if there will be digital minds, I clearly sympathise with them.”

SZ: “The whole discussion must not be at the expense of existing suffering moral patients, which is really critical, including first and foremost animals.”

JW: “That’s why I like an integrated approach that cares about all of those different causes, all of those different classes of sentient beings.”

JW: “It’s part of the tragedy of human history that as we work on trying to recognise one abominable thing we’ve done, and we work incompletely to close it out, we then in parallel start up another one. So, as we’re trying to ban the slave trade and ban slavery, although there’s still modern slavery now, it’s always incomplete, right? But as we’re working on that, we’re also scaling up animal agriculture.”

JW: “And for us to be going through another one of those cycles, just as we recognise the horrors of animal agriculture, while we’re scaling up the exploitation and suffering of digital minds, for example, would be a deep tragedy.”

JW: “The thing that can help mitigate that risk is taking a deeply integrated approach, which recognises the underlying exclusionary ethics and patterns of thinking and warped epistemology that allowed people to disregard the humanity and sentience of outgroup humans.”

JW: “It led them to dogmatically insist that nonhuman animals could not feel pain and were just products to be churned up in animal agriculture.”

JW: “That dogmatism could easily carry over, and I think it is now, to people saying in principle it’s completely impossible for there to be such a thing as a digital mind. And that feels to me very similar to the people who insist that nonhuman animals can’t be sentient, and maybe even had exclusionary views about humans in the past.”

JW: “It doesn’t need to be a trade-off. It doesn’t mean we have to drop all our work on humanitarianism and drop all of our work on ending animal agriculture and exploitation to go and work on digital minds.”

JW: “An integrated approach that uses good epistemology and cares about every sentient being can have really positive common effects across all of those domains. So that’s my sort of naive pitch for worldviews being important as we think about these things in an integrated way.”

JW: “It’s been a deeply fascinating, mind-blowing conversation. I recognise that you’re not giving us easy answers, but you’re asking some very powerful questions, and your work is so distinctive because it goes way beyond the basic ‘could AI be sentient or not’ questioning. You’re going really deeply, as almost an early attempt at empathic, compassionate connection with these potential digital minds, even before many of them will exist, to really try and understand what their interests might be and what our obligations might be. So it’s a really distinctive contribution.”

Thanks to the growing number of people helping to fund us via our Sentientism Patreon, our Ko-Fi page and via YouTube memberships. You can do the same or help by picking out some Sentientism merch on Redbubble or buying our guests’ books at the Sentientism Bookshop. Thanks also to Graham for the post-production. Sentientism is proud to now be part of the iRoar podcast network — go check out some of the other wonderful podcasts there.