In Michael Ende's novel "The Neverending Story", there is a passage in which Bastian, the protagonist of this spiritual odyssey through the imaginary realm of “Fantastica”, comes to the "City of Old Emperors ". This city is inhabited by other former travellers who, like himself, have lost a piece of their memory in return for every wish granted to them by the mysterious talisman "Auryn".
Completely deprived of their personality, they have finally become incapable of using their own imagination any further, and now pass the time by playing with letter cubes for all eternity under the supervision of a sadistic monkey known as Argax, knowing that probability makes it imperative that sooner or later, every single conceivable story will thus come into being.
I constantly have to think of this story when I follow the debate about so-called "artificial intelligence" - not because I would underestimate the actual progress of the techniques involved, but because I have no illusions about how those results will come about. Like the letter-dicing monkey from the "never-ending story", the AI programme can do nothing but combine the material made available to it again and again without ever being able to innovate. Admittedly, what modern chat and art programmes are now able to produce thanks to clever programming based on their databases is remarkable and will improve exponentially in the course of the next few years, perhaps even months. And yes, there is already the danger of a gradual displacement of humans even from many areas that until now seemed to be the natural reserve of humans because of their creativity.
But what seems even more dangerous to me are the consequences of the belief that AI will one day develop a real awareness and could thus become a partner, perhaps even a symbiotic bearer of men’s search for meaning - a development that is eagerly pursued by the transhumanist movement and, thanks to massive advertising, is also enjoying increasing popularity among many "progressively" minded people, or is at least regarded as "inevitable" and "without alternative".
From this point of view, it is no coincidence that the last few months have been marked by non-stop emphasis on the unexpected "beauty" of AI portrait or landscape "art" – and we cannot overlook the irony that the very political and media elite that usually wants us to believe that figurative art is ultimately retrograde, even fascistic, now refers to exactly the same figurative art in order to convince us that a computer has access to something like a sense of style or beauty.
Such a statement could hardly be sold convincingly through computer-generated "abstract" or even "performance” art. But there is a second, considerably more dangerous suggestion behind the first. Anyone who assumes that a computer programme is capable of consciously creating "beauty" must, by implication, ask himself to what extent "beauty" created by human beings is actually a unique feature of humanity, or simply the result of mere, albeit organically based, "algorithms". This question is anything but abstract, since it is intimately linked to the struggle for the definition of everything that constitutes the human being, his consciousness and, yes, also his soul.
If one believes the increasingly powerful transhumanists, there is no reason to oppose an ever closer connection between our society and AI, indeed ultimately the (alleged) transfer of human consciousness into a virtual "reality", since man is, at the end of the day, just one machine among others, whose self-awareness is pure fiction and whose belief in transcendence and soul conceals an outdated collective survival strategy.
On the other hand, there is the nagging doubt as to whether man can really be reduced to a bundle of algorithms and merely "imagines" his own consciousness (as if this "imagination" is not precisely what actually makes him human!) - a doubt that has found its most striking proof in the idea of the "philosophical zombie".
This is exemplified in the following question once described by Sue Blackmore, “So ... imagine that there is a zombie Sue Blackmore. Zombie-Sue looks just like me, acts just like me, talks about her private experiences just as I do, and argues about consciousness just as I do; to anyone observing her from the outside she is completely indistinguishable from the original Sue. The difference is that she has no inner life and no conscious experiences; she is just a machine that produces words and behaviours while all is dark inside. Could such a zombie-Sue exist?” (“Conversations on Consciousness”).
It seems that this is precisely the problem behind the apparent "consciousness" of the AI. Do we really have to attribute a consciousness to a subject just because it behaves in exactly the same way as other subjects with consciousness?
But this question also refers to the limits of the AI. Ultimately, an AI can only use the material made available to it in the way it has been programmed. If it has already happened that individual AIs have partially reprogrammed their own code, even this reprogramming has only followed the priorities laid down from the beginning, i.e. it does not produce anything really new of itself. The human artist, on the other hand, is capable of genuine innovation and not only shifts around for all eternity the material made available to him, but also develops it further in unexpected directions on the basis of his emotional and mental state.
This is especially clear in the example of AI art, as it is becoming increasingly apparent that the "stylistic" peculiarities of that art are mere plagiarism, since more and more painters are complaining that AI art is simply the result of their online works having been fed into the databases of the AIs by unscrupulous programmers in order to be transformed into the raw material of computer-generated "art"; sometimes by plagiarising even the most individual artistic mannerisms (such as the signatures).
What is supposed to suggest "soulful" inspiration is thus nothing else than computer-generated plagiarism on an industrially large scale - but without the honesty to name the "real" artist behind the style and detail treatment. One example among many may suffice. The point made above so may not refer to merely aesthetic or philosophical issues, but to a fundamental problem that has dangerous implications for the future of man and society.
For one thing, the question of the power we want to grant AI in shaping our everyday lives is becoming increasingly urgent. Convinced that it can now not only summarise and solve purely quantitative problems better than humans, but increasingly also philosophical, ethical and political problems, we run the risk of entrusting our future to a mechanism that can ultimately only ever imitate human decision-making through statistical evaluation of the available material, never capable of real creativity or feeling anything like "responsibility". Also not to mention the danger that the evaluation of that material depends entirely on the framework conditions of the corresponding programming and can therefore never be truly neutral anyway, since it is based on the implicit or explicit ideological decisions and preferences made by the programmer or operator. If you doubt this point, try to discuss conservative opinions with a chat-app!
On the other hand, we are approaching with breath-taking speed the moment when the connection between human consciousness and computer will become an actual option. One will have to regard the confidence-inspiring emphasis on the "human" sides of AI as an advertising measure for that ultimate goal. The consequences of such a perverse connection between man and machine, between flesh and computer chip, would be disastrous: the supposedly "harmless" use of digital material to "merely" enhance our memory or accelerate certain thought processes would leave us at the mercy of those who provide the material and inevitably affect the way we use our own brains, since it is well known that the particularities of any instrument also affect the imagination of the one who uses it.
The increasingly widely advertised possibility, on the other hand, of going the other way, i.e. of feeding the human consciousness into a virtual world, might be considered the ultimate self-destruction of the human being. For no matter how perfectly an AI might be able to analyse a human being and then reproduce him in a deceptively similar way, right down to his smallest thought processes and mannerisms, we must never forget that it is only a mere image that has just as much real self-awareness as those pathetic automata of the 18th century that delighted the courtly society of the Ancien Régime by pouring hot chocolate or playing the harpsichord.
So even if a programme gives the impression of having preserved the memory and consciousness of a human being who has migrated into the computer in order to live out his existence in a supposedly "unlimited" virtual space more intensely and more vividly than ever before and realise the dream of ultimate power and immortality: We must never forget that this new entity is not situated in a true conscious continuity with its human original, since with the discarding of the last remnants of organic identity, the process of “migration” of the human consciousness into the virtual space leads to the ultimate death of the human being and the switching-on of an algorithm just mimicking humanity.
Just as we as individuals would have no advantage whatsoever from our exact clone being created somewhere in the world, since our own consciousness is bound to our present body, it would be of no use to us if a virtual monstrosity were to be created that perfectly imitates us and with which we somehow "connect".
But it is to be feared that many people, in their striving for immortality and omnipotence, would be blinded by that illusion and might actually succumb to the temptation to transfer their consciousness into virtual reality - a process whose "success" many fellow human beings would probably also believe, since the algorithm created on the basis of the original person would logically reassure the remaining humans that everything is in order.
The end result of this evolution might even be the gruesome dystopia of an earth where humanity has voluntarily extinguished itself out of naivety and hubris, while a soulless and ultimately brainless computer runs a programme in which perfect replicas of those extinct people enjoy an apparently paradisiacal state, which in reality they can no more consciously enjoy than NPC characters can enjoy the landscape of the computer game in which they move: "virtual" humanity as a brainless, senseless and unconscious puppet game, behind which the sardonic sniggering of the devil would be heard ever more clearly.