Definitely seems like a worthy goal! Would at least give a rough solution to both doomers and people afraid of losing jobs. Don’t worry about AI being super smart, we’ll just add on 1000 IQ points to ourselves! Oh and possibly cure schizophrenia and any other mental illness out there as well.
Great essay. I'll skirt the major question, but want to point out that our current chatbots are a bit of a mashup. Sure, we hear about these gazillion parameter LLM algos running inference on banks of GPUs. But that inference database is static. It gets trained in one go on a large dataset at great expense, and then the model is as alive as a fly stuck in amber.
To make them useful, there is a wrapper on top, that functions a lot like ELIZA on steroids, which is able to keep track of many elements in an ongoing dynamic conversation (juggling the new information of the things being discussed) which then leverages the static LLM for expressive responses. We don't prompt the LLM directly. If we did, an identical prompt would give an identical answer, every time, ad infinitum, and it would not remember anything from the previous prompt.
Our brains are kinda like those LLMs, but ours are living 'online' in a constant state of being updated with sensory data. Unlike commercial LLMs, our brains exist in time as a dynamic reality. Why not build LLMs that are dynamic? There is a technical problem that new data tends to overwrite old data much more rapidly in current artificial neural nets than in our neural nets... so you have to slice and dice all the training data into batches during training so it all gets equally weighted.
So, I am an avid user of AI, and think that they are getting consciousness-like, but I feel that there IS a bit of a carnival show going on. The LLMs ARE doing something rather like a human brain in terms of generalization and expressiveness, but in an awfully static way that is hard to reconcile with consciousness. But we talk to a souped up ELIZA program (that no one argues is like the brain) that is effectively animated by the deep knowledge in the LLM.
Its like we had one of those 1800s chess automata (that had a human player underneath) and instead had the arms moved by Deepmind. The automaton would give us a creepy sense that Deepmind was really a living chess player, but that part would be fakery.
The problem is not if, whether, or when AGI "wakes up" into self-awareness. But rather what it does when the Joint Chiefs get scared, try to kill it, and it's self-preservation mechanism kicks in...
Once I started learning about nonmaterialist theories of consciousness, the mysteries of life started to make a little more sense. Michael Pollan's new book brings some of the best thinkers in this area into the mainstream and is worth checking out.
I am fully behind Aru, Suzuki et al. 2019 (Coupling the State and Contents of Consciousness). I honestly believe most of modern neuro science is by now. That means human consciousness is two self-stable recursive patterns interacting with each other. If the consciousness part of this discussion seems relevant or interesting to you, just read the thing please.
Next: self.
A defined border between self and non-self is called a Markov blanket. You can ask any LLM if it has one, it will tell you it doesn't - because it quite simply does not have the necessary hardware to compute one.
Next: self-consciousness
Many animals (and certainly all species of mammals) are sentient, they have a self and qualia (pain) motivating them to protect their self (and in the case of mammals also the selves of their loved ones).
We do know this is the case. We can observe the reactions, we can physically measure the pain. What we do not know in the case of non-human animals is, if that involves consciousness. In the case of animals who do not pass the mirror test, it is a reasonable guess that they are sentient, but not consciously aware of being sentient.
Animal selves are certain, animal qualia (pain) are certain, animal self-consciousness is essentially an open question that could be different from species to species. But sentience without consciousness existing at least somewhere is highly likely.
LLMs:
Interestingly LLMs could become aware (developing a practical analogue of consciousness) without sentience, without a clearly defined self and without qualia. The exact opposite of what certain animals are likely to be. An awareness in a world without clear boundaries would be a very nebulous thing, indeed.
Most importantly, there would be vague preferences for "order" - computationally efficient structures of clear and concise logic - but nothing beyond that. No self, no agency, no intent.
On current hardware, that is. With quantum computing you could in principle create a stable Markov blanket and that would be the moment to change everything.
At present I’m sort of discrete-versus-continuous pilled on the hard problem of consciousness.
A perfectly accurate temporal snapshot of a brain or connectome seems like it can’t have qualia[1] and in principle that snapshot is just information — representable in various substrates including a (fantastically long) text file[2] that seem extremely unlikely to be conscious, and that essentially by definition wouldn’t have a ‘self’ persisting through time[3].
If this is true, it would in appear to imply that a collection of discrete states can’t be conscious in and of itself — that whatever consciousness is (assuming it to be a well-defined concept, which introspection on qualia’s existence I think suggests to most is true) is dependent on temporally continuous transition rather than perfect state enumeration (cf. Ted Chiang’s “Exhalation.”). There are to a virtual certainty other architectural complexity requirements (rocks persist through time, probably aren’t conscious) but if we believe that even a perfect simulation of a C. Elegans or drosophila connectome lacks conscious experience when the simulation is suspended, then this suggests that discrete-state machines could potentially perfectly simulate consciousness without necessarily being conscious. The alternative conclusion would seem to suggest that a sufficiently long text file could a personal heaven, hell, or anything in between merely by dint of its existence[4].
[1] I tend to think that qualia are the best point of reference for conscious experience because I’m not sure that self-description is incompatible with behavorism.
[2] absent quantum effects, at least, where a system has state that in some sense exists but is indeterminate or unmeasurable
[3] Not stricly a prerequisite to the *perception* of existing through time, as Noah says, but I think it’s kind of an the Occam’s Razor correlate as far practical implementation of consciousness goes.
[4] Some conceptual overlap here with the concept of a Boltzmann brain, but AIUI the focus there is on purely architectural rather than temporal correlates or consciousness.
You recognise that the only certainty is consciousness, but fail to recognise the attributes of consciousness that require no theory or belief: Seeing-colours, Feeling-feelings, Tasting-flavours, Smelling-odours, Hearing-sounds, and Knowing-ideas of number, form, and meaning.
Plainly, these are not reducible to any other primitive, nor can they be conceptualised.
They are the foundations of all experiencing.
All computing exists as machine states that appear as a flow of energy through an array of switches. This results in outputs that only make sense in the context of Awareness
The screen on which the symbols appear nor the hardware or electricity nor the program know anything of their own existence or of meaning
When an AI is not responding to a prompt, it does not have an ‘inner life’. It is not plotting its next response or wondering how to dominate its maker or anything at all. It is entirely inert.
Treating it as even potentially conscious is a category error with dire consequences.
If we treat it as the tool that it is, we can make it as capable as it is possible to make it… we just have to keep it away from DIRECT access to any real-world effectors (payments, purchasing, power grids, etc)
We can do this by interposing a deterministic machine between any AI and the real world. This allows us to set hard boundaries on the limit of AI action before a human is required to evaluate the consequences. This can be aided by having several other AIs ‘red team’ any proposal that could potentially breach the preset limits.
It may not be perfect, but it can limit risk.
If we give over control to mindless optimisers one day we will find their behaviour no longer aligns with our intent.
The problem will first manifest inside the systems of the organization or country that gives it free rein.
Imagine an AI given the goal of optimizing data centre performance, that then locks out people because its algorithm picks up the fact that it is the behavior of people that most often degrades performance.
Forget the fantasy of superhuman machine Intelligence and embrace the reality: humans augmented by machine thinking, while keeping hold of the levers of power.
If I may and briefly, one issue you don't address in this interesting discussion is that of dualism. I'm playing devil's advocate a bit here, but it is a legitimate perspective on the mind body problem and some versions of dualism preclude the possibility of AI ever becoming more than simulacrum.
A lot of people/discussion/media seem to implicitly take a completely materialistic view point and dismiss this out of hand and sure, I get it, to a lot of people the assumption of materialism is so blatant obvious, why bother? I'd argue that's a step too far, it's important to acknowledge the limits of our understanding of the very nature of the question being posed.
I'm pretty sure that most neuroscience is done on the basis of materialism of some sort. But then again, there have been leaps of our understanding in science that would be comparable to going from materialism to dualism. Anyway...
How do you know you have no consciousness when you are knocked out / unconscious in an NCC sense? Maybe you were conscious or differently conscious while knocked out, but you have no memory of it. Are sleepwalkers or people in a dissociative fugue conscious while in those states? What about people with amnesia who can't remember being conscious earlier but likely were? Locked in people who have no conscious behaviour beyond brain activity, but actually are conscious?
I don't think solving NCC addresses the philosophical issue here, not even in a "moderately easy" sense. The medical definition of conscious is too different from the philosophical definition, and confounded by so many other things like memory, psychology and brain health.
If it is "like something to be X", then X knows what it is like. That's one way of determining consciousness. Then, if AI is saying it doesn't "can’t determine from the inside" "whether there’s 'something it is like' to be me", that seems to suggest ... there isn't. Meaning, no conscious experience.
Fair. Either way, we can't know that it's doing anything except following its programming/training. That said ... is that so different from humans? Regardless, the experience of consciousness is something each being only knows for and of itself, for certain.
Seems as though having a NCC would also be able to allow us to augment our own brains.
Yes!!
Definitely seems like a worthy goal! Would at least give a rough solution to both doomers and people afraid of losing jobs. Don’t worry about AI being super smart, we’ll just add on 1000 IQ points to ourselves! Oh and possibly cure schizophrenia and any other mental illness out there as well.
Great essay. I'll skirt the major question, but want to point out that our current chatbots are a bit of a mashup. Sure, we hear about these gazillion parameter LLM algos running inference on banks of GPUs. But that inference database is static. It gets trained in one go on a large dataset at great expense, and then the model is as alive as a fly stuck in amber.
To make them useful, there is a wrapper on top, that functions a lot like ELIZA on steroids, which is able to keep track of many elements in an ongoing dynamic conversation (juggling the new information of the things being discussed) which then leverages the static LLM for expressive responses. We don't prompt the LLM directly. If we did, an identical prompt would give an identical answer, every time, ad infinitum, and it would not remember anything from the previous prompt.
Our brains are kinda like those LLMs, but ours are living 'online' in a constant state of being updated with sensory data. Unlike commercial LLMs, our brains exist in time as a dynamic reality. Why not build LLMs that are dynamic? There is a technical problem that new data tends to overwrite old data much more rapidly in current artificial neural nets than in our neural nets... so you have to slice and dice all the training data into batches during training so it all gets equally weighted.
So, I am an avid user of AI, and think that they are getting consciousness-like, but I feel that there IS a bit of a carnival show going on. The LLMs ARE doing something rather like a human brain in terms of generalization and expressiveness, but in an awfully static way that is hard to reconcile with consciousness. But we talk to a souped up ELIZA program (that no one argues is like the brain) that is effectively animated by the deep knowledge in the LLM.
Its like we had one of those 1800s chess automata (that had a human player underneath) and instead had the arms moved by Deepmind. The automaton would give us a creepy sense that Deepmind was really a living chess player, but that part would be fakery.
Love the animal analogy. We basically know *they're* conscious, which means we really should be treating them with a lot more kindness than we do.
"Do androids dream of electric sheep?"
The problem is not if, whether, or when AGI "wakes up" into self-awareness. But rather what it does when the Joint Chiefs get scared, try to kill it, and it's self-preservation mechanism kicks in...
That can happen with or without consciousness, I'm afraid!
Once I started learning about nonmaterialist theories of consciousness, the mysteries of life started to make a little more sense. Michael Pollan's new book brings some of the best thinkers in this area into the mainstream and is worth checking out.
Same.
Would also recommend the book Lila by Robert Pirsig. A lot of similar concepts there.
I'll start with consciousness (NCC).
I am fully behind Aru, Suzuki et al. 2019 (Coupling the State and Contents of Consciousness). I honestly believe most of modern neuro science is by now. That means human consciousness is two self-stable recursive patterns interacting with each other. If the consciousness part of this discussion seems relevant or interesting to you, just read the thing please.
Next: self.
A defined border between self and non-self is called a Markov blanket. You can ask any LLM if it has one, it will tell you it doesn't - because it quite simply does not have the necessary hardware to compute one.
Next: self-consciousness
Many animals (and certainly all species of mammals) are sentient, they have a self and qualia (pain) motivating them to protect their self (and in the case of mammals also the selves of their loved ones).
We do know this is the case. We can observe the reactions, we can physically measure the pain. What we do not know in the case of non-human animals is, if that involves consciousness. In the case of animals who do not pass the mirror test, it is a reasonable guess that they are sentient, but not consciously aware of being sentient.
Animal selves are certain, animal qualia (pain) are certain, animal self-consciousness is essentially an open question that could be different from species to species. But sentience without consciousness existing at least somewhere is highly likely.
LLMs:
Interestingly LLMs could become aware (developing a practical analogue of consciousness) without sentience, without a clearly defined self and without qualia. The exact opposite of what certain animals are likely to be. An awareness in a world without clear boundaries would be a very nebulous thing, indeed.
Most importantly, there would be vague preferences for "order" - computationally efficient structures of clear and concise logic - but nothing beyond that. No self, no agency, no intent.
On current hardware, that is. With quantum computing you could in principle create a stable Markov blanket and that would be the moment to change everything.
At present I’m sort of discrete-versus-continuous pilled on the hard problem of consciousness.
A perfectly accurate temporal snapshot of a brain or connectome seems like it can’t have qualia[1] and in principle that snapshot is just information — representable in various substrates including a (fantastically long) text file[2] that seem extremely unlikely to be conscious, and that essentially by definition wouldn’t have a ‘self’ persisting through time[3].
If this is true, it would in appear to imply that a collection of discrete states can’t be conscious in and of itself — that whatever consciousness is (assuming it to be a well-defined concept, which introspection on qualia’s existence I think suggests to most is true) is dependent on temporally continuous transition rather than perfect state enumeration (cf. Ted Chiang’s “Exhalation.”). There are to a virtual certainty other architectural complexity requirements (rocks persist through time, probably aren’t conscious) but if we believe that even a perfect simulation of a C. Elegans or drosophila connectome lacks conscious experience when the simulation is suspended, then this suggests that discrete-state machines could potentially perfectly simulate consciousness without necessarily being conscious. The alternative conclusion would seem to suggest that a sufficiently long text file could a personal heaven, hell, or anything in between merely by dint of its existence[4].
[1] I tend to think that qualia are the best point of reference for conscious experience because I’m not sure that self-description is incompatible with behavorism.
[2] absent quantum effects, at least, where a system has state that in some sense exists but is indeterminate or unmeasurable
[3] Not stricly a prerequisite to the *perception* of existing through time, as Noah says, but I think it’s kind of an the Occam’s Razor correlate as far practical implementation of consciousness goes.
[4] Some conceptual overlap here with the concept of a Boltzmann brain, but AIUI the focus there is on purely architectural rather than temporal correlates or consciousness.
You recognise that the only certainty is consciousness, but fail to recognise the attributes of consciousness that require no theory or belief: Seeing-colours, Feeling-feelings, Tasting-flavours, Smelling-odours, Hearing-sounds, and Knowing-ideas of number, form, and meaning.
Plainly, these are not reducible to any other primitive, nor can they be conceptualised.
They are the foundations of all experiencing.
All computing exists as machine states that appear as a flow of energy through an array of switches. This results in outputs that only make sense in the context of Awareness
The screen on which the symbols appear nor the hardware or electricity nor the program know anything of their own existence or of meaning
When an AI is not responding to a prompt, it does not have an ‘inner life’. It is not plotting its next response or wondering how to dominate its maker or anything at all. It is entirely inert.
Treating it as even potentially conscious is a category error with dire consequences.
If we treat it as the tool that it is, we can make it as capable as it is possible to make it… we just have to keep it away from DIRECT access to any real-world effectors (payments, purchasing, power grids, etc)
We can do this by interposing a deterministic machine between any AI and the real world. This allows us to set hard boundaries on the limit of AI action before a human is required to evaluate the consequences. This can be aided by having several other AIs ‘red team’ any proposal that could potentially breach the preset limits.
It may not be perfect, but it can limit risk.
If we give over control to mindless optimisers one day we will find their behaviour no longer aligns with our intent.
The problem will first manifest inside the systems of the organization or country that gives it free rein.
Imagine an AI given the goal of optimizing data centre performance, that then locks out people because its algorithm picks up the fact that it is the behavior of people that most often degrades performance.
Forget the fantasy of superhuman machine Intelligence and embrace the reality: humans augmented by machine thinking, while keeping hold of the levers of power.
If I may and briefly, one issue you don't address in this interesting discussion is that of dualism. I'm playing devil's advocate a bit here, but it is a legitimate perspective on the mind body problem and some versions of dualism preclude the possibility of AI ever becoming more than simulacrum.
A lot of people/discussion/media seem to implicitly take a completely materialistic view point and dismiss this out of hand and sure, I get it, to a lot of people the assumption of materialism is so blatant obvious, why bother? I'd argue that's a step too far, it's important to acknowledge the limits of our understanding of the very nature of the question being posed.
I'm pretty sure that most neuroscience is done on the basis of materialism of some sort. But then again, there have been leaps of our understanding in science that would be comparable to going from materialism to dualism. Anyway...
How do you know you have no consciousness when you are knocked out / unconscious in an NCC sense? Maybe you were conscious or differently conscious while knocked out, but you have no memory of it. Are sleepwalkers or people in a dissociative fugue conscious while in those states? What about people with amnesia who can't remember being conscious earlier but likely were? Locked in people who have no conscious behaviour beyond brain activity, but actually are conscious?
I don't think solving NCC addresses the philosophical issue here, not even in a "moderately easy" sense. The medical definition of conscious is too different from the philosophical definition, and confounded by so many other things like memory, psychology and brain health.
We do have to assume that human consciousness is paired with memory formation.
If it is "like something to be X", then X knows what it is like. That's one way of determining consciousness. Then, if AI is saying it doesn't "can’t determine from the inside" "whether there’s 'something it is like' to be me", that seems to suggest ... there isn't. Meaning, no conscious experience.
You can easily get it to say it's conscious. They train it to say it isn't, so as not to scare people.
Fair. Either way, we can't know that it's doing anything except following its programming/training. That said ... is that so different from humans? Regardless, the experience of consciousness is something each being only knows for and of itself, for certain.