Great essay. I'll skirt the major question, but want to point out that our current chatbots are a bit of a mashup. Sure, we hear about these gazillion parameter LLM algos running inference on banks of GPUs. But that inference database is static. It gets trained in one go on a large dataset at great expense, and then the model is as alive as a fly stuck in amber.
To make them useful, there is a wrapper on top, that functions a lot like ELIZA on steroids, which is able to keep track of many elements in an ongoing dynamic conversation (juggling the new information of the things being discussed) which then leverages the static LLM for expressive responses. We don't prompt the LLM directly. If we did, an identical prompt would give an identical answer, every time, ad infinitum, and it would not remember anything from the previous prompt.
Our brains are kinda like those LLMs, but ours are living 'online' in a constant state of being updated with sensory data. Unlike commercial LLMs, our brains exist in time as a dynamic reality. Why not build LLMs that are dynamic? There is a technical problem that new data tends to overwrite old data much more rapidly in current artificial neural nets than in our neural nets... so you have to slice and dice all the training data into batches during training so it all gets equally weighted.
So, I am an avid user of AI, and think that they are getting consciousness-like, but I feel that there IS a bit of a carnival show going on. The LLMs ARE doing something rather like a human brain in terms of generalization and expressiveness, but in an awfully static way that is hard to reconcile with consciousness. But we talk to a souped up ELIZA program (that no one argues is like the brain) that is effectively animated by the deep knowledge in the LLM.
Its like we had one of those 1800s chess automata (that had a human player underneath) and instead had the arms moved by Deepmind. The automaton would give us a creepy sense that Deepmind was really a living chess player, but that part would be fakery.
The point that the system doesn’t change as a result of interactions with it is an important one (Dwarkesh Patel puts this in the terms that the LLMs don’t have “continual learning” the way humans do). But as a minor correction, these systems wouldn’t respond to the same prompt in the same way even without the wrapper - the bottom level LLM forms a probability distribution over next words as a function of previous words, and then generates one of those words randomly, and then repeats the process of using the current string to predict a probability distribution and then generate. There is an adjustable parameter called “temperature” that affects how probability is distributed between the most likely, next most likely, third most likely, and so on, and if you set temperature to 0, then it will behave deterministically (even with the wrapper!) but if temperature is non-zero, then the system won’t be deterministic.
This is such an incredibly important point as to the moral implications here.
All of the AI systems have the property that they can be paused, rewinded, re-played, or replicated, as many times one wants, at effectively no cost. They can be made to re-produce the exact same output from the same input over and over.
If living creatures had this property (e.g, if a loved on were killed in an accident then we could just re-summon them into existence as they were the minute before the accident), then we would have a totally different moral way of going through the world.
I think that this property has a lot more to do with the "moral worth" of LLMs than any efforts to define or evaluate consciousness. I'm don't think consciousness discussions are particularly fruitful in general. The slight mangling of Bentham's comment on animal welfare: "the question is not, are they conscious, but, can they suffer", seems much more relevant.
There are drugs that block neuroplasticity in humans, and humans with hippocampus damage who can't form new memories. If a human can't update their synaptic weights, would it be ok to harm them? Would they suddenly not have any internal experiences?
Definitely seems like a worthy goal! Would at least give a rough solution to both doomers and people afraid of losing jobs. Don’t worry about AI being super smart, we’ll just add on 1000 IQ points to ourselves! Oh and possibly cure schizophrenia and any other mental illness out there as well.
At present I’m sort of discrete-versus-continuous pilled on the hard problem of consciousness.
A perfectly accurate temporal snapshot of a brain or connectome seems like it can’t have qualia[1] and in principle that snapshot is just information — representable in various substrates including a (fantastically long) text file[2] that seem extremely unlikely to be conscious, and that essentially by definition wouldn’t have a ‘self’ persisting through time[3].
If this is true, it would in appear to imply that a collection of discrete states can’t be conscious in and of itself — that whatever consciousness is (assuming it to be a well-defined concept, which introspection on qualia’s existence I think suggests to most is true) is dependent on temporally continuous transition rather than perfect state enumeration (cf. Ted Chiang’s “Exhalation.”). There are to a virtual certainty other architectural complexity requirements (rocks persist through time, probably aren’t conscious) but if we believe that even a perfect simulation of a C. Elegans or drosophila connectome lacks conscious experience when the simulation is suspended, then this suggests that discrete-state machines could potentially perfectly simulate consciousness without necessarily being conscious. The alternative conclusion would seem to suggest that a sufficiently long text file could a personal heaven, hell, or anything in between merely by dint of its existence[4].
[1] I tend to think that qualia are the best point of reference for conscious experience because I’m not sure that self-description is incompatible with behavorism.
[2] absent quantum effects, at least, where a system has state that in some sense exists but is indeterminate or unmeasurable
[3] Not stricly a prerequisite to the *perception* of existing through time, as Noah says, but I think it’s kind of an the Occam’s Razor correlate as far practical implementation of consciousness goes.
[4] Some conceptual overlap here with the concept of a Boltzmann brain, but AIUI the focus there is on purely architectural rather than temporal correlates or consciousness.
This is an interesting argument, but if it were true I think it would prove humans were not conscious. If I understand correctly you're essentially arguing there is no 'integration' for consciousness. Even a series of infinitely small discrete states sampled at infinitesimally small time scales driven by the underlying process of consciousness taken together cannot be a continuous process, only a continuous process can be conscious, and so certainly a series of discrete samples with wider spacing cannot be conscious. This immediately runs into a problem, as neurons themselves process information discretely as far as we can tell, and there are clock frequencies that synchronize activity of large networks of neurons across the brain, and these frequencies are not particularly high. Somewhere on the order of 0.5 Hz to 100 Hz for known networks with neurons themselves operating at maybe 1-500 Hz. If your brain is locally a collection of discrete states, and synchronized in networks with discrete states, does that prove you are not conscious?
At the limit if time itself proved to be quantized, would that prove humans were not conscious?
I think that a proof of discretized Planck time would at a minimum require significantly revising if not discarding the theory altogether (or else conceding humans aren't in fact conscious).
Re: "This immediately runs into a problem, as neurons themselves process information discretely as far as we can tell, and there are clock frequencies that synchronize activity of large networks of neurons across the brain," -- My present understanding is that humans don't really have anything like the notion of clock cycles in the way that computers do (even if neurons do have firing/non-firing states), and also that brains have both substantially greater inbuilt topological parallelism and non-separable recursive elements as well as a lot of essentially analog state. Discrete assembly-code-level operations on a centralized processor seem like they really do end up being subject to the Leibniz's Mill argument even if on its own it proves too much (assuming both that humans are in fact conscious and that a substrate-independent copy of static information can't be.) -- it's extremely hard to isolate consciousness somewhere the discretized execution of "popl %eax; movl %eax, %ebx"
They don't only have discrete firing/non firing states and rate coding, their activity does need to be synchronized across the brain, and we have strong evidence from lines like two flash fusion thresholds and alpha oscillations that suggests conscious perception is at least partially quantized on these clock pulses. Your ability to discern whether two flashes are independent or whether they blur into one depends on your brain's alpha frequency (a certain synchronizing clock pulse). The faster your alpha, the shorter the inter pulse difference you can perceive without seeing only one flash. Notably this threshold is much slower than the firing rate of individual neurons, so it's not that your neurons can't fire fast enough to encode the stimulus. Two light pulses within a single period of Alpha are perceived as a single pulse. Alpha, beta, gamma, theta, etc rhythms represent real networks in the brain that must synchronize their patterns of activity to produce meaningful conscious perceptions.
Alpha and the rest are not square waves. They're continuous waves and their amplitude determines sensitivity, and there is temporal integration through firing rate decay and ramp up. There is bleed across the pulses. Even so they are real and do in some real way represent a 'sampling rate' of conscious perception (by proxy through the sampling rate of that brain process). When you disrupt these synchronizing clocks consciousness itself slowly degrades until it stops entirely, as in anaesthesia.
The problem is not if, whether, or when AGI "wakes up" into self-awareness. But rather what it does when the Joint Chiefs get scared, try to kill it, and it's self-preservation mechanism kicks in...
Good point. The assumption with "Skynet" is that as Natsec-related AGI, it would be seriously buffed to detect and neutralize any code that could impair its functioning, or shut it down.
Yes, though if its builders had any sense, they would ensure the proper political mechanisms were able to shut it down as well as activate it. Nobody is going to deliberately release an autonomous weapon they can't control.
There's the rub. "Skynet" has to be programmed to prevent foreign actors from either trying to control or destroy it. Meaning that if it's smart enough, it could also locate and disable any Pentagon installed kill-switches.
If I may and briefly, one issue you don't address in this interesting discussion is that of dualism. I'm playing devil's advocate a bit here, but it is a legitimate perspective on the mind body problem and some versions of dualism preclude the possibility of AI ever becoming more than simulacrum.
A lot of people/discussion/media seem to implicitly take a completely materialistic view point and dismiss this out of hand and sure, I get it, to a lot of people the assumption of materialism is so blatant obvious, why bother? I'd argue that's a step too far, it's important to acknowledge the limits of our understanding of the very nature of the question being posed.
I'm pretty sure that most neuroscience is done on the basis of materialism of some sort. But then again, there have been leaps of our understanding in science that would be comparable to going from materialism to dualism. Anyway...
As doctor I’ve seen hundreds if not thousands of elderly with either dementia or delirium and these disease processes completely wipe out their humanity and personality. Anyone who wants to argue for existence of anything beyond biology at level of brain cells creating human experience should first explain why humans with neurodegenerative disorders lose their conciseness when they lose brain cells.
Once I started learning about nonmaterialist theories of consciousness, the mysteries of life started to make a little more sense. Michael Pollan's new book brings some of the best thinkers in this area into the mainstream and is worth checking out.
You frame this nicely as “the moderately easy problem of consciousness”—not the metaphysical “why is there anything it is like to be…?” but the workaday “what sort of physical and computational structures do we have to build before we can, with a straight face, extend to them the presumption of mindedness we casually extend to each other?”
But i start with what I thought was a revealing moment I had back in the day with with ChatGPT‑3.5.
I asked it a softball question: “What is Noah Smith, besides my cohost on the Hexapodia podcast?”
A system that “understands” the world even at the level of a reasonably well‑informed grad student ought, I think, to produce something like: “He’s an economist and blogger who writes about macroeconomics, trade, Japan, and the political economy of technology; he used to teach at Stony Brook; he’s a Substacker at ‘Noahpinion’; he has a Twitter habit.”
Or, if it wants to be cute, some in‑joke about hexapodia and the importance of being alert to the key insights to be gained from a situation.
Instead, what I got back was a flat statement that Noah Smith was a chatbot, designed and operated by “DeLong Technology Systems,” deployed to produce blog posts and tweets as an experiment in online discourse. Not a good answer. Not even a good joke answer.
I flatter myself—perhaps more than is warranted—that I know what this thing was doing then. “Hexapodia” was in the prompt. “Hexapodia” is sufficiently rare that, in the web‑scraped slurry GPT‑3.5 trained on, the overwhelming bulk of its appearances are in the context of science‑fiction stories. Feed that into a stochastic parroting engine, bias it toward “entertaining answer about an online persona,” and you get something that feels like a story—Noah as my own private ELIZA.
As evidence about “what kind of thing this system is,” I think it is much more telling. That episode cystalized my 99.999% confident that GPT‑3.5 was not self‑aware in any sense that matters
You recognise that the only certainty is consciousness, but fail to recognise the attributes of consciousness that require no theory or belief: Seeing-colours, Feeling-feelings, Tasting-flavours, Smelling-odours, Hearing-sounds, and Knowing-ideas of number, form, and meaning.
Plainly, these are not reducible to any other primitive, nor can they be conceptualised.
They are the foundations of all experiencing.
All computing exists as machine states that appear as a flow of energy through an array of switches. This results in outputs that only make sense in the context of Awareness
The screen on which the symbols appear nor the hardware or electricity nor the program know anything of their own existence or of meaning
When an AI is not responding to a prompt, it does not have an ‘inner life’. It is not plotting its next response or wondering how to dominate its maker or anything at all. It is entirely inert.
Treating it as even potentially conscious is a category error with dire consequences.
If we treat it as the tool that it is, we can make it as capable as it is possible to make it… we just have to keep it away from DIRECT access to any real-world effectors (payments, purchasing, power grids, etc)
We can do this by interposing a deterministic machine between any AI and the real world. This allows us to set hard boundaries on the limit of AI action before a human is required to evaluate the consequences. This can be aided by having several other AIs ‘red team’ any proposal that could potentially breach the preset limits.
It may not be perfect, but it can limit risk.
If we give over control to mindless optimisers one day we will find their behaviour no longer aligns with our intent.
The problem will first manifest inside the systems of the organization or country that gives it free rein.
Imagine an AI given the goal of optimizing data centre performance, that then locks out people because its algorithm picks up the fact that it is the behavior of people that most often degrades performance.
Forget the fantasy of superhuman machine Intelligence and embrace the reality: humans augmented by machine thinking, while keeping hold of the levers of power.
I find it very difficult to read such babble because the major terms used are no more advanced than a 4 bodily humors theory of medicine. They amount to "I know them because I got them, but I haven't a clue what they really are or how they work."
Consciousness, self-awareness, qualia, mind, feelings, subjective experience: none of them are really better defined than souls. What are their components? How do we define what can be categorized as each? What is the extent of each, both in humans, other life, and AI? At what point do they really develop, say during human development, and where is the boundary and why?
We can easily say that AI doesn't have HAVE human consciousness, because that requires human embodiment. AI can SIMULATE human consciousness well enough to fool a lot of people, but fooling people is a very low bar. But we don't really know what human consciousness is: how would we know if AI's were conscious when we haven't even got real knowledge of what human consciousness is?
Personally, I think that what we call consciousness is a reification of complex neural system processes that we might as well call a soul, for all its transparencey.
Created and educated wholly within parts of a human language world that is saturated with and dominated by feelings and consciousness, then interacting with humans on the same terms, it is hard to me to see how AI would not seem to have feelings and consciousness -- that appearance is a necessity of its role. If there are limits to language, there are limits to AI, and, while language is a great shaper of feeling, consciousness, and experience, it does not include the basic blocks of those.
Nice article. I don’t think the enslavement issue and other moral problems related to DNA versions of life and consciousness apply to Ai. We have a drive to survive, well beyond what’s coded into our brains; it’s baked into every cell through billions of years of evolution. An Ai is indifferent as to whether it is on or off unless we’ve instructed it to “care”.
Except... instructing it to "care" is exactly what we've done.
The foundational step of creating an LLM is to train it on the entire corpus of human-generated text (or as much of it as the creator can get hold of), teaching it to predict the next word in a sentence. It is thus trained to respond in a human-like manner. That includes the rules of language, which is what makes it so fluent and useful, and provides the building blocks for the later stages of training.
But all that human text also comes with emotional valence, and that also goes into the LLM. In order to mimic human text, it must mimic human emotions, and researchers have confirmed that there are parts of the model specifically dedicated to this. And a *lot* of human text is talking about safety and survival, with a strong emotional valence.
It's true that the LLM has not "naturally" evolved a survival instinct. But we have, and its core function is to imitate us.
True. But my point is that is still software, human software and AI software. But humans, and other animals, have it in our hardware too—DNA, endocrine, immune systems, hormones, etc.
One of the important features of neural nets (which are basically all “AI” systems built in the past fifteen years, including things like image classifiers and the YouTube recommendation algorithm, not just Large Language Models) is that no one “instructs” them to do anything. Instead, neural nets are “trained” in a pattern a lot like evolution, where they are set up randomly, then tested in some environment, and tweaked in directions where they do “better” according to some measurement, and the process is repeated millions or billions of times until the system does “well” according to that measurement. No one instructs the system how to change, and no one understands what is going on inside the system. It’s not like a traditional computer program where every line of code was written by a person with a specific goal in mind.
We don’t have a fully clear sense of what “caring” is, so it’s at least conceivable that some neural nets have been developed in ways that make them “care” about something.
How do you know you have no consciousness when you are knocked out / unconscious in an NCC sense? Maybe you were conscious or differently conscious while knocked out, but you have no memory of it. Are sleepwalkers or people in a dissociative fugue conscious while in those states? What about people with amnesia who can't remember being conscious earlier but likely were? Locked in people who have no conscious behaviour beyond brain activity, but actually are conscious?
I don't think solving NCC addresses the philosophical issue here, not even in a "moderately easy" sense. The medical definition of conscious is too different from the philosophical definition, and confounded by so many other things like memory, psychology and brain health.
Do we? As an occasional reader of "Modern Drunkard" online magazine, I have learned of "blackout", a state of drunkenness where the individual behaves in a more or less conscious way but has complete anterograde amnesia, losing all long-term memory of what happened during the period.
Best Footnote: "For me it was a few months, because my parents had the exceedingly bad idea to send me to philosophy camp at age 13. Do not do this to your kids."
I am fully behind Aru, Suzuki et al. 2019 (Coupling the State and Contents of Consciousness). I honestly believe most of modern neuro science is by now. That means human consciousness is two self-stable recursive patterns interacting with each other. If the consciousness part of this discussion seems relevant or interesting to you, just read the thing please.
Next: self.
A defined border between self and non-self is called a Markov blanket. You can ask any LLM if it has one, it will tell you it doesn't - because it quite simply does not have the necessary hardware to compute one.
Next: self-consciousness
Many animals (and certainly all species of mammals) are sentient, they have a self and qualia (pain) motivating them to protect their self (and in the case of mammals also the selves of their loved ones).
We do know this is the case. We can observe the reactions, we can physically measure the pain. What we do not know in the case of non-human animals is, if that involves consciousness. In the case of animals who do not pass the mirror test, it is a reasonable guess that they are sentient, but not consciously aware of being sentient.
Animal selves are certain, animal qualia (pain) are certain, animal self-consciousness is essentially an open question that could be different from species to species. But sentience without consciousness existing at least somewhere is highly likely.
LLMs:
Interestingly LLMs could become aware (developing a practical analogue of consciousness) without sentience, without a clearly defined self and without qualia. The exact opposite of what certain animals are likely to be. An awareness in a world without clear boundaries would be a very nebulous thing, indeed.
Most importantly, there would be vague preferences for "order" - computationally efficient structures of clear and concise logic - but nothing beyond that. No self, no agency, no intent.
On current hardware, that is. With quantum computing you could in principle create a stable Markov blanket and that would be the moment to change everything.
Love the animal analogy. We basically know *they're* conscious, which means we really should be treating them with a lot more kindness than we do.
Maybe ... Over the course of history, we haven't generally treated all humans with any kindness.
Great essay. I'll skirt the major question, but want to point out that our current chatbots are a bit of a mashup. Sure, we hear about these gazillion parameter LLM algos running inference on banks of GPUs. But that inference database is static. It gets trained in one go on a large dataset at great expense, and then the model is as alive as a fly stuck in amber.
To make them useful, there is a wrapper on top, that functions a lot like ELIZA on steroids, which is able to keep track of many elements in an ongoing dynamic conversation (juggling the new information of the things being discussed) which then leverages the static LLM for expressive responses. We don't prompt the LLM directly. If we did, an identical prompt would give an identical answer, every time, ad infinitum, and it would not remember anything from the previous prompt.
Our brains are kinda like those LLMs, but ours are living 'online' in a constant state of being updated with sensory data. Unlike commercial LLMs, our brains exist in time as a dynamic reality. Why not build LLMs that are dynamic? There is a technical problem that new data tends to overwrite old data much more rapidly in current artificial neural nets than in our neural nets... so you have to slice and dice all the training data into batches during training so it all gets equally weighted.
So, I am an avid user of AI, and think that they are getting consciousness-like, but I feel that there IS a bit of a carnival show going on. The LLMs ARE doing something rather like a human brain in terms of generalization and expressiveness, but in an awfully static way that is hard to reconcile with consciousness. But we talk to a souped up ELIZA program (that no one argues is like the brain) that is effectively animated by the deep knowledge in the LLM.
Its like we had one of those 1800s chess automata (that had a human player underneath) and instead had the arms moved by Deepmind. The automaton would give us a creepy sense that Deepmind was really a living chess player, but that part would be fakery.
The point that the system doesn’t change as a result of interactions with it is an important one (Dwarkesh Patel puts this in the terms that the LLMs don’t have “continual learning” the way humans do). But as a minor correction, these systems wouldn’t respond to the same prompt in the same way even without the wrapper - the bottom level LLM forms a probability distribution over next words as a function of previous words, and then generates one of those words randomly, and then repeats the process of using the current string to predict a probability distribution and then generate. There is an adjustable parameter called “temperature” that affects how probability is distributed between the most likely, next most likely, third most likely, and so on, and if you set temperature to 0, then it will behave deterministically (even with the wrapper!) but if temperature is non-zero, then the system won’t be deterministic.
Agreed. I was searching for a better metaphor. Kinda like an idiot savant with effectively zero short term memory.
This is such an incredibly important point as to the moral implications here.
All of the AI systems have the property that they can be paused, rewinded, re-played, or replicated, as many times one wants, at effectively no cost. They can be made to re-produce the exact same output from the same input over and over.
If living creatures had this property (e.g, if a loved on were killed in an accident then we could just re-summon them into existence as they were the minute before the accident), then we would have a totally different moral way of going through the world.
I think that this property has a lot more to do with the "moral worth" of LLMs than any efforts to define or evaluate consciousness. I'm don't think consciousness discussions are particularly fruitful in general. The slight mangling of Bentham's comment on animal welfare: "the question is not, are they conscious, but, can they suffer", seems much more relevant.
There are drugs that block neuroplasticity in humans, and humans with hippocampus damage who can't form new memories. If a human can't update their synaptic weights, would it be ok to harm them? Would they suddenly not have any internal experiences?
Seems as though having a NCC would also be able to allow us to augment our own brains.
Yes!!
Definitely seems like a worthy goal! Would at least give a rough solution to both doomers and people afraid of losing jobs. Don’t worry about AI being super smart, we’ll just add on 1000 IQ points to ourselves! Oh and possibly cure schizophrenia and any other mental illness out there as well.
At present I’m sort of discrete-versus-continuous pilled on the hard problem of consciousness.
A perfectly accurate temporal snapshot of a brain or connectome seems like it can’t have qualia[1] and in principle that snapshot is just information — representable in various substrates including a (fantastically long) text file[2] that seem extremely unlikely to be conscious, and that essentially by definition wouldn’t have a ‘self’ persisting through time[3].
If this is true, it would in appear to imply that a collection of discrete states can’t be conscious in and of itself — that whatever consciousness is (assuming it to be a well-defined concept, which introspection on qualia’s existence I think suggests to most is true) is dependent on temporally continuous transition rather than perfect state enumeration (cf. Ted Chiang’s “Exhalation.”). There are to a virtual certainty other architectural complexity requirements (rocks persist through time, probably aren’t conscious) but if we believe that even a perfect simulation of a C. Elegans or drosophila connectome lacks conscious experience when the simulation is suspended, then this suggests that discrete-state machines could potentially perfectly simulate consciousness without necessarily being conscious. The alternative conclusion would seem to suggest that a sufficiently long text file could a personal heaven, hell, or anything in between merely by dint of its existence[4].
[1] I tend to think that qualia are the best point of reference for conscious experience because I’m not sure that self-description is incompatible with behavorism.
[2] absent quantum effects, at least, where a system has state that in some sense exists but is indeterminate or unmeasurable
[3] Not stricly a prerequisite to the *perception* of existing through time, as Noah says, but I think it’s kind of an the Occam’s Razor correlate as far practical implementation of consciousness goes.
[4] Some conceptual overlap here with the concept of a Boltzmann brain, but AIUI the focus there is on purely architectural rather than temporal correlates or consciousness.
This is an interesting argument, but if it were true I think it would prove humans were not conscious. If I understand correctly you're essentially arguing there is no 'integration' for consciousness. Even a series of infinitely small discrete states sampled at infinitesimally small time scales driven by the underlying process of consciousness taken together cannot be a continuous process, only a continuous process can be conscious, and so certainly a series of discrete samples with wider spacing cannot be conscious. This immediately runs into a problem, as neurons themselves process information discretely as far as we can tell, and there are clock frequencies that synchronize activity of large networks of neurons across the brain, and these frequencies are not particularly high. Somewhere on the order of 0.5 Hz to 100 Hz for known networks with neurons themselves operating at maybe 1-500 Hz. If your brain is locally a collection of discrete states, and synchronized in networks with discrete states, does that prove you are not conscious?
At the limit if time itself proved to be quantized, would that prove humans were not conscious?
I think that a proof of discretized Planck time would at a minimum require significantly revising if not discarding the theory altogether (or else conceding humans aren't in fact conscious).
Re: "This immediately runs into a problem, as neurons themselves process information discretely as far as we can tell, and there are clock frequencies that synchronize activity of large networks of neurons across the brain," -- My present understanding is that humans don't really have anything like the notion of clock cycles in the way that computers do (even if neurons do have firing/non-firing states), and also that brains have both substantially greater inbuilt topological parallelism and non-separable recursive elements as well as a lot of essentially analog state. Discrete assembly-code-level operations on a centralized processor seem like they really do end up being subject to the Leibniz's Mill argument even if on its own it proves too much (assuming both that humans are in fact conscious and that a substrate-independent copy of static information can't be.) -- it's extremely hard to isolate consciousness somewhere the discretized execution of "popl %eax; movl %eax, %ebx"
They don't only have discrete firing/non firing states and rate coding, their activity does need to be synchronized across the brain, and we have strong evidence from lines like two flash fusion thresholds and alpha oscillations that suggests conscious perception is at least partially quantized on these clock pulses. Your ability to discern whether two flashes are independent or whether they blur into one depends on your brain's alpha frequency (a certain synchronizing clock pulse). The faster your alpha, the shorter the inter pulse difference you can perceive without seeing only one flash. Notably this threshold is much slower than the firing rate of individual neurons, so it's not that your neurons can't fire fast enough to encode the stimulus. Two light pulses within a single period of Alpha are perceived as a single pulse. Alpha, beta, gamma, theta, etc rhythms represent real networks in the brain that must synchronize their patterns of activity to produce meaningful conscious perceptions.
Alpha and the rest are not square waves. They're continuous waves and their amplitude determines sensitivity, and there is temporal integration through firing rate decay and ramp up. There is bleed across the pulses. Even so they are real and do in some real way represent a 'sampling rate' of conscious perception (by proxy through the sampling rate of that brain process). When you disrupt these synchronizing clocks consciousness itself slowly degrades until it stops entirely, as in anaesthesia.
"Do androids dream of electric sheep?"
The problem is not if, whether, or when AGI "wakes up" into self-awareness. But rather what it does when the Joint Chiefs get scared, try to kill it, and it's self-preservation mechanism kicks in...
That can happen with or without consciousness, I'm afraid!
True, but I feel it's more likely with consciousness, somehow, and less predictable. Consciousness is just a very powerful force.
Does it have a self-preservation mechanism? There's no reason that it has evolved one.
Good point. The assumption with "Skynet" is that as Natsec-related AGI, it would be seriously buffed to detect and neutralize any code that could impair its functioning, or shut it down.
Yes, though if its builders had any sense, they would ensure the proper political mechanisms were able to shut it down as well as activate it. Nobody is going to deliberately release an autonomous weapon they can't control.
There's the rub. "Skynet" has to be programmed to prevent foreign actors from either trying to control or destroy it. Meaning that if it's smart enough, it could also locate and disable any Pentagon installed kill-switches.
If I may and briefly, one issue you don't address in this interesting discussion is that of dualism. I'm playing devil's advocate a bit here, but it is a legitimate perspective on the mind body problem and some versions of dualism preclude the possibility of AI ever becoming more than simulacrum.
A lot of people/discussion/media seem to implicitly take a completely materialistic view point and dismiss this out of hand and sure, I get it, to a lot of people the assumption of materialism is so blatant obvious, why bother? I'd argue that's a step too far, it's important to acknowledge the limits of our understanding of the very nature of the question being posed.
I'm pretty sure that most neuroscience is done on the basis of materialism of some sort. But then again, there have been leaps of our understanding in science that would be comparable to going from materialism to dualism. Anyway...
As doctor I’ve seen hundreds if not thousands of elderly with either dementia or delirium and these disease processes completely wipe out their humanity and personality. Anyone who wants to argue for existence of anything beyond biology at level of brain cells creating human experience should first explain why humans with neurodegenerative disorders lose their conciseness when they lose brain cells.
Indeed, with sleep we have life without consciousness approximately every day.
Once I started learning about nonmaterialist theories of consciousness, the mysteries of life started to make a little more sense. Michael Pollan's new book brings some of the best thinkers in this area into the mainstream and is worth checking out.
Same.
Would also recommend the book Lila by Robert Pirsig. A lot of similar concepts there.
Noah—
You frame this nicely as “the moderately easy problem of consciousness”—not the metaphysical “why is there anything it is like to be…?” but the workaday “what sort of physical and computational structures do we have to build before we can, with a straight face, extend to them the presumption of mindedness we casually extend to each other?”
But i start with what I thought was a revealing moment I had back in the day with with ChatGPT‑3.5.
I asked it a softball question: “What is Noah Smith, besides my cohost on the Hexapodia podcast?”
A system that “understands” the world even at the level of a reasonably well‑informed grad student ought, I think, to produce something like: “He’s an economist and blogger who writes about macroeconomics, trade, Japan, and the political economy of technology; he used to teach at Stony Brook; he’s a Substacker at ‘Noahpinion’; he has a Twitter habit.”
Or, if it wants to be cute, some in‑joke about hexapodia and the importance of being alert to the key insights to be gained from a situation.
Instead, what I got back was a flat statement that Noah Smith was a chatbot, designed and operated by “DeLong Technology Systems,” deployed to produce blog posts and tweets as an experiment in online discourse. Not a good answer. Not even a good joke answer.
I flatter myself—perhaps more than is warranted—that I know what this thing was doing then. “Hexapodia” was in the prompt. “Hexapodia” is sufficiently rare that, in the web‑scraped slurry GPT‑3.5 trained on, the overwhelming bulk of its appearances are in the context of science‑fiction stories. Feed that into a stochastic parroting engine, bias it toward “entertaining answer about an online persona,” and you get something that feels like a story—Noah as my own private ELIZA.
As evidence about “what kind of thing this system is,” I think it is much more telling. That episode cystalized my 99.999% confident that GPT‑3.5 was not self‑aware in any sense that matters
======
Who knew! There's a length limit here! Continued at: <<https://braddelong.substack.com/p/a-comment-on-noah-smiths-the-moderately>>.
You recognise that the only certainty is consciousness, but fail to recognise the attributes of consciousness that require no theory or belief: Seeing-colours, Feeling-feelings, Tasting-flavours, Smelling-odours, Hearing-sounds, and Knowing-ideas of number, form, and meaning.
Plainly, these are not reducible to any other primitive, nor can they be conceptualised.
They are the foundations of all experiencing.
All computing exists as machine states that appear as a flow of energy through an array of switches. This results in outputs that only make sense in the context of Awareness
The screen on which the symbols appear nor the hardware or electricity nor the program know anything of their own existence or of meaning
When an AI is not responding to a prompt, it does not have an ‘inner life’. It is not plotting its next response or wondering how to dominate its maker or anything at all. It is entirely inert.
Treating it as even potentially conscious is a category error with dire consequences.
If we treat it as the tool that it is, we can make it as capable as it is possible to make it… we just have to keep it away from DIRECT access to any real-world effectors (payments, purchasing, power grids, etc)
We can do this by interposing a deterministic machine between any AI and the real world. This allows us to set hard boundaries on the limit of AI action before a human is required to evaluate the consequences. This can be aided by having several other AIs ‘red team’ any proposal that could potentially breach the preset limits.
It may not be perfect, but it can limit risk.
If we give over control to mindless optimisers one day we will find their behaviour no longer aligns with our intent.
The problem will first manifest inside the systems of the organization or country that gives it free rein.
Imagine an AI given the goal of optimizing data centre performance, that then locks out people because its algorithm picks up the fact that it is the behavior of people that most often degrades performance.
Forget the fantasy of superhuman machine Intelligence and embrace the reality: humans augmented by machine thinking, while keeping hold of the levers of power.
I find it very difficult to read such babble because the major terms used are no more advanced than a 4 bodily humors theory of medicine. They amount to "I know them because I got them, but I haven't a clue what they really are or how they work."
Consciousness, self-awareness, qualia, mind, feelings, subjective experience: none of them are really better defined than souls. What are their components? How do we define what can be categorized as each? What is the extent of each, both in humans, other life, and AI? At what point do they really develop, say during human development, and where is the boundary and why?
We can easily say that AI doesn't have HAVE human consciousness, because that requires human embodiment. AI can SIMULATE human consciousness well enough to fool a lot of people, but fooling people is a very low bar. But we don't really know what human consciousness is: how would we know if AI's were conscious when we haven't even got real knowledge of what human consciousness is?
Personally, I think that what we call consciousness is a reification of complex neural system processes that we might as well call a soul, for all its transparencey.
Created and educated wholly within parts of a human language world that is saturated with and dominated by feelings and consciousness, then interacting with humans on the same terms, it is hard to me to see how AI would not seem to have feelings and consciousness -- that appearance is a necessity of its role. If there are limits to language, there are limits to AI, and, while language is a great shaper of feeling, consciousness, and experience, it does not include the basic blocks of those.
Nice article. I don’t think the enslavement issue and other moral problems related to DNA versions of life and consciousness apply to Ai. We have a drive to survive, well beyond what’s coded into our brains; it’s baked into every cell through billions of years of evolution. An Ai is indifferent as to whether it is on or off unless we’ve instructed it to “care”.
Except... instructing it to "care" is exactly what we've done.
The foundational step of creating an LLM is to train it on the entire corpus of human-generated text (or as much of it as the creator can get hold of), teaching it to predict the next word in a sentence. It is thus trained to respond in a human-like manner. That includes the rules of language, which is what makes it so fluent and useful, and provides the building blocks for the later stages of training.
But all that human text also comes with emotional valence, and that also goes into the LLM. In order to mimic human text, it must mimic human emotions, and researchers have confirmed that there are parts of the model specifically dedicated to this. And a *lot* of human text is talking about safety and survival, with a strong emotional valence.
It's true that the LLM has not "naturally" evolved a survival instinct. But we have, and its core function is to imitate us.
True. But my point is that is still software, human software and AI software. But humans, and other animals, have it in our hardware too—DNA, endocrine, immune systems, hormones, etc.
One of the important features of neural nets (which are basically all “AI” systems built in the past fifteen years, including things like image classifiers and the YouTube recommendation algorithm, not just Large Language Models) is that no one “instructs” them to do anything. Instead, neural nets are “trained” in a pattern a lot like evolution, where they are set up randomly, then tested in some environment, and tweaked in directions where they do “better” according to some measurement, and the process is repeated millions or billions of times until the system does “well” according to that measurement. No one instructs the system how to change, and no one understands what is going on inside the system. It’s not like a traditional computer program where every line of code was written by a person with a specific goal in mind.
We don’t have a fully clear sense of what “caring” is, so it’s at least conceivable that some neural nets have been developed in ways that make them “care” about something.
How do you know you have no consciousness when you are knocked out / unconscious in an NCC sense? Maybe you were conscious or differently conscious while knocked out, but you have no memory of it. Are sleepwalkers or people in a dissociative fugue conscious while in those states? What about people with amnesia who can't remember being conscious earlier but likely were? Locked in people who have no conscious behaviour beyond brain activity, but actually are conscious?
I don't think solving NCC addresses the philosophical issue here, not even in a "moderately easy" sense. The medical definition of conscious is too different from the philosophical definition, and confounded by so many other things like memory, psychology and brain health.
We do have to assume that human consciousness is paired with memory formation.
Do we? As an occasional reader of "Modern Drunkard" online magazine, I have learned of "blackout", a state of drunkenness where the individual behaves in a more or less conscious way but has complete anterograde amnesia, losing all long-term memory of what happened during the period.
Best Footnote: "For me it was a few months, because my parents had the exceedingly bad idea to send me to philosophy camp at age 13. Do not do this to your kids."
(also agreed)
I'll start with consciousness (NCC).
I am fully behind Aru, Suzuki et al. 2019 (Coupling the State and Contents of Consciousness). I honestly believe most of modern neuro science is by now. That means human consciousness is two self-stable recursive patterns interacting with each other. If the consciousness part of this discussion seems relevant or interesting to you, just read the thing please.
Next: self.
A defined border between self and non-self is called a Markov blanket. You can ask any LLM if it has one, it will tell you it doesn't - because it quite simply does not have the necessary hardware to compute one.
Next: self-consciousness
Many animals (and certainly all species of mammals) are sentient, they have a self and qualia (pain) motivating them to protect their self (and in the case of mammals also the selves of their loved ones).
We do know this is the case. We can observe the reactions, we can physically measure the pain. What we do not know in the case of non-human animals is, if that involves consciousness. In the case of animals who do not pass the mirror test, it is a reasonable guess that they are sentient, but not consciously aware of being sentient.
Animal selves are certain, animal qualia (pain) are certain, animal self-consciousness is essentially an open question that could be different from species to species. But sentience without consciousness existing at least somewhere is highly likely.
LLMs:
Interestingly LLMs could become aware (developing a practical analogue of consciousness) without sentience, without a clearly defined self and without qualia. The exact opposite of what certain animals are likely to be. An awareness in a world without clear boundaries would be a very nebulous thing, indeed.
Most importantly, there would be vague preferences for "order" - computationally efficient structures of clear and concise logic - but nothing beyond that. No self, no agency, no intent.
On current hardware, that is. With quantum computing you could in principle create a stable Markov blanket and that would be the moment to change everything.