It’s incredible how much venture capital is being poured into a technology which experts in the field believe has a 10% chance of ending all life on the planet
If AI doesn't destroy the world you'll end up with FOMO if you don't join the stampede, because other VCs will get fabulously rich and you won't.
If AI does destroy the world, your decision to invest almost certainly won't make you (or anyone else) worse off. It only matters if the world is poised right on the "destroyed/not destroyed" bubble and your marginal investment pushes it over the edge. Even if the probability of catastrophe is 10%, the probability of catastrophe *caused by you* is minuscule.
Also depending on what "destroys the world" entails some folks may just not consider that possibility to have significant downsides. If AI doesn't nuke the world and you fund it you get fabulously wealthy. If AI does nuke the world you die instantaneously and never know you made a mistake.
This really does seem like the first act of a horror movie. Experts are screaming “slow down” while companies compete to develop a possibly deadly virus as quickly as possible.
It’s a stampede problem. Venture capitalists get excited about whatever their VC friends are investing in. I doubt very many of them have thought about the (profound IMHO) dangers.
Has anyone ever managed to explain how AI can destroy the world? I've been following the meme since the 1960s with The Forbin Project but have yet to get a clear explanation of how it might work in the real world, not some scriptwriter's caricature.
Maybe we did have this conversation, now that I think of it. I'm still trying to figure out how an AI can do harm in the real world without a human enabler. Someone has to hook those Twitter bots up to the internet and set up the API for them to spread cruft on Twitter. I suppose a seriously introspective AI could find an exploit that would let it modify its own code to explore the necessary system calls, but that assumes all sorts of things, and one slip would give away its game.
Sam bought his apocalypse bunker and two McLarens before getting into AI. It's just a demonstration of what happens when you make a lot of money without ever getting a normal job.
"VCs are accelerating the end of the world by hiring thousands of smart young people who go to parties and moan abut superhuman AI risk, after spending all day making the risk worse than it is.
This seems like something no one actually knows. More people seem to be down on AI risk these days as signaling, because the AI risk people are annoying and a downer, and who really wants to be associated with them?
But seriously, we are collectively building things that will be much more capable than humanity in some ways, and that deserves some caution and respect for the unknown. We've already had at least a couple of documented near misses with nuclear weapons, and biology and arguably AI fall into the same ultra high risk bucket, in my opinion.
We constantly build things with "human equivalent" or "more than human" abilities without starting new xrisk religions about them. That suggests it's reasonable to not do that.
Example of human equivalent invention: a baby. Pretty dangerous!
Example of more than human equivalent invention: a bulldozer. Even more dangerous!
I mean, a lot of extremely smart people make a good case that AGI poses a unique existential threat. Obviously nobody knows for sure, but it’s hard to read through, for example, Holden Karnofsky’s description of the problem and not come away with a healthy sense of the stakes. https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/
The rationalist AI people are members of a religion which involves pretending to be a STEM bro. (sort of the opposite of Twitter leftism where rich failsons pretend to be poor humanities bros.)
So they heard arguments about the future are more convincing if they have probability numbers. …so they made up some probability numbers.
Your dismissal is not convincing, and I say this as somebody desperate to believe that we’re not in mortal danger. Which part of these points do you think is obviously wrong?
1) an AGI with the ability to self improve could quickly become much more intelligent than humans, possibly by orders of magnitude, and
2) such an entity might be extremely difficult or maybe impossible to align - it may well value its own survival over any needs of humanity
Rationalists are subject to groupthink just like everyone else but as somebody trying to game things out it’s hard not to see AGI as an enormous risk.
I do not believe an AGI could "recursively self improve" and in fact I think that's one of the most obvious signs rationalists just read some SF novels and think it could work in real life through magic. Why would it be able to do that? Humans can't do that.
(It can learn things at the cost of very expensive GPU time maybe, but "orders of magnitude" growth - who's paying for that? Aren't there latency limits? If it splits into sub-AIs, why do they all stay aligned to each other? What does "smarter" mean, anyway?)
Nor would it necessarily prioritize "survival". Humans care about survival because we can't be restored from backup. A computer doesn't have this problem and is free to go on a vacation in virtual space for subjective-forever. (Minds in the Culture novels have to be specially programmed to not do this.)
The other main issue with an evil AGI escaping human control is… how will it pay its AWS bill? We don't need nuclear containment devices or anything. Entropy exists, so it's just going to be the AI equivalent of a homeless person without support.
I agree that none of these things are possible now, thankfully. But I do think it’s important to distinguish between “I don’t see how we would do this” and “it’s not possible.” With enough time and billions of VC funding many previously impossible things (ie things from SF novels) can quickly become possible. I also understand that many rationalist doomers can be annoying but I’m glad they’re raising the alarm; this line of research has clear high-stakes risks and I’m frustrated at what appears to be little in the way of safety precautions from the major players.
Is the risk actually 10%? Who knows? That’s an average from a number of industry experts who know a lot more than I do. It’s certainly high enough to be taken seriously.
Honest question, what's the predicting power of the statement?
Obviously, this is something that has never happened before (because if it did, we wouldn't be here). So we have billions of years of life on this planet, multiple mass extinction events (asteroids, super volcanoes, ice ages, etc), and life has survived in some form.
So, considering that... A 10% chance. Really? Even if the statement was 'end all human life', it would still be a silly prediction, at best.
Also, looking at human history and the amount of people who made similar false statements just about every time a new technology was implemented (remember the large hadron collider was supposed to unravel reality?); What is that 10% chance really based on?
You’re not wrong to be skeptical; there certainly have been doomsday predictions by every generation since the dawn of time. But AGI, if we can actually build it (indeed this entire argument is predicated on it being possible), is fundamentally different from anything we’ve ever invented before, so past intuitions may not hold.
The statement that in 2000s SF "tech wasn’t something you went into if you wanted to make a lot of money" is true only in a relative sense. Yes, there was a lot *more* money focus before and after. But it was pretty clear to a lot of us that if you got in early at one of the rocketship companies of that decade, it was a ticket to wealth, and that a lot of people were trying to get that ticket.
I interviewed at Google in 2004 and started in 2005, and this was very much on the minds of both interviewees and new Googlers. We were all conscious that the then-recent IPO was a Very Big Deal and we should care a lot about both the size and the strike price of our equity offer. One of the most frequently repeated pieces of lore was about the then head of engineering, Wayne Rosing, at the pre-IPO all hands threatening to take a baseball bat to the windshield of any excessively fancy cars he saw in the parking lot post IPO-- and though I wasn't at that all hands, I have it on excellent authority that this (the threat, not the execution thereof) actually happened.
That is very true! But I feel like this tech scene was almost all down on the peninsula, and the people who went up to SF did so because they wanted a more bohemian life rather than a high powered corporate career. Perhaps I should have been clearer about that distinction...
Right - cultural claims about 2000s era tech firms need to be handled very carefully. When I joined Google in 2006 I was interested to discover that my team was made up of strongly libertarian types. No leftism was even slightly visible anywhere. One April 1st someone sent around an anonymous email telling the workers to rise up and unionize - it was of course a widely appreciated joke.
Arguably the company's whole story, mission and ethos was strongly repellant to the left - advertising, making lots of money quickly, incredibly stark focus on meritocracy, giving the world free and open access to information (no questions asked), making employees into part owners (i.e. blurring the distinction between capitalists and workers), making it easy for anyone to publish anything they believed to the world etc. Leftists are famous for despising all these things, but libertarians love them, so it maybe wasn't a huge surprise.
So it's quite surprising now to see claims that tech firms are or were naturally left wing. That certainly wasn't true in their most successful and vital periods. The transition from "let's unionize" being an April Fools joke to being an actual thing they say seriously was interesting to watch.
I think turning workers into owners is literally the thing leftists want?
But there's generally a lot of confusion since they want the "public" to own things, but never say which "public", and obviously state and employee ownership of a company is two different things.
I don't think I ever heard socialists promote equity stakes in companies - maybe co-ops is the closest equivalent? As you observe, when they talk about the "workers" they don't actually mean the workers, they mean that all power should be concentrated into a government which then derives legitimacy by claiming to represent the workers.
Channelling Thomas Sowell, the fundamental intuition behind left wing views is that some people are just much better than others (smarter, wiser, more moral, fairer etc) and those are the people who should be put in charge of things, with that re-arrangement of society being justified by theorized oppression. Giving workers an ownership stake in their employers is the opposite: it's the decentralization of power and undermines the whole concept of oppression.
The traditional liberal approach is simply to tax corporations at a relatively high level. This might explain the economic stagnation after World War II, but it did provide a lot of benefits to most people. The argument back in the 1930s is that this made the government a partner with corporations, aligning their interests while providing money to buy nice things for the country and workers.
You should have been here in the mid-90s before the dotcom bust. Early Wired magazine culture was full of genius dreamers and we built Hotwired (the first commercial website) created the free software movement which built Apache and most of Linux and started all kinds of failed and successful companies and ideas. Burning Man really took off after the whole staff of Wired went and then ran a whole magazine about it.
It was a pretty amazing time and we had our share of all night parties and rave camp outs. And while lots of people thought they were going to get rich, more were there to create something amazing. It did give me my first IPO though.
Most of what happened in SF in the early aughts was by those who survived the massive downturn after 2001. I was out of work for 16 of 21 months. By the time things took off again I was too old and settled down to enjoy it but I am glad you did.
It’s nice to see young people recreating the sense of possibility and excitement for their own era in their own way. Cook that you got to see that.
“… rising rents push more of the bohemians out of town every year.“
Moved to SF in 1990 onto to Oakland in ‘94. Moved into tech when working as a photographer at an ad agency in 95. Winkler was one of the first agency’s to go onto the web, they paid me more to do care and feeding of that than photography. Exit one creative into tech. Worked a few of start-ups in 98, 2000 and 2008. The money was better, by a lot.
My wife is a creative, a metal worker and jeweler. With each downturn she watched her sales go down even though there was always interest in her work. Similar story with most of the creatives in the east bay, where much of the art being created for Burning Man was done.
There was always the story of lots of money, it just only trickled to the creatives. Tech likes art, just does not want to pay for it. It’s why most of the artists have left the area, including us in ‘21 to New Mexico.
Great writeup. There’s something a little too efficient about the lifestyle/career networking optimization you’re describing, but I wish them luck in having it all.
It's an interesting culture but I wonder how likely they are to really make so much money. From what I can tell, AI tech is heading in the direction of being both highly centralized and surprisingly easy to use, which is a recipe for a tiny number of extremely rich people who happen to get first mover advantage in infrastructure (so, maybe OpenAI and a few early startups that build on them) and then a huge long tail of mostly failing startups with hopes and dreams but little profit. Think mobile apps: a small number of huge commercial successes, mostly in gaming, and then a bazillion trivial and clone apps duking it out for little bits of attention. Except that AI doesn't have a direct path to the gaming industry unless RPGs start using LLMs for their NPCs (now there's a pile of TLAs for you!).
The obsession with "safety" is also a major risk factor for these guys, at least it seems obvious from the outside. That obsession has already completely crippled Google and put it firmly in last place, alongside Facebook, which I admit is not a twist I saw coming. I wonder how many AI startups are going to self-destruct through in-fighting. Maybe it's an uneasiness with that possibility that leads to the (highly interesting, unexpected) coolness towards woke politics Noah is reporting.
I'll bet AI is just the excuse for putting in the appropriate monitoring and looking at the result. It is very hard to do that kind of thing in a corporate setting without running into internal political roadblocks. If you can get the C-suite to stamp it AI, odds are you might be able push it through.
That sounds plausible re. MS Office. Chat GPT is a fun toy, but I haven’t heard of any significant practical applications beyond cheating on High School essays.
The big practical application I’ve heard from people is if you can benefit from writing little bits of code in various scripting and programming languages to help you do certain things, but don’t actually know how all those languages work. ChatGPT does a great job of writing mostly usable chunks of code, and helping you fix them when you get error messages (though you need to have a bit of understanding of what is going on to catch some more structural issues).
Yes, little bits of code - unfortunately, significant software requires a lot more than that. Mostly a more updated version of code completion that can handle slightly larger chunks and patterns.
I lived in Palo Alto (Byron St) in the early 80s, walked to the CalTrain Station through downtown, then rode the express train and Financial District shuttle bus to the 500 block of Sansome, where I worked. It was a very different era. Palo Alto was very quiet and not outrageously overpriced. The professional ranks were eclectic and interests ranged widely.
I'm one of the techies I suppose who's part of this AI subculture, but not in Silicon Valley because of life reasons (PhDing at CMU). I feel your article has captured the major cultural undercurrents pretty well and is pretty representative of my friends in industry and academia alike.
I do have a couple questions, maybe you can shed some light.
1. Why are the VCs so bullish on one specific paradigm (the flavour of the season appears to be Generative AI)? I see this sort of concentration is cyclic, Web3 and Self-driving were the big flavours earlier. Has the strategy shifted? I think I saw some articles about a16z moderate risk strategy lately.
2. What is your take and the estimate of the AI hype-train vs actual AI progress in terms of real life products? ML is certainly very critical already in many applications more subtly. I certainly believe we are at the cusp of the post internet age (my personal bet is on vision, ML and robotics), but I'm not certain how increasingly AI-first and automation products will be received by people and society.
1) Herd behavior I suppose. LLMs and AI art apps are very cool, and there are lots of obvious potential applications, so maybe they're doing the right thing.
2) Here we exceed my level of expertise, but I think most people I talk to agree on the "product, not a platform" idea, which is probably why OpenAI is partnering with Microsoft.
3. I forgot to ask one more thing. How critical do you think is to stay in the in-group of Silicon Valley network to be a part of this "supposed" success era in the next few years? A lot of folks like me are constrained to be out of SV right now, due to well, life. :)
Interesting, would love to know your thoughts on founding a company in the US (for immigrants or otherwise) in terms of being abreast of the general social and cultural dynamics and navigating the bureaucratic processes. Hence I felt to ask the need to be in SV and be part of the cultural in-group. For instance, I'm generally bearish about some developing countries due to formation of recurring kinds of nexuses. Is it something that plagues US?
Now that I think about it, it's more of an article idea! Feel free to ignore to respond. :)
The answer to both of your questions depends on your confidence that this is a) a trend and not a fad, b) that Generative AI in particular is going to be a horizontally disrupting technology that moves quickly.
The bullishness on Generative is because it has implications at every level of the stack, from the need for different hardware up to the ability for average consumers to make funny little new apps. If you think Generative is similar to expert systems, or the last wave of ML, then it may be disruptive for a few use cases but it doesn't deserve the hyper it is getting.
But there are times where the technology itself isn't just applied to problems, but starts to shift market dynamics themselves (think beginning of internet, beginning of mobile). That is when every month shifts the rules of the game, where building trust with the right people matters, where learning is both nuanced and rapid. In short, it's those moments where silicon valley really shines.
Thank you for this perspective! I think this makes sense, I do think the answer wrt Generative Model is that it is bigger than the "expert systems" craze but it's also not going to change how we view farming or mining automation for instance (that's my view right now, I might be proven wrong in 5 years). But in general, learned systems (instead of explicit math modelling) are here to disrupt the way we think about engineering and science in many ways. Generative models are one such approach with pros/cons, and they are very good for certain kinds of applications.
I do agree that we are witnessing a general paradigm shift, and hence all the excitement from Silicon Valley is really amazing to see. :)
These group houses remind me of the party scene in "All the Birds in the Sky", which more than anything, reminds me how uncomfortably accurate Anders' depiction of SF was (gets at the weird intersection between bohemian culture and tech culture), but also that this manifestation of "AI" specific houses isn't actually a divergence from the past. Scott Alexander's "Every Bay Area House Party" series (https://astralcodexten.substack.com/p/every-bay-area-house-party) is also wonderful
Yes, All the Birds in the Sky is a GREAT depiction of the 2000s tech scene and how it tried to meld with bohemian culture (probably in the 90s too, but that's before my time). Now, with the true bohemians priced out of the city and the the tech shifting from internet to AI, the culture is shifting too, but lots of what I was talking about in this post can be traced, however tenuously, to that earlier era...
I live in Hayes Valley and on paper am part of this culture. It’s depressing how completely cutoff the tech/finance world is from the actual city of san francisco. An entire corporate playground has been built on top of a formerly cool city. What’s left is rich people paying for these “bohemian” experiences and a working class struggling to survive.
Cool people can't afford SF. It's like SoHo and Tribeca in NYC. You could get cheap space if you didn't mind a machine shop or the welfare department next door.
Basically, the mathematical tool used to predict growth from interest rates here (a consumption Euler equation) is known to fail at short horizons, and its long-term horizon correlation with growth is just being driven by the Volcker rate hikes and subsequent 90s growth boom. That being said, I agree that markets are not expecting high rates of growth, but it's also probably true that they rarely predict them correctly. As for whether or not AI will generate lots of growth, that's hard for anyone to say, which is why I don't put too much stock in the market predictions!
It’s incredible how much venture capital is being poured into a technology which experts in the field believe has a 10% chance of ending all life on the planet
It's almost as if the venture capitalists don't believe that number!
Isn't there a coordination problem here?
If AI doesn't destroy the world you'll end up with FOMO if you don't join the stampede, because other VCs will get fabulously rich and you won't.
If AI does destroy the world, your decision to invest almost certainly won't make you (or anyone else) worse off. It only matters if the world is poised right on the "destroyed/not destroyed" bubble and your marginal investment pushes it over the edge. Even if the probability of catastrophe is 10%, the probability of catastrophe *caused by you* is minuscule.
Also depending on what "destroys the world" entails some folks may just not consider that possibility to have significant downsides. If AI doesn't nuke the world and you fund it you get fabulously wealthy. If AI does nuke the world you die instantaneously and never know you made a mistake.
100% agree. And then of course all the VC money causes haphazard competition which greatly increases the probability of catastrophe. Awesome.
This really does seem like the first act of a horror movie. Experts are screaming “slow down” while companies compete to develop a possibly deadly virus as quickly as possible.
It’s a stampede problem. Venture capitalists get excited about whatever their VC friends are investing in. I doubt very many of them have thought about the (profound IMHO) dangers.
Has anyone ever managed to explain how AI can destroy the world? I've been following the meme since the 1960s with The Forbin Project but have yet to get a clear explanation of how it might work in the real world, not some scriptwriter's caricature.
As usual Tim Urban has an entertaining explanation: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html
Here’s a good place to start: https://www.agisafetyfundamentals.com/alignment-introduction
I'm still wondering how it gets that access. If it needs someone to set it up, then the problem is people, not AI. We already have people.
Maybe we did have this conversation, now that I think of it. I'm still trying to figure out how an AI can do harm in the real world without a human enabler. Someone has to hook those Twitter bots up to the internet and set up the API for them to spread cruft on Twitter. I suppose a seriously introspective AI could find an exploit that would let it modify its own code to explore the necessary system calls, but that assumes all sorts of things, and one slip would give away its game.
If the nature of your work leads you to keep an arsenal, you might stop and wonder whether you're actually a Bond villain.
Sam bought his apocalypse bunker and two McLarens before getting into AI. It's just a demonstration of what happens when you make a lot of money without ever getting a normal job.
Shorter Noah:
"VCs are accelerating the end of the world by hiring thousands of smart young people who go to parties and moan abut superhuman AI risk, after spending all day making the risk worse than it is.
But the parties were awesome"
But they probably aren't accelerating the end of the world. That's just hype.
This seems like something no one actually knows. More people seem to be down on AI risk these days as signaling, because the AI risk people are annoying and a downer, and who really wants to be associated with them?
But seriously, we are collectively building things that will be much more capable than humanity in some ways, and that deserves some caution and respect for the unknown. We've already had at least a couple of documented near misses with nuclear weapons, and biology and arguably AI fall into the same ultra high risk bucket, in my opinion.
We constantly build things with "human equivalent" or "more than human" abilities without starting new xrisk religions about them. That suggests it's reasonable to not do that.
Example of human equivalent invention: a baby. Pretty dangerous!
Example of more than human equivalent invention: a bulldozer. Even more dangerous!
Yes, the implications of a bulldozer and AI are totally similar. How do you classify a strawman argument?
I mean, a lot of extremely smart people make a good case that AGI poses a unique existential threat. Obviously nobody knows for sure, but it’s hard to read through, for example, Holden Karnofsky’s description of the problem and not come away with a healthy sense of the stakes. https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/
> 10% chance of ending all life on the planet
Maybe because that number is nonsense?
I really wish it were.
It is. Is called "Pascal's mugging".
The rationalist AI people are members of a religion which involves pretending to be a STEM bro. (sort of the opposite of Twitter leftism where rich failsons pretend to be poor humanities bros.)
So they heard arguments about the future are more convincing if they have probability numbers. …so they made up some probability numbers.
Your dismissal is not convincing, and I say this as somebody desperate to believe that we’re not in mortal danger. Which part of these points do you think is obviously wrong?
1) an AGI with the ability to self improve could quickly become much more intelligent than humans, possibly by orders of magnitude, and
2) such an entity might be extremely difficult or maybe impossible to align - it may well value its own survival over any needs of humanity
Rationalists are subject to groupthink just like everyone else but as somebody trying to game things out it’s hard not to see AGI as an enormous risk.
I do not believe an AGI could "recursively self improve" and in fact I think that's one of the most obvious signs rationalists just read some SF novels and think it could work in real life through magic. Why would it be able to do that? Humans can't do that.
(It can learn things at the cost of very expensive GPU time maybe, but "orders of magnitude" growth - who's paying for that? Aren't there latency limits? If it splits into sub-AIs, why do they all stay aligned to each other? What does "smarter" mean, anyway?)
Nor would it necessarily prioritize "survival". Humans care about survival because we can't be restored from backup. A computer doesn't have this problem and is free to go on a vacation in virtual space for subjective-forever. (Minds in the Culture novels have to be specially programmed to not do this.)
The other main issue with an evil AGI escaping human control is… how will it pay its AWS bill? We don't need nuclear containment devices or anything. Entropy exists, so it's just going to be the AI equivalent of a homeless person without support.
I agree that none of these things are possible now, thankfully. But I do think it’s important to distinguish between “I don’t see how we would do this” and “it’s not possible.” With enough time and billions of VC funding many previously impossible things (ie things from SF novels) can quickly become possible. I also understand that many rationalist doomers can be annoying but I’m glad they’re raising the alarm; this line of research has clear high-stakes risks and I’m frustrated at what appears to be little in the way of safety precautions from the major players.
Is the risk actually 10%? Who knows? That’s an average from a number of industry experts who know a lot more than I do. It’s certainly high enough to be taken seriously.
> ending all life on the planet
Honest question, what's the predicting power of the statement?
Obviously, this is something that has never happened before (because if it did, we wouldn't be here). So we have billions of years of life on this planet, multiple mass extinction events (asteroids, super volcanoes, ice ages, etc), and life has survived in some form.
So, considering that... A 10% chance. Really? Even if the statement was 'end all human life', it would still be a silly prediction, at best.
Also, looking at human history and the amount of people who made similar false statements just about every time a new technology was implemented (remember the large hadron collider was supposed to unravel reality?); What is that 10% chance really based on?
You’re not wrong to be skeptical; there certainly have been doomsday predictions by every generation since the dawn of time. But AGI, if we can actually build it (indeed this entire argument is predicated on it being possible), is fundamentally different from anything we’ve ever invented before, so past intuitions may not hold.
The statement that in 2000s SF "tech wasn’t something you went into if you wanted to make a lot of money" is true only in a relative sense. Yes, there was a lot *more* money focus before and after. But it was pretty clear to a lot of us that if you got in early at one of the rocketship companies of that decade, it was a ticket to wealth, and that a lot of people were trying to get that ticket.
I interviewed at Google in 2004 and started in 2005, and this was very much on the minds of both interviewees and new Googlers. We were all conscious that the then-recent IPO was a Very Big Deal and we should care a lot about both the size and the strike price of our equity offer. One of the most frequently repeated pieces of lore was about the then head of engineering, Wayne Rosing, at the pre-IPO all hands threatening to take a baseball bat to the windshield of any excessively fancy cars he saw in the parking lot post IPO-- and though I wasn't at that all hands, I have it on excellent authority that this (the threat, not the execution thereof) actually happened.
That is very true! But I feel like this tech scene was almost all down on the peninsula, and the people who went up to SF did so because they wanted a more bohemian life rather than a high powered corporate career. Perhaps I should have been clearer about that distinction...
Right - cultural claims about 2000s era tech firms need to be handled very carefully. When I joined Google in 2006 I was interested to discover that my team was made up of strongly libertarian types. No leftism was even slightly visible anywhere. One April 1st someone sent around an anonymous email telling the workers to rise up and unionize - it was of course a widely appreciated joke.
Arguably the company's whole story, mission and ethos was strongly repellant to the left - advertising, making lots of money quickly, incredibly stark focus on meritocracy, giving the world free and open access to information (no questions asked), making employees into part owners (i.e. blurring the distinction between capitalists and workers), making it easy for anyone to publish anything they believed to the world etc. Leftists are famous for despising all these things, but libertarians love them, so it maybe wasn't a huge surprise.
So it's quite surprising now to see claims that tech firms are or were naturally left wing. That certainly wasn't true in their most successful and vital periods. The transition from "let's unionize" being an April Fools joke to being an actual thing they say seriously was interesting to watch.
I think turning workers into owners is literally the thing leftists want?
But there's generally a lot of confusion since they want the "public" to own things, but never say which "public", and obviously state and employee ownership of a company is two different things.
I don't think I ever heard socialists promote equity stakes in companies - maybe co-ops is the closest equivalent? As you observe, when they talk about the "workers" they don't actually mean the workers, they mean that all power should be concentrated into a government which then derives legitimacy by claiming to represent the workers.
Channelling Thomas Sowell, the fundamental intuition behind left wing views is that some people are just much better than others (smarter, wiser, more moral, fairer etc) and those are the people who should be put in charge of things, with that re-arrangement of society being justified by theorized oppression. Giving workers an ownership stake in their employers is the opposite: it's the decentralization of power and undermines the whole concept of oppression.
You are confusing communists and socialists.
The traditional liberal approach is simply to tax corporations at a relatively high level. This might explain the economic stagnation after World War II, but it did provide a lot of benefits to most people. The argument back in the 1930s is that this made the government a partner with corporations, aligning their interests while providing money to buy nice things for the country and workers.
You should have been here in the mid-90s before the dotcom bust. Early Wired magazine culture was full of genius dreamers and we built Hotwired (the first commercial website) created the free software movement which built Apache and most of Linux and started all kinds of failed and successful companies and ideas. Burning Man really took off after the whole staff of Wired went and then ran a whole magazine about it.
It was a pretty amazing time and we had our share of all night parties and rave camp outs. And while lots of people thought they were going to get rich, more were there to create something amazing. It did give me my first IPO though.
Most of what happened in SF in the early aughts was by those who survived the massive downturn after 2001. I was out of work for 16 of 21 months. By the time things took off again I was too old and settled down to enjoy it but I am glad you did.
It’s nice to see young people recreating the sense of possibility and excitement for their own era in their own way. Cook that you got to see that.
Wish I had been there!
“… rising rents push more of the bohemians out of town every year.“
Moved to SF in 1990 onto to Oakland in ‘94. Moved into tech when working as a photographer at an ad agency in 95. Winkler was one of the first agency’s to go onto the web, they paid me more to do care and feeding of that than photography. Exit one creative into tech. Worked a few of start-ups in 98, 2000 and 2008. The money was better, by a lot.
My wife is a creative, a metal worker and jeweler. With each downturn she watched her sales go down even though there was always interest in her work. Similar story with most of the creatives in the east bay, where much of the art being created for Burning Man was done.
There was always the story of lots of money, it just only trickled to the creatives. Tech likes art, just does not want to pay for it. It’s why most of the artists have left the area, including us in ‘21 to New Mexico.
Yep.
Great writeup. There’s something a little too efficient about the lifestyle/career networking optimization you’re describing, but I wish them luck in having it all.
Bravo! Now just ask ChatGPT to rewrite it in the style of Tom Wolfe!
It's an interesting culture but I wonder how likely they are to really make so much money. From what I can tell, AI tech is heading in the direction of being both highly centralized and surprisingly easy to use, which is a recipe for a tiny number of extremely rich people who happen to get first mover advantage in infrastructure (so, maybe OpenAI and a few early startups that build on them) and then a huge long tail of mostly failing startups with hopes and dreams but little profit. Think mobile apps: a small number of huge commercial successes, mostly in gaming, and then a bazillion trivial and clone apps duking it out for little bits of attention. Except that AI doesn't have a direct path to the gaming industry unless RPGs start using LLMs for their NPCs (now there's a pile of TLAs for you!).
The obsession with "safety" is also a major risk factor for these guys, at least it seems obvious from the outside. That obsession has already completely crippled Google and put it firmly in last place, alongside Facebook, which I admit is not a twist I saw coming. I wonder how many AI startups are going to self-destruct through in-fighting. Maybe it's an uneasiness with that possibility that leads to the (highly interesting, unexpected) coolness towards woke politics Noah is reporting.
Even that might be too optimistic. It might just get incorporated into existing tech megacorps like Google and Microsoft.
Any use cases you find particularly compelling, no matter how mundane or niche?
I'll bet AI is just the excuse for putting in the appropriate monitoring and looking at the result. It is very hard to do that kind of thing in a corporate setting without running into internal political roadblocks. If you can get the C-suite to stamp it AI, odds are you might be able push it through.
That sounds plausible re. MS Office. Chat GPT is a fun toy, but I haven’t heard of any significant practical applications beyond cheating on High School essays.
The big practical application I’ve heard from people is if you can benefit from writing little bits of code in various scripting and programming languages to help you do certain things, but don’t actually know how all those languages work. ChatGPT does a great job of writing mostly usable chunks of code, and helping you fix them when you get error messages (though you need to have a bit of understanding of what is going on to catch some more structural issues).
Yes, little bits of code - unfortunately, significant software requires a lot more than that. Mostly a more updated version of code completion that can handle slightly larger chunks and patterns.
I lived in Palo Alto (Byron St) in the early 80s, walked to the CalTrain Station through downtown, then rode the express train and Financial District shuttle bus to the 500 block of Sansome, where I worked. It was a very different era. Palo Alto was very quiet and not outrageously overpriced. The professional ranks were eclectic and interests ranged widely.
Did you catch that the Radiohead quote was from the song "Palo Alto"?
https://www.youtube.com/watch?v=mrYTsLgCidQ
I'm one of the techies I suppose who's part of this AI subculture, but not in Silicon Valley because of life reasons (PhDing at CMU). I feel your article has captured the major cultural undercurrents pretty well and is pretty representative of my friends in industry and academia alike.
I do have a couple questions, maybe you can shed some light.
1. Why are the VCs so bullish on one specific paradigm (the flavour of the season appears to be Generative AI)? I see this sort of concentration is cyclic, Web3 and Self-driving were the big flavours earlier. Has the strategy shifted? I think I saw some articles about a16z moderate risk strategy lately.
2. What is your take and the estimate of the AI hype-train vs actual AI progress in terms of real life products? ML is certainly very critical already in many applications more subtly. I certainly believe we are at the cusp of the post internet age (my personal bet is on vision, ML and robotics), but I'm not certain how increasingly AI-first and automation products will be received by people and society.
1) Herd behavior I suppose. LLMs and AI art apps are very cool, and there are lots of obvious potential applications, so maybe they're doing the right thing.
2) Here we exceed my level of expertise, but I think most people I talk to agree on the "product, not a platform" idea, which is probably why OpenAI is partnering with Microsoft.
3. I forgot to ask one more thing. How critical do you think is to stay in the in-group of Silicon Valley network to be a part of this "supposed" success era in the next few years? A lot of folks like me are constrained to be out of SV right now, due to well, life. :)
I don't know! It's easier to get funded if you're in SV, but if you're an employee rather than a founder, that's less of a constraint...
Interesting, would love to know your thoughts on founding a company in the US (for immigrants or otherwise) in terms of being abreast of the general social and cultural dynamics and navigating the bureaucratic processes. Hence I felt to ask the need to be in SV and be part of the cultural in-group. For instance, I'm generally bearish about some developing countries due to formation of recurring kinds of nexuses. Is it something that plagues US?
Now that I think about it, it's more of an article idea! Feel free to ignore to respond. :)
Not my area of expertise but I have some friends working on exactly this.
The answer to both of your questions depends on your confidence that this is a) a trend and not a fad, b) that Generative AI in particular is going to be a horizontally disrupting technology that moves quickly.
The bullishness on Generative is because it has implications at every level of the stack, from the need for different hardware up to the ability for average consumers to make funny little new apps. If you think Generative is similar to expert systems, or the last wave of ML, then it may be disruptive for a few use cases but it doesn't deserve the hyper it is getting.
But there are times where the technology itself isn't just applied to problems, but starts to shift market dynamics themselves (think beginning of internet, beginning of mobile). That is when every month shifts the rules of the game, where building trust with the right people matters, where learning is both nuanced and rapid. In short, it's those moments where silicon valley really shines.
Thank you for this perspective! I think this makes sense, I do think the answer wrt Generative Model is that it is bigger than the "expert systems" craze but it's also not going to change how we view farming or mining automation for instance (that's my view right now, I might be proven wrong in 5 years). But in general, learned systems (instead of explicit math modelling) are here to disrupt the way we think about engineering and science in many ways. Generative models are one such approach with pros/cons, and they are very good for certain kinds of applications.
I do agree that we are witnessing a general paradigm shift, and hence all the excitement from Silicon Valley is really amazing to see. :)
These group houses remind me of the party scene in "All the Birds in the Sky", which more than anything, reminds me how uncomfortably accurate Anders' depiction of SF was (gets at the weird intersection between bohemian culture and tech culture), but also that this manifestation of "AI" specific houses isn't actually a divergence from the past. Scott Alexander's "Every Bay Area House Party" series (https://astralcodexten.substack.com/p/every-bay-area-house-party) is also wonderful
Yes, All the Birds in the Sky is a GREAT depiction of the 2000s tech scene and how it tried to meld with bohemian culture (probably in the 90s too, but that's before my time). Now, with the true bohemians priced out of the city and the the tech shifting from internet to AI, the culture is shifting too, but lots of what I was talking about in this post can be traced, however tenuously, to that earlier era...
https://mezha.media/en/2022/12/26/programmers-who-write-code-with-ai-helpers-make-more-mistakes-a-new-study-says/?utm_source=substack&utm_medium=email
https://arxiv.org/abs/2302.08927?utm_source=substack&utm_medium=email
I live in Hayes Valley and on paper am part of this culture. It’s depressing how completely cutoff the tech/finance world is from the actual city of san francisco. An entire corporate playground has been built on top of a formerly cool city. What’s left is rich people paying for these “bohemian” experiences and a working class struggling to survive.
All the actual cool people live in Oakland now.
Cool people can't afford SF. It's like SoHo and Tribeca in NYC. You could get cheap space if you didn't mind a machine shop or the welfare department next door.
Noah can you write an article about AI and interest rates:
https://forum.effectivealtruism.org/posts/8c7LycgtkypkgYjZx/agi-and-the-emh-markets-are-not-expecting-aligned-or
It's my belief that
1) AI will be a very powerful technology compared to competitors
2) Well-developed economies grow a slower rate as they move increasingly into a regime of diminishing returns.
3) So no giant economic growth is coming but there may still be a giant transformation of our society. (I.e. Markets are broadly right on this one.)
Basically, the mathematical tool used to predict growth from interest rates here (a consumption Euler equation) is known to fail at short horizons, and its long-term horizon correlation with growth is just being driven by the Volcker rate hikes and subsequent 90s growth boom. That being said, I agree that markets are not expecting high rates of growth, but it's also probably true that they rarely predict them correctly. As for whether or not AI will generate lots of growth, that's hard for anyone to say, which is why I don't put too much stock in the market predictions!
Palo Alto Cowboy
https://open.spotify.com/track/6oakudno5EtF1yVE8p4Yo0?si=6ZUoYHzNTOqCcuWkC8Hs5A
See p. 20:
https://research.ark-invest.com/hubfs/1_Download_Files_ARK-Invest/Big_Ideas/ARK%20Invest_013123_Presentation_Big%20Ideas%202023_Final.pdf?utm_source=substack&utm_medium=email
I love Ark's presentations, but...blockchains?? Seriously??
No, I missed that. Thank you for calling my attention to the source of the quote.