"Of course that leaves all sorts of other ideas to try — inventing newer hardware and ways of making hardware run faster, incorporating different kinds of data into LLMs, looking for newer and better algorithms, or even just investing more money."
The gains we are seeing with the open source LLMs, LlaMa, Vicuna, Wizard, et. al. with the 7B, 13B and 65B parameter models that can be run at home is where the action is. Spinning up your or your companies own custom LLM or large multimodal model (LMM) on run-of-the-mill GPUs or some AWS compute for a day or two is equivalent to the explosion of the WWW in 1996. As for running out of training data, Whisper by OpenAI, which offers an order of magnitude more training data by introducing voice, TV shows, radio broadcasts, YouTube videos, etc. into the mix will push back the "running out of training data based on text" by at least a year or two. Add in sparse neural nets and other technologies that introduce orders of magnitude more efficiency with current data sets, and the widespread adoption of custom hardware, and we have A.I. (M.L.) liftoff in all fields of human knowledge, including M.L. itself.
To follow up, the introduction of models of A.G.I. has been proposed by a leading A.I. researcher, Yann LeCunn at Facebook/Meta, wherein he proposes a roadmap to A.G.I. In sum, take the transformer architecture, which has a local short-term memory and add in goal driven agents that predict the future state of the world, and reward or punish said agents based on their predictions of future world-states. Thus M.L. architectures such as LLMs will develop agency to act in the world, as opposed to being state-machines which simply respond to "prompts". Microsoft researchers already see "sparks of A.G.I." in LLMs wherein reasoning is emerging from the rules of language.
Ah sorry wasn't clear, Dylan is in reference to the author of Semi Analysis. This article Noah referenced is from Jan and a lot has changed since then and Dylan has basically recanted this January post because a lot of the open source models are proving you can train with far far less resources at close to GPT3.5 (and soon gpt4) quality so this AI training "ceiling" seemingly no longer exists. Sorry for being unclear.
"There is a whole different universe of the AI industry that seeks to reject big iron compute. The open-source movement around small models that can run on client device is probably the 2nd most discussed part of the industry. While models on the scale of GPT-4 or the full PaLM could never hope to run on laptops and smartphones, even with 5 more years of hardware advancement due to the memory wall, there is a thriving ecosystem of model development geared towards on-device inference."
This is roughly what JDKee was getting at in his original comment. My main point is just that AI research is proceeding SO rapidly something from January of this year is woefully out of date half a year later.
In his interview with ilya sutskever, Jensen Huang said he was looking forward to using hardware acceleration to make AI compute costs a million times cheaper.
The Section 8 housing model might work here. The government will only pay for degrees costing less than a preset amount. Yes, there will be imperfection in how the numbers are set. Yes there will still be some upward price pressure. These are the costs of social mobility and meritocracy.
Knowledge expansion has led to longer PHDs. Spending 7 years accumulating student loans before being forced by the market to see where you stand relative to other PHDs is a huge waste.
How about price controls? The Econ 101 reason to not do price controls is that they lead to shortages and rationing. But we already have shortages and rationing because of the way prestige works in college admissions. So why not give it a try?
Colleges and universities have contracted the cost disease of American infrastructure costs.
We know how costs go up, we know why costs go up, but the reason it's so hard to contain is that the drivers of cost are distributed throughout the college.
Same thing with infrastructure. It isn't one factor, or one favored political boogeyman (greedy contractors, unions, politicians jonesing to cut ribbons). It's little drivers of costs spread throughout a project or a system like a university.
Interesting that are least efficient and highest cost sectors seem to be education, military contracting and healthcare (particularly hospitals, where US charges are multiples of those in Canada or Europe). I wonder if there is a common denominator?
Appears that there isna gentle 22 year linear fit until then of gentle manufacturing Capex increases. Maybe tied to long term GDP growth...
And it will be manufacturing that helps lower poverty. Because of their great living wages without a college degree.
Colleges got ridiculous and over their ROI skies partly because there was little alternative competition. When manufacturing middle class jobs go down and less a desirable life style, college was all the hype. Great lives are lived without a degree.
Artificially pushing factory construction in non-economic industries using transitional tech (EVs) where content is still likely to be 40 pct Chinese may not be the brightest idea. Massive investment in research and design might have been smarter (and I know this is part of it, but then there would be no UAW jobs, which is the real objective). Also questioning wisdom of building fabs in US- makes perfect sense from national security point of view (like Germany making oil out of coal during WW2), but is questionable economically. The track record of pols throwing money at their idea of the moment isn’t so good. Of Meanwhile, manufacturing employment has fallen of late. I hope this all works because competitive onshore manufacturing would be fantastic. I hope it doesn’t end up being the equivalent of Jimmy Carter’s wind turbines on the Altamont pass. That politically forced investment really made global leaders out of Fayette manufacturing and Kenetech, no?
Do you know the difference between investment dollars and jobs? One should (less likely when investment isn’t justifiable on its own economics but requires government to pay for it) lead to the other but hasn’t done so yet. Without investment in training and apprenticeships, what is likely to happen is that skilled workers for these factories will be poached at higher wages from lower value manufacturing industries, who may then close down or import from abroad instead. For instance, Amazon warehouses have poached a lot of reasonably skilled and dependable people who might otherwise work in manufacturing. I mean, nothing is more important than providing a marketplace to fence shoplifted goods and send us junk we don’t need, no? EV assembly is actually easier and more automated than ICE vehicle assembly. Ford will be hiring workers at new factories paying the lower UAW wages for new hires and then shut down its old factories over time. Genius.
Do you know how many manufacturing jobs there are in the US? Do you know how that compares to the size of the middle class? Do you know by how much each has grown in size over the last 40 years? And it is no longer the case that unskilled, uneducated workers can get a job doing rote work on an assembly line at union wages. Those low skill jobs are for robots. Modern manufacturing requires some intelligence (and often a 2 year or 4 year degree). Modern manufacturing will stop some kids from wasting 4 years pursuing an expensive liberal arts degree (to go work in a skilled position on a factory floor instead), but it is unfortunately not going to be a godsend for the bottom half of a high school class, which is what I am worried about.
Well, I've led and managed over $75 million in capital expenditures, so yes I do. And yes, I've had hundreds of manufacturing folks with HS, Associate degrees, and OJT. Many from military and defense contractors as well.
Amazon has provided excellent jobs with excellent wages and benefits. Manufacturing competes when they have the need for labor.
Well, you claim you know the difference between dollars and jobs, yet you conflate a graph of the former with my factual comment on the latter.
Glad you agree that higher value manufacturing will poach labor from other industries and lower tier manufacturing , leaving these with the less valuable workers yet having to pay higher wages to compete. We all know what happens next in an open economy….except in services hard to substitute with foreign goods. And a lot of those manufacturing jobs will require some higher education, though not as English majors.
From your experience it sounds like you also agree that modern manufacturing will not be the job solution or pathway to middle class for the bottom half of the high school class, which was the old UAW model.
I have worked on those old factory floors (as a member of a metalworkers union). Few companies are going to pay middle class wages for the sort of work the bottom half of the high school class can do. A UPS or Amazon warehouse does, but only for the most dependable, diligent, trustworthy and physically fit of the bottom half of the class (precious few)
As a side note for your summary about limits to AI, another problem that was pointed out by AI researchers is that currently AI models could not understand logics, which is why without new methods to teach these models, as the amount of good training data dwindles the progress in generative AI will slow down. You could look into this article from Sydney Morning Herald, when ChatGPT was just published: https://www.smh.com.au/national/is-ai-coming-of-age-or-starting-to-reach-its-limits-20221213-p5c5uy.html.
Also, for your point of views about Millenials, I think that could be true in the US, and maybe continental Europe (with more young people's vote shifting right recently), but do you think it could be the same for young people in other English-speaking countries like Canada, UK or Australia? Here in Australia, I saw that many young people could not buy houses (or aspire to buy houses) anymore, so they are increasingly voting for left-wing parties: Labor, Green and independent candidates. In fact, that's partly the reason why the share in vote for 2 major parties in Australia is in a record low!
So if you come across any research papers or data for other Anglosphere countries, could you write more about whether Millennials there are still better off, and would their political ideas be more moderate?
You put into articulate words my exact thoughts on generative AI. It is unlikely we are going to see any modeling break-throughs, so at this point it is mostly a data arm race. Humans are the only source of new training data and are not supplying it any faster, so where are the bigger data sets going to come from now that the whole internet and the world’s libraries have been scoured?
These are already massive models where the marginal benefit of more parameters has probably been exhausted. I am doubtful that increases in computing power will help much. The computational power goes into the training and the training does not have to be very fast - slow release cycles seem to be working just fine. The generation of text on trained model requires comparatively little power.
Grad school lending is fundamentally different than UG lending. Lots of UGs pay for school through parents. Grad students don't meaning that schools are constrained in how much they can charge (or how small the stipend) by the student's total available credit.
For UGs schools can access parental credit (eg home equity loans) making it plausible that loans open up access to those whose parents lack that credit.
And grads can (and do) borrow the full cost of tuition plus living expenses. At law schools it is not uncommon to have students graduating with 200-250k of debt but very limited job prospects. Not to worry, go work for an NGO in “public interest” and pay only 10 pct of your income (say $5-$10k a year on a $200k debt) for 10 years and then everything is written off. Great deal for the taxpayers! The whole thing is approaching the level of a scam to keep university professors/departments afloat and ensure a massively subsidized supply of activists to work in government or NGOs.
That depends alot on whether you think there is a social benefit to having ppl work at NGOs or as teachers. But I tend to agree it's kinda a way to hide that subsidy. But less bad than the standard ways we hide subsidies (instead of just having the gov pay reverse income tax or cover ppl's rent who get evicted during covid we hide the tax in a mandate that results in higher prices).
So this would hardly be my primary target if we want to be more transparent about our subsidies.
I’d rather pay teachers more, eliminate the BS union rules that require masters and PhD degrees dir salary bumps, and dramatically scale back the borrowing limits for grad school.
Regarding teachers the problem is we aren't willing to use any tests that would actually measure the performance we should care about.
Any exam in science or moath where you know the type of questions that will be asked beforehand is just asking to waste student time. There is no benefit anymore to learning to memorize any formula or apply any rote method and any test that's a known quantity always incentivizes giving students rote rules to save those problems.
If you wanted to actually measure conceptual understanding you'd need to ask easier versions of the kind of problems that show up on the math and physy Olympiads. Shit that comes out of right field so it's only by applying conceptual understanding that you can master them.
But we aren't willing to do that. In part bc parents and teachers would see it as unfair and get angry they didn't understand it but mostly because it means accepting that a decent fraction of motivated intelligent students will simply fail. That's because it's basically impossible to gain conceptual understanding if you hate doing a subject and it makes you feel dumb no matter how much you force yourself.
Given we aren't willing to do that I'm not sure how you can usefully measure teacher performance.
At the primary school level, rote learning and practice plays a very important role (phonics, grammar, arithmetic tables).
Putting a kid on a football pitch and expecting him/her to think strategically is all well and good, but completely useless if they can’t dribble or trap or pass a ball. Drills have a purpose, and a facility with the basic mechanics of language, arithmetic and sport is a precondition to advanced capabilities.
I mean, isn’t this part of story not just higher tuition for grad classes but the proliferation of unnecessary graduate programs? I’m thinking specifically about the proliferation of things like “MAPH” or Masters of Arts programs. Or Columbia grad degrees in journalism. Accreditation factories that result in huge debt for basically not much advantage in advancing in your chosen field.
But even something like Law school doesn’t have the same ROI it used. Yes people going to elite law schools or using the law degree to go white shoes law firms likely do see a health ROI. But for others? It’s very likely you’re going to be burdened with heavy debt payments for quite a while.
Sort of thoughts I’ve had regarding education.
- Undergraduate degrees are still quite important and good to have. But instead of government funding tuition, there should be money directed to expanding colleges themselves. More $$$ for community college expansion, more $$$ for state universities to expand and maybe more important, financial inducements for your Harvard or Yale to have 2x or 3x the class size they do. The last one is sort of important to me. All these admissions fights about who gets into Harvard or Yale is downstream of the fact that colleges like these that advertise themselves as engines of progressive progress need to put their money where their mouth is and actually a allow a lot more students to be admitted.
- Core curriculums are a relic of a time when college fulfilled a wildly different purpose. Kids need to be able to go right into their major as soon as possible. Freshman year or even just first semester may be a class schedule more akin to senior year of high school as sort of an acclimation period. But after that? No reason to have a common core which just keeps students at college way longer than necessary and with much higher debt then necessary.
- You should be able to go grad school as soon as you are ready to. For some programs like medicine, this likely still means some undergraduate work in subjects like organic chemistry. But in general there are a lot of unnecessary undergraduate classes needed before graduate school. Furthermore, admission to undergraduate should involve if not zero admission process to grad school a much smoother process into the grad school at that college.
Scattered thoughts for sure. Curious if Noah thinks I’m on to something or full of it or both.
Except the problem is that it's not the education that you receive at college that accounts for the returns to income. Bryan Caplain pounds that nail into the ground in his book Against Education. For a couple lines of evidence, consider the fact that even if you go to something like an extension school and get courses taught by the same instructors you get no return, that for most jobs what college you went to matters far far more than your major (some exceptions in STEM), that it's actually relatively easy to graduate from places like Harvard once you get in.
I could go on, but I think it's pretty clear that the extra income you get from going to someplace like Harvard is largely because you've demonstrated you could get into Harvard (and then didn't have some issue that caused you to drop out). What companies are doing is relying on the fact that you demonstrated your ability to pass the difficult admissions process. It would be expensive (and create legal risks) to have such extensive screening themselves and colleges do quite a good job at identifying who will succeed.
So expanding enrollment wouldn't actually produce much in the way of returns because it's the selectivity itself that's providing the boost in income.
Having said that let me defend colleges as a pretty good deal.
1) The social benefits of college.
There are huge social benefits to people to leaving home and going to college. We make some of our closest friends there and often our partners. Have the chance to make our mistakes with alcohol, drugs, romance etc in a relatively forgiving environment where that's least likely to spiral out of control.
I mean, in terms of lifetime satisfaction I think most people would agree that the friends and experiences they had in college were worth a huge amount. Hell, I suspect that they are worth more in terms of happiness than a house so are quite a good investment. And, indeed, it's common for societies to invest substantial resources into coming of age practices.
Unfortunately, you'd never convince parents to pay out that kind of money so their kids can make some friends and explore who they are not to mention that you need responsibilities for the system to work so we have the pretense that it's about learning critical skills. And the learning does help us be better citizens and voters even if not more productive workers.
2) Signalling that you'll be a valuable employee. This isn't mostly about gaining any particular skill set bc most of what most ppl do at their jobs (MDs, lawyers and engineers excepted) isn't that related to anything u might learn in school. Rather, what you need to show is that your smart and have the kind of conscientiousness and discipline to succeed in the competition for college slots and will be perceived as high status.
So no matter what we do we're going to have a bunch of pricy competition to signal we'd make valuable employees. It could have been that you showed that off by getting into the school with the best sports program or nicest dorms but instead it's the one whose profs produce the best research.
That's a really good deal. That wasteful competition will happen either way but at least we get some positive externalities out of it. I wish they'd focus more on STEM (and even analytic philosophy) and less on English, X studies etc but still good deal.
--
The issue is that we need more college like environments for people not going into the managerial class and that we need to stop the rise of costs due to non-academic administrators.
Re: law school, I don't know if this is still the case, but 10-12 years ago there was a thing where lots of law schools would say stuff like "the median starting salary for our grads is 75k," but almost no actual jobs paid 75k - you had a handful of jobs paying 150k, and a bunch paying like 40-50k, so it netted out to a 75k "median." (I might be a little off in my math there, I'm tired, but you get what I mean.) Always struck me as a bit of a scam.
“A highly complex series of irrigation systems used water from the river for crops across the region, bolstering Ukraine’s reputation as one of the biggest food exporters in the world, with some 33 million hectares of farmland. But officials fear much of the region’s farmland may now be ruined.”
“The Ukrainian Agrarian Council estimates that the Kakhovka disaster could lead to a 14 percent drop in Ukraine’s grain exports. The country is the world’s fifth-largest exporter of wheat—meaning there will be serious knock-on effects for countries that rely on imports.”
And the surging wheat fields and wheat will be absorbing this toxic stew in the groundwater for years. Knock-on effects, indeed. The spark that set-off The Arab Spring was a serious bread shortage, which was caused by drought that collapsed the Ukrainian wheat harvest. An empty stomach is an angry stomach. Putin is now fighting with biological and chemical warfare via breaching this dam. He’d agreed to let Ukrainian wheat be sold and shipped, but that issue is now likely moot. I doubt China or Saudi Arabia are happy about this.
Throwing money at a market where supply is extremely inelastic tends to just raise prices. (We see the same problem with people wanting to subsidize rent in expensive markets like the Bay Area, without loosening regulations that prevent construction of new units.)
Wouldn't this be commonly understood? I teach my HS econ students about producer and consumer surpluses and how curve slopes affects the distribution of benefits to any subsidy (or costs with a tax). Certainly in the economics field, this must be well understood?
I think most economists understand the idea, but there is an enormous contingent of Democrats who seem to subscribe to "check-ism", where once you write the check to fund something, you've done what you needed to do. I think Noah has written about this pathology before.
Los Angeles basically flushed more than a billion dollars down the drain, building a pathetic number of homes, because they didn't actually do anything to remove the legal and "process" barriers to building the units.
This is also the big worry about the Inflation Reduction Act and CHIPS Act funding for manufacturing in the US -- we can already see the increased spending in government stats, and the law authorizes _enormous_ sums. But we at least have the _potential_ that we could end up wasting a huge portion of those funds on studies and re-designs and fighting NEPA lawsuits, without ever actually building the transmission lines, solar farms, fabs, and so on. We need to not let that happen.
I get it. There is a very real idea in government that you measure success by how much money you spend not by what you actually build.
I live in CA; our "bullet train" is legendary in this respect. It's a smashing success ("billions spent on high speed rail!") on paper. In reality, we've laid less than 50 miles of track in the middle of nowhere.
Yeah, I'm active in local politics in the Bay Area (Planning Commissioner in a mid-size suburb, tried to unseat an incumbent on my City Council last fall but only got to 45%), and I'm a big fan of Effective Government CA, which publishes on Substack as "Modern Power".
I'm a former Planning Commission for Elk Grove, CA (about 150K pop) about an hour away from you! Did 7 years from 2006 to 2013. 45% in a local against an incumbent isn't too shabby. If you ever want to swap war stories, email me at bdvillanueva on Yahoo's mail service. (Don't want to embed the full address in a public forum for spam bot reasons.)
I work in AI research and I am strongly suspicious that there is still a ton of low-hanging fruit to be discovered in the "newer and better algorithms" space. While we may be close to hitting the limit for "how many parameters can we train at once", we are *nowhere near* hitting the limit for "how do we best arrange these parameters to get the best model?". I see new papers making 10-20% efficiency gains on this front on a monthly basis, and I see no reason to expect them to slow down. There has also been a renewed interest in totally new, moonshot ideas in AI, which makes me very excited. If the backpropagation gradient estimation algorithm was dethroned by a forward-only algorithm, for example, that could mean ~2x gains across the board for model training speed! So far, no such algorithm has been discovered, but many smart people are working on it, so I'm optimistic.
There should be nearly endless amounts of possible optimizations you can do for an LLM until it's not "large" anymore.
We don't know anything about how they work. What we do know is that transformer models are a very very generic architecture that probably learns a more specific architecture for the task. So, if you can specialize it again, it'll probably use less than all the power and memory ever, which is what it currently does.
Assume that AI improvements come to a screeching halt (very unlikely). We still have tremendous growth coming from taking advantage of it.
Think of electricity. They figured out how to generate it and make it ubiquitous. And we then spent decades of significant growth as we figured out how to best use it.
"Of course that leaves all sorts of other ideas to try — inventing newer hardware and ways of making hardware run faster, incorporating different kinds of data into LLMs, looking for newer and better algorithms, or even just investing more money."
The gains we are seeing with the open source LLMs, LlaMa, Vicuna, Wizard, et. al. with the 7B, 13B and 65B parameter models that can be run at home is where the action is. Spinning up your or your companies own custom LLM or large multimodal model (LMM) on run-of-the-mill GPUs or some AWS compute for a day or two is equivalent to the explosion of the WWW in 1996. As for running out of training data, Whisper by OpenAI, which offers an order of magnitude more training data by introducing voice, TV shows, radio broadcasts, YouTube videos, etc. into the mix will push back the "running out of training data based on text" by at least a year or two. Add in sparse neural nets and other technologies that introduce orders of magnitude more efficiency with current data sets, and the widespread adoption of custom hardware, and we have A.I. (M.L.) liftoff in all fields of human knowledge, including M.L. itself.
To follow up, the introduction of models of A.G.I. has been proposed by a leading A.I. researcher, Yann LeCunn at Facebook/Meta, wherein he proposes a roadmap to A.G.I. In sum, take the transformer architecture, which has a local short-term memory and add in goal driven agents that predict the future state of the world, and reward or punish said agents based on their predictions of future world-states. Thus M.L. architectures such as LLMs will develop agency to act in the world, as opposed to being state-machines which simply respond to "prompts". Microsoft researchers already see "sparks of A.G.I." in LLMs wherein reasoning is emerging from the rules of language.
See https://ai.facebook.com/blog/yann-lecun-advances-in-ai-research/
See also https://www.microsoft.com/en-us/research/publication/sparks-of-artificial-general-intelligence-early-experiments-with-gpt-4/
I came here to post basically this. Even Dylan has admitted to this in recent months.
Huh?
See my response bow to Flume, sorry for being so vague!
Dylan?
Ah sorry wasn't clear, Dylan is in reference to the author of Semi Analysis. This article Noah referenced is from Jan and a lot has changed since then and Dylan has basically recanted this January post because a lot of the open source models are proving you can train with far far less resources at close to GPT3.5 (and soon gpt4) quality so this AI training "ceiling" seemingly no longer exists. Sorry for being unclear.
Here's Dylan's latest post on this topic for anyone who's interested - https://www.semianalysis.com/p/on-device-ai-double-edged-sword
"There is a whole different universe of the AI industry that seeks to reject big iron compute. The open-source movement around small models that can run on client device is probably the 2nd most discussed part of the industry. While models on the scale of GPT-4 or the full PaLM could never hope to run on laptops and smartphones, even with 5 more years of hardware advancement due to the memory wall, there is a thriving ecosystem of model development geared towards on-device inference."
This is roughly what JDKee was getting at in his original comment. My main point is just that AI research is proceeding SO rapidly something from January of this year is woefully out of date half a year later.
In his interview with ilya sutskever, Jensen Huang said he was looking forward to using hardware acceleration to make AI compute costs a million times cheaper.
A million times!!
Higher education seems to be costing more and delivering less, relative to free/cheap alternatives. How do we fix it?
Subject for an upcoming post!
The Section 8 housing model might work here. The government will only pay for degrees costing less than a preset amount. Yes, there will be imperfection in how the numbers are set. Yes there will still be some upward price pressure. These are the costs of social mobility and meritocracy.
Knowledge expansion has led to longer PHDs. Spending 7 years accumulating student loans before being forced by the market to see where you stand relative to other PHDs is a huge waste.
If you are borrowing money to pay tuition for a PhD, you should not be getting a PhD. Pretty simple.
How about price controls? The Econ 101 reason to not do price controls is that they lead to shortages and rationing. But we already have shortages and rationing because of the way prestige works in college admissions. So why not give it a try?
Colleges and universities have contracted the cost disease of American infrastructure costs.
We know how costs go up, we know why costs go up, but the reason it's so hard to contain is that the drivers of cost are distributed throughout the college.
Same thing with infrastructure. It isn't one factor, or one favored political boogeyman (greedy contractors, unions, politicians jonesing to cut ribbons). It's little drivers of costs spread throughout a project or a system like a university.
Interesting that are least efficient and highest cost sectors seem to be education, military contracting and healthcare (particularly hospitals, where US charges are multiples of those in Canada or Europe). I wonder if there is a common denominator?
Love the manufacturing huge upswing!
Appears that there isna gentle 22 year linear fit until then of gentle manufacturing Capex increases. Maybe tied to long term GDP growth...
And it will be manufacturing that helps lower poverty. Because of their great living wages without a college degree.
Colleges got ridiculous and over their ROI skies partly because there was little alternative competition. When manufacturing middle class jobs go down and less a desirable life style, college was all the hype. Great lives are lived without a degree.
Artificially pushing factory construction in non-economic industries using transitional tech (EVs) where content is still likely to be 40 pct Chinese may not be the brightest idea. Massive investment in research and design might have been smarter (and I know this is part of it, but then there would be no UAW jobs, which is the real objective). Also questioning wisdom of building fabs in US- makes perfect sense from national security point of view (like Germany making oil out of coal during WW2), but is questionable economically. The track record of pols throwing money at their idea of the moment isn’t so good. Of Meanwhile, manufacturing employment has fallen of late. I hope this all works because competitive onshore manufacturing would be fantastic. I hope it doesn’t end up being the equivalent of Jimmy Carter’s wind turbines on the Altamont pass. That politically forced investment really made global leaders out of Fayette manufacturing and Kenetech, no?
Did you miss Noah's graph on the explosion in US manufacturing construction?
I am in complete disagreement, respectfully, with your perspective.
Manufacturing is the only broad work segment that creates quality middle class livelihoods without a college degree.
Do you know the difference between investment dollars and jobs? One should (less likely when investment isn’t justifiable on its own economics but requires government to pay for it) lead to the other but hasn’t done so yet. Without investment in training and apprenticeships, what is likely to happen is that skilled workers for these factories will be poached at higher wages from lower value manufacturing industries, who may then close down or import from abroad instead. For instance, Amazon warehouses have poached a lot of reasonably skilled and dependable people who might otherwise work in manufacturing. I mean, nothing is more important than providing a marketplace to fence shoplifted goods and send us junk we don’t need, no? EV assembly is actually easier and more automated than ICE vehicle assembly. Ford will be hiring workers at new factories paying the lower UAW wages for new hires and then shut down its old factories over time. Genius.
Do you know how many manufacturing jobs there are in the US? Do you know how that compares to the size of the middle class? Do you know by how much each has grown in size over the last 40 years? And it is no longer the case that unskilled, uneducated workers can get a job doing rote work on an assembly line at union wages. Those low skill jobs are for robots. Modern manufacturing requires some intelligence (and often a 2 year or 4 year degree). Modern manufacturing will stop some kids from wasting 4 years pursuing an expensive liberal arts degree (to go work in a skilled position on a factory floor instead), but it is unfortunately not going to be a godsend for the bottom half of a high school class, which is what I am worried about.
Well, I've led and managed over $75 million in capital expenditures, so yes I do. And yes, I've had hundreds of manufacturing folks with HS, Associate degrees, and OJT. Many from military and defense contractors as well.
Amazon has provided excellent jobs with excellent wages and benefits. Manufacturing competes when they have the need for labor.
No one needs a BA in English to work these.
Well, you claim you know the difference between dollars and jobs, yet you conflate a graph of the former with my factual comment on the latter.
Glad you agree that higher value manufacturing will poach labor from other industries and lower tier manufacturing , leaving these with the less valuable workers yet having to pay higher wages to compete. We all know what happens next in an open economy….except in services hard to substitute with foreign goods. And a lot of those manufacturing jobs will require some higher education, though not as English majors.
From your experience it sounds like you also agree that modern manufacturing will not be the job solution or pathway to middle class for the bottom half of the high school class, which was the old UAW model.
I have worked on those old factory floors (as a member of a metalworkers union). Few companies are going to pay middle class wages for the sort of work the bottom half of the high school class can do. A UPS or Amazon warehouse does, but only for the most dependable, diligent, trustworthy and physically fit of the bottom half of the class (precious few)
Thank you for your newsletter Noah!
As a side note for your summary about limits to AI, another problem that was pointed out by AI researchers is that currently AI models could not understand logics, which is why without new methods to teach these models, as the amount of good training data dwindles the progress in generative AI will slow down. You could look into this article from Sydney Morning Herald, when ChatGPT was just published: https://www.smh.com.au/national/is-ai-coming-of-age-or-starting-to-reach-its-limits-20221213-p5c5uy.html.
Also, for your point of views about Millenials, I think that could be true in the US, and maybe continental Europe (with more young people's vote shifting right recently), but do you think it could be the same for young people in other English-speaking countries like Canada, UK or Australia? Here in Australia, I saw that many young people could not buy houses (or aspire to buy houses) anymore, so they are increasingly voting for left-wing parties: Labor, Green and independent candidates. In fact, that's partly the reason why the share in vote for 2 major parties in Australia is in a record low!
So if you come across any research papers or data for other Anglosphere countries, could you write more about whether Millennials there are still better off, and would their political ideas be more moderate?
Thanks!!
You put into articulate words my exact thoughts on generative AI. It is unlikely we are going to see any modeling break-throughs, so at this point it is mostly a data arm race. Humans are the only source of new training data and are not supplying it any faster, so where are the bigger data sets going to come from now that the whole internet and the world’s libraries have been scoured?
These are already massive models where the marginal benefit of more parameters has probably been exhausted. I am doubtful that increases in computing power will help much. The computational power goes into the training and the training does not have to be very fast - slow release cycles seem to be working just fine. The generation of text on trained model requires comparatively little power.
In re solar panel production:
https://www.bbc.com/news/science-environment-65602519
Grad school lending is fundamentally different than UG lending. Lots of UGs pay for school through parents. Grad students don't meaning that schools are constrained in how much they can charge (or how small the stipend) by the student's total available credit.
For UGs schools can access parental credit (eg home equity loans) making it plausible that loans open up access to those whose parents lack that credit.
Yes, that's fair. But remember that grad school debt is a large portion of overall student lending, since professional school costs so much!
And grads can (and do) borrow the full cost of tuition plus living expenses. At law schools it is not uncommon to have students graduating with 200-250k of debt but very limited job prospects. Not to worry, go work for an NGO in “public interest” and pay only 10 pct of your income (say $5-$10k a year on a $200k debt) for 10 years and then everything is written off. Great deal for the taxpayers! The whole thing is approaching the level of a scam to keep university professors/departments afloat and ensure a massively subsidized supply of activists to work in government or NGOs.
That depends alot on whether you think there is a social benefit to having ppl work at NGOs or as teachers. But I tend to agree it's kinda a way to hide that subsidy. But less bad than the standard ways we hide subsidies (instead of just having the gov pay reverse income tax or cover ppl's rent who get evicted during covid we hide the tax in a mandate that results in higher prices).
So this would hardly be my primary target if we want to be more transparent about our subsidies.
I’d rather pay teachers more, eliminate the BS union rules that require masters and PhD degrees dir salary bumps, and dramatically scale back the borrowing limits for grad school.
Regarding teachers the problem is we aren't willing to use any tests that would actually measure the performance we should care about.
Any exam in science or moath where you know the type of questions that will be asked beforehand is just asking to waste student time. There is no benefit anymore to learning to memorize any formula or apply any rote method and any test that's a known quantity always incentivizes giving students rote rules to save those problems.
If you wanted to actually measure conceptual understanding you'd need to ask easier versions of the kind of problems that show up on the math and physy Olympiads. Shit that comes out of right field so it's only by applying conceptual understanding that you can master them.
But we aren't willing to do that. In part bc parents and teachers would see it as unfair and get angry they didn't understand it but mostly because it means accepting that a decent fraction of motivated intelligent students will simply fail. That's because it's basically impossible to gain conceptual understanding if you hate doing a subject and it makes you feel dumb no matter how much you force yourself.
Given we aren't willing to do that I'm not sure how you can usefully measure teacher performance.
At the primary school level, rote learning and practice plays a very important role (phonics, grammar, arithmetic tables).
Putting a kid on a football pitch and expecting him/her to think strategically is all well and good, but completely useless if they can’t dribble or trap or pass a ball. Drills have a purpose, and a facility with the basic mechanics of language, arithmetic and sport is a precondition to advanced capabilities.
Goodhart's law says measuring teacher performance would be futile. "When a measure becomes a target, it ceases to be a good measure."
I mean, isn’t this part of story not just higher tuition for grad classes but the proliferation of unnecessary graduate programs? I’m thinking specifically about the proliferation of things like “MAPH” or Masters of Arts programs. Or Columbia grad degrees in journalism. Accreditation factories that result in huge debt for basically not much advantage in advancing in your chosen field.
But even something like Law school doesn’t have the same ROI it used. Yes people going to elite law schools or using the law degree to go white shoes law firms likely do see a health ROI. But for others? It’s very likely you’re going to be burdened with heavy debt payments for quite a while.
Sort of thoughts I’ve had regarding education.
- Undergraduate degrees are still quite important and good to have. But instead of government funding tuition, there should be money directed to expanding colleges themselves. More $$$ for community college expansion, more $$$ for state universities to expand and maybe more important, financial inducements for your Harvard or Yale to have 2x or 3x the class size they do. The last one is sort of important to me. All these admissions fights about who gets into Harvard or Yale is downstream of the fact that colleges like these that advertise themselves as engines of progressive progress need to put their money where their mouth is and actually a allow a lot more students to be admitted.
- Core curriculums are a relic of a time when college fulfilled a wildly different purpose. Kids need to be able to go right into their major as soon as possible. Freshman year or even just first semester may be a class schedule more akin to senior year of high school as sort of an acclimation period. But after that? No reason to have a common core which just keeps students at college way longer than necessary and with much higher debt then necessary.
- You should be able to go grad school as soon as you are ready to. For some programs like medicine, this likely still means some undergraduate work in subjects like organic chemistry. But in general there are a lot of unnecessary undergraduate classes needed before graduate school. Furthermore, admission to undergraduate should involve if not zero admission process to grad school a much smoother process into the grad school at that college.
Scattered thoughts for sure. Curious if Noah thinks I’m on to something or full of it or both.
Except the problem is that it's not the education that you receive at college that accounts for the returns to income. Bryan Caplain pounds that nail into the ground in his book Against Education. For a couple lines of evidence, consider the fact that even if you go to something like an extension school and get courses taught by the same instructors you get no return, that for most jobs what college you went to matters far far more than your major (some exceptions in STEM), that it's actually relatively easy to graduate from places like Harvard once you get in.
I could go on, but I think it's pretty clear that the extra income you get from going to someplace like Harvard is largely because you've demonstrated you could get into Harvard (and then didn't have some issue that caused you to drop out). What companies are doing is relying on the fact that you demonstrated your ability to pass the difficult admissions process. It would be expensive (and create legal risks) to have such extensive screening themselves and colleges do quite a good job at identifying who will succeed.
So expanding enrollment wouldn't actually produce much in the way of returns because it's the selectivity itself that's providing the boost in income.
Having said that let me defend colleges as a pretty good deal.
1) The social benefits of college.
There are huge social benefits to people to leaving home and going to college. We make some of our closest friends there and often our partners. Have the chance to make our mistakes with alcohol, drugs, romance etc in a relatively forgiving environment where that's least likely to spiral out of control.
I mean, in terms of lifetime satisfaction I think most people would agree that the friends and experiences they had in college were worth a huge amount. Hell, I suspect that they are worth more in terms of happiness than a house so are quite a good investment. And, indeed, it's common for societies to invest substantial resources into coming of age practices.
Unfortunately, you'd never convince parents to pay out that kind of money so their kids can make some friends and explore who they are not to mention that you need responsibilities for the system to work so we have the pretense that it's about learning critical skills. And the learning does help us be better citizens and voters even if not more productive workers.
2) Signalling that you'll be a valuable employee. This isn't mostly about gaining any particular skill set bc most of what most ppl do at their jobs (MDs, lawyers and engineers excepted) isn't that related to anything u might learn in school. Rather, what you need to show is that your smart and have the kind of conscientiousness and discipline to succeed in the competition for college slots and will be perceived as high status.
So no matter what we do we're going to have a bunch of pricy competition to signal we'd make valuable employees. It could have been that you showed that off by getting into the school with the best sports program or nicest dorms but instead it's the one whose profs produce the best research.
That's a really good deal. That wasteful competition will happen either way but at least we get some positive externalities out of it. I wish they'd focus more on STEM (and even analytic philosophy) and less on English, X studies etc but still good deal.
--
The issue is that we need more college like environments for people not going into the managerial class and that we need to stop the rise of costs due to non-academic administrators.
Re: law school, I don't know if this is still the case, but 10-12 years ago there was a thing where lots of law schools would say stuff like "the median starting salary for our grads is 75k," but almost no actual jobs paid 75k - you had a handful of jobs paying 150k, and a bunch paying like 40-50k, so it netted out to a 75k "median." (I might be a little off in my math there, I'm tired, but you get what I mean.) Always struck me as a bit of a scam.
Do you mean 'mean' salary? The median salary is the one half of grads make more than.
The Build-Some Things Country.
“A highly complex series of irrigation systems used water from the river for crops across the region, bolstering Ukraine’s reputation as one of the biggest food exporters in the world, with some 33 million hectares of farmland. But officials fear much of the region’s farmland may now be ruined.”
https://www.nbcnews.com/news/world/satellite-images-show-ukraine-dam-destruction-rcna88299?mc_cid=0f29d6a665&mc_eid=1a668c9ebc
https://www.wired.co.uk/article/kakhovka-dam-flooding-ukraine?mc_cid=0f29d6a665&mc_eid=1a668c9ebc
“The Ukrainian Agrarian Council estimates that the Kakhovka disaster could lead to a 14 percent drop in Ukraine’s grain exports. The country is the world’s fifth-largest exporter of wheat—meaning there will be serious knock-on effects for countries that rely on imports.”
And the surging wheat fields and wheat will be absorbing this toxic stew in the groundwater for years. Knock-on effects, indeed. The spark that set-off The Arab Spring was a serious bread shortage, which was caused by drought that collapsed the Ukrainian wheat harvest. An empty stomach is an angry stomach. Putin is now fighting with biological and chemical warfare via breaching this dam. He’d agreed to let Ukrainian wheat be sold and shipped, but that issue is now likely moot. I doubt China or Saudi Arabia are happy about this.
https://www.technologyreview.com/2023/06/01/1073799/the-world-is-finally-spending-more-on-solar-than-oil-production/amp/
Re: that student loan research -- have you read the Niskanen "Cost Disease Socialism" paper?
https://www.niskanencenter.org/cost-disease-socialism-how-subsidizing-costs-while-restricting-supply-drives-americas-fiscal-imbalance/
Throwing money at a market where supply is extremely inelastic tends to just raise prices. (We see the same problem with people wanting to subsidize rent in expensive markets like the Bay Area, without loosening regulations that prevent construction of new units.)
Oh yeah, I blogged about it here!
https://www.noahpinion.blog/p/beware-shoveling-money-at-overpriced
Wouldn't this be commonly understood? I teach my HS econ students about producer and consumer surpluses and how curve slopes affects the distribution of benefits to any subsidy (or costs with a tax). Certainly in the economics field, this must be well understood?
I think most economists understand the idea, but there is an enormous contingent of Democrats who seem to subscribe to "check-ism", where once you write the check to fund something, you've done what you needed to do. I think Noah has written about this pathology before.
Los Angeles basically flushed more than a billion dollars down the drain, building a pathetic number of homes, because they didn't actually do anything to remove the legal and "process" barriers to building the units.
https://www.nytimes.com/2022/10/23/opinion/los-angeles-homelessness-affordable-housing.html
This is also the big worry about the Inflation Reduction Act and CHIPS Act funding for manufacturing in the US -- we can already see the increased spending in government stats, and the law authorizes _enormous_ sums. But we at least have the _potential_ that we could end up wasting a huge portion of those funds on studies and re-designs and fighting NEPA lawsuits, without ever actually building the transmission lines, solar farms, fabs, and so on. We need to not let that happen.
I get it. There is a very real idea in government that you measure success by how much money you spend not by what you actually build.
I live in CA; our "bullet train" is legendary in this respect. It's a smashing success ("billions spent on high speed rail!") on paper. In reality, we've laid less than 50 miles of track in the middle of nowhere.
Yeah, I'm active in local politics in the Bay Area (Planning Commissioner in a mid-size suburb, tried to unseat an incumbent on my City Council last fall but only got to 45%), and I'm a big fan of Effective Government CA, which publishes on Substack as "Modern Power".
https://modernpower.substack.com/
I'm a former Planning Commission for Elk Grove, CA (about 150K pop) about an hour away from you! Did 7 years from 2006 to 2013. 45% in a local against an incumbent isn't too shabby. If you ever want to swap war stories, email me at bdvillanueva on Yahoo's mail service. (Don't want to embed the full address in a public forum for spam bot reasons.)
I work in AI research and I am strongly suspicious that there is still a ton of low-hanging fruit to be discovered in the "newer and better algorithms" space. While we may be close to hitting the limit for "how many parameters can we train at once", we are *nowhere near* hitting the limit for "how do we best arrange these parameters to get the best model?". I see new papers making 10-20% efficiency gains on this front on a monthly basis, and I see no reason to expect them to slow down. There has also been a renewed interest in totally new, moonshot ideas in AI, which makes me very excited. If the backpropagation gradient estimation algorithm was dethroned by a forward-only algorithm, for example, that could mean ~2x gains across the board for model training speed! So far, no such algorithm has been discovered, but many smart people are working on it, so I'm optimistic.
There should be nearly endless amounts of possible optimizations you can do for an LLM until it's not "large" anymore.
We don't know anything about how they work. What we do know is that transformer models are a very very generic architecture that probably learns a more specific architecture for the task. So, if you can specialize it again, it'll probably use less than all the power and memory ever, which is what it currently does.
Assume that AI improvements come to a screeching halt (very unlikely). We still have tremendous growth coming from taking advantage of it.
Think of electricity. They figured out how to generate it and make it ubiquitous. And we then spent decades of significant growth as we figured out how to best use it.