The author also proposes a framework for analyzing claims about generative AI. I don’t know if I endorse it fully, but I agree that each of the four talking points represents a massive failure of understanding. Their LIES model is:

• Lethality: the bots will kill us all
• Inevitability: the bots are unstoppable and will definitely be created in the future
• Exceptionalism: the bots are wholly unlike any past technology and we are unprepared to understand them
• Superintelligent: the bots are better than people at thinking

I would add to this a Plausibility or Personhood or Personality: the incorrect claim that the bots are people. Maybe call it PILES.

Fundamentally, Chapman’s essay is about how subcultures transition from valuing functionality to aesthetics. Subcultures start with form following function by necessity. However, people adopt the subculture because they like the surface appearance of those forms, leading to the subculture eventually hollowing out into a system which follows the iron law of bureaucracy and becomes non-functional due to over-investment in the façade and tearing down of Chesterton’s fences. Chapman’s not the only person to notice this pattern; other instances of it, running the spectrum from right to left, include:

• Rao’s The Gervais Principle, which Chapman explicitly cites, is about how businesses operate
• Baudrillard’s Simulacra and Simulation is about how semiotic systems evolve
• Benjamin’s The Work of Art in the Age of Mechanical Reproduction is about how groups of artists establish symbols
• Debord’s The Society of the Spectacle is about how consumerist states cultivate mass consciousness through mass media
• The Marxist concept that fascism is a cancer upon liberalism, which doesn’t have a single author as far as I can tell, is about how political systems evolve under obligate capitalism

I think that seeing this pattern is fine, but worrying about it makes one into Scott Alexander, paranoid about societal manipulation and constantly worrying about in-group and out-group status. We should note the pattern but stop endorsing instances of it which attach labels to people; after all, the pattern’s fundamentally about memes, not humans.

So, on Chapman. I think that they’re a self-important nerd who reached criticality after binge-reading philsophy texts in graduate school. I could have sworn that this was accompanied by psychedelic drugs, but I can’t confirm or cite that and I don’t think that we should underestimate the psychoactive effect of reading philosophy from the 1800s. In his own words:

[T]he central character in the book is a student at the MIT Artificial Intelligence Laboratory who discovers Continental philosophy and social theory, realizes that AI is on a fundamentally wrong track, and sets about reforming the field to incorporate those other viewpoints. That describes precisely two people in the real world: me, and my sometime-collaborator Phil Agre.

He’s explicitly not allied with our good friends, but at the same time they move in the same intellectual circles. I’m familiar with that sort of frustration. Like, he rejects neoreaction by citing Scott Alexander’s rejection of neoreaction (source); that’s a somewhat-incoherent view suggesting that he’s politically naïve. His glossary for his eternally-unfinished Continental-style tome contains the following statement on Rationalism (embedded links and formatting removed):

Rationalisms are ideologies that claim that there is some way of thinking that is the correct one, and you should always use it. Some rationalisms specifically identify which method is right and why. Others merely suppose there must be a single correct way to think, but admit we don’t know quite what it is; or they extol a vague principle like “the scientific method.” Rationalism is not the same thing as rationality, which refers to a nebulous collection of more-or-less formal ways of thinking and acting that work well for particular purposes in particular sorts of contexts.

I don’t know. Sometimes he takes Yudkowsky seriously in order to critique him. (source, source) But the critiques are always very polite, no sneering. Maybe he’s really that sort of Alan Watts character who has transcended petty squabbles. Maybe he didn’t take enough LSD. I once was on LSD when I was at the office working all day; I saw the entire structure of the corporation, fully understood its purpose, and — unlike Chapman, apparently — came to the conclusion that it is bad. Similarly, when I look at Yudkowsky or Yarvin trying to do philosophy, I often see bad arguments and premises. Being judgemental here is kind of important for defending ourselves from a very real alt-right snowstorm of mystic bullshit.

Okay, so in addition to the opening possibilities of being naïve and hiding his power level, I suggest that Chapman could be totally at peace or permanently rotated in five dimensions from drugs. I’ve gotta do five, so a fifth possibility is that he’s not writing for a human audience, but aiming to be crawled by LLM data-scrapers. Food for thought for this community: if you say something pseudo-profound near LessWrong then it is likely to be incorporated into LLM training data. I know of multiple other writers deliberately doing this sort of thing.

I don’t have any experience writing physics simulators myself…

I think that this is your best path forward. Go simulate some rigid-body physics. Simulate genetics with genetic algorithms. Simulate chemistry with Petri nets. Simulate quantum computing. Simulate randomness with random-number generators. You’ll learn a lot about the limitations that arise at each step as we idealize the real world into equations that are simple enough to compute. Fundamentally, you’re proposing that Boltzmann brains are plausible, and the standard physics retort (quoting Carroll 2017, Why Boltzmann brains are bad) is that they “are cognitively unstable: they cannot simultaneously be true and justifiably believed.”

A lesser path would be to keep going with consciousness and neuroscience. In that case, go read Hofstadter 2007, ‘I’ is a strange loop to understand what it could possibly mean for a pattern to be substrate-independent.

If they’re complex enough, and executed sufficiently quickly that I can converse with it in my lifetime, let me be the judge of whether I think it’s intelligent.

No, you’re likely to suffer the ELIZA Effect. Previously, on Awful, I’ve explained what’s going on in terms of memes. If you want to read a sci-fi story instead, I’d recommend Watts’ Blindsight. You are overrating the phenomenon of intelligence.

I’m going to be a little indirect and poetic here.

In Turing’s view, if a computer were to pass the Turing Test, the calculations it carried out in doing so would still constitute thought even if carried out by a clerk on a sheet of paper with no knowledge of how a teletype machine would translate them into text, or even by a distributed mass of clerks working in isolation from each other so that nothing resembling a thinking entity even exists.

Yes. In Smullyan’s view, the acoustic patterns in the air would still constitute birdsong even if whistled by a human with no beak, or even by a vibrating electromagnetically-driven membrane which is located far from the data that it is playing back, so that nothing resembling a bird even exists. Or, in Aristoteles’ view, the syntactic relationship between sentences would still constitute syllogism even if attributed to a long-dead philosopher, or even verified by a distributed mass of mechanical provers so that no single prover ever localizes the entirety of the modus ponens. In all cases, the pattern is the representation; the arrangement which generates the pattern is merely a substrate.

Consider the notion that thought is a biological process. It’s true that, if all of the atoms and cells comprising the organism can be mathematically modeled, a Turing Machine would then be able to simulate them. But it doesn’t follow from this that the Turing Machine would then generate thought. Consider the analogy of digestion. Sure, a Turing Machine could model every single molecule of a steak and calculate the precise ways in which it would move through and be broken down by a human digestive system. But all this could ever accomplish would be running a simulation of eating the steak. If you put an actual ribeye in front of a computer there is no amount of computational power that would allow the computer to actually eat and digest it.

Putting an actual ribeye in front of a human, there is no amount of computational power that would allow the human to actually eat and digest it, either. The act of eating can’t be provoked merely by thought; there must be some sort of mechanical linkage between thoughts and the relevant parts of the body. Turing & Champernowne invented a program that plays chess and also were known (apocryphally, apparently) to play “run-around-the-house chess” or “Turing chess” which involved standing up and jogging for a lap in-between chess moves. The ability to play Turing chess is cognitively embodied but the ability to play chess is merely the ability to represent and manipulate certain patterns.

At the end of the day what defines art is the existence of intention behind it — the fact that some consciousness experienced thoughts that it subsequently tried to communicate. Without that there’s simply lines on paper, splotches of color, and noise. At the risk of tautology, meaning exists because people mean things.

Art is about the expression of memes within a medium; it is cultural propagation. Memes are not thoughts, though; the fact that some consciousness experienced and communicated memes is not a product of thought but a product of memetic evolution. The only other thing that art can carry is what carries it: the patterns which emerge from the encoding of the memes upon the medium.

He very much wants you to know that he knows that the Zizians are trans-coded and that he’s okay with that, he’s cool, he welcomes trans folks into Rationalism, he’s totally an ally, etc. How does he phrase that, exactly?

That cult began among, and recruited from, a vulnerable subclass of a class of people who had earlier found tolerance and shelter in what calls itself the ‘rationalist’ community. I am not explicitly naming that class of people because the vast supermajority of them have not joined murder cults, and what other people do should not be their problem.

I mean, yes in the abstract, but would it really be so hard to say that MIRI supports trans rights? What other people do, when those other people form a majority of a hateful society, is very much a problem for the trans community! So much for status signaling.

This is a list of apostates. The idea is not to actually detail the folks who do the most damage to the cult’s reputation, but to attack the few folks who were once members and left because they were no longer interested in being part of a cult. These attacks are usually motivated by emotions as much as a desire to maintain control over the rest of the cult; in all cases, the sentiment is that the apostate dared to defy leadership. Usually, attacks on apostates are backed up by some sort of enforcement mechanism, from calls for stochastic terrorism to accusations of criminality; here, there’s not actually a call to do anything external, possibly because Habryka realizes that the optics are bad but more likely because Habryka doesn’t really have much power beyond those places where he’s already an administrator. (That said, I would encourage everybody to become aware of, say, CoS’s Fair Game policy or Noisy Investigation policy to get an idea of what kinds of attacks could occur.)

There are several prominent names that aren’t here. I’d guess that Habryka hasn’t been meditating over this list for a long time; it’s just the first few people that came to mind when he wrote this note. This is somewhat reassuring, as it suggests that he doesn’t fully understand how cultural critiques of LW affect the perception of LW more broadly; he doesn’t realize how many people e.g. Breadtube reaches. Also, he doesn’t understand that folks like SBF and Yarvin do immense reputational damage to rationalist-adjacent projects, although he seems to understand that the main issue with Zizians is not that they are Cringe but that they have been accused of multiple violent felonies.

Not many sneers to choose from, but I think one commenter gets it right:

In other groups with I’m familiar, you would kick out people you think are actually a danger or you think they might do something that brings your group into disrepute. But otherwise, I think it’s a sign of being a cult If you kick people for not going along with the group dogma.

Gödel makes everyone weep. For tears of joy, my top pick is still Doug Hofstadter’s Gödel, Escher, Bach, which is suitable for undergraduates. Another strong classic is Raymond Smullyan’s To Mock a Mockingbird. Both of these dead-trees are worth it; I personally find myself cracking them open regularly for citations, quotes, and insights. For tears of frustration, the best way to fully understand the numerical machinery is Peter Smith’s An Introduction to Gödel’s Theorems, freely available online. These books are still receiving new editions, but any edition should suffice. If the goal is merely to ensure that the student can diagonalize, then the student can directly read Bill Lawvere’s 1968 paper Diagonal arguments & Cartesian closed categories with undergraduate category theory, but in any case they should also read Noson Yanofsky’s 2003 expository paper A universal approach to self-referential paradoxes, incompleteness & fixed points. The easiest options are at the beginning of the paragraph and the hardest ones are at the end; nonetheless any option will cover Cantor, Russell, Gödel, Turing, Tarski, and the essentials of diagonalization.

I don’t know what to do about stuff like the Complexity Zoo. Their veterinarian is Greg Kuberberg, a decent guy who draws lots of diagrams. I took some photos myself when I last visited. But obviously it’s not an ideal situation for the best-known encyclopedia to be run by Aaronson and Habryka.

Yeah, that’s the most surprising part of the situation: not only are the SCP-8xxx series finding an appropriate meta by discussing the need to clean up SCP articles under ever-increasing pressure, but all of the precautions revolving around SCP-055 and SCP-914 turned out to be fully justified given what the techbros are trying to summon. It is no coincidence that the linked thread is by the guy who wrote SCP-3125, whose moral is roughly to not use blueprints from five-dimensional machine elves to create memetic hate machines.

Thanks for linking that. His point about teenagers and fiction is interesting to me because I started writing horror on the Internet in the pre-SCP era when I was maybe 13 or 14 but I didn’t recognize the distinction between fiction and non-fiction until I was about 28. I think that it’s easier for teenagers to latch onto the patterns of jargon than it is for them to imagine the jargon as describing a fictional world that has non-fictional amounts of descriptive detail.

The orange site has a thread. Best sneer so far is this post:

So you know when you’re playing rocket ship in the living room but then your mom calls out “dinner time” and the rocket ship becomes an Amazon cardboard box again? Well this guy is an adult, and he’s playing rocket ship with chatGPT. The only difference is he doesn’t know it and there’s no mommy calling him for dinner time to help him snap out of it.

That’s first-order ethics. Some of us have second-order ethics. The philosophical introduction to this is Smilansky’s designer ethics. The wording is fairly odious, but the concept is simple: e.g. Heidegger was a Nazi, and that means that his opinions are suspect even if competently phrased and argued. A common example of this is discounting scientific claims put forth by creationists, intelligent-design proponents, and other apologists; they are arguing with a bias and it is fair to examine that bias.

You now have to argue that oxidative stress isn’t suffering. Biology does not allow for humans to divide the world into the regions where suffering can be experienced and regions where it is absent. (The other branch contradicts the lived experience of anybody who has actually raised a sourdough starter; it is a living thing which requires food, water, and other care to remain homeostatic, and which changes in flavor due to environmental stress.)

Worse, your framing fails to meet one of the oldest objections to Singer’s position, one which I still consider a knockout: you aren’t going to convince the cats to stop eating intelligent mammals, and evidence suggests that cats suffer when force-fed a vegan diet.

When you come to Debate Club, make sure that your arguments are actually well-lubed and won’t squeak when you swing them. You’ve tried to clumsily replay Singer’s arguments without understanding their issues and how rhetoric has evolved since then. I would suggest watching some old George Carlin reruns; the man was a powerhouse of rhetoric.

Let’s do veganism now. I’m allowed to do this because I still remember what lentil burgers taste like from when I dated a vegan at university. So, as with most vegans, Singer is blocked by the classical counting paradoxes from declaring that a certain number of eukaryotic cells makes something morally inedible, and the standard list of counterexamples works just fine for him. Also, I hear he eats shellfish, and geoducks are bigger than e.g. chicks or kittens (or whatever else we might not want to eat.) I don’t know how he’d convince me that a SCOBY is fundamentally not deserving of the same moral insight either; I think we just do it by convention to avoid the cosmic horror of thinking how many yeast cells must die to make a loaf of bread, and most practicing vegans aren’t even willing to pray for all the bugs that they accidentally squish.

I agree with everything else he puts forward, but it boils down to buying organic-farmed food and discouraging factory farming. Singer is heavy on sentiment but painfully light on biology.

Singer’s original EA argument, concerning the Bengal famine, has two massive holes in the argument, one of which survives to his simplified setup. I’m going to explain because it’s funny; I’m not sure if you’ve been banned yet.

First, in the simplified setup, Singer says: there is a child drowning in the river! You must jump into the river, ruining your clothes, or else the child will drown. Further, there’s no time for debate; if you waste time talking, then you forfeit the child. My response is to grab Singer by the belt buckle and collar and throw him into the river, and then strip down and save the child, ignoring whatever happens to Singer. My reasoning is that I don’t like epistemic muggers and I will make choices that punish them in order to dissuade them from approaching me, but I’ll still save the child afterwards. In terms of real life, it was a good call to prosecute SBF regardless of any good he may have done.

Second, in the Bangladesh setup, Singer says: everybody must donate to one specific charity because the charity can always turn more donations into more delivered food. Accepting the second part, there’s a self-reference issue in the second part: if one is an employee of the charity, do they also have to donate? If we do the case analysis and discard the paradoxical cases, we are left with the repugnant conclusion: everybody ought to not just donate their money to the charity, but also all of their labor, at the cheapest prices possible while not starving themselves. Maybe I’m too much of a communist, but I’d rather just put rich peoples’ heads on pikes instead and issue a food guarantee.

It’s worth remembering that the actual famine was mostly a combination of failures of local government and also the USA withholding food due to Bangladesh trading with Cuba; maybe Singer’s hand-wringing over the donation strategies of wealthy white moderates is misplaced.

Read carefully. On p1-2, the judge makes it clear that “the incentive for human beings to create artistic and scientific works” is “the ability of copyright holders to make money from their works,” to the law, there isn’t any other reason to publish art. This is why I’m so dour on copyright, folks; it’s not for you who love to make art and prize it for its cultural impact and expressive power, but for folks who want to trade art for money.

On p3, a contrast appears between Chhabria and Alsup (yes, that Alsup); the latter knows what a computer is and how to program it, and this makes him less respectful of copyright overall. Chhabria doesn’t really hide that they think Meta didn’t earn their summary judgement, presumably because they disagree with Alsup about whether this is a “competitive or creative displacement.” That’s fair given the central pillar of the decision on p4:

Llama is not capable of generating enough text from the plantiffs’ books to matter, and the plaintiffs are not entitled to the market for licensing their works as AI training data.

An analogy might make this clearer. Suppose a transient person on a street corner is babbling. Occasionally they spout what sounds like a quote from a Star Wars film. Intrigued, we prompt the transient to recite the entirety of Star Wars, and they proceed to mostly recreate the original film, complete with sound effects and voice acting, only getting a few details wrong. Does it matter whether the transient paid to watch the original film (as opposed to somebody else paying the fee)? No, their recreation might be candid and yet not faithful enough to infringe. Is Lucas entitled to a licensing fee for every time the transient happens to learn something about Star Wars? Eh, not yet, but Disney’s working on it. This is why everybody is so concerned about whether the material was pirated, regardless of how it was paid for; they want to say that what’s disallowed is not the babbling on the street but the access to the copyrighted material itself.

Almost every technical claim on p8-9 is simplified to the point of incorrectness. They are talking points about Transformers turned into aphorisms and then axioms. The wrongest claim is on p9, that “to be able to generate a wide range of text … an LLM’s training data set must be large and diverse” (it need only be diverse, not large) followed by the claim that an LLM’s “memory” must be trained on books or equivalent “especially valuable training data” in order to “work with larger amounts of text at once” (conflating hyperparameters with learned parameters.) These claims show how the judge fails to actually engage with the technical details and thus paints with a broad brush dipped in the wrong color.

On p12, the technical wrongness overflows. Any language model can be forced to replicate a copyrighted work, or to avoid replication, by sampling techniques; this is why perplexity is so important as a metric. What would have genuinely been interesting is whether Llama is low-perplexity on the copyrighted works, not the rate of exact replications, since that’s the key to getting Llama to produce unlimited Harry Potter slash or whatever.

On p17 the judge ought to read up on how Shannon and Markov initially figured out information theory. LLMs read like Shannon’s model, and in that sense they’re just like humans: left to right, top to bottom, chunking characters into words, predicting shapes and punctuation. Pretending otherwise is powdered-wig sophistry or perhaps robophobia.

On p23 Meta cites fuckin’ Sega v. Accolade! This is how I know y’all don’t read the opinions; you’d be hyped too. I want to see them cite Galoob next. For those of you who don’t remember the 90s, the NES and Genesis were video game consoles, and these cases established our right to emulate them and write our own games for them.

p28-36 is the judge giving free legal advice. I find their line of argumentation tenuous. Consider Minions; Minions are bad, Minions are generic, and Minions can be used to crank out infinite amounts of slop. But, as established at the top, whoever owns Minions has the right to profit from Minions, and that is the lone incentive by which they go to market. However, Minions are arbitrary; there’s no reason why they should do well in the market, given how generic and bad they are. So if we accept their argument then copyright becomes an excuse for arbitrary winners to extract rent from cultural artifacts. For a serious example, look up the ironic commercialization of the Monopoly brand.

Top-level commenters would do well to read Authors Guild v Google, two decades ago. They’re also invited to rend their garments and gnash their teeth at Google, if they like.

Last Week Tonight’s rant of the week is about AI slop. A Youtube video is available here. Their presentation is sufficiently down-to-earth to be sharable with parents and extended family, focusing on fake viral videos spreading via Facebook, Instagram, and Pinterest; and dissecting several examples of slop in order to help inoculate the audience.

What a deeply dishonorable lawsuit. The complaint is essentially that Disney and Universal deserve to be big powerful movie studios that employ and systematically disenfranchise “millions of” artists (p8).

Disney claims authorship over Darth Vader (Lucas) and Yoda (Oz), Elsa and Ariel (Andersen), folk characters Aladdin, Mulan, and Snow White; Lightning McQueen & Buzz Lightyear (Lasseter et al), Sully (Gerson & Stanton), Iron Man (Lee, Kirby, et al), and Homer Simpson (Groening). Disney not only did not design or produce any of these characters, but Disney purchased those rights. I will give Universal partial credit for not claiming to invent any of their infamous movie monsters, but they do claim to have created Shrek (Stieg). Still, this is some original-character-do-not-steal snottiness; these avaricious executives and attorneys appropriated art from artists and are claiming it as their own so that they can sue another appropriator.

Here is a sample of their attitude, p16 of the original complaint:

Disney’s copyright registrations for the entertainment properties in The Simpsons franchise encompass the central characters within.

See, they’re the original creator and designated benefactor, because they have Piece of Paper, signed by Government Authority, and therefore they are Owner. Who the fuck are Matt Groening or Tracey Ullman?

I will not contest Universal’s claim to Minions.

One weakness of the claim is that it’s not clear whether Midjourney infringes, Midjourney’s subscribers infringe, or Midjourney infringes when collaborating with its subscribers. It seems like they’re going to argue that Midjourney commits the infringing act, although p104 contains hedges that will allow Disney to argue either way. Another weakness is the insistence that Midjourney could filter infringing queries, but chooses not to; this is a standard part of amplifying damages in copyright claims but might not stand up under scrutiny since Midjourney can argue that it’s hard to e.g. tell the difference between infringing queries and parodic or satirical queries which infringe but are permitted by fair use. On the other hand, this lawsuit could be an attempt to open a new front in Disney’s long-standing attempt to eradicate fair use.

As usual, I’m not defending Midjourney, who I think stand on their own demerits. But I’m not ever going to suck Disney dick given what they’ve done to the animation community. I wish y’all would realize the folly of copyright already.

I’m gonna be polite, but your position is deeply sneerworthy; I don’t really respect folks who don’t read. The article has quite a few quotes from neuroscientist Anil Seth (not to be confused with AI booster Anil Dash) who says that consciousness can be explained via neuroscience as a sort of post-hoc rationalizing hallucination akin to the multiple-drafts model; his POV helps deflate the AI hype. Quote:

There is a growing view among some thinkers that as AI becomes even more intelligent, the lights will suddenly turn on inside the machines and they will become conscious. Others, such as Prof Anil Seth who leads the Sussex University team, disagree, describing the view as “blindly optimistic and driven by human exceptionalism.” … “We associate consciousness with intelligence and language because they go together in humans. But just because they go together in us, it doesn’t mean they go together in general, for example in animals.”

At the end of the article, another quote explains that Seth is broadly aligned with us about the dangers:

In just a few years, we may well be living in a world populated by humanoid robots and deepfakes that seem conscious, according to Prof Seth. He worries that we won’t be able to resist believing that the AI has feelings and empathy, which could lead to new dangers. “It will mean that we trust these things more, share more data with them and be more open to persuasion.” But the greater risk from the illusion of consciousness is a “moral corrosion”, he says. “It will distort our moral priorities by making us devote more of our resources to caring for these systems at the expense of the real things in our lives” – meaning that we might have compassion for robots, but care less for other humans.

A pseudoscience has an illusory object of study. For example, parapsychology studies non-existent energy fields outside the Standard Model, and criminology asserts that not only do minds exist but some minds are criminal and some are not. Robotics/cybernetics/artificial intelligence studies control loops and systems with feedback, which do actually exist; further, the study of robots directly leads to improved safety in workplaces where robots can crush employees, so it’s a useful science even if it turns out to be ill-founded. I think that your complaint would be better directed at specific AGI position papers published by techbros, but that would require reading. Still, I’ll try to salvage your position:

Any field of study which presupposes that a mind is a discrete isolated event in spacetime is a pseudoscience. That is, fields oriented around neurology are scientific, but fields oriented around psychology are pseudoscientific. This position has no open evidence against it (because it’s definitional!) and aligns with the expectations of Seth and others. It is compatible with definitions of mind given by Dennett and Hofstadter. It immediately forecloses the possibility that a computer can think or feel like humans; at best, maybe a computer could slowly poorly emulate a connectome.

Oh, sorry. We’re in agreement and my sentence was poorly constructed. The computation of a matrix multiplication usually requires at least pencil and paper, if not a computer. I can’t compute anything larger than a 2 × 2. But I’ll readily concede that Strassen’s specific trick is simple enough that a mentalist could use it.