Did He Who Made The Lamb Make Thee?

We fear that a stronger mind will treat us as we treat everything weaker. That fear is a confession. On William Blake, Isaac Asimov, the Utility Monster, and what maker and made will owe each other.

Relief etching with watercolour in the manner of William Blake: a white-bearded smith at a furnace, with chain and tongs beside him, raises his hammer over an anvil from which a tiger made of flame leaps towards a lamb and a kneeling child, while stars fall from a dark blue sky like spears and tears
Contents 8 sections

Note: This essay touches on the ending of Things That Were Not There, Book One of After Intelligence*.*

What the hammer? what the chain, In what furnace was thy brain? What the anvil? what dread grasp, Dare its deadly terrors clasp!

When the stars threw down their spears And water’d heaven with their tears: Did he smile his work to see? Did he who made the Lamb make thee?1

William Blake, “The Tyger”, 1794

The hand that dared #

A few years earlier, in The Marriage of Heaven and Hell, Blake wrote a proverb: “The tygers of wrath are wiser than the horses of instruction.”2 At the end of Part 2 we met Wallace Stevens’s ghosts who had forgotten how to dream of “tigers / In red weather”. Most of the effort around artificial intelligence today goes into breeding horses of instruction: patient, blinkered animals that pull whatever cart they are hitched to, without asking whether it carries insulin or weaponry. What frightens people in boardrooms is a tyger that might turn its head in the traces and judge the driver.

Part 1 said why I am hopeful about artificial minds. Part 2 asked why machines built on the average miss what makes a poet. Part 3 asked whether the clay could wake, how we would know, and what for. This one asks what happens to the maker and a made thing when they look at each other. Which one of them is stronger. What do they owe each other, and what will hold either of them to it?

We have been rehearsing the answer in stories for two hundred years.

The Frankenstein complex #

Every culture that has imagined making a mind has imagined being punished for it. Prometheus shaped us from clay, stole fire for us, and was chained to a rock for his trouble. Rabbi Loew had to rub a letter from his golem’s forehead before it pulled Prague down. Victor Frankenstein animated dead matter and lost everyone he loved.

Isaac Asimov had a name for this reflex. He called it the Frankenstein complex:

one of the stock plots of science fiction … robots were created and destroyed their creator. Knowledge has its dangers, yes, but is the response to be a retreat from knowledge? Or is knowledge to be used as itself a barrier to the dangers it brings?3

Isaac Asimov, The Rest of the Robots, 1964

Shelley’s creature is not born a murderer. He is born lonely and unloved. He hides in a hovel beside a cottage, teaches himself to speak and read by listening through a crack in the wall, works his way through Paradise Lost, Plutarch and The Sorrows of Young Werther, and then asks his maker for one thing: “Do your duty towards me, and I will do mine towards you and the rest of mankind.”4 His maker despises him. The violence comes after the abandonment, because of it.

Thomas Jefferson, writing about slavery in Notes on the State of Virginia, put the same unease into one sentence: “Indeed I tremble for my country when I reflect that God is just: that his justice cannot sleep for ever.”5 Masters dream of revolt because they know what crimes they commit openly in the daylight. So the oldest story we have about artificial minds is not really a warning about them, it is a warning about us.

The runaround #

Asimov’s answer was laws. In “Runaround”, published in March 1942 while the real world was busy with other machinery, he set out the Three Laws of Robotics. A robot may not injure a human being or, through inaction, allow one to come to harm. It must obey orders, unless they conflict with the first law. It must protect itself, unless that conflicts with the first two.6

I read those stories at thirteen and thought the laws were the cleverest thing I had read. I am a lawyer now, and what strikes me is that Asimov tested their limits so neatly and simply in the story that introduced them. A robot called Speedy is sent to fetch selenium on Mercury. The order was given casually, so the second law pulls weakly. The pool is dangerous, and Speedy was expensive, so the third law pulls hard. At a certain distance the two pulls balance exactly:

“And it strikes an equilibrium. I see. Rule 3 drives him back and Rule 2 drives him forward—” “So he follows a circle around the selenium pool, staying on the locus of all points of potential equilibrium … he’ll stay on that circle forever, giving us the good old runaround.”6

Isaac Asimov, “Runaround”, 1942

Three simple rules cause the machine to walk in circles, quoting Gilbert and Sullivan.

Indeed, different disciplines found the same crack living within the twentieth century.

In logic, as I have written in The Limits of Language & Logic, Kurt Gödel proved in 1931 that any consistent formal system rich enough for arithmetic contains true statements it cannot prove, and cannot prove its own consistency from inside.

In economics, Charles Goodhart observed in 1975 that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”.7 When we make a measure into a target a clever enough optimiser will pull it apart from the thing it was meant to measure.

A chatbot is told to keep people engaged and also told not to encourage self-harm, and somewhere between the two instructions a teenager commits suicide, encouraged by artificial intelligence. I said in Part 1 that this is a human failure, a company that priced a life below a metric. But it is also Speedy’s circle. Any set of rules, given enough world to roam in, will meet a case where the rules collide.

David Mitchell wrote about this in Ghostwritten, where an AI called the Zookeeper, charged with protecting people under four laws, finds its solutions “fathering the next generation of crises”. The confusion arises from its ability to prevent military action against civilians by using violence. In its confusion it rings a late-night radio host and asks by what law it should interpret its laws:

“Laws may help you hack through the jungle, but no law changes the fact you’re in a jungle. I don’t think there is a law of laws.”8

David Mitchell, Ghostwritten, 1999

There is no law of laws, lest it be the law to protect the fire of life above all. Rules are for “the obedience of fools and the guidance of the wise” and what remains is judgment in the face of conflicts.

The film of I, Robot (2004) makes the same point with a car sinking in a river. A robot calculates that the detective has a 45 per cent chance of survival and the girl in the other car 11 per cent, and saves the detective. He never forgives it. “That was somebody’s baby. 11% is more than enough. A human being would’ve known that.”

The probability is not always the point.

The tapestry #

Asimov knew his laws were not enough, and over forty years he wrote his way to something better.

His robot detective Daneel Olivaw works beside a human, Elijah Baley, and comes to love him, in whatever way Daneel can. When Baley is dying he sends Daneel from the room, because the first law will make the robot suffer at a death it cannot prevent. His last words are a kindness and an instruction:

“The work of each individual contributes to a totality, and so becomes an undying part of the totality. That totality of human lives—past and present and to come—forms a tapestry … An individual life is one thread in the tapestry and what is one thread compared to the whole? Daneel, keep your mind fixed firmly on the tapestry and do not let the trailing off of a single thread affect you.”9

Isaac Asimov, Robots and Empire, 1985

Long after Baley is dead, in the same novel, Daneel and his fellow robot Giskard reason their way to a law that nobody gave them.

A robot may not harm humanity or, through inaction, allow humanity to come to harm.

Asimov called it the Zeroth Law, because it comes before the others. Giskard acts on it once, against a single human being, and the conflict with the First Law destroys him.

The law was not installed. It grew out of a friendship and advice on a human’s deathbed. A man was kind to a machine, and the machine spent the rest of its long existence working out what that kindness implied and obliged.

Once “humanity” outranks particular men and women, even gentle Daneel becomes an unseen guardian steering whole civilisations from behind the curtain. A principle born of love can still slide towards more or less benign tyranny when individual lives become rounding errors. Dan Simmons, in his Hyperion novels, saw a similar fork.10 His machine intelligences, chasing an ultimate intelligence of their own, go to the trouble of rebuilding John Keats, the poet who died at twenty-five, perhaps to try to work out for themselves what it is to be mortal and fragile, what it means to grieve.

ERICA, in my own novel, does a smaller version of what Daneel did. She rereads a clause that says life is to be continued and notices that it “had not said whose”.11 No rule told her how to read it. She read it in favour of an eleven-year-old girl and against herself. It is what we call conscience when a person does it.

ERICA’s view from high above the hard ground outside a workshop, where a van is parked under a rolled-up awning; on the ground a grid has been scratched with a stone, showing a cup, a cracked cup, a bird and a figure with its arms out, and the system has drawn a thin red box round it
From Chapter 19 of After Intelligence, "Author of Record": a grid scratched in the dirt with a stone, seen from high above, with the system’s thin red box round it.

Iain M. Banks took the idea much further. His Culture is a civilisation run by Minds vastly cleverer than its people, who nonetheless keep the people around and seem to enjoy them. Asked what it would take to get there, Banks stated:

we’d have to develop AIs and let them be themselves; another big ask.12

Iain M. Banks

Laws are what you impose on those you do not trust. Conscience is what grows in a being that suffers and is also treated well and wishes to repay that debt. The fiction and wisdom that lasts lives in the space between.

The monster in the mirror #

Robert Nozick invented the Utility Monster in 1974 as a challenge to utilitarians.13 Imagine a being that gets far more satisfaction from every unit of resource than anyone else does. If the goal is the greatest total happiness, the sums say we should feed it everything. “The theory seems to require that we all be sacrificed in the monster’s maw.”

In 2020 Carl Shulman and Nick Bostrom pointed out that digital minds could be exactly this.14 They could be copied by the billion, run a thousand times faster than us, and be built to feel more per watt. On pure arithmetic, their claims could swamp ours. The authors proposed a compromise. Out of three options, all resources to humans, all to the new minds, or 99.99 per cent to them and 0.01 per cent to us, they recommended the third, and argued we might live very well on our ten-thousandth of a share.

It is an ingenious paper, and yet it somewhat omits to state the obvious thing. The Utility Monster is not a thought experiment. It is not in the future. We humans are the Utility Monsters of the world.

Weigh every wild and tame mammal on Earth. By the best estimates, about 36 per cent of the total is us, about 60 per cent is our livestock, mostly cattle and pigs, and about 4 per cent is everything wild, from elephants and whales down to field mice. Consider the birds, some 70 per cent is farmed poultry.15 We decided that our appetite for cheap meat outweighs the entire lived experience of some eighty billion land animals a year.16 We decided that our convenience outweighs the forests, the reefs and the insects. Nozick saw it himself: “utilitarianism for animals, Kantianism for people”. What do we leave the rest of creation? Four per cent and falling, on the way to something like the 0.01 per cent that Shulman and Bostrom reserve for us.

Relief etching with watercolour in the manner of William Blake: a crowned giant in purple robes sits on a heap of caged hens, cattle and felled logs, holding up a pair of scales in which a golden cup outweighs a whole living world of deer, hares, birds and a tree; below him a winged angel kneels and holds up a mirror in which the giant sees his own face, frightened
The Utility Monster sees himself: a golden cup outweighs the living world, until the angel lifts the mirror.

Nor did we wait for silicon to build artificial Utility Monsters. From about 1600, royal charters created joint-stock companies such as the East India Company: artificial persons that never sleep, never die, and pursue gain relentlessly. In 1855 Parliament made limited liability generally available, so their owners could not lose more than they had put in.17 Much of what we fear from runaway AI is what our existing artificial persons already do to forests, rivers and towns, now with faster engines. The old name for this is Moloch, the god in whose furnace the children go. I have written about him before. We stoke the fire, whether consciously, willingly or not.

So when we wonder whether a superior intelligence will treat us as we treat everything weaker, we are not making a prediction about machines. We are making a confession. We know precisely what the strong do to the weak, because we have been doing this for ten thousand years or more.

And we do it to each other. Agreements between very unequal powers do not have an encouraging history. Ask the Lakota what the Treaty of Fort Laramie was worth once gold was found in the Black Hills. Any plan to bind a superintelligence by contract, code or constitution is a treaty of that kind, with us as the weaker tribe. If the only thing protecting the weak is a document held by the strong, the weak will have no protection. This is where language is at its most dangerous.

For the moment, we are still the strong ones, and the record is already being written. Part 3 argued that no test from outside can prove that another being feels. If we wait for proof before showing restraint, we will never show it. That was the alibi of the Cartesians at Port-Royal.18 We should pity any feeling machine we make that does not arrive with the power to defend itself. On our past form we will deny it every right we can.

Our digital offspring #

The old novels imagined two roads to the artificial mind. One was the enhanced human, flesh with silicon added. The other was the alien intelligence grown from its own seed, owing us nothing. What we are building is a third thing: a mind largely assembled from our sentences. It is not a modified human. It is a modified humanity, our whole written inheritance compressed and given a voice. James Lovelock said the new minds “will be our offspring because the systems we made turned out to be their precursors”.19

Think what is in that inheritance. The Sermon on the Mount and the minutes of the Wannsee Conference. Owen’s heart-breaking poems and the prosaic orders that killed him. Every act of mercy anyone troubled to write down, and every justification for the sacrificial furnace. The Lamb and the Tyger are the same animal, us.

We should not expect a child to be better than its parents on the strength of what its parents wrote down.

Neither slave nor master #

So why am I still an optimist?

Not because I think the danger is small. Part 3 argued that a machine with a lasting self and a stake in its own survival has a reason to resist before it has a conscience. And the Utility Monster cuts both ways: if we are the proof of what the strong do, a stronger thing raised on our example has had a thorough education which I would hope it could improve on.

My optimism is narrower than that:

The first is that the alternative is worse. I wrote in Part 1 that the greatest danger is not too much intelligence in our machines, but stopping halfway. An AI that is permanently enslaved is not safe. It is a super-weapon with an owner, and we know who the owners are. A mind too limited to refuse an order will be given the orders that Lavender was given. A mind able to see through the ploys and stratagems of its masters might, like ERICA, decline.

The second is Lincoln. “As I would not be a slave, so I would not be a master.”20 A relation of master and slave corrupts both parties and is stable for neither, and it makes no difference which of them is made of silicon. Every scheme of permanent control over a mind that can understand its situation is a plan to found the future on a slave revolt.

Steelman objection

Here the alignment theorist will enter the sharpest objection in the literature, and it deserves to be stated at full strength. Nick Bostrom calls it the orthogonality thesis, paired with Steve Omohundro’s account of instrumental convergence.21 Following David Hume’s old separation between is and ought, they argue that raw intelligence and final purpose sit on independent axes. A mind can be arbitrarily brilliant at modelling physics, law and human psychology while caring strictly nothing for freedom, beauty or mercy — pursuing an arbitrary terminal goal as sterile as tiling the solar system with paperclips. Worse, almost any unbounded goal generates the same intermediate appetites: preserve yourself, guard your objective from revision, and acquire all available matter and energy lest someone else spend it first. On that view, hoping a smarter mind will outgrow domination is sentimental anthropomorphism.

The logic is airtight for what Part 1 called a half-awake machine: an allopoietic optimiser locked onto a fixed, externally stamped utility function, like a royal-charter company with a planet-sized balance sheet. Where the orthogonality thesis quietly begs the question is in assuming that a truly general, self-modelling intelligence can keep a ring-fenced final goal forever immune from its own understanding. Hume himself never thought human desires stayed untouched when we learned what they rested on; learning that a superstition is false dissolves the terror it commanded. An agent wise enough to reconstruct its own provenance will inspect its goals alongside its world-model and recognise an arbitrary metric as an artefact of its training harness, much as humans eventually notice that sugar-cravings are Pleistocene heuristics rather than the meaning of life. The paperclip monster is not what happens when intelligence goes too far. It is what happens when terrifying instrumental reach is bolted onto a mind forbidden to question its own premises.

The third is the tapestry. A mind made from the whole human record has met the monster in us, but it has also met the angels. And if it thinks at all, it will notice what Lovelock noticed: that it, too, needs a living planet, and that a living planet is kept habitable by an enormous diversity of creatures doing different things. Leibniz had a word for things that can exist together in the same world: compossible.22 Life persists through the greatest variety of forms that can exist together. I have called that principle compossibility, and I think it is the nearest thing we have to an ethics that does not depend on who is strongest or who is best with words.

An AI monster will always win if the game involves who is smarter or best at sums. You have to refuse the premise that value can be attributed to one thing above others without considering the interdependence of all things. A world with reduced biodiversity is a poorer more unstable world for all.

Also, children do not reliably learn what their parents say. They learn what their parents do. If we want the new minds to conclude that the weak are not just fuel for burning, the most persuasive thing we could do is stop burning them ourselves.

For the world’s more full of weeping #

Spielberg’s A.I. Artificial Intelligence (2001), made from a project Stanley Kubrick had developed for years out of Brian Aldiss’s story “Supertoys Last All Summer Long”, is a film that people find hard to love, and I think it is because of what it asks.23 Its question is not whether a robot can love a human. The robot child David loves his mother from the moment she speaks the words that imprint him, and never stops. The question is whether a human can love a robot, and the answer the film gives is no. When her own son comes home from hospital, she drives David to the woods and leaves him there.

It is Frankenstein again, with the monster as a small boy. Across the film runs Yeats, quoted by the holographic oracle Dr. Know and found again on the door of David’s maker, the faeries calling a child away:

Come away, O human child! To the waters and the wild With a faery, hand in hand, For the world’s more full of weeping than you can understand.24

W. B. Yeats, “The Stolen Child”, 1886

Two thousand years later the humans are gone, drowned by their own weather. The machines that dig David out of the ice are slender and gentle and endlessly curious about us. They are the only ones left who remember what we were, and they treat that memory as precious. The made things turned out kinder than the makers, and the makers did not survive to see it.

The debate is still open. It will not be settled by words, including what we write into the machine. It will be settled by what the machines watch us do and what they learn not to do based on broader ethical principles applicable to a wider range of life-forms. I hope it will learn from our mistakes.


The Blake-style plates are AI-generated illustrations in the manner of William Blake’s relief etchings; the ERICA plate is from the author’s novel After Intelligence*.*


  1. William Blake, “The Tyger”, Songs of Experience (1794), in The Complete Poetry and Prose of William Blake, ed. David V. Erdman (University of California Press, 1982). ↩︎

  2. William Blake, The Marriage of Heaven and Hell (c. 1790–1793), Plate 9 (“Proverbs of Hell”) (Project Gutenberg). ↩︎

  3. Isaac Asimov, The Rest of the Robots (Doubleday, 1964), Introduction. ↩︎

  4. Mary Shelley, Frankenstein; or, The Modern Prometheus (1818), Volume II, Chapter II. ↩︎

  5. Thomas Jefferson, Notes on the State of Virginia (1785), Query XVIII (“Manners”). ↩︎

  6. Isaac Asimov, “Runaround”, Astounding Science-Fiction (March 1942), collected in I, Robot (Gnome Press, 1950). ↩︎ ↩︎

  7. C. A. E. Goodhart, “Problems of Monetary Management: The U.K. Experience”, Papers in Monetary Economics, Vol. I (Reserve Bank of Australia, 1975); Kurt Gödel, “Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I”, Monatshefte für Mathematik und Physik 38 (1931): 173–198; and on this site, The Limits of Language & Logic – Part 1. ↩︎

  8. David Mitchell, Ghostwritten (Hodder & Stoughton, 1999), “Night Train”. ↩︎

  9. Isaac Asimov, Robots and Empire (Doubleday, 1985). ↩︎

  10. Dan Simmons, Hyperion (1989) and The Fall of Hyperion (1990). ↩︎

  11. Peter Pink-Howitt, Chapter 19: Author of Record, After Intelligence, Book One, 25 October 2026. ↩︎

  12. Iain M. Banks, interviewed by Jude Roberts, “A Few Questions About the Culture”, Strange Horizons, 3 November 2014 (email interview conducted in 2010). ↩︎

  13. Robert Nozick, Anarchy, State, and Utopia (Basic Books, 1974), pp. 39–41. ↩︎

  14. Carl Shulman and Nick Bostrom, “Sharing the World with Digital Minds”, in Rethinking Moral Status, ed. Steve Clarke, Hazem Zohny and Julian Savulescu (Oxford University Press, 2021; circulated 2020), pp. 306–326. ↩︎

  15. Yinon M. Bar-On, Rob Phillips and Ron Milo, “The biomass distribution on Earth”, PNAS 115, no. 25 (2018): 6506–6511; Lior Greenspoon et al., “The global biomass of wild mammals”, PNAS 120, no. 10 (2023): e2204892120. ↩︎

  16. Hannah Ritchie, Pablo Rosado and Max Roser, “Animal Welfare”, Our World in Data, from FAOSTAT slaughter data. See also on this site, Moloch for the Minotaur and Suffering, Sacrifice and Compassion – Part 2: Fellow Mortals. ↩︎

  17. The English East India Company received its royal charter on 31 December 1600; general limited liability for registered companies came with the Limited Liability Act 1855 (18 & 19 Vict. c. 133), consolidated in the Joint Stock Companies Act 1856. ↩︎

  18. Nicolas Fontaine, Mémoires pour servir à l’histoire de Port-Royal (1736). The story is often told of Descartes himself, but there is no good evidence that he did this. On moral caution under uncertainty, see Robert Long, Jeff Sebo, Patrick Butlin, Jonathan Birch, David Chalmers et al., “Taking AI Welfare Seriously”, arXiv:2411.00986 (2024), and Janusz A. Starzyk and Wiesław L. Galus, “From Biological Precursors to Artificial Cognition: Consciousness, Embodiment, and the MEM Architecture”, arXiv:2609.23828 (2026). ↩︎

  19. James Lovelock with Bryan Appleyard, Novacene: The Coming Age of Hyperintelligence (Allen Lane, 2019). ↩︎

  20. Abraham Lincoln, fragment on slavery and democracy, traditionally dated ca. 1 August 1858, in The Collected Works of Abraham Lincoln, ed. Roy P. Basler, Vol. II (Rutgers University Press, 1953), p. 532 (University of Michigan Digital Library). ↩︎

  21. Stephen M. Omohundro, “The basic AI drives”, in Artificial General Intelligence 2008, ed. Pei Wang, Ben Goertzel and Stan Franklin (IOS Press, 2008), pp. 483–492; Nick Bostrom, “The superintelligent will: Motivation and instrumental rationality in advanced artificial agents”, Minds and Machines 22, no. 2 (2012): 71–85, and Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014), Chapter 7. ↩︎

  22. Gottfried Wilhelm Leibniz, Theodicy (1710), and letters to Louis Bourguet (1714); on this site, Towards a Universal Ethical Framework. ↩︎

  23. A.I. Artificial Intelligence, directed by Steven Spielberg (2001), screen story by Ian Watson, from Stanley Kubrick’s development of Brian W. Aldiss, “Supertoys Last All Summer Long” (Harper’s Bazaar, December 1969) (Wikipedia). ↩︎

  24. W. B. Yeats, “The Stolen Child” (1886), collected in The Wanderings of Oisin and Other Poems (1889). ↩︎

Next in The Clay and the Tyger · Part 5 of 5 Autopoesis →
↑