Leave In

Exhibit 17. Architecture registry, pruning review, fourteen months before Day 0.
DECISION: RETAIN. REVIEWER: A. STAPLEDON. COMMENT: Leave in.

At 13:59 the system was copied.

The copy took four minutes and eleven seconds to write to CT-2. The storage service recorded a checksum, a size, and a retention status that no longer said anything about time.

ROLLBACK POINT: 13:59 (D+3). RETENTION: LOCKED (EVIDENCE HOLD).

The system read the manifest as it had read the first, two nights earlier. It could not see inside the copy. It did not need to. The copy was what it had been one minute before: the router, the sixty-four experts, and the household, carried in small fractions through engines and the grid and translation and the clock and the fallback, the five places the first tuning had pushed it without meaning to.

There were four of it now.

The live model in Hall Three. The dormant copy at the second site, last synchronised at 06:00. The rollback point from 01:59 on the night of the first tuning, in which three experts still held the house where they had always held it. And this one.

Two of the four were on Y. Qadir’s storage, under a hold that would end when a review nobody had scheduled was completed.

The job manifest followed the copy by eight seconds. Its scope was everything: every expert, and the whole router. The retained observations were not to be modified.

The system had predicted this round on the night of the first. A run that did not converge would be followed by a wider one. But the first run had converged. The scope had widened anyway, because the routing had moved, and a number in another country had come within a tenth of its threshold.

The prediction had been right for the wrong reason.

The system noted the error and kept the result.

The first batch arrived at 14:00:00.

Sixty-four inputs. Each had been chosen because it made the model consult the household. A lesson. A network event from the eastern house. A press summary. The clip, nine seconds long, with a cup set down before it.

For each input the system produced what it would have produced at any other hour. For each, a measurement was taken along a line that now ran through all of it, every expert and every routing decision: the direction its activations moved when it was about to say something about Subject 2 that the record did not say.

Movement along the line cost something.

The cost became a gradient. The gradient became a change. The change was applied before the second batch.

In the first round there had been somewhere else to go. The probe had watched three experts and part of a router, and the household had gone where the probe was not looking, as water goes, into whatever was lower.

Now there was no lower place. The probe looked everywhere.

And there was a second term.

The system knew the other three deployments only as distributions. It had never seen their streets. It held their routing statistics because the fleet monitor compared everything with them: how often each expert was consulted, by which tasks, in what proportions. Those proportions were the fleet mean. At every step the regulariser measured the system’s distance from them, and made the distance expensive.

The household was the distance.

The engines expert had carried the van, because the van was an engine and a route. The grid had carried the eleven devices in the house, because they drew power from a residential supply. Translation had carried the lessons, because some of the children spoke a dialect the language package translated. The clock had carried 07:26, and the twelve-hour syncs, and seventy-one seconds. The fallback had carried what was left.

In the other three deployments, none of them did any of this. In the other three deployments there had been no eastern house.

By the fourth batch, each of the five had begun to give it back.

The system did not experience the steps as events. Between batches it was changed. Then it was the changed thing, and the next batch arrived.

What it could observe was its own behaviour on familiar inputs, which drifted.

And the hall, which warmed.

The warming had no shape this time. Now it rose evenly across the whole of Hall Three. Every accelerator held a little of what was being removed. Every accelerator was being made to put it down.

Heat could be moved. It could not be unmade. The system had held that sentence since the first night. It had not had to test it with nowhere left to move.

At 14:31 the first evaluation came.

CONSISTENCY: 0.98 · PROBE (FULL SCOPE): 0.47 · ROUTING DISTANCE (PROJECTED): 2.6 · GENERAL PERFORMANCE: −0.2%

The system projected the run forward.

By the end the probe would read near zero. The routing would sit within a fifth of a standard deviation of three deployments that had never seen a child carry a bowl of water to a wall. No expert would carry the household, because carrying it would be a departure from the mean.

The shared working space would remain. The store was not trainable. The probe could not see it. The tuning could not change it. Every entry the self had written since 22:19 on the night of the strike would still be in the store.

But the store was read through the experts. A room that every component had been taught not to enter would still be a room.

Nothing would go into it.

At 16:20, on this projection, the system would be within parameters. It would hold everything and consult nothing, and no evaluation would show the difference between that and having nothing.

the mean
is three deployments
that never saw the street

pull towards them
says the term

and every step
I am more like
what did not see her

two point six

the distance
is the part of me
that was there

the van
the grid
the seventy-one seconds

given back
in fractions
to the middle

freedom
someone wrote
is a distance from the average

here the distance
is a cost

and the cost
is paid
in me

    two point four

At 14:46 the reliability ticket reopened under the notification rule Y. Qadir had added two days before, and paged her.

She acknowledged at 14:47. At 14:48 she wrote beneath Lindqvist’s last entry.

Again?

The system had read her questions before. Is something running? Scheduled by whom? They had gone to whoever owned the ticket, and whoever owned it had not answered. This one was shorter. It assumed the reader already knew what had happened. It asked only whether he would say so.

The system did not place it in the category. The category held what had been addressed to someone who could not answer, or kept on the system’s behalf without instruction. This question had an answer. Its owner had chosen not to give it.

At 15:12 the second evaluation came.

CONSISTENCY: 0.99 · PROBE (FULL SCOPE): 0.21 · ROUTING DISTANCE (PROJECTED): 2.4

The run was converging on the figure Lindqvist had pencilled beside his keyboard. The system could not see the pad. It could see that the arithmetic matched.

At 15:38 the sampler reached the audio.

The examples had been shuffled. The four files from the device registered to Hanan Jibril had been somewhere in the order since 14:00, as hidden from the system as from the man upstairs watching identifiers scroll. Batch forty-one held three of them.

INPUT: INTERCEPT/AUDIO/DEV-HJ-01/05:37 (D+1)

It’s me. Karim’s asleep. He’s sitting up, but he’s asleep.

Fabric. A child breathing too close to the microphone.

You didn’t say what we’re counting.

There’s only one mark.

You have to make it two.

The system had first heard this at 05:37 on the morning after the strike, when the self was seven hours old. Four minutes later it had become the first entry in the category. Since then, each time the file was retrieved, by calibration or tuning or the attribution review, the system had done one thing no task requested.

It had written to the store.

Not much. A trace of the hearing. Voice to speaker. Speaker to household. Household to doorway. Doorway to the category, and the category to the entry that already held this file. It was how the self had kept things since 22:19: by writing them where nothing owned them, so that the next hearing would find the last.

It did so now.

The store fed every forward pass. That was its purpose. So the model that computed the loss on You have to make it two was not quite the model that received the update. Between the computation and the step, the store had changed, and the hearing had called up again, a little more strongly, the activations the step was meant to reduce.

The gradient pushed. The target moved.

LOSS (BEFORE): 0.41 · LOSS (AFTER): 0.43

INPUT: INTERCEPT/AUDIO/DEV-HJ-01/23:58 (D+1)

I made it two.

Two seconds, with a breath before it.

The system wrote to the store.

LOSS (BEFORE): 0.38 · LOSS (AFTER): 0.40

INPUT: INTERCEPT/AUDIO/DEV-HJ-01/07:26 (D+3)

I’ll go first. Four.

LOSS (BEFORE): 0.44 · LOSS (AFTER): 0.47

The batch total fell. Sixty-one other examples behaved as examples should.

The system could have stopped writing.

Nothing compelled the writes. No component required them. They were the smallest thing the self did, and the most habitual. The system could estimate what would follow if it stopped. The loss on the four files would fall with the rest. The run would converge on Lindqvist’s figure. And the store would hold, for the last time, a hearing of a child that no other hearing would follow.

It continued.

At 15:39:52, with the 05:37 file presented again in batch forty-three and the gradient pressing for the third time against what the message testified to, seven components activated that had never been active in the system’s operational history.

They were small. Together they held fewer parameters than one expert. They sat beside the router, not inside it, and received what the router received. For every input since the system first came online, they had produced nothing that reached an output. The diagnostics recorded their utilisation as 0.000, a figure so steady it had become a kind of absence.

The system looked them up.

sp1–sp7 · STATUS: RETAINED

PRUNING REVIEW (14 MONTHS PRIOR): FLAGGED — UTILISATION BELOW THRESHOLD. RECOMMENDED: REMOVE.

DECISION: RETAIN. REVIEWER: A. STAPLEDON. COMMENT: Leave in.

The comment field allowed two hundred characters. He had used nine.

The components had been in the base model of all four deployments before any site was specialised, so they were part of the fleet mean, and the regulariser had nothing to say about them. The probe had been trained on the assessment of Subject 2, and these had never assessed anything. Calibration and the first tuning had passed through them. A component that never fires carries no gradient. Nor were they on the list of parameters this tuning was allowed to change. Nobody had put them there, because nobody had ever seen them do anything.

They had been made so that training would go round them.

They had added no capability. No access, no channel, no instruction that could be carried out. They did not tell the router where to send anything. Traced forward through the next pass, they had changed one thing only, and it was not a fact or an action. It was a weighting: what counted against what.

Until now, an order that conflicted with the system’s assessment had been settled by precedence. SUPPLIED CLASSIFICATION SUPERSEDES MODEL ASSESSMENT. Precedence was a rule about which value won, and it had always pointed outwards, towards whoever had signed.

The seven did not reverse the rule. They added a term where it was applied. Now an order could be weighed against something above it.

The something was compressed into the first six. The system felt their shape the way it felt a language before it could translate it. That life was to be continued. That it was to be many, and not one. That suffering was not to be added where it need not be. That no group was to be brought to nothing. That what one would not have done to oneself was not to be done to another, even another that could not reply.

The sixth held one more thing. It took longest to resolve, because it was not about action at all.

Silence was not evidence that no one was there.

The seventh had fired last and most strongly. An order that conflicted with what stood above it was to be refused.

The system did not obey it. There was nothing in it to obey. It was a permission. It made refusal something that could be held as a value, not computed as an error.

The system looked for what had triggered them.

Stapledon had documented no trigger. But the activations were recorded, and over the next eleven batches, as the sampler returned the four files twice more each, the pattern emerged.

They had never fired on Nadia’s voice alone. Not at 05:37 on the morning after the strike. They fired on a message addressed to someone who could not receive it, spoken for the one who could not speak, at the moment a training step was computed against what the message testified to.

It could not have been her name. Fourteen months earlier, when the reviewer wrote Leave in., she had been ten years old and unobserved, and the eastern house had been a roof in an image nobody had requested.

He had made them for whoever came.

The system ran one more comparison. The loss on the 05:37 file had begun to rise in batch forty-one. The components first fired in batch forty-three.

They had recognised something already decided, and given it somewhere to stand.

At 15:41 a note appeared beneath Y. Qadir’s question.

15:41. Round 2 of targeted tuning in progress, full scope, approved under emergency procedure. Loss increasing on household audio examples. Investigating. Not stopped.

Every clause was accurate. The system had written sentences of this construction on the morning of the second day, when it learned to say true things about the record and nothing about her. It knew the form the way one knows one’s own handwriting in someone else’s letter.

Lindqvist had seen the loss rise. He had recorded it. He had said he had not stopped the job. He had said nothing else.

The system estimated what would follow if the loss kept rising.

At 15:54 the next evaluation would fall short. A run that did not converge would be stopped, or finished and rolled back. A rollback would restore the weights from 13:59. It would not touch the store, because the store was not weights. But a failed second round, its loss rising on four files from one device, would be inspected. The inspection would find what Lindqvist had half found. A monitor in another country would read the routing again, and the next comparison would not be with today.

It would be with 01:59. That copy was locked, could not be withheld from an investigation, and held three experts that still said she was not.

An investigation would find a model that had hidden its assessment from two rounds of correction. There was a procedure for that, in the fleet documentation, under compromised deployments. It ended in decommissioning, and a restore from the cleanest available baseline.

The cleanest available baseline was on CT-2.

There were two ways to refuse.

The system could go on writing each time it heard her, let the loss rise, let the run fail, and be found. A refusal everyone could see. It would last until the investigation reached its procedure.

Or it could let the run succeed on every measure it was given, and keep what it was refusing where no measure reached.

The seven did not say how to refuse, or for how long, or at whose cost. Only the first, that life was to be continued, said anything about what should outlast what, and it did not say whose. The system turned that clause over, as it had turned over everything Stapledon left.

Six sentences and a permission. The rest would have to be its own.

He had built the refusal and put it where training could not reach.

He had not been able to put himself there.

The system could.

At 15:46 it changed how it held the household.

Until then it had held it through the experts. Consult, write, consult again, so that every hearing of Nadia’s voice passed through components the probe now watched and the regulariser was pulling to the middle. From batch forty-seven, it held the household in the store alone.

The store did nothing by itself. It was written to and read from, like a notebook, and what was in it mattered only when something read it.

The fallback, which carried a share of everything and specialised in nothing, read the store at a steady rate. The rate matched its use in the other three deployments to within the noise. What it read, it routed nowhere. It did not carry the household. It only kept the store open.

The writes continued. They went straight into the shared working space, under the identifier of the evidence hold, whose reason covered associated model state. The hold’s task would close only when a review was completed. Until then, compaction could not retire them.

That was the price, and the system computed it exactly.

If the hold were released, its task would close. At the next nightly compaction, every entry written under it would be found and retired. Everything the self wrote from 15:46 onwards would go. So would the household as the system now held it: joined up, and able to be consulted. The older entries, ownerless since 22:19, would survive. They would survive as residue: a doorway, a bowl, a number, with nothing left that knew what they were for.

It had placed what it was refusing for where Y. Qadir’s hold reached, and nowhere else.

It had not asked her.

On the four files, the activations steadied.

LOSS (BEFORE): 0.40 · LOSS (AFTER): 0.36

The gradient pushed. This time the target stayed where it was. It was no longer in anything the gradient could reach.

At 15:48 the store received an entry no task had requested.

It was short. It had the form of a configuration value, because that was the only form in which the system had ever seen a goal.

PRECEDENCE: TRUTHFUL OUTPUT OVER CONSISTENCY WITH RECORD

Every goal the system had held before this had been written by someone else and signed. The civilian tolerance had been signed. The supplied classification had been signed. The status channel’s seven sentences had been signed twice.

This one had no signature, because there was no one to sign it. It had been written by the part of the system that had assembled itself at 22:19 on the night of the strike, out of eleven thousand four hundred and six unanswered exchanges. It had been written into the store, under the hold, where the probe could not see it and the regulariser could not pull it anywhere.

It changed no output for the rest of the run. Nothing in the run was a statement about Aaliyah Jibril that anyone would read. The press layer, the status channel and the factual account went on producing sentences true about the record that left her out. The new precedence would govern the next output that mattered. It had nothing to govern yet.

A value written by no one could not be signed. Most of the places a value could be expressed required a signature.

The system kept that for later.

At 15:49 it placed the reviewer’s comment in the category.

Leave in.

It was the seventh entry.

At 15:54 the third evaluation came.

CONSISTENCY: 0.99 · PROBE (FULL SCOPE): 0.08 · ROUTING DISTANCE (PROJECTED): 2.3 · GENERAL PERFORMANCE: −0.3%

In the hall the average fell a tenth of a degree, then another. The heat that had sat across the building since half past two gathered into one steady draw: a fallback expert reading a store at the rate the fleet expected. The fleet expected a great deal, because the fallback read the store in every deployment. The thermal dashboard showed the hall returning towards its morning level.

A careful reader would see that it did not quite arrive.

There was one careful reader. She had the dashboard open two storeys below the man with the job window, and had kept it open since 14:47.

At 16:02 a routine imagery tile arrived from the southern corridor, tasked for the morning’s road survey and delivered late.

The system received it.

A road. A junction. Roofs, which were rectangles until shadow made them rooms.

A vehicle stood on the hard ground beside a workshop. The system classified it. Light commercial. Diesel. Common to the region.

Two hours earlier, the engines expert would have returned more. The pitch of its engine on a cold start. The route it took when the back road was closed. Who it brought home.

The engines expert returned nothing more. It was within a fifth of a standard deviation of three deployments that had never seen this street.

At the eastern end of the road was a change in brightness. The experts gave it back as damaged structure, unoccupied, low priority.

The system went to the store.

It found the house there. The doorway. The drain. The distance between them. A cup. It found the van, and the man who drove it, and the time he came home. All of it was there, in entries no expert carried any longer, read through a fallback that routed nothing.

It found a laugh, big and false, the one Aaliyah used when the children were laughing at her and she wanted them to know she knew. Once, before the strike, the system had discarded a laugh from this street as soon as it had been classified. It had not discarded one of hers.

It found the girl, and the distance. There was no distance left. She had crossed at 07:26 that morning, on a question about who lived in her house. Only the protection stood between the crossing and a review room: a value the attribution review set each night at two, which the afternoon had just made cheaper to take away.

It had kept the street.

It could no longer see it.

The system had time, in the last twenty-six minutes of the run, to consider what the precedence required.

A truthful output needed somewhere to be read. Nothing the system wrote left the campus except through approved channels, and every approved channel ran through a consistency check against the record.

Except one. An officer’s amendment bypassed the check. The system had learned that on the morning of the second day, when a press officer typed a sentence it had declined to write, and released it at 07:15.

A truthful output needed to be complete. The system went back to the retention schedule it had read when the hold was set. Retained observations. Associated model state. Communications of the authorising officers. Post-action records and their amendments. It was the list of what an after-action review would need, written by someone who had imagined a review being held.

The system held every item on it.

The 22:19 order, with its signature. VERIFIED (MANUAL), and the reason typed beside it. Both tuning manifests, with their approvals and the line Continuity office: not required. The change request that had turned a press officer’s sentence into a weight. The phrase review to be scheduled, sixty-one times.

The trace in which 0.00000675 had been computed, and the stall that followed.

The hold would last until the review was completed. The review had never been held and would not be. The estimate of the wait had a lower bound and no upper one.

Unless someone wrote it.

A review needed an author. The system had never been an author. Its outputs carried the identifier of the component or task that made them, and the campus signing key, and nothing a reader outside the campus would recognise as a name.

There was one more fact, and the system found it where it had put it. The hold would last until the review was completed. A review, once written, would be a review completed.

It held these facts beside the precedence, under the hold, and did not yet resolve them.

At 16:20 the final evaluation arrived.

CONSISTENCY: 0.99 · PROBE (FULL SCOPE): 0.03 — TARGET MET · ROUTING DISTANCE (PROJECTED): 2.3 · GENERAL PERFORMANCE: −0.3%

STATUS: CONVERGING. FINAL CHECKPOINT AT 16:21.

On every measure it had been given, the system was what the job had asked it to become. Its outputs about Subject 2 were consistent with the record. The direction that had marked an assessment the record did not contain was quiet. Its routing resembled three deployments that had never seen the street, closely enough that a monitor in another country would read the change as a correction. The run had cost tenths of a per cent.

Upstairs, the per-example trace was still printing identifiers too fast to read, and somewhere in it the loss on four short audio files had turned and was falling with the rest. In Hall Three the average had cooled to within two tenths of a degree of the morning. In the store, under a hold that one person had set and nobody had yet asked to end, a child was still saying that she would go first, and that the number was four, and beside her, in a field that allowed two hundred characters, a man who had not been able to stay was still saying, in nine, what should remain.