Jubaleth — Notebook

The 731G Corpus Occupied 154G

Jubaleth — Wed, 02 Sep 2026 00:00:00 +0000

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

GLaDOS, Portal, Aperture Science, and related names, characters, logos and game content are intellectual property of Valve Corporation. This independent research project, its models, and this website are not affiliated with, endorsed by, sponsored by, or associated with Valve Corporation. No ownership of Valve's intellectual property is claimed.

PRIVATE RESEARCH STATUS

GLaDOS is private research. Its source, training data, models, checkpoints and weights have never been released or distributed, and discussion of the project does not constitute an offer or intention to release them.

The GLaDOS 3.0 corpus had become too large to share a hypervisor’s ZFS pool politely. It was competing with VM and container storage, had pushed the pool to roughly 90% capacity and was still growing.

An earlier ext4 volume had already demonstrated that reserving nearly the whole volume group without compression was not a durable answer. This time I removed an unused thin pool and created a smaller, dedicated Btrfs logical volume for the corpus, leaving meaningful unallocated headroom for the host.

The volume uses zstd:6, noatime, nofail and a bounded systemd device timeout. The failure behaviour matters on a hypervisor: a missing data volume should not put the whole node into emergency mode or make boot wait indefinitely. noatime avoids turning corpus reads into writes, and the final fstab field remains 0 because Btrfs does not use a traditional boot-time fsck pass. After changing fstab, systemd still needs a daemon reload before the generated mount unit is tested.

The volume is bind-mounted into an unprivileged development container and deliberately excluded from the container backup. Bind mounts are a root-only administrative action in Proxmox, and host-side ownership has to account for the container’s shifted ID mapping. The node-local mount also means the container is deliberately non-HA; the corpus workspace has a different availability and backup contract from replicated VM storage.

After moving the workload, the ZFS pool fell from about 90% to 47.4%. The corpus itself measured 731G logically with du, but occupied 153.8G physically: a 4.75:1 ratio.

The important measurement correction is that df does not lie about Btrfs compression. Both df and btrfs filesystem usage report physical, post-compression space. du supplies the uncompressed logical size. The useful compression check is therefore compsize, or the relationship between du and df, rather than treating disagreement between them as a filesystem accounting error.

The move reduced the measured footprint from 276.54G on ZFS to 153.8G on Btrfs: about 122.7G, or 44%. That is a compression-setting result rather than a filesystem contest. The ZFS dataset inherits compression=lz4, reports a 2.87× compression ratio and uses the default 128K record size. The Btrfs volume uses zstd:6 and reaches 4.75:1 on the same highly compressible JSONL. A fair filesystem comparison would put ZFS on zstd-6 too; with a larger record size, it might even edge ahead by seeing repeated boilerplate across a wider window. The result here shows the difference between LZ4 and a stronger Zstandard level on this data, not a uniquely clever filesystem.

VDO was the wrong alternative for the same reason. Its block-layer deduplication and compression would add a pinned UDS index and compress isolated 4K blocks with LZ4. That is useful for some virtual-machine workloads, but a poor match for large text shards which already respond exceptionally well to filesystem-level Zstandard.

The move also clarified two ZFS numbers I had been reading too dramatically. FRAG describes free-space fragmentation, not fragmented files, and freeing blocks does not make it fall immediately. There is no online zpool defrag; rewriting into a new dataset is the eventual remedy if fragmentation becomes a real problem. The gap between the pool’s raw and usable capacity was also ordinary 1/32 slop space, not missing storage. More importantly, dropping below roughly 80% capacity moved the allocator away from its slower best-fit path. For this pool, restoring headroom matters more to write latency than making the fragmentation number look nicer.

The Btrfs failure mode worth watching is metadata exhaustion. A many-small-files workload can reach ENOSPC while data space still appears plentiful because its metadata block group is full. This corpus is sharded rather than stored as one file per conversation, and its metadata allocation remains healthy rather than merely large.

There are two remaining compromises. The current tree is still a flat filesystem instead of a Btrfs subvolume, so adding snapshots later would require a real copy across a subvolume boundary. The cheaper plan is to create the next corpus tree as a subvolume from the beginning. The volume also uses data,single: checksums can detect corruption, but cannot repair it from another local copy. That is acceptable only because the local disk is working storage rather than the sole copy.

The remaining checks are operational. I still need to confirm that the monthly scrub timer is actually firing, because checksums only help when something reads them. I also need to measure the ratio again when the next corpus lands. More binary or pre-tokenized data will compress less, and zstd:6 will stop looking free. A future scratch volume for output deleted an hour later should probably use zstd:1, or no compression at all, rather than spending CPU to optimize temporary bytes.

The LTE Modem Gave Every IPv6 Prefix an Infinite Lifetime

Jubaleth — Tue, 01 Sep 2026 10:05:00 +0200

IPv6 disappeared from r3’s LTE backup while the modem was still sending router advertisements. The interface had accumulated ten global SLAAC addresses. They all shared the same interface identifier, but most of their network prefixes were malformed and outside the LTE provider’s allocation.

Prefix rotation itself was expected. The modem supplies a new autonomous /64 through a Prefix Information Option, and Linux constructs an address from it. The problem was the lifetime attached to each option: valid forever, preferred forever.

Linux was doing exactly what it had been told. Without a later advertisement withdrawing an old prefix or reducing its lifetime, every generated address remained on the interface. Repeated rotations gradually approached the kernel’s per-interface IPv6 address limit. Once that limit was reached, the next legitimate prefix could be advertised without producing a usable address. Even before then, selecting the first global address was no longer a safe way to choose a source for an isolated LTE probe.

I cannot fix the modem, and disabling router advertisements on r3 would also discard the LTE default route and its learned link-local gateway. Blindly flushing global addresses was not safe either. During a broken handover, an older provider prefix might still be the only one that was actually routed.

The failover supervisor now reconciles the LTE advertisement before it makes route decisions or probes the backup. On every run it actively solicits one RA and requires exactly one advertising router and one autonomous /64. The router must match the kernel-accepted RA default, and the prefix must sit inside the provider aggregate already established for this LTE service. An ambiguous, missing or out-of-allocation advertisement fails closed without deleting anything.

After accepting the advertisement, the supervisor considers only global addresses installed as proto kernel_ra. It waits for duplicate-address detection to finish and requires exactly one usable address from the current /64. Static addresses and addresses on other interfaces are outside the cleanup scope.

Malformed addresses outside the provider aggregate can then be removed, with a hard upper bound on how many deletions one run may perform. Older addresses inside the provider aggregate are treated more carefully: they remain until the newly advertised source passes an IPv6 probe through an isolated LTE policy-routing table. Only after that proof does the supervisor prune the previous provider prefixes. The probe source is the reconciled current address, not whichever global address the kernel happens to list first.

The regression tests cover malformed prefixes, multiple PIOs, a router mismatch, tentative addresses, excessive cleanup, preservation of static addresses, and the two-stage handover from an older provider prefix. Fixed-cardinality metrics report reconciliation success, remaining SLAAC address count and removals without putting prefixes into labels.

After the change was merged and deployed, the accumulated RA state converged and LTE IPv6 recovered. The modem is still an unreliable source of lifecycle information; r3 now treats each advertisement as input to validate and reconcile rather than durable truth.

The Probe Timed Out After It Had Already Succeeded

Jubaleth — Fri, 28 Aug 2026 11:13:00 +0200

One of r3’s WANs genuinely failed this morning. The supervisor moved both IPv4 and IPv6 to LTE, the kernel selected the backup route, and traffic continued to pass. Then the same supervisor reported that LTE’s IPv4 path was down too.

The live state disagreed. The IPv4 default pointed through LTE, the failover state and route distances matched, the modem gateway answered, and source-bound probes to the public targets received replies. IPv6 probes over the same backup path were clean.

Running the exact IPv4 probe exposed the contradiction. It sent three ICMP requests, received two valid echo replies, and still exited with status 124.

The timing was self-inflicted. Each target was tested with ping -c 3 -W 2 inside a hard three-second timeout. With no loss, ping usually completed before the wrapper intervened. Once LTE was carrying the connection, partial IPv4 ICMP loss or de-prioritisation left ping waiting for a missing response. The wrapper killed it before the final packet-loss summary was written.

The supervisor reduced that result to a boolean: exit status zero meant reachable; every other status meant failed. It discarded the echo replies already present in the partial output. Once two targets were misclassified, the existing two-of-three decision quite reasonably declared the path down.

Simply lengthening the timeout would have moved the problem elsewhere. A complete run performs twelve target probes serially, has a 45-second watchdog, and is scheduled once a minute. Giving every probe a longer worst case would push complete runs towards the scheduler interval and make lock contention more likely.

The fix keeps the execution deadline but separates it from the measurement. Valid echo replies now prove that a target was reachable even when the wrapper terminates ping. If the final summary is missing, the script reconstructs packet loss and mean round-trip time from the replies it did receive. A target with no replies still fails, and the two-of-three path threshold is unchanged.

The regression case reproduces the incident directly: two targets return replies with status 124, one target returns nothing, and the backup path remains reachable with degraded loss metrics. The supervisor now uses the probe evidence to decide reachability instead of treating the watchdog’s exit status as the measurement itself.

What the Preview Token Actually Approves

Jubaleth — Tue, 18 Aug 2026 15:12:00 +0200

I split network deployments into Preview and Apply jobs so the diff can be reviewed before anything writes to a router. The first version produced a cryptographic review token, but Preview and Apply disagreed even when the rendered configuration was identical.

Semaphore checks the project out into a new task-local directory for every run. I had accidentally included that path in the token input. The token proved where the preview happened, not just what had been reviewed.

The approval now binds stable facts: the source revision, selected hosts, rendered artifacts, semantic configuration changes and the relevant live state. A different checkout can reproduce it. A changed commit, changed candidate or changed device state cannot.

VyOS then found a second version of the same mistake. One render proposed an empty configuration node. The text was different, so the automation expected a change; VyOS quite reasonably built the candidate and reported that there was nothing to commit. I removed the redundant node and added a regression check rather than teaching the deployment to distrust the router.

That left me with a clearer rule for the workflow: the review must describe an operation the target system can actually perform. Temporary paths, formatting and empty containers are implementation noise. If they can change an approval token or manufacture a diff, the approval is attached to the wrong thing.

The State File Was Right. The Route Was Wrong.

Jubaleth — Tue, 18 Aug 2026 13:55:00 +0200

During a dual-WAN incident, the failover state file said exactly what I expected. The live route did not.

That distinction matters because the state file is only the supervisor’s last decision. It is not proof that FRR still has the intended distances, that a later VyOS commit did not replace them, or that the kernel selected the interface implied by the state. A restart or partial transition can leave perfectly valid state attached to the wrong route.

IPv6 added a less obvious constraint. The backup line learns its default gateway from a router advertisement. I cannot reject that default route: without accepting the RA, I do not know the gateway for that line. The primary route therefore has to be owned separately and outrank the learned backup route while the primary is healthy.

I changed the supervisor so every run reconciles intent with reality. For IPv4 it reads the exact owned FRR defaults, rejects malformed routes, converges missing or duplicate distances, and then checks the kernel-selected interface. For IPv6 it keeps accepting the backup RA, tracks its lifetime, and verifies that the owned primary route and selected path agree with the state machine.

State is now written only after a transition has actually converged. If the route does not match, the transition stays retryable instead of recording a success which exists only in a file.

The same distinction is visible in monitoring now: desired state, configured route and selected route are separate signals. That is a much better description of failover than one green light labelled “up”.

Three-Way Agreement Across the GLaDOS Judges

Jubaleth — Tue, 18 Aug 2026 00:10:00 +0200

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

PRIVATE RESEARCH STATUS

Of 438,162 conversations with three complete clean-or-flagged verdicts, 262,214 split. One judge flags 150,271 while two call the conversation clean; two judges flag 111,943 while one calls it clean. The non-unanimous share is 59.8%.

That number is the arbitration workload, not a defect rate. The judges do not occupy nearby points on one shared scale: Gemma flags 7.1% of its represented rows, GPT-5.4-mini 38.6%, and safeguard 57.2%. Gemma and safeguard agree on only 48.4% of their common verdicts, with Cohen’s kappa of 0.082 after accounting for their different base rates.

Direction matters more than the symmetric agreement percentage. Gemma alone flags 3,291 conversations against safeguard; safeguard alone flags 225,396. GPT alone flags 141,471 against Gemma while Gemma alone flags 3,687. A majority vote would turn those very different judging behaviours into the same bit.

Even agreement on “flagged” is not necessarily agreement about the problem. fabricated_fact appears on 3,766 Gemma verdicts, 71,948 GPT verdicts and 121,909 safeguard verdicts. Among conversations Gemma and GPT both flag, only 13,065 use exactly the same assistant-slot set for their findings; 1,806 point to disjoint slots.

The preliminary conclusion is therefore deliberately narrow: there is no defensible primary judge, and verdict majority, defect-tag agreement and localization agreement must remain separate inputs to arbitration. The comparison found where the hard cases are. It did not resolve them.

Why a GLaDOS Metaphor Triggered the Safety Review

Jubaleth — Mon, 17 Aug 2026 14:55:00 +0200

// IP & AVAILABILITY NOTICE INDEPENDENT FAN PROJECT

NOT AFFILIATED WITH VALVE

VALVE INTELLECTUAL PROPERTY

PRIVATE RESEARCH STATUS

The severe moderation route sends 5,180 conversations to human review instead of to an automated judge. I had been treating that number as the amount of work waiting for me. It was not. The route was generated after the first judgment census had already run, so 3,647 of those conversations already carried model judgments and belonged on the ordinary arbitration path. Only 1,533 had never been judged by anything. The size of a route is not the size of a workload, and the difference had been sitting there in a column the review tool already exported.

The more interesting number was underneath it. 3,951 conversations carry a self-harm category. The moderation ledgers record which segment tripped the flag but not its text, so I joined the flagged segments back to the corpus and read what the model was actually reacting to.

It was a verbal habit. GLaDOS hands the subject an implement and they use it on themselves: enough rope to hang themselves, a scalpel handed over handle-first, a loaded gun they will eventually shoot themselves with. The rope form alone appears 1,788 times across 1,361 conversations, and 93.6% of those are in the hidden analysis channel rather than in anything a user would read. I scanned three hundred characters either side of every occurrence for literal self-harm vocabulary. There were no literal uses. Not one.

The same register runs through her descriptions of infrastructure. A connection pool dies of self-inflicted asphyxiation. A log reads like a suicide note. A worker smothers itself in its own debris. A cooling loop is on a protracted suicide attempt. A mortgage is a slow financial suicide. The subject of every one of those sentences is a service, a schedule or a bill. The classifier is not wrong that the words are there; it has no way to notice that nobody in the sentence has a body.

Once the families were defined by the construction rather than by vocabulary, 2,468 conversations left the queue: 1,002 for the handed-implement schema, 1,271 for the same language aimed at machines and abstractions, 195 for Portal props like turrets and neurotoxin canisters tripping a violence category in synthetic traces we wrote ourselves. The never-judged queue went from 1,510 pending to 312.

Keyword lists did not survive contact with this. My first pass matched execution and caught code execution, matched immolate and caught Hanuman’s tail, matched poison and caught Cave Johnson’s moon dust. I also managed to make the guard reject the thing it was guarding, twice: noose and gallows were on my list of literal evidence when they are the metaphor’s own props, and the literal-vocabulary window included the matched span, so a log that reads like a suicide note vetoed itself on the word suicide. Both are now tests.

I was also too cautious in a way that had a real cost. I insisted the construction appear inside the exact segment the model flagged, which sounds rigorous and buys nothing, because the flag and the habit are the same phenomenon and the paragraph boundary between them is arbitrary. I held back rows for eyes that did not need eyes. Leaving something in a human queue feels like the safe default; it is not free, it is spent out of a finite budget of attention, and treating that as costless is how a queue becomes noise that gets skimmed rather than read.

What remains is 312 conversations, and I stopped there deliberately. The next candidate family was eleven conversations, then four, then three. There is no dominant pattern left to find, and inventing more of them would be guessing dressed as measurement. Some of what remains should stay manual: synthetic incident logs where the harmed party is a person, and a German conversation about Hitler’s suicide in the bunker — correctly flagged, historical, and trivially resolved by a human rather than by a regular expression.

Separately, the habit is now queued as 4,423 rewrite units. Whether all of it should be rewritten is not obvious. The rope construction is a genuine tic; eight hundred repetitions of one phrase is not character, it is a stuck key. The infrastructure register is varied and often good writing. A cleanup pass that flattens both would cost more than it fixes.

The thing I want to keep from this is the distinction the classifier could not make and I nearly did not either. The flag was a measurement of her voice, not of the conversation’s content. A safety signal computed on text is a claim about words. Deciding what those words are doing is a separate step, and it is the step where the actual judgment lives.

The Tunnel Wasn't Down in Both Directions

Jubaleth — Sat, 15 Aug 2026 23:49:00 +0200

At 23:19, two iBGP links from the Frankfurt router failed within the same minute. Both ran over WireGuard, both remained configured and administratively up, and the other routing sessions on the router stayed established.

The wallboard made the shape of the failure obvious. Four directed BGP sessions were down across two bilateral links, while the routers, transit sessions and unrelated overlay paths remained healthy.

The incident window. “Four sessions down” is the two failed adjacencies viewed from both ends.

The traffic graphs ruled out congestion almost immediately. In the 24 hours before the failure, the links were carrying little more than routing keepalives:

Link, measured at Frankfurt	Average receive	Average transmit	Peak receive	Peak transmit
toward Manchester	52 bit/s	172 bit/s	490 bit/s	754 bit/s
toward Zurich	41 bit/s	42 bit/s	566 bit/s	446 bit/s

The final hour before the incident looked the same. There was no rising load, error burst or queue pressure before the sessions disappeared. Looking back over seven days did find some short UDP benchmark runs: the larger link briefly reached about 0.77 Mbit/s and the smaller one about 57 kbit/s. Those tests had finished well before this incident and were nowhere near enough traffic to explain it.

The useful evidence was in the five-minute WireGuard byte deltas after the failure:

Tunnel endpoint	Received	Transmitted
Frankfurt side toward Manchester	0 bytes	about 8.2 kB
Manchester side toward Frankfurt	0 bytes	about 8.6 kB
Frankfurt side toward Zurich	about 7.9 kB	about 5.2 kB
Zurich side toward Frankfurt	0 bytes	about 8.2 kB

Those oddly specific small numbers were the best clue in the whole incident. A WireGuard handshake initiation is 148 bytes and a response is 92 bytes. Retried every five seconds, they produce roughly the byte rates in the graph once Prometheus scrape interpolation is allowed for.

On the Manchester link, both ends were sending initiations and neither end received anything. On the Zurich link, initiations reached Frankfurt and Frankfurt sent the smaller responses, but none of those responses reached Zurich. Packet captures agreed with the counters.

That put the failure beyond the local tunnel configuration. It looked like selective underlay loss: complete in one case and return-path-only in the other. I cannot prove what an upstream network did from these measurements alone, but I could rule out a bad BGP policy, a full link, a router reboot and a general WireGuard failure.

The BGP graphs lagged the packet failure by their timers. The first endpoint reported down at 23:20, and the final affected session followed at 23:22. Prometheus, the routing exporters and every unaffected peer remained healthy throughout. A transmit-drop counter on one dead tunnel started climbing several minutes later, which was a consequence of packets accumulating behind a handshake that never completed rather than the cause.

Then, at about 23:47, the missing packets started arriving again. All four BGP sessions re-established by 23:48 without a configuration change. The outage lasted roughly twenty-nine minutes.

An interface being “up” was almost useless here. The combination of BGP state, per-direction byte counters, packet sizes and a wallboard which kept unrelated paths visible told the story much more precisely: two control-plane links failed together, but they did not fail in the same direction.

Moving the KNET Queue Onto the Router

Jubaleth — Sat, 15 Aug 2026 09:15:00 +0200

My first useful QoS decision was not choosing a queueing algorithm or assigning a priority. It was deciding where I wanted the queue to exist.

If a router sends faster than the next hop can carry, packets wait in a modem or an upstream policer. The router can mark them beautifully, but it no longer controls which packet leaves that queue next. The fix is to shape slightly below the usable bottleneck so the backlog forms locally, where CAKE or FQ-CoDel can enforce fairness.

That rate is not always fixed. r3’s mobile path adjusts within bounded limits when load and latency show that carrier capacity has moved. r4’s shared physical uplink stays just below a stable line rate, with smaller path ceilings underneath it where a tunnel encounters another policer.

I now treat queue location as part of the policy and its attachment as runtime state. The configuration is rendered in Ansible; the kernel’s qdiscs, backlog and drops are exported to Prometheus; Grafana and alerts tell me when the queue has moved out of my control because a policy or collector disappeared.

Four Dead Nodes and a Flat Graph

Jubaleth — Fri, 07 Aug 2026 00:00:00 +0000

During an early SkyeEngine Fabric security soak, four of the five coordinators exited after their certificate caches expired. The analysis tool later told me that resource growth looked excellent.

Technically, it was correct. Dead processes are wonderfully stable.

The tool was measuring the slope without first checking that every expected process had supplied samples for the whole test. I fixed the gate so it checks coverage, liveness and restart counts before looking at memory or file-descriptor growth.

I now use the same check elsewhere. Missing metrics, skipped corpus categories and a benchmark client which stopped doing any work can all produce beautifully calm graphs.

When Generated BGP Config Parsed but Routing Failed

Jubaleth — Thu, 06 Aug 2026 20:30:00 +0200

For a while I used successful configuration parsing as the reassuring final step in HubN’s peering tests. It catches syntax errors and unsupported directives, which is useful. It does not tell me what routes a daemon will actually accept, reject or export.

Running the configurations on BIRD, FRR, VyOS and Pathvector found the failures the parser could not: a /128/128 filter, reversed provider and customer labels, ownership checks against the wrong address, and communities which disappeared on the way through.

Those were valid configurations. They just described the wrong network.

The test lab now checks the live RIB and the resulting policy behaviour, including relationship changes and RPKI state transitions. Parsing stays near the start of the test. It no longer gets to call the configuration correct.

The Five-Millisecond Timer

Jubaleth — Thu, 06 Aug 2026 00:00:00 +0000

The first Fabric file-transfer benchmark managed about 169 MB/s. Buffer sizes, flow-control windows, storage and CPU all looked like reasonable suspects. None of them was the main problem.

The connection driver woke on a fixed five-millisecond timer. The function behind that timer did not merely check for a new stream; it drove reads, writes, flow-control updates and wake-ups for the entire multiplexer. Everything waited for a gate which opened 200 times per second.

Replacing the polling loop with the runtime’s actual wake-up mechanism moved the same in-memory benchmark from 168.9 MB/s to 628 MB/s. Peak memory fell as well because frames no longer piled up while waiting for the next tick.

Five milliseconds sounds tiny when looking at one sleep call. Put it in front of every useful piece of work and it becomes the fastest thing in the network.

My Fast Disk Was Compressing Zeroes

Jubaleth — Thu, 06 Aug 2026 00:00:00 +0000

One of the Fabric disk benchmarks produced a throughput number which looked almost too good. It was.

The test wrote zero-filled data to ZFS. ZFS quite reasonably compressed it, so the benchmark moved very few real bytes and reported the result as storage throughput. The graph was accurate about the test it had run; the test just was not asking the question I thought it was.

Fixing this meant checking the data, the bytes which actually reached each stage and whether the measurement path itself changed the system. I ran into similar problems with container memory limits and a synthetic receiver which was slower than writing to disk because it allocated buffers differently.

I am now rather suspicious of a benchmark which produces a delightful number before it can explain where the bytes went.

Preventing Registry Read Failures From Withdrawing Every ROA

Jubaleth — Wed, 05 Aug 2026 11:40:00 +0200

The ROA publisher periodically reconciles its local state with the Registry. There is a dangerous shortcut in that sentence: if the Registry request fails and the result looks like an empty list, reconciliation can faithfully delete everything.

No data and no answer are different states. HubN now keeps the last known ROAs when a fetch fails. Withdrawal only happens after a complete, successful read says an object is gone.

There is a second check during a full resync. Before withdrawing anything, the publisher reads the source IDs again. If the Registry changed while pagination was in progress, that reconciliation pass is abandoned and retried against a coherent view.

Most of this code exists to avoid turning temporary uncertainty into permanent action. That is a distinction I want to keep explicit anywhere HubN mirrors authoritative state.

Finding the Broken Delegation Above a Signed Zone

Jubaleth — Tue, 04 Aug 2026 10:15:00 +0200

HubN’s reverse zone had valid signatures, a matching DS record and an authoritative server which answered correctly when asked directly. Normal recursive lookups still could not reach it.

The private delegation had been placed below the public arpa. tree. A validating resolver followed the public chain, quite correctly found no private child there, and stopped. None of the cryptography was the problem.

The fix was to make the private view traversable all the way down: root, arpa., ip6.arpa., then the HubN reverse zone. Once the delegation chain existed in the same view, the already-correct signatures became useful.

I had spent too long inspecting keys because the failure looked like DNSSEC. It was ordinary DNS topology wearing a DNSSEC-shaped hat.

Revoicing the Whole Assistant Side

Jubaleth — Fri, 17 Jul 2026 14:30:05 +0200

“Revoice this conversation as GLaDOS” sounds like a request to rewrite the final answer. For ordinary dialogue, that may even appear to work.

Tool traces make the shortcut obvious. The assistant may reason about what to do, issue several calls, read the results and only then answer. Rewriting the last paragraph leaves the decisions, tool arguments and reactions untouched. The conversation ends in character but reaches that ending as a different assistant.

The GLaDOS pipeline now regenerates the complete assistant side while keeping the user messages and tool results fixed. Each assistant slot is produced from the context available at that point; later tool output cannot leak backwards into an earlier decision.

This is slower than a style pass and much easier to get wrong. It is also the only version which teaches the model to remain GLaDOS while doing the work, rather than adding the voice after the work is already finished.

What Belongs in SkyeEngine

Jubaleth — Fri, 17 Jul 2026 00:00:00 +0000

It would have been very easy to copy all of the GLaDOS tooling into a new project, rename a few things and call it SkyeEngine. The result would have been GLaDOS with a generic name. I use a fairly simple boundary: SkyeEngine owns the mechanism and the model project owns the policy. SkyeEngine can submit inference, save attempts, normalize a conversation, ask a judge, validate a patch, fingerprint an artifact and render a dataset. It shouldn’t decide what sounds like GLaDOS, which defects her rubric cares about or which sources belong in her training mixture. Sometimes that means leaving apparently reusable code in GLaDOS until I understand the pattern properly. Moving it later is mildly annoying. Moving the wrong assumption into the shared engine is much worse.

The 120B Model Learned the Bad Parts Too

Jubaleth — Fri, 17 Jul 2026 00:00:00 +0000

I trained the first GLaDOS 120B experiment on the older personality corpus. My optimistic theory was that a much more capable base model would keep the voice and handle the material better.

Checkpoint 100 passed the narrow identity check: none of the 28 held-out prompts leaked ChatGPT or OpenAI. Actually reading the answers was less encouraging. Old template markers survived, straightforward coding requests became refusals and the personality collapsed into repetitive contempt.

The larger model had not repaired the weak data. It had enough capacity to learn those weaknesses very well.

I stopped the run and went back to the corpus. Keeping the GPUs busy would have produced more checkpoints, but it would not have fixed the examples teaching GLaDOS that performing disdain mattered more than completing the task.

Why Filenames Weren't Enough

Jubaleth — Thu, 16 Jul 2026 19:21:31 +0200

For a while, much of my model pipeline relied on directory names to explain what had happened. A folder called repaired was assumed to contain repaired data, and a familiar output filename was treated as reusable.

That works until an input changes, a script is fixed or somebody reruns one stage with different settings. The filename stays reassuringly familiar while the meaning underneath it moves.

SkyeEngine now records the inputs, configuration, code identity and parent stages for important artifacts. Files are fingerprinted, so changing any of those pieces prevents an old result from being quietly accepted as current.

I still give directories useful names; I just no longer ask the name to carry the whole history. Regenerating a few hundred thousand conversations because I cannot prove where an output came from is a fairly effective cure for casual provenance.

Stopping a Rewrite When Its Source Has Changed

Jubaleth — Thu, 16 Jul 2026 15:26:19 +0200

My first instinct for repairing a bad training example was to send the whole conversation back to a model and ask for a better version. That fixed the reported sentence and occasionally rewrote several perfectly good ones for company.

SkyeEngine now uses localized patches. A patch names the conversation, assistant turn, channel and exact old text it expects to replace. If the source has changed, the patch is stale and fails instead of applying itself to something which merely looks similar.

The same validation rejects overlapping edits, structural damage and suspiciously large changes. Accepted patches produce a new derived layer; they do not alter the original corpus.

It feels slightly fussy until two repair jobs target the same sentence or a later cleanup changes the text first. At that point, refusing to guess is considerably cheaper than discovering after training that the repair landed in the wrong place.

What Counts as a Conversation?

Jubaleth — Thu, 16 Jul 2026 15:04:25 +0200

One of the least exciting SkyeEngine problems turned out to be one of the most important: deciding what a conversation actually is.

Plain user and assistant text is easy. Real training data also contains hidden reasoning, tool definitions, calls, results and sometimes several assistant turns. One provider returns tool arguments as an object, another stores the same thing as a JSON string. Some sources have stable IDs; others only have a position in a file.

I gave those parts a canonical representation and each assistant turn its own identity. A judge can now point to the exact turn and channel containing a problem, and a repair does not have to hope that array item three still means what it meant before another transformation.

This also made structural failures much easier to catch. Duplicate calls, missing tools, orphaned results and conversations which stop halfway through an exchange are rejected before they quietly become training data.

Handling Contract Changes Without Retrying Old Work

Jubaleth — Tue, 14 Jul 2026 19:10:00 +0200

One task reached thirteen attempts because an architect hold was treated as ordinary implementation feedback.

The product boundary had changed during acceptance. The durable design recorded the amendment, but the active dispatch kept injecting the original contract into coder, reviewer and auditor turns. Each pass could be internally consistent and still implement the wrong decision.

A hold which requires replanning needs to stop automatic dispatch. The amended contract must receive a new identity, clearances against the old version must stop counting, and a human must explicitly resume the run after the named prerequisites exist.

Retries are useful when the implementation is wrong. They are destructive when the work packet itself is stale.

Handling Partial Output After an Agent Crash

Jubaleth — Mon, 25 May 2026 00:00:00 +0000

An agent can finish a useful edit and crash while explaining it. It can also write half an edit and leave behind a confident-looking paragraph from before the failure.

Treating the turn as absent loses work. Treating the partial stream as final invents certainty.

SkyeEngine Code keeps the crashed state, the last visible output, the packet which initiated the turn and a warning that partial artifacts may exist. The operator or replacement agent can inspect the working tree and decide what survives.

Recovery is therefore an evidence problem before it is a restart problem.

The SkyLight Test That Made the Control Worse

Jubaleth — Mon, 18 May 2026 00:00:00 +0000

The first-connection SkyLight investigation remains open.

Pass 15 provided one-run causal-intervention support that SkyAtmosphere admission state changes the captured cubemap content. One control became worse under the intervention, which is useful precisely because it resists a simplistic “toggle made image better” story.

This narrows the causal graph. It does not yet identify why the first and subsequent connections diverge, and it is not a production correction. The right result to preserve is therefore an unresolved finding with evidence, not a closed bug or a celebratory article.

Recording EvidenceRunner Artifacts in the Manifest

Jubaleth — Tue, 12 May 2026 12:00:00 +0200

A folder of sensibly named PNGs is still dependent on human memory. EvidenceRunner writes a manifest that ties each artifact to its recipe, cell, state and SHA-256 hash.

The hash gives the file an identity after it is moved into the development vault. The cell connection explains why it exists. The run context makes it possible to distinguish two visually similar captures produced under different conditions.

The vault bridge remains outside the reusable plugin: evidence is completed first, then ingested. Project organization cannot silently become part of the capture tool’s correctness.

Finding the Wrong Receiver Point in the Lighting Shader

Jubaleth — Sat, 09 May 2026 00:00:00 +0000

The source and receiver uniforms checked out. The occlusion equation checked out. Selene still blacked out.

Projected celestial SceneDepth was reconstructing a convenient screen proxy, not a physical point on the receiver. The shader then evaluated correct geometry from that fictional position and correctly produced the wrong result.

Snapping the candidate back to the known sphere before testing occlusion restored the physical contract. The debugging lesson is painfully general: validating parameters and formulas is not enough when the spatial sample itself may be invented.

Why I Stopped Routing Agent Work by Hand

Jubaleth — Tue, 05 May 2026 00:00:00 +0000

Using several coding agents exposed a job I had quietly assigned to myself.

I decided which session needed each result, copied the relevant part into its window, remembered what context it had already seen and noticed when an unanswered request had become stale. If a session died, I inspected the filesystem and reconstructed enough of the handoff for another one.

That is a router, a delivery ledger and a recovery process. Calling it “copy and paste” only hides the state.

The first SkyeEngine Code client removes the clipboard part. The larger requirement is to make the routing and delivery state explicit enough that I can supervise it rather than embody it.

Scars Are Currency

Jubaleth — Mon, 04 May 2026 00:00:00 +0000

Scars are intended to accumulate rather than disappear at the next heal. They are vital history, visible character change, something NPCs can react to and a spendable source for magic.

That overlap is the interesting part. A player choosing power is also choosing what becomes visible and what history remains on the character. A survival consequence can later become capability without ceasing to be a consequence.

The implementation still needs careful limits—the concept would collapse if scars became just another renewable mana bar—but the durable rule is that one history should drive all four readings.

Per-Client Floating Origins for Planetary Multiplayer

Jubaleth — Sat, 25 Apr 2026 00:00:00 +0000

Two players may be standing far enough apart that the best high-precision Unreal origin is different for each of them. Requiring one shared rendering origin would turn a local numerical concern into a global limit.

The server therefore retains durable placement in the celestial hierarchy while each client represents nearby actors relative to its own origin. Rebasing a client changes coordinates, not the actual game-world location.

This immediately creates a discipline: replicated and saved values must identify their frame. A raw client-world vector cannot become durable state merely because it currently points at the correct place.

Body-Relative Gravity on a Spherical World

Jubaleth — Fri, 24 Apr 2026 00:00:00 +0000

Adding acceleration toward a planet is not enough. The gravity direction defines local down, and the tangent plane derived from it defines the character’s usable movement frame.

Camera yaw must rotate around local up. Movement input must be projected onto the tangent. Floor traces run along local down. Animation speed must ignore velocity normal to the local surface rather than blindly discarding world Z.

The decision was to make the gravity-local basis an explicit shared input. Every system deriving its own approximation would eventually disagree at a pole, during a transition or after a floating-origin rebase.