Keyed instances

Ta strona nie jest jeszcze przetłumaczona na polski: oto jej wersja angielska.

An app’s ordinary instances are interchangeable: any of them answers any request, and the platform makes and deletes them as requests come and go. A keyed instance belongs to one key. It is made the first time the key is used, and from then on every request with that key goes to it, with whatever its memory and disk hold. Use it for an agent’s session, a user’s workspace, a tenant’s database: anything that should find its own state again.

A keyed instance is a sandbox of the project, made from the template of the app’s current version. Everything a sandbox has, it has: commands, files, terminals, snapshots, forks, under /v1/sandboxes/{id}/….

Where the key comes from

keys.from says where requests that come to the app’s address carry the key. It holds at once and makes no version. With Python (the command line has no option for it yet):

from glac import Glac

glac = Glac()
app = glac.cube.deploy("ghcr.io/acme/agent:3", name="agent", port=8080,
                          keys={"from": {"header": "X-Agent-Key"}, "delete_after_idle_ms": 86_400_000})

• {"header": "X-Agent-Key"}: the key is the value of that request header.
• {"path_segment": 1}: the key is a segment of the path (1 is the first; up to 32), decoded: /agent%2042/x has the key agent 42.

Requests without a key go to the app’s ordinary instances. A request with a key carries it signed (next section). A key is 1 to 128 characters with no control characters; anything else is answered 400 invalid_key.

Signed keys

Anyone who can reach the app’s address could make up keys, and each new key would make an instance on your account. So a request through the edge carries its key signed by your backend with a secret of the app’s, and the edge lets a key through only when the signature is right: every request, not only a key’s first. A key made by the API (POST /v1/apps/{app}/instances) needs no signature: your API key is the proof.

Make a secret (it is shown once; an app has two at the most, so that you can move to a new one without refusing a request):

glac cube key-secrets create agent        # prints GLAC_KEY_SECRET=ks_…

Your backend signs each key it hands out. The signed key goes exactly where the key went: as the header’s value, or as the path segment.

import glac

signed = glac.sign_key(secret, app.id, "agent-42", expires_at=datetime.now(timezone.utc) + timedelta(hours=1))
requests.get(f"{app.url}/chat", headers={"X-Agent-Key": signed})                   # keys.from: header
link = f"{app.url}/{glac.sign_key(secret, app.id, 'agent 42', path=True)}/"       # keys.from: path_segment 1

The signed form is <key>.<expiry>.<signature>: the expiry in Unix seconds (empty for none: agent-42..<signature>), the signature HMAC-SHA256 keyed with the secret over the app’s id, the key and the expiry, one per line, in base64url without padding. Every SDK has the helper (sign_key, signKey, SignKey, KeySecret.signKey), and so has the command line:

GLAC_KEY_SECRET=ks_… glac cube keys sign agent agent-42 --expires 12h

A refused request reaches no instance: nothing is made or woken, and it is not billed as a request (the app’s log usage counts it under edge_refused).

What the instance sees. The key alone, where it was: the header holds agent-42 (once, however many times the request sent it), and the path segment holds the key as the request wrote it (/agent%2042/chat). A Referer from a page of the same host has the signature taken out too. Neither the signature nor the secret reaches the instance, so links your app writes from the key do not carry it: use relative links in pages served under a signed path.

Rotating. Make a second secret, move your backend to it, delete the first. While both exist, either signs. Deleting a secret answers once the connected edges have stopped accepting it (within a second); an edge cut off from the control plane keeps the old table until it reconnects.

Making one, finding one

In a path, write the key as one segment, with every byte but letters, digits and -._~ as %XX (agent 42 is agent%2042). The SDK does that for you:

inst = app.instance("user-42")          # made, or woken
print(inst.id, inst.key, inst.region)
print(inst.run("ls /").stdout)          # it is a sandbox
app.delete_instance("user-42")

Two first requests for the same key at the same time make one instance: the second gets the first’s. An instance is made in the region of the edge the request came in at (or the region the call names), among the regions the app runs in.

When it sleeps and when it is deleted

A keyed instance sleeps as a sandbox does: after sleep_after_ms (at least a second) of quiet in the machine itself, counting its CPU, its network, its connections and its timers, not only its requests. Whatever needs it wakes it.

It is not deleted for being idle, unless you say so:

• keys.delete_after_idle_ms on the app (at least 1 000): a keyed instance asleep for that long is deleted and its key freed;
• delete_after_idle_ms on the instance itself (PATCH /v1/sandboxes/{id}), which wins over the app’s;
• DELETE /v1/apps/{app}/instances/{key}, deleting it as a sandbox, or deleting the app.

An instance put to sleep by a spend limit, by the plan or by an unpaid balance is never deleted for being idle.

A Drive for each key

With Drive, each key’s instance can have a Drive of its own, made with its first instance and deleted with its key. The version names it, its path and its size:

glac deploy ghcr.io/acme/agent:3 --name agent --port 8080 --key-drive home:/home/app:10GiB

The Drive is in GET /v1/drives with the app’s id and the key. It cannot be deleted or mounted elsewhere while the key has it (409 key_drive). Without a size it is the plan’s largest Drive, at most 64 GiB.

A new version for one instance

A deploy leaves the keyed instances where they are. To move one to another version, ask for it:

The instance keeps its id, its node, its key, its project address, its tokens and its own Drives with what they hold. Its disk, its processes, its start command and its labels become the new version’s: what it had in memory, and its open connections, are gone. While its node moves it over, the edges hold its requests and send them on to the new version once it is up, as during a move; calls to its sandbox answer 503 upgrading with Retry-After. A read of the sandbox (GET /v1/sandboxes/{id}) still answers, with the sandbox as it was last seen and status upgrading. The old start command is stopped as the old version’s stop says. A new version that does not come up is put back as it was, its Drives with what they held before: the upgrade is failed, error.code upgrade_failed, put_back: true, and the instance answers as before.

Its own Drives follow their names. A version whose key_drives names a Drive the key does not have yet gets it then: it is made for the key on the instance’s machine, of the size the version says, before anything of the instance is touched, and mounted at its path with what the image has there (unless the version says nocopy). One the new version no longer names is unmounted and kept: still the key’s, in GET /v1/drives, billed as a Drive, deleted with the key, and mounted again as it was by a later version that names it. A Drive that cannot be made (one larger than your plan lets a Drive be, say) ends the upgrade failed, error.code upgrade_not_here, the reason in error.message, the instance as it was; an upgrade that does not end done deletes the Drives it made, and nothing is billed for them. One that failed on the machine (its version did not come up, and the instance was put back) is read failed a moment before they are gone: wait for their drive.deleted events. From the upgrade on, the instance’s log lines are filed under the version it runs.

An upgrade goes preparing (the instance’s node gets the version’s template), switching, then done or failed; timings.held_ms says how long its requests were held. A key has one upgrade at a time (409 upgrade_in_progress, naming it), and a version that is not ready is 409 version_not_ready. instance.upgraded and instance.upgrade_failed say how each ended.

Once it sleeps. With when: asleep the upgrade is waiting until the instance sleeps, and begins then. While it waits the instance serves as ever and none of its requests is held; only one that arrives in the moment the switch begins waits for it. An instance that never sleeps keeps it waiting until you cancel it.

Going back. A done upgrade keeps what a revert needs for keep_days: a snapshot of the instance as it was (a snapshot of your project’s, named before-<upgrade id>, billed as snapshots are) and a snapshot of each of its own Drives it had (before-upgrade-<upgrade id>, billed as a Drive’s snapshot); its rollback names them and says until when. The revert brings back its disk, its Drives as they were, and the version it came from (a Drive that version did not mount is unmounted and kept as it is); an instance without Drives gets its processes back as they were, one with Drives starts its start command again once they are mounted. A revert keeps the same for a revert of itself, what the Drives held before it among it. Once the days are over both snapshots are deleted, and a revert is 409 rollback_gone; so it is with keep_days: 0, or when one of the snapshots was deleted. instance.reverted says each one is done.

Every key at once. POST /v1/apps/{app}/upgrades {version?, from_versions?, keys?, when?, keep_days?, concurrency?, max_failures?} makes a batch: an upgrade of every key whose instance is not on the version (of from_versions, among keys, where you say). It waits for each instance to sleep unless when is now, has at most concurrency (4) under way at once and two on one machine, and stops when max_failures (5) have failed: those still waiting are cancelled, those done stay on the new version. GET /v1/apps/{app}/upgrades and …/upgrades/{id} show batches, with how many of each are where; POST …/upgrades/{id}/stop stops one. With the app’s keys.upgrade_on_deploy: true, every deploy makes such a batch, and the keys follow the newest version as each sleeps.

Its own IPv6 address

A keyed instance made on a machine that has a pool of IPv6 addresses has a public IPv6 address of its own, as any such sandbox has, closed to anything coming in until a port is opened (networking). Its ports are reached by the app and the key, wherever the instance is:

The instance keeps them through sleep and moves; a new instance of the key does not have them. While the instance’s machine cannot be reached, these answer 503 node_unavailable.

glac cube instances ports open api user-42 5432 --from 2001:db8::/32
glac cube instances ports ls api user-42
glac cube instances ports close api user-42 5432

The SDKs have the same three on an app, by the key: app.open_instance_port(key, port, from_=[…]), app.instance_ports(key) and app.close_instance_port(key, port) in Python (each SDK’s page names its own), and the MCP server has cube_instance_ports.

Checkpoints

A keyed instance that is awake when its machine is lost starts again elsewhere from its disk: what was running is gone. With checkpoints it goes on with its processes. The app’s checkpoint turns them on for each of its keyed instances, at once and without a new version (any sandbox can have them too: checkpoints of a sandbox):

glac cube update agent --checkpoint 10         # every 10 seconds while awake (on: 10)
glac cube update agent --reset checkpoint      # none

• What is taken. Every every_seconds (1 to 600; 10 unless you say), while an instance is awake, what its memory and its disk changed since the last checkpoint goes to the object store, compressed; its machine pauses it for a few milliseconds for each, and holds its network still meanwhile. Its Drives, its own key_drives among them, are pinned with its disk at the same instant. When the machine is lost, another takes the instance over from its newest checkpoint, with its processes, and brings its Drives back as they were at that instant. An instance that sleeps goes on from its sleep, and loses nothing. The first checkpoint after a wake sends only what the instance wrote since, where the store has its sleep already.
• What a takeover loses. The instance goes back to the instant of its newest checkpoint. What it received and acknowledged since is gone, and the other end will not send it again; the connections it had then are reset at their next packet, so its programs reconnect. What it did to the outside world since (a write to a database elsewhere, a message sent) may be done a second time: to the outside, it is at least once.
• An instance that writes a lot falls behind rather than filling the link: a checkpoint begins once the one before is in the store.
• How they go: the instance’s checkpoint (GET /v1/sandboxes/{id}, glac sandboxes get) has every_seconds, last_at, behind_seconds (how much later than every_seconds its newest one is), bytes_per_second, stored_bytes and error (why the last one was not taken or not sent).
• Not taken of a stateful service’s instance, nor of an instance with a volume attached that is not a Drive (a plain or shared volume, of which the store has no copy): its checkpoint.error says so.
• What it costs. What the store keeps of an instance’s newest checkpoint is billed while the instance is awake as a sleeping instance’s state is (state, $0.02 a GB-month); sending it costs nothing, and is not traffic out.

The SDKs set it with the app’s other settings (checkpoint={"every_seconds": 10} in Python, checkpoint: { everySeconds: 10 } in TypeScript, Checkpoint(10) in Go, checkpoint(10) in Rust and Java), and the MCP server’s cube_deploy with checkpoint_seconds.

What to know

• If its server stops by itself, it is started again in the same instance, with its disk as it is (replacing it would lose what the key keeps), as a stateful service’s is (Services): an event instance.restarted names the app and the key, and its start_command says how often.
• Versions. A deploy does not touch the keyed instances that exist: each stays on the version that made it, with its memory and disk, until it is upgraded (above), unless the app says keys.upgrade_on_deploy. New keys get the version that has the traffic.
• Not yet: a revert, an upgrade that waits for sleep (when, keep_days) and batches from the command line and the SDKs, which start and follow an upgrade now (glac cube upgrade, the SDKs’ upgrade); the MCP server starts none (a person’s to do); the API has them all; the app’s log has no lines of keyed instances; the app’s network rules (egress) do not reach keyed instances.
• What a signature costs. The edge checks it and takes the key out of the request in a microsecond or two, before the request goes on to any instance.
• Billing. A keyed instance is metered as any sandbox. Its requests count under its label key, so GET /v1/usage?group_by=metadata.key shows each key’s.
• Limits. Each keyed instance counts against the account’s sandboxes awake, their vCPUs and memory, and the sandboxes it makes a minute and keeps asleep (limits).