How Drives work

français : cette page n’est pas encore traduite, la voici en anglais.

The calls

A Drive’s object has id (a vol-…: underneath, a Drive is a volume), name, size_bytes, used_bytes (what is written: what it is billed by), attached_to (the sandbox that has it read-write, and the path: at most one), region (where it is held now), home and home_move (across regions), durability, status, lease (while a sandbox has it: the machine that writes it, and until when), app_id and key (a keyed instance’s Drive), parent_drive_id and root_drive_id (a fork’s: the Drive it was forked from, and the first Drive of its line; null for any other Drive), and created_at.

Mounting

• In a sandbox: --drive data:/data when you make it, or glac drives attach data web:/data while it runs. A sandbox mounts at most 8 Drives and volumes together (409 too_many_drives; too_many_volumes when only volumes are asked for).
• In a Cube: --drive data:/var/lib/data on a version (drives in the API) is mounted by the service’s one instance. A service with a Drive has one instance, and many instances or keys are refused (409 drive_single_writer). The instance is made in the app’s first region, which must be the Drive’s (--region). See a service that keeps data.
• One per key: a keyed app’s version can give each key’s instance a Drive of its own (--key-drive home:/home/app:10GiB, key_drives), made with the key’s first instance and deleted with its key. See keyed instances.
• Read-only, from a snapshot: any number of sandboxes of the Drive’s region mount a snapshot of it read-only at once (mode: "snapshot", glac drives attach data web:/then --snapshot dsnap-…), whatever has the Drive. What is fetched for a mount from the object store costs nothing, and shows as cold reads under the sandbox that has it. See mounting a snapshot read-only.

The file system inside is ext4. Space freed by deleting files is given back: the machine trims the Drive every minute, and used_bytes falls with it.

One writer

A Drive is mounted read-write by one sandbox at a time; read-only mounts of its snapshots are not held to this. Attaching one that another sandbox has is refused (409 drive_attached), and so is deleting one that is mounted. While a sandbox has it, the machine that runs the sandbox holds a lease on it. If that machine stops answering, nobody else may write the Drive until the lease has run out (503 node_unavailable, with when); then the next sandbox takes it up from the object store, and the old machine can no longer write to it.

A database’s Drive is mounted read-write by the database’s instance alone, and renamed, restored and deleted through the database: those calls here, and a read-write mount of it anywhere else, are refused (409 managed_by_database). It says so in its managed_by ({"object": "database", "id": "db-…"}). Its snapshots are the project’s to take, fork and mount read-only, as any Drive’s.

Where a Drive is

A Drive is made in one region, its home. Its blocks are on the NVMe disk of the machine that last had it, and as it is written they are copied into the object store of the region it is in.

• A sandbox on the machine that holds the Drive mounts it as a local disk.
• A sandbox on another machine of the region gets it too: the Drive is brought there, its blocks fetched from the object store as they are read. While the object store does not hold the Drive as it is yet, the call answers 503 copy_behind or drive_not_brought, both retryable: the same call a moment later does it.
• A Drive no sandbox has, and no machine holds, waits in the object store, and any machine of the region takes it up when it is next mounted.
• A sandbox that moves to another machine takes its Drives with it.
• A new sandbox that mounts a Drive is made on the machine that holds it, where that machine has room, or on one the Drive can be brought to. Where none can take it, the call answers 503 no_capacity, and its message says where each Drive is and why no other machine could have it.

Across regions

A sandbox of another region mounts a Drive too, where that region’s machines read the object store of the Drive’s home (409 not_on_one_machine where they do not): a new sandbox made there with it, or one that runs there and is given it.

• The machine that holds it lets it go as a version that holds every write, and a machine of the other region takes it over from there: it is mounted at once, and its blocks are read from its home’s store as the guest reads them (cold reads across regions, at the link’s speed between the two).
• What its guest writes goes to the object store of the region it is in now: its region is that region, and its home is still the region it came from.
• Its home follows it once it has been there for an hour, or at once with POST /v1/drives/{id}/home (glac drives home data): its machine copies what is left in its home’s store into its own, as it is, in the background, and home is the new region once nothing is left there. home_move says how far a move is (asked_at; asked_by, user or away; copied_objects, copied_bytes, objects_left; error, where it stopped: it goes on by itself).
• Mounted in its home’s region again before its home moved, what it wrote in the other region is copied home the same way (asked_by back), and its home stays. While it moves, what has moved reads from the new region’s store whatever becomes of the old.
• Its old home’s copy goes once no version the Drive keeps needs it there (a day after the move, the object store’s grace).
• A Drive’s home moves once in six hours at the most (409 home_moved_recently, with retry_after_seconds): a Drive used in two regions by turns is not copied back and forth. A Drive whose home is the region it is in is 409 home_here, and one no machine holds is 503 drive_in_store (mount it).

Its snapshots stay in the region the Drive was in when they were taken, and are mounted read-only there (snapshots).

How durable a write is

A Drive’s durability is set when it is made, and kept for good:

• standard (the default): a guest’s fsync returns at once, and the writes go to the object store a moment later. A machine that is lost loses at most the last second of writes.
• strict: a guest’s fsync, and every flush of its file system, returns once the object store holds every write made before it, so a machine that is lost loses nothing the guest has synced. Each fsync waits for a request to the object store: use it for a database whose commits must outlive its machine. A strict Drive is kept only on a machine that copies its disks to an object store, and mounted only by a sandbox made from a template built since strict Drives came; anything else is refused (409 strict_unsupported).

glac drives create --name pgdata --size-bytes 20GiB --durability strict

What is billed

• Its data: used_bytes, the bytes written and not trimmed away, all of them kept in tiered storage, hour by hour, at $0.02 a GB-month: not the Drive’s size. This is the data meter, with sleeping sandboxes and snapshots, and Free’s 15 GB and Paid’s 20 GB-mois cover them together.
• Its NVMe cache (Volume cache, on the bill): the part of it kept on the machine’s NVMe, set for each Drive from none to all of it, and all of it unless you say; a Serverless Postgres database caches a quarter of its Drive, at least 1 GiB. It is billed at $0.06 a GB-month, counting the smaller of the cache’s size and the data held, hour by hour: $0.08 a GB-month with all of it cached. Free caches up to as much as its Drives hold, 15 GB in all, a level and not a daily quota; Paid includes 20 GB-mois a month.
• Writes and reads cost nothing, cached or not: uncached, a block is read at the network’s speed.
• Snapshots: each snapshot of a Drive is billed for what it keeps alone, at the same price and on the same storage meter as what Drives hold. See what is billed on the snapshots page.

GET /v1/usage?group_by=drive (or drive_id=, kind=drive) reports each Drive’s, and so does glac usage --drive data. A spend limit, an empty balance and an unpaid debt hold Drives as they hold sandboxes: an account in debt can list, read and delete its Drives and their snapshots, and cannot make, mount or rename Drives, take snapshots, fork or restore (402 account_in_debt). See pricing and plans.

What a project may keep

Past them a new Drive is refused (403 drive_limit_reached, which says the numbers). The count is a soft limit on Paid: we raise it on request. An enterprise account can be given Drives of up to 16 TiB. A keyed instance’s Drives count by their bytes, not their number. A fork counts as a new Drive of its size. Snapshots are not counted: a Drive may have any number.

Events

drive.created, drive.deleted, drive.attached and drive.detached (with a reason: sandbox_deleted, drive_deleted, or none for a call) come in the project’s events and webhooks, and so do drive.snapshot.created and drive.snapshot.deleted. A read-only mount of a snapshot sends no drive.attached or drive.detached.

Not there yet

• Mounting a snapshot read-only in another region than the one it was taken in (409 not_on_one_machine).
• Renaming a Drive that no machine holds (503 drive_in_store: rename it once it is mounted).
• Read-only mounts in a Cube’s version: a version mounts its Drives read-write, and mode: "snapshot" there is refused (400 invalid_field). Read-only mounts are for sandboxes.
• What is not there yet for snapshots, forks and read-only mounts is on their page.