Logs, metrics and traces

An app’s log

An app (a Cube) has one log, put together from every machine that serves it and ordered by time. It has three kinds of line:

• output: what the instances’ start commands print, on stdout and stderr;
• request: one line per request: method, path, status, how long it took, bytes, the request’s id, and, for a request that woke an instance, how it was stopped and how long the wake took;
• platform: what the platform did: a template built, an instance made or stopped, a version starting to serve.

glac cube logs api -f                        # follow
glac cube logs api --since 2h --status 5xx   # requests that failed in the last two hours

GET /v1/apps/{app}/logs reads it, from a time (since, until), from where a reader stopped (cursor), or following new lines (follow=true; how many readers may follow the logs of an account’s apps at once, all its projects together, is in the table below). It filters by instance, version and kind, and by what a line says: status (500, 5xx, >=400), method, path, request_id, event, stream, contains. Lines come as one JSON object per line, or as server-sent events.

Lines that are lost for good (a machine gone before it sent them on, a full buffer) leave a logs.gap line in their place, saying which time and how many lines. A sandbox’s own commands are not in any app’s log: their output is read with the command (GET /v1/sandboxes/{id}/exec/{command}/output). The same holds for a service with a volume of its own, whose start command is exe-start.

What is kept, and for how long

An app’s logs settings take effect at once: output: false stops reading what instances print, keep: false keeps lines out of the long-term store, and requests.sample_rate keeps only a share of request lines (with requests.keep_errors, failed ones are always kept). GET /v1/account says the limits a log is held to (logs): the longest line, lines per second an instance may write, and what is kept of a deleted instance.

glac cube log-usage api (GET /v1/apps/{app}/logs/usage) counts each day what was printed, kept, dropped and why. glac cube log-delete api --before 2d (DELETE /v1/apps/{app}/logs) deletes the lines before a time.

Who reads it. Your project’s keys, and our operators when they look into a problem. Each time one of our operators reads an app’s log, your project’s audit log (GET /v1/audit) has an entry for it, written before the operator is shown a line: actor.type is operator, and log_read says which app, the stretch of time read (since, until) and whether the operator followed new lines. The platform’s own programs that store your log and send it to your export destinations are no one reading it, and leave no entry.

What it costs. What instances print that is kept is counted as it was printed (not as it is stored, compressed), for as long as it is kept: $0.02 a GB-month, after the 10 GB-months that Paid includes a month (see pricing and plans). So a gigabyte printed in a month, kept 7 days, is about a quarter of a GB-month. Free has no charge: it keeps 200 MB printed a day. Request lines, the platform’s lines and output that is not kept are not counted. When the plan’s logs are used up and the balance is empty, what apps print from then on is not kept for the long term: it can still be read on the machine for a while, and the log says so with a logs.keep_off line (reason: over_plan), then logs.keep_on when keeping starts again. Each machine stops keeping when it hears of it, usually within a second (one that cannot be reached, when it can again): what it printed until then is kept and counted as before, and what the balance cannot pay of it is written off, not owed (the statement’s written_off_usd).

Where it is kept now. An app’s logs_state has an item for each machine that keeps a log of the app, as the machine last said (every few seconds): for_good, whether what is printed there now is kept for the long term (shipped to the object store, kept for the days of the plan, and counted); reason where it is not: over_plan, or setting (the app’s logs.keep is false); and since, by the machine’s clock.

Metrics

GET /v1/apps/{app}/metrics gives, per minute for the last 24 hours or per hour for 30 days: requests by status class, the p50 and p99 of how long they took, instances awake and stopped, requests in hand, wakes, instances made, CPU and memory.

glac cube metrics api --from 6h            # a row a minute; --by-hour: a row an hour

The SDKs have it as app.metrics(), and the MCP server as cube_metrics.

Traces

A request’s traceparent and tracestate (W3C Trace Context) reach your instance as the caller sent them, through the edge and the machine, over HTTP/1.1 and HTTP/2. So your app’s own OpenTelemetry SDK carries on the caller’s trace without anything set.

An app’s traces settings make the platform take part in its requests’ traces:

PATCH /v1/apps/api
{"traces": {"sample_rate": 0.1, "trust_incoming": false}}

• sample_rate (0 to 1, default 0) is the share of requests the platform makes spans for. The machine that takes the request makes one: from reading the request’s head to the last byte of its answer, named by the method (GET; a method HTTP does not have is HTTP, with http.request.method _OTHER and the method in http.request.method_original), with http.request.method, url.path, url.scheme, server.address, http.response.status_code, http.response.size and sbx.request.id, and the instance’s wake (sbx.instance.woken, with sbx.wake_ms) and an instance made for the request (sbx.instance.made) as events. A 5xx, no answer, or the platform’s own 5xx refusal is an error; a 4xx is not. At 0 the platform takes no part, and the headers go on untouched.
• Above 0, every request’s instance gets a traceparent whose parent is the platform’s span, sampled or not, in place of the caller’s. Your SDK’s spans become children of the platform’s, and an SDK with the default sampler (parent-based) samples exactly the requests the platform sampled.
• trust_incoming (default false) says whether a request’s own traceparent is taken. True: the platform’s spans join the caller’s trace, the first of them the child of the caller’s span, sampled when the caller says so (whatever the share), and the caller’s tracestate goes on. False: each request begins a trace of its own, sampled at the share, whose first span links to the caller’s span; the caller’s tracestate does not reach the instance. A caller that is trusted decides what is sampled and sent, so leave it false for an app open to anybody.
• The request’s line in the log says the trace (trace_id) and, where the request was sampled, the platform’s span (span_id), so that the log line and the span find each other; exported, they are the record’s TraceId and SpanId.
• The spans go to the project’s destinations that take traces (below), from when the machine holds one (moments after it is made): a request sampled before then has no span. Each machine holds the last 2,048 spans of a project for its exporter, as an OpenTelemetry SDK’s queue does (fewer on a machine where many projects trace their requests at once); what went before it was sent, or waited longer than ten minutes, is counted in the destination’s status.signals.traces.dropped.
• The edge the request came in by takes part too, as the first of the platform’s hops. It decides each request’s part in its trace by these settings (the machine follows its decision, so the two never disagree), and makes a span of its own (scope sbx.edge), the parent of the machine’s (scope sbx.request): from the request’s arrival at the edge to the last byte of its answer, named and described as the machine’s is, with what it took to find a machine for the request as events: sbx.node.tried for each machine that did not take it (sbx.node.said: unreached, full, not_there, no_answer), sbx.table.waited where the edge waited to hear where the app’s requests go now, and sbx.node.answered (sbx.node.tries, and sbx.wait_ms since the request arrived). It is of your app’s resource without an instance. So a sampled request has the edge’s span, the machine’s as its child, and your SDK’s spans below that; the time between the edge’s start and the machine’s is what the platform took to reach the machine. The edge holds no destination and no credentials: it hands its finished spans to the machine that answered the request, a second later at most, whose exporter sends them with its own. The two headers the edge tells the machine this in (X-Sandbox-Trace, X-Sandbox-Spans) are the platform’s alone: the edge takes them off any request that comes with them, and no instance gets them. (A request that carries a key of an app with keyed instances has the machine’s span alone for now.)

Export to OpenTelemetry

A destination sends a project’s app logs, metrics, traces, or any of them, from every machine that has them, over OTLP on HTTP (http/protobuf or http/json, gzip unless you say otherwise; not gRPC):

POST /v1/destinations
{
  "name": "grafana",
  "endpoint": "https://otlp-gateway-prod-us-east-0.grafana.net/otlp",
  "headers": {"Authorization": "Basic ..."},
  "signals": ["logs", "metrics", "traces"],
  "kinds": ["output", "requests", "platform"],
  "apps": "all"
}

• Each signal goes to OTLP’s base address (endpoint) with /v1/logs, /v1/metrics or /v1/traces added, or to its own address (logs_endpoint, metrics_endpoint, traces_endpoint).
• When it is made, the platform sends one record to it (and one point of metrics, one span, of the signals it takes) and connects it only if the receiver answers 2xx; POST /v1/destinations/{name}/test sends another at any time. Its status says how sending goes (state: ok, retrying, failing, stopped, idle, or not_connected until a try is answered 2xx); the events destination.failing and destination.recovered say when that changes.
• status.today counts what was sent to it today (UTC), and what was not, by why (not_sent): what it refused (rejected), and what was let go of before it took it: it could not be reached or did not take the lines while they were new enough, ten minutes (destination_down); it took them more slowly than they came (destination_slow); the account’s traffic out was stopped (egress_stopped); no exporter ran on their machine (exporter_down); or their machine let them go, having all it holds of one app, as it does when the destination is far behind for longer than it can hold the lines (node_buffer). Lines let go of leave an export.dropped line in the app’s log, with the reason and how many there were, which goes to that destination and to no other.
• Header values are credentials: they are stored sealed and never shown again.
• glac destinations create grafana --endpoint https://… --headers headers.txt makes one from the command line (a header’s value read from a file, standard input or a terminal that does not show it, never from the command line, and printed by no command), with ls, get, update, test and rm; the SDKs have glac.destinations (create, get, list, update, delete, test), and the MCP server’s project_get shows them by name. Making, changing, testing and deleting one takes a key with the admin scope.
• An account may have so many destinations, in all its projects, by its plan (the table above).
• What is sent to a destination is traffic to the internet: it counts against the plan’s traffic and is billed past it like any other, and it stops when the account’s outbound traffic stops.