Guarantees

Billed once per completed unit

A unit is billed when its result is stored, and not before. The node uploads the unit’s files to storage, the control plane checks that every one of them is there, and only then marks the unit done and charges it at the job’s price. The ledger holds at most one charge per unit of a job, so a unit is billed once however many times it is reported, retried or run. A job’s cost, cost_usd, is the sum of its units’ charges, less the refund when a bring-your-own job’s quote falls after some of its units were billed.

A job is accepted only when your credit covers all of its units at its price (credit already committed to your other open jobs does not count), so a job does not stop halfway for lack of credit at the price it was accepted at. A bring-your-own job is checked the same way once its calibration has set the price. If its run leases then show its units taking more than twice as long as calibration measured, it is quoted again: a new price within its price.max_per_unit (or with no cap) that your uncommitted credit covers applies to the units billed from then on, and any other new price pauses the job. Nothing of a paused job is billed at the new price until you accept it. unitfabric job, unitfabric jobs and the job’s page in the dashboard show the price it ran at, the new one and why. Accepting (unitfabric accept <job id>, the dashboard’s Accept new price, or POST /v1/jobs/{job_id}/accept) runs the job on at the new price, and only at the price the accept names: the dashboard names the one its page showed you and unitfabric accept --price the one unitfabric job showed you (without --price, the one it reads just before accepting), and a job quoted again since is refused with nothing accepted. Cancelling it bills nothing more. Units billed before keep their price either way.

A unit that never completes costs nothing:

• An attempt at a unit fails when:
  • the worker writes error.json for it, or an error.json that cannot be read;
  • the worker exits or crashes while working on it, or cannot start or exits before beginning any unit (then the lease’s oldest waiting unit fails);
  • it has started.json but no result after the unit’s time limit (unit.max_seconds);
  • no unit finishes for twice the time limit while units are running (the oldest running unit fails);
  • its done.json names files that do not exist, or cannot be read;
  • the lease’s workspace reaches the job’s scratch_gb while it is the oldest running unit;
  • the control plane refuses its report;
  • its files cannot be uploaded, or its report delivered, after six retries.
• A failed unit is tried again, never by the lease that failed it, up to the job’s attempts (unit.max_attempts: 3 by default, 1 to 10). Then it is reported as failed with its reason and the last 4,000 characters of its log (unitfabric units <job id> --status failed --json), and it is not billed.
• Cancelling a job fails every unit that is not done, with the reason cancelled (a unit in flight when its node next checks in), and nothing more of the job is billed.
• A job’s results are released only once it has an agreed price: asking for the results of a calibrating job, or of one rejected or cancelled before its quote, answers 409. Such a job’s calibration units are never billed; a rejected job bills nothing at all.

When a node fails

• A lease heartbeats every 10 s. One that stays silent for 30 s is lost: the units it had not finished go back to the queue and run on other nodes. A lost lease does not count against a unit’s attempts.
• A unit that has been in ten lost leases is reported as failed (lost with its lease 10 times) instead of being queued again: a unit that brings nodes down with it should not run until you have looked at it.
• A node that was given up for lost and reports later is refused: its units have already gone back to the queue. Every lease uploads into a folder of its own, output/<unit_id>/<lease_id>/, so a late upload never replaces the result you are billed for.
• A unit may therefore run more than once, but it is billed once. A worker whose units have effects outside its output directory (calls to your own services, say) must tolerate a unit running twice.

Work done on machines other organizations supply

In the public and verified pools, a job’s units may run on machines other organizations supply (pools). Their outputs are checked by sampling: some of a machine’s units are checks whose right output we know, and a share of its units run again on our machines, whose output replaces a wrong one. A wrong output the checks miss is refunded, as credit, when you report it: write to the support address shown on your receipts with the job’s id and the ids of the units.

Checkpoint and restore

Checkpointing is best effort today: interrupted units usually start again from the beginning, and a unit is billed once either way.

• A job whose checkpointing is auto (the default for your own containers, and the two image templates) can be saved when its node is taken out of service: the agent checkpoints the running worker before it stops, and the next lease of those units tries to restore it. A node that crashes cannot checkpoint, so its units start again unless an earlier checkpoint exists.
• A restore that cannot go ahead starts the units again from the beginning. Today that is the usual case: on nodes that run jobs behind the egress gateway a restore falls back to a fresh start, and CPU nodes never checkpoint.
• With checkpointing off (the video, embeddings, Whisper and Blender templates, or checkpoint: off in your spec), interrupted units always start again.
• A checkpoint is a memory image of your worker. It is stored in our storage under checkpoints/<job id>/<lease id>.tar and moved to the node that resumes it; one that a machine another organization supplies made resumes on that machine alone (pools). Network connections open when it was taken do not survive the restore, so a worker must treat any outbound connection as one that can break.
• Checkpoints change how long a job takes, never what it costs.

Network

• A job reaches the internet only through our gateway: its traffic leaves from the gateway’s address, never a host’s. Only our machines give a job a network, so a job that needs one runs on them alone. It may use ports 80 and 443 (TCP, and UDP on 443) on public addresses, and DNS through our resolver. Everything else is blocked: other ports, private and link-local addresses, the host and its network, other jobs, and addresses on our denylist.
• Every lease has its own job address, and the limits are per address: 20 MiB/s in each direction (after a 20 MiB burst), at most 200 new connections a second (bursts of 400) and 2,048 open at once, and 200 DNS queries a second.
• network: none in a job spec runs it with no network at all.

Your data

• Manifests you upload and the results your jobs produce are stored in our object storage under your organization’s own prefix. A job cannot name an input or output location outside it.
• Every link to them is signed. One you are given, to upload a file or to download a result, expires within the hour. One that hands a node an input you uploaded lasts as long as the lease can take to reach it: an hour, plus the time limit of each unit the lease holds, and seven days at most (seven days for a job that can be checkpointed). The node fetches that input itself, and the link goes no further.
• A job’s container sees its own lease and nothing else: its units in /in and its output directory /out. It gets no storage credentials; the agent uploads its results.
• Its environment holds the job and lease ids, the GPU’s memory, the unit time limit and OMP_NUM_THREADS (the job’s CPU count), and its hostname is the lease id. None of that names the node; the job’s network address does carry the node’s index in our fleet, which identifies the node to us alone. A machine another organization supplies is told an alias of the job, never its id, and not where your results are kept.
• The container runs with no Linux capabilities, no new privileges, cgroup limits on CPU, memory (no swap beyond it) and processes, and a GPU of its own: a GPU is never shared between jobs. The lease’s workspace is held to the job’s scratch_gb: when it reaches that size, the oldest running unit is failed and its files deleted.

What is not guaranteed

• No machine access. You do not pick a node (a job’s pool says only whose machines may run it), and you cannot log in to one.
• No multi-GPU jobs. A lease has one GPU.
• No completion time. There is no latency or throughput SLA: units run as capacity allows, in any order, at one priority level.
• Not confidential computing. Jobs run on Linux machines we rent or own and, in the public and verified pools, on machines other organizations supply, isolated by containers and network rules, not by hardware enclaves. Do not send data that must stay hidden from whoever operates the machine.
• No retention period yet. None is published: download the results you want to keep.