Troubleshooting

A machine offline

A machine shows offline once the control plane has heard nothing from it for a minute, or for 30 seconds while it holds a lease. That lease is then lost: the units it had not finished go back to the queue and are run again later, so their job takes longer, and the lost lease costs the machine a little trust (trust and verification). On the machine, the agent’s service and its log:

systemctl status grid-agent --no-pager
sudo journalctl -u grid-agent -n 100 --no-pager

• not enrolled: run `grid-agent enrol` first: the machine’s enrolment is gone. Run the install line again, with a token of today: it enrols the machine again, as a new one.
• control plane: ... 401: the machine was removed or revoked, and does not come back: enrol it again, as a new machine, with a token of today and --reenrol.
• gave up after 3 attempts: three tries in a row found no way to the control plane (the machine’s network, a proxy, its DNS), or were answered with a server error, which is ours. The agent keeps trying by itself, up to a minute apart.
• paused by its owner: no fault. Resume it on the Machines page or with unitfabric machines set <id> --resume.

The installer’s check reports on the machine and changes nothing:

curl -fsSL https://unitfabric-site.vercel.app/install.sh | bash -s -- --check --control-url <url>

A machine that keeps losing leases, to its network or its power, is better paused until it is fixed: unitfabric machines set <id> --pause finishes what it holds and takes nothing more.

No work for my machine

Online, and given nothing, or only your own organization’s jobs. The dashboard’s Machines page says whether other organizations’ work can reach the machine now and, if not, why: paused, private, not benchmarked, or a class we cannot check yet. In unitfabric machines (or on the Machines page), in this order:

• Pending: flags has pending. A machine you have not confirmed takes none of your organization’s jobs: confirm it on the Machines page or with unitfabric machines confirm <id>.
• Paused: flags has paused. It takes nothing until you resume it (unitfabric machines set <id> --resume).
• Private: flags has private. It takes only your organization’s jobs (--public opens it to others’).
• A class we cannot check yet: another organization’s job comes only once we can check your machine’s work on it, on a machine of ours of the same class: your machine’s GPU class for a job that needs a GPU, a CPU machine for one that does not (trust and verification). A machine of a GPU class we do not run yet takes your organization’s jobs, and can take other organizations’ jobs that need no GPU; the Machines page says when it is held back.
• A rate above the work: a machine whose cost per unit, its rate over the units an hour it was measured at, is above 80% of a job’s price is not offered the job. The Machines page shows, for each template, below which rate it would be: lower the rate there, or with unitfabric machines set <id> --rate <usd>.
• No benchmarks: installed with --no-bench, or no benchmark ran. Run them with the agent stopped:

  sudo systemctl stop grid-agent && sudo /opt/grid/agent/.venv/bin/grid-agent bench; sudo systemctl start grid-agent

• Jobs it cannot take: a supplier’s machine runs no job that needs the network; it runs another organization’s own container image only if you allow it (unitfabric machines set <id> --accept-byo: the image comes from its owner’s registry, which sees your machine’s address, and runs as root inside its container, under the agent’s rootful Podman like every job, in a user namespace of its own, with every capability dropped and no network); and a machine that cannot checkpoint (any in WSL2, any CPU machine, and a Linux one without CRIU) runs only jobs whose units take at most 10 minutes or that never checkpoint.
• No work waiting that your machine could do more cheaply than the machines already on it.

A machine whose leases of one kind of work did nothing, many times over (a failing GPU, a full disk), counts as doing none of that work, so it is offered no other organization’s jobs of it. Once it is fixed, enrol it again.

Enrolling a machine again

An install line with --reenrol, or any install line on a machine whose enrolment is gone, enrols the machine as a new one, with a token of today. What that changes:

• The new machine is pending until you confirm it, on the Machines page or with unitfabric machines confirm <id>, as a first one is.
• It starts without the old one’s record of how its leases went, the rates its work ran at among them, so a machine whose failures priced it out of a kind of work is offered that work again. Its trust does not start afresh: it starts at your organization’s trust, the lowest any of its confirmed machines reached, and 50 at most (trust and verification).
• The old machine stays on the Machines page, offline, until you remove it there or with unitfabric machines remove <id>, as the installer says. Removing a machine whose trust is under the trust it started at voids what it earned that no payout covered (how earnings work).
• An organization whose machines fell under the trust floor enrols no machine at all (revocation).

A token that no longer enrols

A supply token enrols machines for a day after it is made. Later, or once it is revoked, the install stops at install.sh: the control plane did not enrol this machine: the line above says why, the line above being enrolment refused: bad supply token. Make a new token on the Machines page or with unitfabric supply tokens create <name>, and run its line. A machine already enrolled needs no new one: the install line run again, to update its agent, leaves its enrolment as it is and uses no token (the line without a token asks for one all the same: give it the old one).

Enrolment has limits, which the line above names when one is reached: 50 machines an organization (enrolment refused: ...), and 20 enrolled in an hour, or 5 from one address (enrolment failed: ... 429 ...: try again later); and an organization whose machines fell under the trust floor enrols none (enrolment refused: ...). An organization holds 20 supply tokens at most that have not expired or been revoked: revoke one you no longer need with unitfabric supply tokens revoke <id>.

Windows, after a restart

The installer has Windows start the agent’s distribution, Unitfabric, and so the agent, with Windows (the scheduled task Unitfabric agent (WSL)), or, where Windows does not allow that, when you next sign in: the machine is online again within a few minutes of a restart, or of your signing in. If it is not, in PowerShell, list the distributions (Unitfabric should be Running), and read the agent’s log inside it:

wsl -l -v
wsl -d Unitfabric -u root -- journalctl -u grid-agent -n 100 --no-pager

Running the install line again puts back whatever is missing. A first install on a machine without WSL installs it and asks for a restart: restart Windows, and run the same line again.