The Marketplace Lied 62 Times Out of 100

Between July 11 and July 29 I created one hundred GPU instances on a rental marketplace. Sixty-two of them never delivered a working GPU. Total damage for all sixty-two failures: $4.37. Both halves of that sentence are the story.

The marketplace in question rents you other people's consumer hardware. An RTX 3090 for eleven cents an hour, which prices a 4-billion-parameter QLoRA fine-tune at roughly a quarter. The catch is the phrase "other people's": the host is a gaming rig in a stranger's spare room, and the stranger has opinions about driver updates that your CUDA image does not share.

I spent three years running about 3,000 AWS instances for a push messaging platform, and in all that time Amazon never once handed me a machine that could not compute and billed me while I figured that out. That is what an SLA buys. The marketplace has no SLA. It has a price, and the price is honest about exactly one thing: the price.

Eleven Cents an Hour, From Strangers

The appeal is real, so let me steel-man it properly. The economics of small fine-tuned models have never been better: Predibase's LoRA Land report fine-tuned 310 adapters on small open models and beat GPT-4 by ten points on average, and NVIDIA's research group estimates 40-70% of the LLM calls in real agent frameworks are narrow enough for a small specialist to serve at 10-30x lower cost. A model that does one job well is cheap to train, and the cheapest place to train it is a marketplace card. The arithmetic pulls you in. It pulled me in.

What the arithmetic leaves out is delivery. My ledger says 62 of 100 instance creates died before any training happened: hosts that froze mid-image-pull, hosts that misreported their hardware, hosts whose drivers were too old for the image they accepted anyway. On the two documented platform-wide bad days, the failure rate hit three in four. This is not the pipeline failing. This is the marketplace being what it is.

Nobody puts that number on the pricing page.

What a Lying Host Looks Like

The expensive failures are not the loud ones. A host that refuses to boot costs you minutes. The host that hurt was the one that booted fine, passed nvidia-smi, showed a healthy GPU with healthy VRAM, and could not run a single tensor operation - a stale driver against a newer CUDA image. Every status check said green. The training process was dead. That host idle-burned 105 minutes of billed time before a human noticed.

The fix is embarrassingly small. After an instance reaches running, don't ask the GPU how it feels. Make it do work:

$ python3 -c "import torch; torch.cuda.init(); \
    a = torch.randn(4096, 4096, dtype=torch.bfloat16, device='cuda'); \
    print((a @ a).sum().item())"
-14336.0
✓ host can actually compute - proceed

One real matmul, about ten seconds. A host that fails it gets destroyed and replaced immediately, and its machine id goes into quarantine so the retry loop cannot re-pick the same dud. The 105-minute class of failure now costs about ten seconds.

Here is the perverse incentive underneath: the marketplace's own reliability score measures host uptime. I watched a host throttle its GPU 22% and keep a spotless score, because the machine was up the whole time it was underdelivering. Uptime is what the platform can measure. Delivery is what you are paying for. Nobody in the transaction has more incentive to measure delivery than you, so you have to keep that ledger yourself: which host booted, which host computed, which host finished a run. Renter-side memory is the only score a host can't game.

The Receipt

Eighteen days, every failure included:

Read those two money lines together and the whole procurement conversation changes. A 62% failure rate sounds disqualifying. $4.37 of consequence says otherwise. The hourly rate was never the number that mattered. The number that matters is what a dead host bills you before you notice it is dead.

That number is an engineering choice. Thirteen minutes of mean lifetime per dead host is not luck; it is a watcher that reads the host's boot status during the image pull and kills the attempt when the status line freezes for six minutes or matches a known-fatal pattern. Before that watcher existed, a bad host cost 25 to 30 minutes of polite poll-waiting. Multiply the difference by sixty-two.

Fail Fast Is a Money Policy

Most teams treat fail-fast as an engineering aesthetic. On rented hardware it is a billing policy, because the meter defines the physics of the problem: the GPU bills while the instance runs, storage bills from the moment of creation, and no amount of clever code changes either clock. The only variable you control is time-to-detection. Every guardrail in my pipeline is that one sentence wearing different clothes:

• A utilization watchdog with a grace period. Anything idle past the window gets stopped automatically. A stopped instance bills cents of storage instead of dollars of GPU. The incident class this exists for is the forgotten monster: an 8xH200 node runs $17.88 an hour whether or not anyone remembers it exists.
• Stop and destroy are asymmetric on purpose. Automation may stop an instance; only a human destroys one. A wrong stop costs cents. A wrong destroy loses a training run. Give the cheap mistake to the machine and keep the expensive one for yourself.
• The script cannot leak a rental by dying. An exit trap stops the instance on any abort, including my own Ctrl-C, and disarms only after the intentional destroy. The failure mode where the orchestration crashes and the meter keeps running is closed structurally, not by vigilance.
• Every create, stop, and destroy lands in an append-only ledger with price and reason. "What did we run and what did it cost" always has an answer, even after the instance is gone. Every number in this article comes from that file.

None of this is exotic. It is the same discipline as auditing the services you forgot you were running, applied at rental speed: the host is guilty until it proves it can compute, and the meter is the adversary.

Provisioning Is the Bottleneck, Not Training

The ledger held one more surprise. One campaign burned 22 create/destroy cycles to get 33 minutes of actual training done. I went in assuming training time was the scarce resource and the ledger said no: acquiring a verified, working GPU is the scarce resource. Once you have one, hold it and batch onto it. Run the next fine-tune on the rental you already proved instead of rolling the 62% dice again.

What did eighteen days of this actually produce? A log-triage model for about 35 cents. A voice turn-detector that answers in 140 milliseconds for about 30 cents. The 65,184-parameter character transformer with measured attention that now runs inside my browser LLM lab. Small specialist models, each trained for less than the coffee I drank while it ran. The received wisdom says custom models need an ML team and a six-figure compute budget. The receipt above says twenty dollars, provided you spend engineering effort on the part nobody brags about: surviving the substrate.

When the Marketplace Makes Sense

I'm not telling you to avoid marketplace GPUs. I wrote a whole decision framework arguing the 40-60% savings are real for the right workloads. The marketplace works when:

• The job checkpoints and tolerates death. Fine-tunes, batch inference, renders. If an interruption costs you the run, you have not designed for this substrate.
• Nothing secret touches the host. Pull artifacts off the box; never push credentials onto it. The stranger owns the machine. Behave accordingly.
• You have delivery verification on day one. A real compute preflight, a boot watchdog, an idle killer. Without them, my $4.37 would have been forty dollars and my 105-minute burn would be your weekend.
• The job fits one node. Multi-node training over strangers' Ethernet is a research project, not a rental.

And when the job needs to just work (sustained big-model training, anything with an SLA on the other end), pay for the datacenter tier. SemiAnalysis's rental index has H100 rates climbing again as agentic workloads soak up capacity, but on-demand H100s at specialized clouds still ran $2-4 an hour by mid-2026. The premium over the marketplace is not waste. It is the price of the 62 failures somebody else absorbs, and the post-shortage market now offers that choice at every tier. Know which tier your job needs before the meter starts.

"The hourly rate was never the number that mattered. The number that matters is what a dead host bills you before you notice it is dead."

Cisco Caceres•Coding since the late 1970s. Still learning, still building.