Error codes

When a job fails, its error_code carries a machine-readable reason and result carries a short message. These are the codes the fulfillment pipeline assigns (from curl errno, HTTP status, and response body).

Code reference

Code	Retryable	Meaning	How to resolve
`auth`	yes (critical)	Provider rejected the credentials (HTTP 401 / 403). Bad or missing API key / endpoint.	Fix the provider key or endpoint in `lib/faucet_config.php`. The operator is emailed; the job retries automatically once the key is valid.
`out_of_credits`	yes (critical)	Provider balance exhausted (HTTP 402, or a body mentioning "insufficient" / "credit" / "balance", e.g. RunPod balance fail, OpenAI/Azure 429 insufficient_quota, Anthropic "credit balance too low").	Top up the provider balance. The operator is emailed; the job retries automatically once funded.
`rate_limit`	yes	Provider throttled the request (HTTP 429, not a quota/credit issue).	Nothing needed - the tiered backoff waits it out. Reduce concurrency if it persists.
`timeout`	yes	The call timed out, or the job sat in `running` past its per-mode wall-clock limit and was reaped (text 300 s, image 900 s, video 2400 s).	Transient. It re-queues and retries. Persistent timeouts point at a wedged provider or too-large a request.
`network`	yes	Could not connect / resolve the provider host (DNS or connection failure).	Check the endpoint host and that the provider is reachable from the worker.
`transport`	yes	A lower-level curl transport error not classified as timeout or network.	Usually transient; retries. Inspect the `result` message if it repeats.
`provider_error`	yes	Provider returned a 5xx, or reported the job FAILED / CANCELLED, or returned no usable job id.	Provider-side fault. Retries automatically; check the provider's status if it persists.
`empty_response`	yes	The call succeeded but returned no content (empty inference text, or a COMPLETED provider job with no output).	Retries. If it keeps happening, the prompt or model config is producing empty output.
`bad_request`	no	The request is malformed for the provider (e.g. no media provider configured for the mode, empty prompt, or a model routed to the wrong path).	Fix the request or configuration. This is the only non-retryable code - the job fails permanently.
`error`	yes	Generic uncaught failure (a thrown exception not carrying a specific code, e.g. a failed media adopt / unwritable file).	Read the `result` message. Transient issues retry; a code bug needs a fix.
`http_<code>`	yes	An HTTP status the classifier did not map to a specific code (e.g. `http_404`, `http_400` from a provider).	Inspect the message. Often an endpoint / path / payload mismatch.

Retry behavior

Every code except bad_request is retryable. A retryable job with retries remaining is re-queued with tiered backoff by remaining retries:

Retries left	Backoff before next attempt
3	2 minutes
2	10 minutes
1	1 hour

When retries are exhausted, the job is marked permanently failed with its last error_code. On the retry path the worker clears external_job_id so an async media job is resubmitted fresh.

Critical codes and alerting

auth and out_of_credits are critical: they email the operator (rate-limited to once per code per 30 minutes) and are always appended to an alert log. Every failure - critical or not - is also mirrored to the fleet error_reports bar. Because balance exhaustion is a real-money event on shared paid keys, a proactive balance poller (planned) warns before jobs start failing.

Failed-job shape

{
  "id": 5140,
  "mode": "image",
  "model": "gpt-image-2",
  "status": "failed",
  "error_code": "out_of_credits",
  "result": "Azure gpt-image failed: HTTP 429: insufficient_quota ...",
  "retries_left": 0,
  "completed_at": "2026-07-12T18:10:00Z"
}

While a job is still retrying it returns to status: queued with the error_code and a next_retry_time set, rather than failed.