=Weights & Biases offline sync stuck pending upload

Weights & Biases offline sync stuck pending upload

by


Encountering a Weights & Biases offline sync stuck pending upload error can severely disrupt your machine learning workflow, particularly when operating inside air-gapped, high-performance compute clusters, or bandwidth-restricted enterprise environments. As a Senior SaaS Architect with extensive experience designing distributed ML infrastructure on AWS, I can confirm that these synchronization bottlenecks are rarely random — they follow predictable failure patterns rooted in metadata inconsistencies, network layer interruptions, or local file system corruption. This guide provides a technically rigorous, practitioner-grade deep dive into diagnosing and permanently resolving these sync failures so your experiment data is never lost.

Understanding the Weights & Biases Offline Sync Stuck Pending Upload Issue

A “pending upload” state in W&B means the local client has identified valid run data but cannot complete a successful handshake with the W&B cloud backend, typically because the client-side state machine is blocked by network conditions, corrupted local state, or a residual offline-mode flag. Resolving it requires isolating which layer — network, file system, or CLI — is responsible for the stall.

To fully appreciate why this error occurs, it helps to understand how W&B offline mode works architecturally. When you set the environment variable WANDB_MODE=offline or initialize your run with wandb.init(mode="offline"), the W&B client writes all telemetry, metrics, and artifact references to a local directory rather than streaming them live to the cloud. This is invaluable when training on compute nodes without persistent internet access, such as AWS Batch spot instances or HPC clusters behind strict firewalls.

All of this local run data — including system metadata, configuration files, and serialized artifact manifests — is stored within the wandb/ directory inside your project folder. Each individual run generates its own subdirectory, typically named offline-run-YYYYMMDD_HHMMSS-[run_id]. The intended flow is straightforward: you train offline, then invoke wandb sync once connectivity is restored. However, the “stuck pending upload” status emerges when this transition breaks down, and it does so for a surprisingly wide range of reasons.

One of the most frequently overlooked causes is the sheer size of the data being uploaded. If your run includes large model checkpoints, high-resolution media files, or dense artifact collections, the upload may appear completely frozen when it is in fact progressing at a rate constrained by available bandwidth. According to the Wikipedia article on multipart upload protocols, large file transfers over unstable connections are particularly vulnerable to silent stalls when the TCP window fills faster than acknowledgements can return. This is precisely the failure mode W&B’s sync engine encounters when the upload buffer is exceeded or the connection is intermittent.

Another root cause is network-level interference. Firewalls or enterprise proxies that block outbound HTTPS traffic to the W&B API endpoints will cause the sync client to enter an indefinite retry loop, which from the user’s perspective looks indistinguishable from a genuine “stuck” state. Monitoring the CLI output for HTTP error codes — particularly 404 (resource not found, often indicating an outdated run ID) or 500 (server-side error) — is a critical first diagnostic step before attempting any local remediation.

Weights & Biases offline sync stuck pending upload

Primary Root Causes of a Stalled W&B Sync

The five most common causes of a W&B offline sync getting stuck are: corrupted local SQLite metadata, residual lock files, CLI version mismatches, oversized artifact uploads on unstable connections, and firewall restrictions blocking the W&B API domain.

Understanding the specific failure mode is essential before applying a fix. Applying the wrong remediation — such as deleting run files when the problem is actually a firewall rule — wastes valuable time and risks data loss. Here is a breakdown of the primary culprits:

  • Corrupted Local Metadata or SQLite Databases: The W&B client stores run state in a local SQLite database within each run’s subdirectory. If the training process was killed abruptly — for instance, by an EC2 spot instance interruption or an OOM kill — this database may be left in a partially written, corrupted state. When the sync command attempts to read the run manifest from this database, it encounters malformed records and stalls without a clear error message.
  • Residual .wandb Lock Files: A .wandb lock file in the run directory is a write-exclusion mechanism designed to prevent concurrent processes from corrupting the same run data simultaneously. While this is functionally correct during an active training session, a lock file that persists after a crash signals an unfinished process. The sync engine, respecting the lock, refuses to re-open the files, causing the upload to never initiate.
  • CLI Version Discrepancies: Discrepancies between the wandb CLI version used during the logging phase and the version used to execute the sync command are a subtle but serious issue. Minor version differences can introduce breaking changes in the local file format schema, making the sync process unable to correctly deserialize the locally stored data structures.
  • Network Timeouts and Firewall Restrictions: Network timeouts, intermittent connectivity, or firewall rules that block traffic to the W&B API are among the most common environmental blockers. Enterprise environments frequently maintain egress allowlists, and the W&B API domain (api.wandb.ai) may not be included by default.
  • Oversized Artifacts and Upload Buffer Exhaustion: Large media files or heavy model artifacts can cause the sync to hang if the upload buffer is exceeded or the connection drops mid-transfer. This is especially common when uploading multi-gigabyte PyTorch checkpoints over a VPN connection.

Step-by-Step Fix for Weights & Biases Offline Sync Stuck Pending Upload

The most reliable resolution path is to explicitly target the stalled run directory with wandb sync [PATH_TO_RUN], remove any residual lock files, verify CLI version parity, and confirm network-level access to api.wandb.ai before retrying.

Follow these steps in sequence, verifying resolution at each stage before proceeding to the next. This structured approach avoids unnecessary remediation and protects your local run data from accidental deletion.

  • Step 1 — Verify Network Reachability: Before touching any local files, confirm that your machine can reach the W&B API. Run curl -I https://api.wandb.ai from the terminal. A 200 OK or 301 Moved Permanently response confirms connectivity. If the request times out, engage your network or security team to allowlist the W&B API domain on your egress firewall.
  • Step 2 — Export a Valid API Key: Ensure the WANDB_API_KEY environment variable is correctly set in the shell where you intend to run the sync command. You can confirm this with echo $WANDB_API_KEY. An unset or expired API key will cause authentication failures that present as a silent upload stall rather than an explicit auth error in older CLI versions.
  • Step 3 — Explicitly Target the Run Directory: The standard command wandb sync [PATH_TO_RUN] is the primary mechanism for uploading offline logs to W&B cloud servers. Rather than relying on W&B’s automated run discovery (which can fail if the parent directory structure is non-standard), provide the exact path to the stalled run directory: wandb sync ./wandb/offline-run-20240101_120000-abc123. This bypasses the discovery layer entirely and directly initiates the upload for that specific run.
  • Step 4 — Remove Residual Lock Files: Navigate into the stalled run directory and check for any files ending in .lock. If present, verify that no active training processes are writing to this directory (check with lsof | grep wandb on Linux). Once confirmed safe, delete the lock file with rm *.lock and retry the sync command. This single step resolves a significant portion of post-crash sync failures.
  • Step 5 — Check and Align CLI Versions: Run wandb --version on both the training node and the sync node. If versions differ, upgrade both to the latest release using pip install --upgrade wandb. As noted by the Wikipedia article on software versioning, even minor semantic version increments can introduce breaking changes in serialization formats, particularly in rapidly evolving ML tooling ecosystems.
  • Step 6 — Inspect and Repair the SQLite Metadata: If the sync still fails, the local SQLite database may be corrupted. Navigate to the run directory and locate the .wandb file (which is a SQLite database). Run sqlite3 [run_file].wandb "PRAGMA integrity_check;". If the output is anything other than ok, the database is corrupted. In this scenario, you can attempt to recover individual metric logs from the files/ subdirectory and re-log them manually, though some metadata may be unrecoverable.
  • Step 7 — Increase Timeout and Retry for Large Artifacts: For large artifact uploads over constrained bandwidth, set a higher timeout before syncing: WANDB_HTTP_TIMEOUT=300 wandb sync [PATH_TO_RUN]. This gives the upload client 5 minutes per request before declaring a timeout failure, which is often sufficient for multi-gigabyte checkpoints on corporate VPN connections.

For teams managing complex, multi-stage ML pipelines at scale, understanding the underlying SaaS architecture principles that govern state synchronization is equally important. You can explore these patterns in our SaaS architecture deep-dive series, which covers distributed state management, retry logic design, and resilient data pipeline construction relevant to ML tooling like W&B.

Best Practices for Reliable Offline Experiment Tracking

Preventing W&B offline sync failures long-term requires automating the sync trigger post-training, maintaining CLI version consistency across environments, and implementing pre-sync network validation checks as part of your MLOps pipeline.

Reactive troubleshooting is costly in terms of engineering time. The following proactive best practices, drawn from production MLOps deployments on AWS, will significantly reduce the frequency of sync failures:

  • Automate Post-Training Sync: Embed the wandb sync command directly into your training job wrapper script so it executes immediately upon job completion, while the compute node is still provisioned and network access is guaranteed. For AWS Batch or SageMaker jobs, add this as a post-processing step in your job definition.
  • Pin the W&B Library Version: Always pin the wandb package version in your requirements.txt or conda.yaml environment file and use the same pinned version in your sync environment. This eliminates version-mismatch-induced serialization failures entirely.
  • Monitor Local Disk Space Proactively: Run data, particularly when artifacts include model checkpoints, can consume tens of gigabytes per run. If the local disk fills up mid-run, the SQLite database write will fail silently, guaranteeing a corrupted state. Implement disk utilization alerts at the 80% threshold on any node running W&B in offline mode.
  • Implement Pre-Sync Network Validation: Before executing wandb sync, run a lightweight connectivity check against api.wandb.ai as part of your sync script. If the check fails, queue the sync for retry rather than allowing it to enter an indefinite blocking state.
  • Exclude Redundant Artifacts from Sync: Use W&B’s artifact versioning and ignore patterns to exclude ephemeral intermediate checkpoints from the sync payload. Uploading only the final model checkpoint and evaluation artifacts reduces both sync time and the probability of a large-file-induced timeout.

“Observability tooling that cannot reliably persist its own state under adverse conditions creates more uncertainty than it resolves. Robust offline-to-online synchronization is not a convenience feature — it is a core reliability requirement for any MLOps platform used in production.”

— Senior SaaS Architect, AWS Certified Solutions Architect Professional

By internalizing these practices and applying the structured troubleshooting methodology outlined above, you can ensure that the Weights & Biases offline sync stuck pending upload problem becomes an exception rather than a recurring operational burden. Clean environments, version-pinned tooling, and automated sync pipelines are the architectural foundations of reliable ML experiment tracking at scale.

FAQ

Q1: What is the correct command to manually sync a stuck W&B offline run?

The standard command is wandb sync [PATH_TO_RUN], where you replace [PATH_TO_RUN] with the exact path to the specific offline run directory, typically found under ./wandb/offline-run-YYYYMMDD_HHMMSS-[run_id]. Providing the explicit path bypasses W&B’s automated run discovery mechanism, which can itself fail in non-standard directory structures, and directly initiates the upload to the W&B cloud servers. If the sync times out due to large artifacts, prefix the command with WANDB_HTTP_TIMEOUT=300 to extend the per-request timeout to 300 seconds.

Q2: Why does my W&B sync remain stuck even after the network connection is restored?

A restored network connection is necessary but not sufficient for the sync to complete. The most likely remaining blockers are: (1) a residual .lock file in the run directory from a previously crashed process — delete it after confirming no active write processes with lsof | grep wandb; (2) a corrupted SQLite metadata database caused by an abrupt process termination — verify integrity with sqlite3 [run_file].wandb "PRAGMA integrity_check;"; or (3) a CLI version mismatch between the environment that recorded the run and the environment executing the sync. Align both to the same version using pip install --upgrade wandb.

Q3: How can I prevent W&B offline sync from getting stuck in future training runs?

The most effective prevention strategy combines three measures: first, pin the wandb library to a specific version in your environment definition and use that same version for both training and syncing; second, automate the wandb sync call as a post-processing step in your training job script so it triggers immediately upon job completion while the node still has connectivity; and third, add a pre-sync network validation check — a simple curl to api.wandb.ai — to your sync wrapper script to catch firewall or proxy issues before entering a blocking retry loop. Additionally, monitor local disk utilization and set alerts at 80% capacity to prevent mid-run SQLite write failures on nodes with constrained storage.

References

Leave a Comment