Comprehensive Troubleshooting & FAQ Guide: Lab C

(From Basic File Management to Advanced Cloud Architecture)

This guide organizes every common point of failure in Lab C from the absolute basics up to advanced pipeline engineering, explaining where to find the error, why it happens, and how to fix it.

📁 Level 1: Initial Setup & File Storage Basics

Q1: I created my bucket, but Dataflow can't find my input file or I can't see my output.

• Where to spot the error: The Dataflow Job Builder will display an immediate validation error on the source or sink step, or the completed job will run in 0 seconds and output nothing.

• Why it happens: This is usually caused by a simple naming mismatch, a file extension error, or a region mismatch.

  • If you upload carma_chronicles_labc.csv but type carma_chronicles_labc.txt in your workflow settings, the engine pulls a blank file.
  • Additionally, if your Cloud Storage bucket and your Dataflow job are in entirely different geographic regions, cross-region restrictions can occasionally block or severely slow down data access.

• The Fix: 1. Double-check your bucket path. Use the "Browse" button instead of typing the path manually.

  2. Ensure your input file extension matches exactly what you uploaded (.csv vs .txt).

  3. As a best practice, keep your storage bucket and your Dataflow job in the same region (e.g., both in us-west3 for Salt Lake City or us-central1).

🔐 Level 2: IAM, Permissions & Silent Job Crashes

Q2: I clicked "Run Job," and it failed immediately. There is no execution graph, and the logs are completely blank.

• Where to spot the error: The Dataflow Jobs console will show a red failed icon, but clicking the "Logs" panel at the bottom reveals absolutely no text. The execution graph screen states: "No stage execution graph available."
• Why it happens: This is a classic identity crisis. Dataflow does not run under your personal login; it delegates the workload to the Compute Engine default service account. On brand-new GCP accounts or projects, this background service account is initialized with zero assigned IAM roles. Because it lacks permissions, it cannot write pipeline staging files to your bucket, access logging engines, or even report its own crash back to your console.
• The Fix: 1. Go to IAM & Admin > IAM in the GCP Console.

2. Locate the account formatted as: [YOUR-PROJECT-NUMBER][email protected].
3. Click the pencil icon to edit its roles and add these three crucial permissions:

• Dataflow Worker (roles/dataflow.worker)
• Dataflow Admin (roles/dataflow.admin)
• Storage Object Admin (roles/storage.objectAdmin)

⚠️ Critical Trap – Bucket vs. Project Level: > Do not try to grant these roles solely inside the Cloud Storage bucket permissions tab (Lab B). Dataflow workers don't just read your file; they need project-wide permissions to spin up Compute Engine infrastructure, write telemetry, and create background pipeline logs. You must apply these roles at the Project Level via the main IAM & Admin console, or the job will continue to fail silently.

🌐 Level 3: Cloud Infrastructure & Resource Allocation

Q3: My job failed with a ZONE_RESOURCE_POOL_EXHAUSTED error.

• Where to spot the error: Click on your failed job in the Dataflow console. Bring up the log viewer by clicking the small blue downward-facing arrow at the bottom of the middle pane. Look for the yellow or red text displaying ZONE_RESOURCE_POOL_EXHAUSTED.
• Why it happens: This is an environmental infrastructure issue, not a mistake in your code. It means the specific Google data center zone you selected is temporarily out of the physical virtual machines (Compute Engine workers) required to handle your pipeline workload.
• The Fix: You must change the deployment region of your pipeline.
  1. Open the "Create Job from Builder" menu or reload your YAML workflow.
  2. Scroll down to Pipeline Options or Advanced Options.
  3. Manually change the region drop-down menu to an alternative region.
  4. Tip: For students in Utah, explicitly changing this option to us-west3 (Salt Lake City) bypasses congested default pools and offers lower latency. Other great alternatives include us-west1 or us-central1.

🛠️ Level 4: Configuration, YAML Syntax & Code Strictness