Dataproc Architecture: Ephemeral Clusters, the GCS Connector, and Serverless Spark

The first Dataproc cluster I ever saw in a client's project had been running for fourteen months. Someone had created it during a proof of concept, named it test-cluster-2, and nobody had turned it off since. It was sized for the peak load of a job that ran twice a day and idled at roughly 4% utilization the rest of the time. When I suggested deleting it, the reaction was the one I've now heard a dozen times: "But then where does the data live?"

That question is the whole article. It's the single Hadoop assumption that has to die before any of the rest of Dataproc makes sense, and until it does, teams end up paying cloud prices for on-prem architecture — which is the worst of both worlds. I wrote about the Cloudera-to-Dataproc migration as a war story: what broke, what took twice as long, what surprised us. This piece is the other half, the one that article assumes you already know — what Dataproc actually is under the hood, and why "delete the cluster" is the correct answer rather than a trick question.

What is Dataproc, really?

Dataproc is a managed Hadoop and Spark service whose defining architectural choice is that compute and storage are separate, which makes the cluster itself disposable. That's it. Everything distinctive follows from that one sentence. It runs real open-source Spark, Hive, Flink, and Presto on real VMs — this isn't a reimplementation — but it provisions a cluster in about 90 seconds and expects you to delete it when the job finishes.

Compare that to the mental model you bring from a Cloudera cluster, where the cluster is the data. On-prem, HDFS lives on the same machines that run YARN containers, so the cluster is a permanent, stateful asset you grow carefully, patch nervously, and never turn off. Data locality is the entire premise: move the computation to the data, because moving terabytes across the network is slow. That premise was correct in 2010, and it's the thing Google's network quietly invalidated.

flowchart TB
    subgraph OLD["On-prem Hadoop — cluster IS the data"]
        N1["Node 1
YARN + HDFS blocks"]
        N2["Node 2
YARN + HDFS blocks"]
        N3["Node 3
YARN + HDFS blocks"]
    end
    subgraph NEW["Dataproc — cluster is disposable compute"]
        GCS[("Cloud Storage
the data — permanent")]
        DPMS[("Dataproc Metastore
the schema — permanent")]
        C1["Ephemeral cluster A
job 1 · deleted after"]
        C2["Ephemeral cluster B
job 2 · deleted after"]
        SL["Serverless batch
no cluster at all"]
        GCS --- C1
        GCS --- C2
        GCS --- SL
        DPMS --- C1
        DPMS --- C2
    end
          

Once the data lives in Cloud Storage and the schema lives in a managed metastore, a cluster holds nothing you can't recreate in 90 seconds. Which means the right lifetime for a cluster is the duration of the job, and the right answer to "where does the data live" is "not here, and that's the point."

Why is the GCS connector the thing that bites you?

The GCS connector lets Spark and Hive read gs:// paths as if they were HDFS — it implements the Hadoop Compatible File System interface, so spark.read.parquet("gs://bucket/path") just works with no code change. That "no code change" is what makes lift-and-shift feasible, and it's also what makes the trap so effective: the API is identical, the semantics are not.

Cloud Storage is an object store wearing a filesystem costume. Four differences actually matter in production:

The rename one is the killer, and it's worth being precise because it's where the "why is my job slower on GCP" tickets come from. Spark's default output committer writes task output to a temporary directory and renames it into place on success. On HDFS that rename is a near-instant namespace operation. On GCS it's a full copy of every output file. A job producing 50,000 small output files spends most of its wall-clock time in a "commit" phase that looks like it's hung. Using the GCS connector's optimized committer and, more importantly, writing fewer, bigger files, is the fix — and it's a fix you never needed on-prem, which is exactly why nobody thinks to look there.

How does autoscaling actually decide?

Dataproc autoscaling compares pending memory (what YARN has been asked for and can't yet satisfy) against available memory, and scales up when there's more demand than capacity, down when there's slack. It's YARN-driven, not CPU-driven, which surprises people expecting a web-app autoscaler.

The knob that matters most is scaleUpFactor — the fraction of pending memory you add per scaling decision. Google's own guidance splits cleanly by workload shape: around 0.05 for MapReduce jobs and Spark jobs with dynamic allocation enabled (they ask for resources incrementally, so reacting to the full pending number overshoots badly), versus 1.0 for Spark jobs with a fixed executor count and Tez jobs (they ask for everything up front, so give them everything). Getting this backwards produces either a cluster that thrashes or one that takes twenty minutes to reach the size it needed immediately.

# Autoscaling policy for Spark with dynamic allocation.
# scaleUpFactor 0.05 because dynamic allocation requests incrementally —
# reacting to the whole pending number would massively overshoot.
workerConfig:
  minInstances: 2
  maxInstances: 100
basicAlgorithm:
  cooldownPeriod: 2m
  yarnConfig:
    scaleUpFactor: 0.05
    scaleDownFactor: 1.0
    gracefulDecommissionTimeout: 1h   # let shuffle data drain, don't kill it

Graceful decommissioning is the setting that separates a working autoscaled cluster from a mysteriously failing one. When Dataproc scales down, a worker being removed may still hold shuffle output that a running stage needs. Yank it and Spark refetches or the stage fails; drain it and the job survives. Set gracefulDecommissionTimeout long enough to cover your longest stage. And the sharpest edge: don't autoscale a cluster that stores anything on HDFS — scaling down primary workers means decommissioning HDFS DataNodes, and that's a data-loss shape you don't want to discover empirically.

Primary vs secondary workers

Dataproc splits workers into two kinds, and the distinction is more useful than it first looks. Primary workers run YARN NodeManagers and HDFS DataNodes. Secondary workers are compute-only — no HDFS — and can be Spot/preemptible VMs at a large discount. Because they hold no durable blocks, losing one to preemption costs you re-execution of some tasks, not data.

The pattern this enables: a small primary group for stability, a large secondary group of Spot VMs for the actual work, and jobs that read and write GCS rather than HDFS so preemption is merely annoying. Get the mix wrong — heavy Spot with a long shuffle-dependent stage — and you'll spend the savings on retries. Spot workers reward short, wide, idempotent stages.

When should I skip the cluster entirely?

Dataproc Serverless for Spark runs a Spark batch workload with no cluster to create, size, tune, or delete — you submit the job and Google runs it. If ephemeral clusters are the answer to "stop paying for idle," Serverless is the answer to "stop thinking about infrastructure at all." For scheduled ETL and variable workloads, it's usually where I'd start now, because the zero-idle-cost property is structural rather than a discipline you have to maintain.

My rule of thumb after a few of these migrations: default to Serverless for batch Spark; use ephemeral clusters when you genuinely need components Serverless doesn't run or tuning it doesn't expose; and reserve long-running clusters for interactive and streaming work where the cluster is doing something continuously. If you can't articulate what a long-running cluster is doing right now, it's test-cluster-2 and it should be deleted.

Where does the schema live?

If clusters are disposable, the Hive metastore can't live on one — and on a stock Hadoop cluster it lives in a MySQL instance on the master node, which is exactly the kind of stateful thing that makes people afraid to delete anything. Dataproc Metastore is the managed, cluster-independent Hive metastore that fixes this: multiple ephemeral clusters and Serverless jobs attach to the same metastore, so table definitions outlive every cluster that ever queried them. If you're coming from the world I described in the Hive Metastore internals piece, this is that same service, finally decoupled from the machines. Skipping it is how teams end up unable to delete a cluster because the schema is trapped inside it.

What does a well-shaped Dataproc job look like?

The pattern that falls out of all of the above — data in GCS, schema in Dataproc Metastore, compute created and destroyed per job:

# Create, run, delete — as one atomic unit. The cluster exists only
# for this job. --max-idle is the seatbelt for when the delete never runs.
gcloud dataproc clusters create etl-${RUN_ID} \
    --region=us-central1 \
    --master-machine-type=n2-standard-4 \
    --worker-machine-type=n2-standard-8 --num-workers=2 \
    --num-secondary-workers=20 --secondary-worker-type=spot \
    --dataproc-metastore=projects/p/locations/us-central1/services/prod-metastore \
    --max-idle=30m

gcloud dataproc jobs submit spark --cluster=etl-${RUN_ID} --region=us-central1 \
    --class=com.example.DailyAggregate --jars=gs://artifacts/etl.jar

gcloud dataproc clusters delete etl-${RUN_ID} --region=us-central1 --quiet

Two details worth stealing. --max-idle auto-deletes the cluster after a period of inactivity — it's the seatbelt for the day your orchestrator dies between the submit and the delete, and it is the single setting that would have prevented test-cluster-2. And the mix of 2 primary workers with 20 Spot secondaries is the shape I reach for by default: enough stable capacity to hold the cluster together, most of the compute bought at a discount, and nothing on HDFS that a preemption could hurt. Dataproc Workflow Templates express this whole create-run-delete sequence as one managed unit if you'd rather not orchestrate it yourself.

What to carry away

Dataproc is not Cloudera with a Google logo, and the teams that struggle with it are almost always the ones running it as though it were. The architecture is one idea — the data lives in Cloud Storage and the schema lives in a managed metastore, so the cluster holds nothing precious and should exist only as long as a job does. Everything else is a consequence: autoscale on YARN's pending memory with graceful decommissioning so you don't murder shuffle data, buy most of your compute as Spot secondary workers that hold no HDFS blocks, and reach for Serverless when you'd rather not think about machines at all. The one habit that costs real money is the one nobody flags in review — writing tens of thousands of tiny files, which was merely untidy on HDFS and is a slow, billable copy storm on an object store where rename isn't free. Fix the file count, delete the cluster, and let the network do the job data locality used to.