Building Production Data Pipelines with AWS Glue and PySpark

Most teams do not have a Spark problem. They have a cluster problem: someone has to size it, patch it, keep it warm enough to be responsive and cold enough to be affordable, and get paged when a driver dies at 3am. AWS Glue removes that job. What it does not remove is the need to understand how Spark actually executes your code — and the gap between a Glue job that works on a sample and one that reliably chews through terabytes is almost entirely in that second category.

This is the guide we wish we'd had: the architecture we deploy, the tuning that actually moves the needle, and the mistakes we keep being called in to fix.

Why serverless Spark changes the trade-off

Glue is Apache Spark with the operational surface removed. You submit a job, AWS provisions workers, runs it, and tears them down. You pay per second of compute with a one-minute minimum. There is no cluster to leave running, no autoscaling group to tune, no AMI to patch.

That changes the economics of bursty workloads dramatically. A nightly batch that needs 100 workers for 40 minutes costs you 40 minutes. The same workload on a persistent EMR or Databricks cluster costs you whatever the cluster's idle policy allows — and idle policies are where budgets go to die.

The trade-off you accept in return:

• Cold start. Glue jobs take roughly a minute to acquire workers. Irrelevant for a 40-minute batch, fatal for a 5-second interactive query. Use Athena for the latter.
• Less control. You choose a worker type and a count, not an instance family, a spot strategy, or a custom Spark build.
• Version cadence. You get the Spark versions AWS ships, when AWS ships them.

For scheduled batch and micro-batch ETL — which is the overwhelming majority of what data platforms actually run — that trade is worth making.

The reference architecture

Almost every pipeline we build follows the same three-hop shape, usually called the medallion architecture. The names matter less than the discipline: each layer has exactly one job, and no layer reaches past its neighbour.

The Glue Data Catalog (and when to skip crawlers)

The Glue Data Catalog is a Hive-compatible metastore: databases, tables, columns, partitions. Athena, Redshift Spectrum, EMR and Glue itself all read from it, which makes it the natural centre of an AWS lakehouse.

Crawlers populate it by sampling S3 and inferring schema. They are excellent for exploration and genuinely risky in production:

• Inference is a guess. A column that is all integers in today's sample becomes bigint; next week's file contains "N/A" and the table silently splits or fails.
• Crawler runs cost money and time proportional to the number of objects scanned.
• A crawler can change a production schema without a code review.

Our default: crawl during discovery, then pin the schema as code and register partitions explicitly. For a table whose partitions follow the Hive convention, one call replaces a crawler entirely:

MSCK REPAIR TABLE analytics.raw_events;

Better still, have the writing job add the partition it just wrote. That is O(1) instead of O(partitions), and it keeps the catalog exactly in step with storage:

sink = glueContext.getSink(
    connection_type="s3",
    path="s3://delantech-lake/modeled/events/",
    enableUpdateCatalog=True,          # register partitions as they are written
    updateBehavior="UPDATE_IN_DATABASE",
    partitionKeys=["event_date"],
)
sink.setCatalogInfo(catalogDatabase="analytics", catalogTableName="events")
sink.setFormat("glueparquet", compression="snappy")
sink.writeFrame(out_dyf)

Storage layout: the decision you cannot undo cheaply

Everything downstream inherits your S3 layout. Get it wrong and you will pay for it on every single query, forever, until someone funds a rewrite.

Use columnar formats

Parquet with Snappy compression is the default and it is rarely the wrong answer. Columnar storage means a query touching 3 of 80 columns reads 3 columns' worth of bytes. On a wide clickstream table that is routinely a 20× reduction in I/O over JSON or CSV — before compression.

Snappy over Gzip is deliberate: Snappy is splittable and much cheaper to decompress, so Spark can parallelize across a single large file. Gzip is not splittable, so one 2 GB gzip file is processed by exactly one core.

Partition on what you filter, at the right cardinality

Partition columns become directory names, and Spark skips whole directories when a query filters on them. This is the single biggest performance lever you have.

s3://delantech-lake/modeled/events/
    event_date=2026-02-17/
        region=us-east-1/
            part-00000-....snappy.parquet

The failure mode is cardinality. Partition by event_date and you get ~365 directories a year: excellent. Partition by user_id and you get millions of directories each holding one tiny file: catastrophic. Listing them alone will take longer than reading the data.

The small-file problem

This is the most common pathology we find in the wild. Streaming ingestion or an over-parallelized job produces thousands of 200 KB files. Spark opens each one, reads a footer, schedules a task. The overhead swamps the work.

The fix is a compaction step that rewrites small files into large ones, sized deliberately:

TARGET_FILE_BYTES = 256 * 1024 * 1024   # 256 MB

# Estimate output size from the input, then pick a file count to match.
input_bytes = sum(o["Size"] for o in list_objects(prefix))
n_files = max(1, math.ceil(input_bytes * COMPRESSION_RATIO / TARGET_FILE_BYTES))

(df
    .repartition(n_files, "event_date")   # shuffle to exactly n_files per write
    .write
    .mode("overwrite")
    .partitionBy("event_date")
    .parquet("s3://delantech-lake/modeled/events/"))

Incremental loads with job bookmarks

Reprocessing all history on every run is the most expensive mistake in ETL, and the easiest to make. Glue's job bookmarks track what a job has already consumed, so each run picks up only new data.

Bookmarks are opt-in and depend on one easily forgotten detail: every source and sink needs a stable transformation_ctx. That string is the key under which state is stored. Change it, and the job silently reprocesses everything.

import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext

args = getResolvedOptions(sys.argv, ["JOB_NAME", "source_database", "target_path"])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)      # bookmark state is loaded here

raw = glueContext.create_dynamic_frame.from_catalog(
    database=args["source_database"],
    table_name="raw_events",
    transformation_ctx="raw_events_source",   # <-- the bookmark key. Never change it.
)

# ... transformations ...

glueContext.write_dynamic_frame.from_options(
    frame=out_dyf,
    connection_type="s3",
    connection_options={"path": args["target_path"], "partitionKeys": ["event_date"]},
    format="glueparquet",
    transformation_ctx="curated_sink",
)

job.commit()      # bookmark advances ONLY if this line runs

Partition pruning beats bookmarks when you can use it

Bookmarks still list the source prefix to work out what is new. If you already know which partitions you want, say so explicitly — Glue then never lists the rest:

raw = glueContext.create_dynamic_frame.from_catalog(
    database="analytics",
    table_name="raw_events",
    push_down_predicate="event_date >= '2026-02-16' and event_date <= '2026-02-17'",
    transformation_ctx="raw_events_source",
)

The predicate is evaluated against partition metadata in the catalog, not against data. On a table with three years of daily partitions this turns a 1,095-directory listing into two.

DynamicFrame or DataFrame?

Glue adds DynamicFrame, its own abstraction, on top of Spark's DataFrame. Both are useful; knowing which to reach for saves a lot of pain.

Our rule: read messy sources as a DynamicFrame, resolve the mess, convert to DataFrame immediately, and do all real work there. You get Glue's schema tolerance at the boundary and Spark's optimizer everywhere else.

# Handle a column that arrives as both string and long, then hand off to Spark.
resolved = raw.resolveChoice(specs=[("user_id", "cast:long")])
df = resolved.toDF()

df = (df
      .filter(F.col("event_ts").isNotNull())
      .withColumn("event_date", F.to_date("event_ts"))
      .dropDuplicates(["event_id"]))

Making PySpark fast at volume

Understand the shuffle

Every groupBy, join, distinct and repartition triggers a shuffle: data is redistributed across the network so that related rows land on the same executor. Shuffles are where Spark jobs go slow, and where they die.

The knob is spark.sql.shuffle.partitions, which defaults to 200. That default is wrong for almost everyone — too high for small data, far too low for large.

# Target ~128-200 MB of shuffled data per partition.
# 800 GB shuffle / 200 MB = ~4000 partitions
spark.conf.set("spark.sql.shuffle.partitions", "4000")

Too few partitions and each task handles too much data, spilling to disk or hitting out-of-memory. Too many and you drown in scheduling overhead. Adaptive Query Execution largely solves the "too many" direction automatically.

Let Adaptive Query Execution do the work

AQE re-plans the query at runtime using actual statistics. It is enabled by default from Spark 3.2 onward (Glue 4.0 and later), and it is the highest-leverage setting in modern Spark:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")   # merges small post-shuffle partitions
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")             # splits skewed partitions automatically

With coalescePartitions enabled you can safely set shuffle.partitions high; AQE collapses the excess after the fact.

Broadcast the small side of a join

Joining a 4 TB fact table to a 2 MB country lookup should not shuffle 4 TB. A broadcast join ships the small table to every executor instead:

from pyspark.sql.functions import broadcast

joined = events.join(broadcast(dim_country), "country_code", "left")

# Or raise the automatic threshold (default 10 MB) so Spark chooses it itself:
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", str(64 * 1024 * 1024))

Handle skew explicitly when AQE cannot

Skew is when one key holds a disproportionate share of rows — a null customer id, a bot account, a default tenant. One task gets 400 GB while 3,999 tasks get 200 MB, and your job's runtime is that one task.

AQE's skewJoin handles many cases. When it does not, salt the key: spread the hot key across N synthetic buckets, and replicate the small side across all N.

from pyspark.sql import functions as F

SALT_BUCKETS = 64

skewed = big.withColumn("_salt", (F.rand() * SALT_BUCKETS).cast("int"))

small_replicated = small.withColumn(
    "_salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))
)

joined = (skewed
          .join(small_replicated, ["join_key", "_salt"], "left")
          .drop("_salt"))

The cost is replicating the small side 64×. That is fine when it is a dimension table and catastrophic when it is not — measure before reaching for this.

Cache deliberately, or not at all

cache() is not a performance setting, it is a memory allocation. It pays off only when a DataFrame is genuinely reused across multiple actions and fits comfortably in memory. Cache a DataFrame you read once and you have spent memory to gain nothing; cache one that does not fit and Spark spills it to disk, making everything slower.

from pyspark import StorageLevel

sessions = build_sessions(events)
sessions.persist(StorageLevel.MEMORY_AND_DISK)   # reused by 3 downstream writes

write_daily(sessions)
write_by_channel(sessions)
write_cohorts(sessions)

sessions.unpersist()      # release it — do not rely on GC

Avoid Python UDFs on hot paths

A Python UDF forces every row out of the JVM, through a Python interpreter, and back — serializing both ways. It is commonly 10–100× slower than the equivalent built-in, and it blocks Catalyst from optimizing across it.

Reach for built-in pyspark.sql.functions first. If you genuinely need custom logic, use a pandas UDF, which operates on vectorized Arrow batches instead of row-by-row:

from pyspark.sql.functions import pandas_udf
import pandas as pd

@pandas_udf("double")
def normalize(v: pd.Series) -> pd.Series:
    return (v - v.mean()) / v.std()      # vectorized, one call per batch

df = df.withColumn("score_z", normalize("score"))

Sizing the job: DPUs, workers, autoscaling

Glue bills in DPUs (Data Processing Units). Worker types map to DPUs and to concrete resources:

Practical sizing method, in order:

1. Start at G.1X with enough workers that total executor cores ≈ your target shuffle partition count / 2.
2. Run it. Read the Spark UI.
3. If tasks are spilling to disk or you see OOM kills, move to G.2X before adding more G.1X workers — you have a memory-per-task problem, not a parallelism problem.
4. If tasks are fast but the job is long because there are many of them, add workers.
5. If one task dominates the stage, you have skew. Adding workers will not help at all.

Glue's auto scaling lets you set a maximum and have Glue add workers only as the DAG demands them, which is a good default for workloads whose volume varies day to day:

--enable-auto-scaling  true
--number-of-workers    64      # upper bound, not a reservation
--worker-type          G.2X

Cost control

At the time of writing, Glue Spark jobs bill around $0.44 per DPU-hour in us-east-1, per second, with a one-minute minimum. Prices vary by region and change — check the current figures before building a business case on them. What does not change is where the waste comes from:

Data quality gates

A pipeline that delivers wrong numbers on time is worse than one that is late, because nobody finds out for a quarter. Assert your assumptions in the pipeline, and fail loudly.

Glue Data Quality expresses rules in DQDL and runs them as part of the job:

Rules = [
    RowCount > 100000,
    IsComplete "event_id",
    IsUnique "event_id",
    IsComplete "user_id",
    Completeness "session_id" >= 0.99,
    ColumnValues "event_type" in ["view", "click", "add_to_cart", "purchase"],
    ColumnValues "revenue" >= 0,
    StandardDeviation "revenue" < 500
]

The rules that earn their keep are rarely the exotic ones. They are: did we get roughly the row count we expected, is the primary key actually unique, and are the enum columns still the enums we think they are. Those three catch most upstream breakage.

Observability

Glue emits to CloudWatch by default, but the defaults are not enough to debug a slow job. Turn these on:

--enable-metrics                        true
--enable-continuous-cloudwatch-log      true
--enable-spark-ui                       true
--spark-event-logs-path                 s3://delantech-logs/glue/spark-events/
--enable-observability-metrics          true

The Spark event logs matter most. With --spark-event-logs-path set, you can start a Spark History Server against that bucket and inspect any past run's DAG, stage timings and task distribution — which is the only reliable way to distinguish skew from under-parallelism after the fact.

The four signals worth alerting on:

• Job duration against a rolling baseline — a job that doubles in runtime is telling you something changed upstream.
• Rows written against expectation — catches silent partial loads.
• Failed task count — retries that eventually succeed still indicate instability.
• Freshness of the target table — the only metric your stakeholders actually feel.

CI/CD and testing

Glue jobs are code. They belong in version control, with tests, deployed by a pipeline — not edited in the console.

resource "aws_glue_job" "events_curate" {
  name              = "events-curate"
  role_arn          = aws_iam_role.glue.arn
  glue_version      = "4.0"
  worker_type       = "G.2X"
  number_of_workers = 24

  command {
    script_location = "s3://delantech-artifacts/jobs/events_curate.py"
    python_version  = "3"
  }

  default_arguments = {
    "--job-bookmark-option"                = "job-bookmark-enable"
    "--enable-metrics"                     = "true"
    "--enable-spark-ui"                    = "true"
    "--spark-event-logs-path"              = "s3://delantech-logs/glue/spark-events/"
    "--enable-continuous-cloudwatch-log"   = "true"
    "--enable-auto-scaling"                = "true"
    "--source_database"                    = "analytics"
  }

  execution_property { max_concurrent_runs = 1 }
}

For testing, keep your transformation logic in plain PySpark functions that take and return DataFrames. Then they run on a local Spark session with no Glue involved at all:

# transforms.py — no Glue imports, therefore testable anywhere
def sessionise(events, timeout_minutes=30):
    w = Window.partitionBy("user_id").orderBy("event_ts")
    gap = F.unix_timestamp("event_ts") - F.unix_timestamp(F.lag("event_ts").over(w))
    return events.withColumn(
        "session_id",
        F.sum((gap > timeout_minutes * 60).cast("int")).over(w)
    )

# test_transforms.py
def test_sessionise_splits_on_gap(spark):
    df = spark.createDataFrame([
        ("u1", "2026-02-17 10:00:00"),
        ("u1", "2026-02-17 10:05:00"),   # same session
        ("u1", "2026-02-17 11:30:00"),   # new session
    ], ["user_id", "event_ts"]).withColumn("event_ts", F.to_timestamp("event_ts"))

    out = sessionise(df).collect()
    assert out[0]["session_id"] == out[1]["session_id"]
    assert out[2]["session_id"] != out[1]["session_id"]

A worked example: 2 TB/day of clickstream

Concrete numbers from a shape we deploy regularly. Roughly 2 TB of raw JSON events per day, landing continuously in S3, needing hourly availability in a curated sessions table.

The decisions that mattered most, in order of impact:

1. Compacting before modelling. Running the model job directly against 4,000 tiny files took 34 minutes. Adding a compaction step cost 3 minutes and brought the model job to 9. Net saving: over 60% of total compute.
2. Predicate pushdown on dt/hr. The naive job listed the whole table on every run. Explicit partition predicates removed a listing that had grown to dominate startup.
3. Broadcasting the dimensions. Four dimension joins, all under 50 MB. Broadcasting them eliminated four full shuffles of the fact table.
4. Salting one skewed key. A single sentinel user_id used for unauthenticated traffic held 11% of all rows. Salting it cut the longest stage from 19 minutes to 4.

Anti-patterns we keep finding

• Transforming on ingest. Raw is your undo button. Give it up and every bug becomes unrecoverable.
• Crawlers on a schedule in production. A schema change should be a pull request, not a surprise.
• collect() on anything large. It pulls the entire dataset into the driver. It will work in dev and kill the driver in prod.
• Bare except around job.commit(). Advances the bookmark past data that was never written.
• overwrite mode without partitionOverwriteMode=dynamic. Silently deletes every partition rather than just the ones you wrote.
• One giant job. A 400-line job that ingests, models and serves cannot be retried in part, tested in part, or reasoned about at all.
• Tuning before measuring. Almost every "Spark is slow" ticket we take is skew or small files, and neither is fixed by config.

That last item is worth dwelling on. Open the Spark UI, find the longest stage, look at the task duration distribution. If the maximum is far above the median, it is skew. If every task is fast but there are 400,000 of them, it is small files. Those two diagnoses cover most of what we see.

Where your own logic goes inside these jobs — DynamicFrame transforms, UDFs, pandas UDFs, Glue Studio custom nodes — is a decision that routinely swings runtime by an order of magnitude. We cover it in How Custom Transformations Work in AWS Glue and PySpark.

The pipeline is only half the picture. Glue runs your compute well; deciding what should orchestrate that compute — Glue's own workflows, Apache Airflow, or Step Functions — is a separate decision with its own trade-offs. We cover it in Apache Airflow vs AWS Glue.