Architecture/CaseStudies/slack-quarry-ssh-to-rest-emr-pipelines

Slack: Eliminating Standing SSH Access from EMR Data Pipelines

Problem + constraints

Slack's data pipelines submitted jobs to Amazon EMR clusters over SSH — Airflow operators opened persistent SSH sessions to cluster nodes to launch and manage Spark/YARN jobs. That design had two compounding problems. Security-wise, it meant standing SSH access to production EMR clusters across eight independent data regions was a baseline requirement just to run routine data jobs — a broad, hard-to-audit attack surface for infrastructure that didn't need interactive shell access, only "run this job and tell me how it went." Operationally, SSH-based execution tied job lifecycle to the client's SSH connection: a dropped connection, a client restart, or network flakiness could leave jobs in an ambiguous state, and there was no server-side, queryable record of what was submitted, running, or failed — observability was only as good as whatever the SSH session happened to capture.

Fixing this meant touching 700+ Airflow operators already in production, spanning every team that ran data pipelines, without breaking critical workloads that couldn't tolerate downtime during the transition.

Solution

Slack built Quarry, a REST-driven orchestration layer that replaced SSH-based job submission across EMR/YARN, Trino, and Snowflake with a unified HTTP API:

Server-side job lifecycle. Instead of an SSH session driving job execution end-to-end, Airflow submits a job over HTTP, gets back a job ID, and polls or is notified of status — submission, tracking, and cancellation are all server-side operations decoupled from whether the original client connection is still alive.
Token-based service-to-service auth replaced SSH key distribution, which had been the actual credential-management burden behind the SSH approach — keys to rotate, distribute, and audit across every node and every team.
REST API logging as the audit trail, giving Slack a queryable, centralized record of what ran, submitted by whom, replacing whatever fragments of an audit trail SSH sessions happened to leave behind.
Incremental rollout — phased operator deprecations and staged validation across environments, tracked via Airflow metadata dashboards showing which operators were still SSH-dependent, executed over three quarters with cross-team coordination to sequence migration order.

The end state: direct SSH access to production EMR clusters was eliminated entirely across all eight regions, with the 700+ operator migration completed without downtime to critical workloads.

What to steal

A standing-access credential (SSH keys to prod) is a security cost even when nothing goes wrong with it. The attack surface Slack closed wasn't "SSH got exploited" — it was that SSH access existed at all as a default way to run routine jobs, when the actual need was narrower: submit a job, track it, cancel it. Replacing an over-broad access pattern with a narrow API that does exactly what's needed is a generalizable security move, independent of whether the broad pattern was ever actually breached.
Decoupling execution from the client connection fixes both security and operational reliability with one change. Server-side job lifecycle wasn't purely a security fix — it also means a dropped connection no longer leaves a job in an ambiguous state, and status becomes a queryable API rather than something reconstructed from session logs. Look for changes that pay down two categories of debt (security posture and operational fragility) with a single piece of work, since coordinating org-wide migrations is what you're actually budget-constrained on.
A dashboard tracking "what's not migrated yet" is what makes a 700-operator migration tractable. Without a live, queryable view of remaining SSH-dependent workflows, a migration this size either stalls (nobody can tell what's left) or gets declared done prematurely (someone assumes it's finished because the loud complaints stopped).
Sequence a large migration in phases with staged validation, not as one org-wide flag day. Three quarters, phased deprecations — the slow pace is what makes "without downtime to critical workloads" possible; a faster timeline would have traded that guarantee away.

Principal Engineer Lens

This case study is a clean instance of the general architecture-review question "does this access pattern grant more than the operation actually needs?" SSH access was never really about SSH — it was standing infrastructure access justified by "that's how we've always submitted jobs," and the fix was to name the actual operations needed (submit, track, cancel) and build an interface that grants exactly those, nothing more. That's the same underlying move as replacing a shared database credential with scoped API tokens, or a broad IAM role with per-action least privilege — the specific technology differs, but the review question is identical: "what's the minimum surface that satisfies the actual use case, and why does the current design grant more than that?" For a Principal-level engineer, the harder and more valuable part of this story isn't the REST API design — it's the migration mechanics of retiring 700+ existing call sites without downtime, which is the actual skill that scales: designing the better interface is usually the easy 20%, and sequencing a safe, observable, multi-quarter migration off the old one is the hard 80% that most designs never get funded to finish.

Reel Script

Setup: Slack's data pipelines ran on Airflow operators that submitted jobs to production EMR clusters over SSH — meaning standing SSH access to production, across eight regions, was baseline infrastructure just to run a routine data job.

Concept walkthrough: Explain the two problems SSH access created at once: a broad, hard-to-audit security surface (SSH keys to prod, distributed and rotated across every node and team), and an operational fragility problem (job lifecycle tied to a client connection that could drop, with no server-side, queryable record of what ran). Then introduce Quarry: a REST layer where Airflow submits a job over HTTP, gets a job ID back, and the job's lifecycle lives server-side from then on — decoupled from whether the original connection survives.

Real example / case study tie-in: Walk the actual migration: 700+ Airflow operators, phased deprecation over three quarters, an Airflow metadata dashboard tracking which operators were still SSH-dependent so the team always knew what was left, token-based service auth replacing SSH key distribution, and REST API logging becoming the audit trail. End state: SSH access to production EMR eliminated across all eight regions, with no downtime to critical workloads during the transition.

Tradeoffs & alternatives: Contrast a fast, all-at-once cutover (would have finished sooner, but risks breaking critical workloads with no rollback window) against Slack's phased, dashboard-tracked migration (slower, but each phase is independently validated and safe to pause). Also note the alternative they didn't take — patching SSH access with better key rotation or bastion hosts — which would have reduced risk incrementally but left the fundamental over-broad access pattern in place.