2. Analytics

Services in this domain enable the processing and analysis of data at scale, ranging from serverless SQL queries to managed big data clusters.

Amazon Athena

Service Introduction: An interactive, serverless query service that allows for the analysis of data directly in Amazon S3 using standard SQL.

Common Usage: Decouples storage from compute; ideal for ad-hoc analysis of logs or "pay-per-query" data exploration without ETL.

Project Examples:

• Building a serverless data lake analysis layer.
• Querying VPC Flow Logs or CloudTrail logs stored in S3 for security auditing.

AWS Data Exchange

Service Introduction: A service that simplifies finding, subscribing to, and using third-party data from various providers in the AWS Cloud.

Common Usage: Streamlines the ingestion of external data sets (e.g., weather, financial) into AWS-native analytics pipelines via S3.

Project Examples:

• Integrating external market data into a SageMaker ML model.
• Aggregating global health statistics for comparative research.

Amazon Data Firehose

Service Introduction: A fully managed service for delivering real-time streaming data to destinations like S3, Redshift, or OpenSearch.

Common Usage: Simplifies streaming ingestion by handling data transformation, compression, and buffering automatically.

Project Examples:

• Streaming application logs to an S3 bucket for long-term archival.
• Real-time loading of IoT telemetry into Amazon Redshift for dashboarding.

Amazon EMR

Service Introduction: A managed cluster platform that simplifies running big data frameworks like Apache Spark, Hive, and Presto.

Common Usage: Used for complex, long-running batch processing and massive-scale data transformations on petabyte-scale datasets.

Project Examples:

• Processing genomic sequencing data using Spark.
• Running massive-scale clickstream analysis for daily business reports.

AWS Glue

Service Introduction: A serverless data integration service that provides ETL, data preparation, and a centralized Data Catalog.

Common Usage: Discovering schema and transforming data between different formats (e.g., CSV to Parquet) to optimize for query performance.

Project Examples:

• Automated schema discovery for an S3-based data lake.
• Building a serverless ETL pipeline to move data from RDS to Redshift.

AWS Glue Studio

Service Introduction: A visual interface within AWS Glue for building, running, and monitoring ETL jobs.

Common Usage: Provides a drag-and-drop canvas for data engineers to construct ETL pipelines, which Glue Studio then generates as Python or Scala code behind the scenes. It's still a code-generating tool aimed at engineers building pipelines, not a code-free "clean my data" tool for business analysts — that's Glue DataBrew.

Project Examples:

• Visually building a job that joins two S3 datasets and writes the result to Redshift.
• Monitoring the run history and performance of existing Glue ETL jobs.

AWS Glue DataBrew

Service Introduction: A visual data preparation tool that lets you clean and normalize data without writing code.

Common Usage: Aimed at business analysts and data scientists (not engineers) — you build a "recipe" of point-and-click transformations (handling missing values, fixing formatting, normalizing columns) applied directly to data in S3, and DataBrew handles execution. This is the standard answer whenever a question asks for a code-free, visual data-prep interface, distinct from Glue Studio (still code-generating, engineer-focused) and Athena/SageMaker (require SQL/Python).

Exam Example (SAA-C03 pattern): "An analytics company needs a code-free, visual interface to allow business analysts to clean and prepare data sets from their S3 data lake for machine learning. Which AWS service is specifically designed for this 'data prep' use case?" Answer choices: Amazon Athena / AWS Glue DataBrew / AWS Glue Studio / Amazon SageMaker Notebooks.

• Correct Answer: AWS Glue DataBrew — A visual data preparation tool that allows analysts and data scientists to clean and normalize data without writing code.
• Why not Athena? — A serverless query service that requires SQL knowledge; no visual, code-free "recipe" interface for data cleaning.
• Why not Glue Studio? — Provides a visual interface for building ETL jobs, but is primarily focused on data engineers creating jobs that generate code (Python/Scala).
• Why not SageMaker Notebooks? — Used by data scientists for experimentation and require writing Python or R code for data transformation.

Project Examples:

• Letting business analysts standardize inconsistent date formats across a raw S3 dataset before it's used in ML training.
• Building a reusable cleaning "recipe" for a recurring, messy CSV export from a partner system.

Amazon Kinesis

Service Introduction: A suite of services (Data Streams, Video Streams) for collecting, processing, and analyzing real-time, streaming data.

Common Usage: Enables low-latency ingestion of data; Data Streams provide "shards" for manual scaling and replayable data records.

Project Examples:

• Real-time monitoring of application performance metrics with millisecond latency.
• Ingesting high-frequency social media feeds for immediate sentiment analysis.

AWS Lake Formation

Service Introduction: A service that simplifies the setup, security, and management of a data lake by defining centralized access control.

Common Usage: Implements fine-grained access control (column/row level) across multiple analytics services like Athena and Redshift.

Project Examples:

• Establishing a secure, multi-tenant data lake for different business units.
• Enforcing PII data masking for specific IAM roles within a query environment.

Analytics: Kinesis vs. Firehose and Integration with AWS Glue

Both Amazon Kinesis and Amazon Data Firehose are core services within the Analytics category used for handling streaming data. While they are often used together, they serve different roles in a data architecture, particularly when integrated with AWS Glue for transformation.

Amazon Kinesis vs. Amazon Data Firehose

The primary difference lies in how they handle data processing and delivery:

• Amazon Kinesis (Data Streams):

  • Processing Model: Designed for real-time, low-latency data ingestion. It uses a "shard" model where you manage capacity by adding or removing shards.
  • Data Retention: It stores data for a set period (typically 24 hours to 7 days), allowing multiple consumers to read the same data simultaneously or replay it if a process fails.
  • Use Case: Best for complex, custom real-time applications where data order within a shard is critical and multiple independent applications need to process the same stream.

• Amazon Data Firehose:

  • Processing Model: A fully managed, serverless service that automatically scales to match data throughput. It is focused on loading data into destinations rather than complex real-time processing.
  • Delivery: It buffers incoming data (by size or time) and delivers it to destinations like Amazon S3, Amazon Redshift, or Amazon OpenSearch Service.
  • Use Case: Best for near real-time ingestion where the primary goal is to get data into a data lake or warehouse for analysis with the least operational effort.

Integration with AWS Glue

AWS Glue is a data transformation service used to prepare and load data for analytics. It integrates with these streaming services to provide structure and transformation to raw data.

• With Amazon Kinesis (Streaming ETL):

  • AWS Glue can run Streaming ETL jobs that consume data directly from a Kinesis Data Stream.
  • Use Case: You use this when you need to perform complex transformations (like joining streams or cleaning data) on-the-fly before the data is saved to its final destination.

• With Amazon Data Firehose (Schema Discovery):

  • Firehose can use the AWS Glue Data Catalog to understand the schema of the incoming data.
  • Use Case: You use this to convert incoming data formats (like converting raw JSON to Apache Parquet or ORC) before Firehose delivers it to Amazon S3. This optimizes the data for faster querying by services like Amazon Athena.

Comparison Summary

Feature	Amazon Kinesis	Amazon Data Firehose
Primary Focus	Real-time ingestion & custom processing.	Near real-time loading & delivery.
Scaling	Manual/Auto-scaling of shards.	Serverless; scales automatically.
Data Replay	Supported (via retention period).	Not supported (once delivered, it's gone).
Glue Role	Glue Streaming ETL for live transformation.	Glue Data Catalog for format conversion.
Destination	Custom consumers (Lambda, EC2).	S3, Redshift, OpenSearch, etc.

In short: Use Kinesis if you need a persistent stream for multiple real-time apps to "talk" to; use Firehose if you simply want to dump a data stream into S3 or a database. Use AWS Glue with either when that data needs to be cleaned, structured, or converted to a different format for long-term storage.

AWS Glue vs. Amazon Data Firehose (exam-critical, easy to conflate)

These are complementary, not competing services — they solve different problems:

• AWS Glue — A serverless ETL and data catalog service. It discovers schema, transforms data (batch or streaming ETL jobs, Spark-based), and maintains the Data Catalog (metadata table definitions used by Athena, Redshift Spectrum, EMR, etc.). The answer whenever a question is about transforming, cleaning, joining, or cataloging data.
• Amazon Data Firehose — A fully managed delivery service for streaming data. It buffers and loads a continuous stream into a destination (S3, Redshift, OpenSearch) with the least possible operational overhead. The answer whenever a question is about getting a stream into a destination with minimal effort.
• Where they overlap: Firehose can call on the Glue Data Catalog to look up schema and convert incoming JSON to Parquet/ORC during delivery — but that's Firehose using Glue for a narrow format-conversion task, not Glue doing general-purpose ETL. If the requirement is actual joins, complex transforms, or processing a live stream on-the-fly, that's a Glue Streaming ETL job consuming directly from Kinesis, not something Firehose does on its own.

Decision rule: "Loading/delivering a stream into S3 or a warehouse with least effort" → Firehose. "Transforming, cataloging, or running complex ETL on data" → Glue.

Exam Example (SAA-C03 pattern): "A data engineer needs to ingest real-time streaming data and deliver it to an Amazon S3 bucket for long-term storage with the least amount of operational effort. The data must be converted from JSON to Apache Parquet during ingestion. Which service is most appropriate?" Answer choices: AWS Glue ETL / Amazon MSK / Amazon Kinesis Data Streams / Amazon Data Firehose.

• Correct Answer: Amazon Data Firehose — Fully managed, serverless delivery service that can automatically convert formats (JSON to Parquet) before landing data in S3.
• Why not Glue ETL? — Glue handles batch or streaming ETL but adds more operational complexity for simple ingestion-and-delivery than a managed Firehose stream.
• Why not MSK? — Provides a managed Kafka environment but requires significant configuration for producers, consumers, and connectors to achieve S3 delivery and format conversion.
• Why not Kinesis Data Streams? — Handles real-time ingestion but requires custom consumers (Lambda, EC2) to handle delivery and transformation to S3.

Amazon Managed Streaming for Apache Kafka (Amazon MSK)

Service Introduction: A fully managed, highly available service for running Apache Kafka clusters on AWS.

Common Usage: Ideal for migrating existing on-premises Kafka workloads to AWS while maintaining compatibility with the Kafka ecosystem.

Project Examples:

• Decoupling microservices using a high-throughput, managed Kafka message bus.
• Building a real-time event-streaming platform for transaction processing.

Amazon OpenSearch Service

Service Introduction: A managed service for search, log analytics, and real-time application monitoring using OpenSearch and legacy Elasticsearch.

Common Usage: Provides rapid full-text search and visualization (via OpenSearch Dashboards) for operational log data.

Project Examples:

• Implementing a search bar for an e-commerce catalog with sub-second response times.
• Building a centralized log analytics platform for cross-account troubleshooting.

Amazon QuickSight

Service Introduction: A fast, cloud-powered business intelligence service that delivers interactive dashboards and ML-powered insights.

Common Usage: Visualizing data from S3, RDS, Redshift, or on-premises sources for executive and operational reporting. SPICE (exam-critical): QuickSight's in-memory, columnar calculation engine ("Super-fast, Parallel, In-memory Calculation Engine"). Data imported into SPICE is cached so dashboards render fast without re-querying the source on every view, and it auto-scales to thousands of concurrent users viewing the same dashboard without hitting the underlying database — the answer whenever a question mentions "dashboards must handle a large number of concurrent viewers without impacting the source database's performance."

Project Examples:

• Creating sales performance dashboards for regional managers.
• Embedding interactive data visualizations into a customer-facing web portal.

Amazon Redshift

Service Introduction: A fast, fully managed petabyte-scale data warehouse using Columnar storage to optimize OLAP workloads.

Common Usage: Centralizing data from multiple sources for complex SQL joins and long-term historical trend analysis.

Project Examples:

• Consolidating multiple transactional databases into a single source of truth for BI.
• Performing historical analysis on ten years of retail transaction data.