AWS Analytics
Contents
AWS Analytic Services
Useful Informations
AWS๋ ๋ถ์์ ์ํด Data Lake๋ฅผ ๋น ๋ฅด๊ณ ์ฝ๊ฒ ๊ตฌ์ถํ๊ณ ๊ด๋ฆฌํ๋๋ฐ ํ์ํ ๋ชจ๋ ๊ฒ์ ์ ๊ณตํ๋ ํตํฉ ์๋น์ค ์ ํ๊ตฐ์ ์ ๊ณตํ๋ค
Data Lake ๋ฐ ๋ถ์ solution ๊ตฌ์ถ์ ์ํ ๊ฐ์ฅ ํฌ๊ด์ ์ด๊ณ ์์ ํ๊ณ ํ์ฅ ๊ฐ๋ฅํ๋ฉฐ ๋น์ฉ ํจ์จ์ ์ธ Service portfolio๋ค
Amazon S3๋ web site, mobile app, enterprise application, IoT sensor, data from device ๋ฑ ์ด๋์๋ ๋ชจ๋ ์ ํ์ data๋ฅผ ์ ์ฅํ๊ณ ๊ฒ์ํ ์ ์๋๋ก ๊ตฌ์ถ๋์๋ค
๋ชจ๋ ์์ data๋ฅผ ์ ์ฅ ๋ฐ ๊ฒ์ํ ์ ์๋๋ก ํ์ํ ๊ฐ์ฉ์ฑ์ ๊ฐ์ถ๊ณ ์ ์๋์์ผ๋ฉฐ 99.999999% ์ ๋ด๊ตฌ์ฑ์ ์ ๊ณตํ๋๋ก ์ ์๋์๋ค
AWS ๋ถ์ ์๋น์ค
Data Lakes and Analytics on AWS
Data Lake ๋ฐ Analytic Solution ๊ตฌ์ถ์ ์ํ ๊ฐ์ฅ ํฌ๊ด์ ์ด๊ณ ์์ ํ๊ณ ํ์ฅ ๊ฐ๋ฅํ๋ฉฐ ๋น์ฉ ํจ์จ์ ์ธ Service Portfolio
AWS๋ ๋ถ์์ ์ํด
Data Lake (์ ์ ๋์ง ์์ raw data)
๋ฅผ ๋น ๋ฅด๊ณ ์ฝ๊ฒ ๊ตฌ์ถํ๊ณ ๊ด๋ฆฌํ๋๋ฐ ํ์ํ ๋ชจ๋ ๊ฒ์ ์ ๊ณตํ๋ ํตํฉ ์๋น์ค ์ ํ๊ตฐ์ ์ ๊ณตํจData Lakes on AWS ๋ ๊ธฐ์กด์
Data Silo (๊ฒฉ๋ฆฌ๋ local data)
๋ฐData Warehouse
๊ฐ ์ํํ ์ ์๋ ๋ฐฉ์์ผ๋ก ๋ค์ํ ์ ํ์ data์ ๋ถ์ ๊ธฐ๋ฒ์ ๊ฒฐํฉํ์ฌ ๋ณด๋ค ์ฌ์ธต์ ์ธ ํต์ฐฐ๋ ฅ์ ์ป๊ธฐ ์ํด ํ์ํ ๊ท๋ชจ, ๋ฏผ์ฒฉ์ฑ ๋ฐ ์ ์ฐ์ฑ์ ์ฒ๋ฆฌํ ์ ์์AWS๋ security & governance๋ฅผ ์ ํํ์ง ์์ผ๋ฉด์ ๋ชจ๋ ๊ด๋ จ data์ ์ฝ๊ฒ access ํ ์ ์๋ ๊ฐ์ฅ ๊ด๋ฒ์ํ ๋ถ์ ๋ฐ machine learning service๋ฅผ ๊ณ ๊ฐ์๊ฒ ์ ๊ณตํจ

Data Movement
: Import your data from on premises, and in real-time
Data Lake
: Store any type of data securely, from gigabytes to exabytes
Analytics
: Analyze your data with the broadest selection of analytics services
Machine Learning
: Predict future outcomes, and prescribe actions for rapid response
Data Lake
Data ๊ฐ Cloud์ ๋ํ ์ค๋น๊ฐ ๋๋ฉด AWS ์์
Amazon S3
๋ฐAmazon Glacier
๋ฅผ ์ฌ์ฉํ์ฌ data๋ฅผ ๋ชจ๋ ํ์์ผ๋ก ์์ ํ๊ฒ ๊ทธ๋ฆฌ๊ณ ๋ฐฉ๋ํ ๊ท๋ชจ๋ก ์ฝ๊ฒ ์ ์ฅํ ์ ์์์ต์ข ์ฌ์ฉ์๊ฐ ๋ถ์์ ์ฌ์ฉํ ๊ด๋ จ ๋ฐ์ดํฐ๋ฅผ ์ฝ๊ฒ ์ฐพ์ ์ ์๋๋ก
AWS Glue
๋ ์ฌ์ฉ์๊ฐ ๊ฒ์ํ๊ณ query ํ ์ ์๋ ๋จ์ผ catalog๋ฅผ ์๋์ผ๋ก ์์ฑํจ
Storage - Amazon S3
Amazon S3
Amazon S3๋ data access๋ฅผ ์ํ ์์ ํ๊ณ ํ์ฅ์ฑ์ด ๋ฐ์ด๋ ๋ฐ๋ฆฌ์ด ์ง์ฐ์๊ฐ์ ๊ฐ์ฒด ์คํ ๋ฆฌ์ง์ด๋ค
S3 Select
๋ data ์ฝ๊ธฐ ๋ฐ ๊ฒ์์ ์ค์ ์ ๋ ์ผ๋ก์จ ์๋ต ์๊ฐ์ ์ต๋ 400%๊น์ง ๋จ์ถํจS3๋ ๊ฐ์ฅ ์๊ฒฉํ ๊ท์ ์๊ตฌ ์ฌํญ๊น์ง๋ ์ถฉ์กฑํ๋ ํฌ๊ด์ ์ธ ๋ณด์ ๋ฐ ๊ท์ ์ค์ ๊ธฐ๋ฅ์ ์ ๊ณตํจ
Backup & Archive - Amazon Glacier
Amazon Glacier
Amazon Glacier๋ data๋ฅผ ์ ๋ถ ๋ด์ access ํ ์ ์๋ ์ฅ๊ธฐ backup ๋ฐ archive๋ฅผ ์ํ ์์ ํ๊ณ ๋ด๊ตฌ์ฑ ์๋ ๋งค์ฐ ์ ๋ ดํ storage ์ด๋ค
Glacier Select
๋ ํ์ํ data๋ง ์ฝ๊ณ ๊ฒ์ํจ๊ณ ๊ฐ์ ์๋ณ GB๋น 0.004 USD์ ์ ๋ ดํ ์๊ธ์ผ๋ก data๋ฅผ ์ ์ฅํ ์ ์์ผ๋ฏ๋ก On-premise solution๊ณผ ๋น๊ตํ๋ฉด ์๋นํ ๋น์ฉ ์ ๊ฐ์ ๊ธฐ๋ํ ์ ์์
Data Catalog - AWS Glue
AWS Glue
AWS Glue๋ data lake์ data๋ฅผ ๊ฒ์ํ ์ ์๋๋ก data catalog๋ฅผ ์ ๊ณตํ๊ณ ๋ถ์์ ์ํด data๋ฅผ ์ค๋นํ๋๋ก **Extract / Transform / Load (ETL)**๋ฅผ ์ํํ ์ ์๋ ์์ ๊ด๋ฆฌํ ์๋น์ค ์ด๋ค
Data catalog๋ ๋ชจ๋ data ์์ฐ์ ๋ํ permanent meta-data storage๋ก ์๋ ์์ฑ๋๋ฏ๋ก ๋ชจ๋ data๋ฅผ ๊ฒ์, query ํ ์ ์์
Amazon Athena (๋ํ์ ๋ถ์)
Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL
Amazon Athena๋ Standard SQL query๋ฅผ ์ฌ์ฉํด S3์ Glacier์ ์๋ data๋ฅผ ์ง์ ๊ฐํธํ๊ฒ ๋ถ์ํ ์ ์๊ฒ ํด์ค
Athena๋ serverless์ด๋ฏ๋ก, ์ค์ ํ๊ฑฐ๋ ๊ด๋ฆฌํ infra๊ฐ ์์
Data๋ฅผ ์ฆ์ queryํ๊ณ , ๋ช ์ด๋ด์ ๊ฒฐ๊ณผ๋ฅผ ์ป๊ณ , ์คํํ query์ ๋ํด์๋ง ๋น์ฉ์ ์ง๋ถํจ
Amazon S3์ ์ ์ฅ๋ data๋ฅผ ๊ฐ๋ฆฌํค๊ณ schema๋ฅผ ์ ์ํ ํ Standard SQL์ ์ฌ์ฉํ์ฌ query๋ฅผ ์์ํ๊ธฐ๋ง ํ๋ฉด ๋จ!
๋๋ถ๋ถ ๊ฒฐ๊ณผ๊ฐ ์ ์ด ์ด๋ด์ ์ ๊ณต๋จ
Amazon CloudSearch (๊ด๋ฆฌํ ๊ฒ์ ์๋น์ค)
Amazon CloudSearch is a managed service in the AWS Cloud that makes it simple and cost-effective to set up, manage, and scale a search solution for your website or application
Amazon CloudSearch supports 34 languages and popular search features such as highlighting, auto-complete, and geospatial search
Amazon EMR
Easily run and scale Apache Spark, Hadoop, HBase, Presto, Hive, and other big data frameworks
Big Data
Amazon EMR ์ ๊ด๋ฆฌํ ์๋น์ค๋ก์ ๋๋์ data๋ฅผ ์ฝ๊ณ ๋น ๋ฅด๋ฉฐ ๋น์ฉ ํจ์จ์ ์ผ๋ก ์ฒ๋ฆฌํ ์ ์์
๋ฐ์ดํฐ ์์ง๋์ด๋ง, ๋ฐ์ดํฐ ๊ณผํ ๊ฐ๋ฐ ๋ฐ ํ์ ์ ์ํ ๊ด๋ฆฌํ EMR Notebook
๊ฐ project๋ version release ํ 30์ผ ์ด๋ด์ EMR์์ update ๋๋ฏ๋ก community๋ก๋ถํฐ ๊ฐ์ฅ ์ต์ ์ ์ต๊ณ project๋ฅผ ์์ฝ๊ฒ ์ป์ ์ ์์
Amazon Elasticsearch Service
Amazon Elasticsearch Service is a fully managed service that makes it easy for you to deploy, secure, and run Elasticsearch cost effectively at scale

์ด์ ๋ถ์
Application monitoring, log ๋ถ์, click stream ๋ถ์๊ณผ ๊ฐ์ ์ด์ ๋ถ์์ ๊ฒฝ์ฐ Amazon Elasticsearch Service๋ฅผ ์ฌ์ฉํ๋ฉด ๊ฑฐ์ ์ค์๊ฐ์ผ๋ก data๋ฅผ ๊ฒ์, ํ์ filtering, ์ง๊ณ ๋ฐ ์๊ฐํ ํ ์ ์์
Amazon Elasticsearch Service๋ Elasticsearch์ ๊ฐํธํ API ๋ฐ ์ค์๊ฐ ๋ถ์ ๊ธฐ๋ฅ๊ณผ ๋๋ถ์ด production workload์ ํ์ํ ๊ฐ์ฉ์ฑ, ํ์ฅ์ฑ, ๋ณด์์ฑ์ ์ ๊ณตํจ
์ค์๊ฐ ๋ถ์
Amazon Kinesis
๋ฅผ ์ฌ์ฉํ๋ฉด IoT telemetry data, application log, web site click stream๊ณผ ๊ฐ์ streaming data๋ฅผ ๊ฐํธํ๊ฒ ์์ง, ์ฒ๋ฆฌ ๋ฐ ๋ถ์ํ ์ ์์๋ชจ๋ data๊ฐ ์์ง๋ ํ์์ผ ์ฒ๋ฆฌ๋ฅผ ์์ํ ์ ์๋ ๊ฒ์ด ์๋๋ผ data lake์ data๊ฐ ์์ ๋๋ ๋๋ก ์ฒ๋ฆฌ ๋ฐ ๋ถ์ํ์ฌ ์ค์๊ฐ์ผ๋ก ๋์ํ ์ ์์
Amazon Managed Streaming for Apache Kafka (Amazon MSK)
Amazon MSK is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data

Apache Kafka is an open-source platform for building real-time streaming data pipelines and applications
With Amazon MSK, you can use native Apache Kafka APIs to populate data lakes, stream changes to and from databases, and power machine learning and analytics applications
Amazon MSK๋ฅผ ์ฌ์ฉํ๋ฉด Apache Kafka infra ๊ด๋ฆฌ์ ๋ํ ์ ๋ฌธ์ฑ ์์ด๋ Apache Kafka์์ ํธ๋ฆฌํ๊ฒ production application์ ๊ตฌ์ถํ๊ณ ์คํํ ์ ์์
Infra ๊ด๋ฆฌ ์๊ฐ์ ์ค์ด๊ณ , ๋ ๋ง์ ์๊ฐ์ application ๊ฐ๋ฐ์ ํ์ฉํ ์ ์์
Streaming data๋ฅผ ์ง์์ ์ผ๋ก ๋ถ์ํ์ฌ ๊ด๋ จ ๋์ ์กฐ์น๋ฅผ ์ทจํ๋ application์ฉ data source๋ก Apache Kafka๋ฅผ ์ฌ์ฉํจ
Amazon Redshift
The most popular and fastest cloud data warehouse

Data warehousing
Amazon Redshift๋ petabite์ ์ ํ ๋ฐ์ดํฐ์ ๋ํด ๋ณต์กํ ๋ถ์ query๋ฅผ ์คํํ ์ ์๋ ๊ธฐ๋ฅ์ ์ ๊ณต
๋ถํ์ํ data ์ด๋ ์์ด S3์ ์ ํ/๋น์ ํ data์ ๋ํด ์ง์ SQL query๋ฅผ ์คํํ๋
Redshift Spectrum
์ ํฌํจํจAmazon Redshift๋ ๊ธฐ์กด solution ๋น์ฉ์ด 1/10๋ ๋์ง ์์!
์๊ฐ๋น 0.25 USD
์ฐ๊ฐ 1,000 USD
Amazon QuickSight
Amazon QuickSight is a fast, cloud-powered business intelligence service that makes it easy to deliver insights to everyone in your organization

๋น ๋ฅธ ๋น์ฆ๋์ค ๋ถ์ ์๋น์ค & ๋์๋ณด๋ ๋ฐ ์๊ฐํ
๋์๋ณด๋ ๋ฐ ์๊ฐํ์ ๊ฒฝ์ฐ, Amazon QuickSight๋ ๋น ๋ฅด๊ณ ๊ฐ๋ ฅํ Cloud ๊ธฐ๋ฐ ๋น์ฆ๋์ค ๋ถ์ ์๋น์ค๋ฅผ ์ ๊ณตํ๋ฏ๋ก ๋ชจ๋ browser or mobile ์ฅ์น์์ access ํ ์ ์๋ ์๊ฐํ ๋ฐ ํ๋ถํ ๋์๋ณด๋๋ฅผ ์ฝ๊ฒ ์์ฑํ ์ ์์
Amazon Data Pipeline
AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals
With AWS Data Pipeline, you can regularly access your data where itโs stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR
AWS Data Pipeline helps you easily create complex data processing workloads that are fault tolerant, repeatable, and highly available
You donโt have to worry about ensuring resource availability, managing inter-task dependencies, retrying transient failures or timeouts in individual tasks, or creating a failure notification system
AWS Data Pipeline also allows you to move and process data that was previously locked up in on-premises data silos.
AWS Glue (Prepare and Load Data
AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics
๊ฐํธํ๊ณ ์ ์ฐํ๋ฉฐ ๋น์ฉ ํจ์จ์ ์ธ ETL
You can create and run an ETL job with a few clicks in the AWS Management Console
You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog
Once cataloged, your data is immediately searchable, queryable, and available for ETL
์๋ ๋ฐฉ์
๋ฐ์ดํฐ ์๋ณธ๊ณผ ๋ฐ์ดํฐ ๋์์ ์ ํํจ
AWS Glue๊ฐ Scala or Python์ผ๋ก ETL code๋ฅผ ์์ฑํ์ฌ ์๋ณธ์ผ๋ก๋ถํฐ data๋ฅผ ์ถ์ถํ๊ณ , ํด๋น data๋ฅผ schema์ ๋ง์ถฐ ๋ณํํ๊ณ , ์ด๋ฅผ ๋์์ผ๋ก loadใ ใ
์ฌ์ฉ์๋ console, ์ ํธํ๋ IDE or notebook์ ์ฌ์ฉํ์ฌ ํด๋น ์ฝ๋๋ฅผ ํธ์ง, debugging, test ํ ์ ์์
Use Cases
1. Queries Against an Amazon S3 Data Lake

2. Analyze Log Data in Your Data Warehouse

3. Unified View of Your Data Across Multiple Data Stores

4. Event-driven ETL Pipelines

AWS Lake Formation
AWS Lake Formation is a service that makes it easy to set up a secure data lake in days

A data lake is a centralized, curated, and secured repository that stores all your data, both in its original form and prepared for analysis.
A data lake enables you to break down
data silos
and combine different types of analytics to gain insights and guide better business decisionsCreating a data lake with Lake Formation is as simple as defining data sources and what data access and security policies you want to apply
Lake Formation
then helps youcollect and catalog data from databases and object storage
move the data into your new Amazon S3 data lake
clean and classify your data using machine learning algorithms
secure access to your sensitive data
Your users can access a centralized data catalog which describes available data sets and their appropriate usage
Your users then leverage these data sets with their choice of analytics and machine learning services, like Amazon Redshift, Amazon Athena, and (in beta) Amazon EMR for Apache Spark
Lake Formation builds on the capabilities available in AWS Glue.
How it works
Identify existing data stores in S3 or relational and NoSQL databases, and move the data into your data lake
Crawl, catalog, and prepare the data for analytics
Then provide your users secure self-service access to the data through their choice of analytics services
Other AWS services and third-party applications can also access data through the services shown
Lake Formation manages all of the tasks in the orange box and is integrated with the data stores and services shown in the blue boxes.
Summary
AWS๋ ๋ถ์์ ์ํด Data Lake๋ฅผ ๋น ๋ฅด๊ณ ์ฝ๊ฒ ๊ตฌ์ถํ๊ณ ๊ด๋ฆฌํ๋๋ฐ ํ์ํ ๋ชจ๋ ๊ฒ์ ์ ๊ณตํ๋ ํตํฉ ์๋น์ค ์ ํ๊ตฐ์ ์ ๊ณตํจ
Amazon S3๋ Data access๋ฅผ ์ํ ์์ ํ๊ณ ํ์ฅ์ฑ์ด ๋ฐ์ด๋๊ณ ๋ฐ๋ฆฌ์ด ์ง์ฐ ์๊ฐ์ ๊ฐ์ฒด ์คํ ๋ฆฌ์ง ์ด๋ค
Last updated
Was this helpful?