AWS Solution Architect Associate — Part 1

Exam Guide

Scenario Based Exam, Multiple choice questions, Multiple response questions,

Design Secure Applications and Architectures, Designed Cost-Optimized Architectures

Design Resilient Architecture: Design a Multi-tier Architecture, Design High availability and Fault tolerant architecture, Design Decoupling mechanism using AWS services, Choose appropriate resilient storage

Design High-Performing Architecture: Identify Elastic and scalable compute solution for workload, High performing and scalable storage solution, High performing networking solution, High performing database solution

Design Secure Architecture: Design Secure access to Aws Services, Secure application tiers, Select appropriate data security options

Design Cost-optimized Architecture: Identify cost-effective storage solution, Cost effective compute and database services , Cost-optimized network architecture

Building Blocks of AWS: Regions and Availability Zones

AWS Regions: Geographical Location, Each Region consists of 3 or more Availability
zones(AZ connected within 100km range). eu-west-1, eu-west-2. Most Services 
are region scoped. 

Availability Zones: Servers, set of AWS Data Centers, isolated from disasters, connected 
with high bandwidth ultra low latency 

Edge Location (Point of Presence): Aws used to caching content
(Cloud Front and Content Delivery Network) over 215 edge locations

Aws Global Infrastructure: 31 regions, 99 Availability Zones

US East (N. Virginia) is the primary region AWS rolls out all it’s new services

How to choose AWS region?

Compliance with data governance and legal requirements, Security, data never leaves region without explicit permission
Proximity, available services, pricing

Who Owns What in the Cloud

You are likely responsible Security groups, IAM users, Patching EC2, Databases running on EC2

AWS likely responsible Management of data centers, cabling, Security camera, Patching RDS

Compute, Storage, Databases and Networking

Compute: You wouldn't be able to build app without compute power
EC2, Lambda, Elastic Beanstalk

Storage: Giant Disk in the Cloud. S3, (Elastic Block Store)EBS, Elastic File Server(EFS)

Database: Store and retrieve information. RDS, DynamoDB

Networking: Connect between compute, storage and database. VPCs, Direct Connect,
 Route 53, API Gatways, AWS global Accelerator

What is Well-Architected Frameworks

Six Pillers
Operational Excellence: Running and monitoring systems to deliver business value,
and continuelly improving processes and procedures

Performance Efficiency: Using IT and computing resources efficiently
Security: Protecting Information and Systems
Cost-Optimization: Avoiding unnecessary cost
Reliability: Ensure workload perform it's intended function correctly
Sustainability: Minimize the environment impacts of running cloud workloads

What is IAM?

Identity and Access Management is global services. IAM allows you to manage users, and their level of access to the AWS . It works in GLOBAL level

Root Account: Full Administrative access, Should not be shared and used.
Users are people within your org and can be grouped
Groups only contains users but not other group
User don't have to belong to group and one user can belong to multiple groups

IAM: Permissions

Users or groups can assign JSON document called policies. How to Secure AWS Root Account?

Enable MFA to root account
Create Admin Group for your administrators and assign appropriate permission to this group
Create user accout for your administrator and assign user to the admin group

Controlling User Actions with IAM Policy Documents

We assign permission using document in JSON format. It consists of:

Version: Policy Language version
Id: Identifier for policy(optional)
Statement: One or more induvidual statement (required)

Statement consists of: 
Sid: an identifier for the statement
Effect:  whether the statements allows or denied access
Principal: acc/usr/role to which the policy applied to
ACtion: list of actions policy allows or denied
Resource: list of resources to which action applied

IAM: Password policy

Strong Password, set minimum length pass, specific chars, allow to change pass, password expiration, prevent password reuse

IAM: MFA

How to access AWS

AWS Management Console (Protected by Password and MFA)
AWS command line interface (Protected by access keys)
AWS SDK (for code , protected by access keys)

AWS CLI allows you to interact with aws services using through Commands in command-in shell. direct access to public APIsof aws services. You can develop script to manage your resources. Open sources and alternative e to AWS management console.

AWS CLI windows

// access keys under IAM/USERS/username/Security Credentials / create Access Key
// in cmd
~ aws configure
// and add id and key
~ aws iam list-users

//get full path
~ pwd
AWS Cloud Shell

IAM roles for service

Some AWS service will need to perform some action on your behalf. So to do we will assign permission to AWS services with IAM roles.

IAM security tools

IAM Credentials Report(account-level): a report that list all your account’s user and status of their various credentials. IAM> Credentials Report will give csv file to show the details

IAM access advisor (User-level): shows service permission granted to user and when thoes service were last accessed. can use this to revise the policy. IAM>Access Advisor>

IAM Guidelines and best practices

Don't use root account accept for account setup
One physical user = one aws account
Assign user to groups and assign permission to groups
create strong password policy
use MFA
create and use roles for giving permission for AWS services
use access keys for ClI SDK
Audit permissions using credentials report and access advisor 

Users, Groups, Policies, Roles, Security, AWS CLI, SDK, Access keys, Audit

AWS Budget Setup

Alarm for budget usage. If you have not permission to view in IAM account. go to root account> account> IAM user and role access to billing information and activate it.

AWS service for Billing and Cost management, Zero cost budget

EC2

Elastic Compute Cloud — Infrastructure as s service. It mainly consist in the capability of:

Renting Virtual Machine(EC2)
Storing data on virtual drives (EBS)
Distributing load accross Machines (ELB)
Scaling services using Auto Scaling Group(ASG)

EC2 configuration options:

OS, Compute Power and Core (CPU), RAM, Storage Space( Network attached EBS &EFS, hardware EC2 instance store), Network card (Speed of card and public IP address) , Firewalls rules (Security Group), Bootstrap Script (EC2 user data configure at first launch)

EC2 Instance Types: instance, vCPU, Mem, Storage, Network Performance, EBS Bandwidth

Create EC2 Instance :

- Instance Name
- Amazon Machine Image (AMI) (64 bit)
- Instance Type
- Key pair Login (Required for SSH): 
    for private key file format
    -    .pem => windows above 10, linux and mac
    -    .ppk => windows below 10
- Network setting (security group), allow the traffic from internet
- Configure storage
- Userdata to execute on first launch
#!/bin/bash
# Use this for your user data (script from top to bottom)
# install httpd (Linux 2 version)
yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd
echo "<h1>Hello World from $(hostname -f)</h1>" > /var/www/html/index.html

- No. of instances
- get public IPV4 and access from internet

EC2 Instance Types

Can use different instances that are optimize for different use case. That are 7 types of instances.

General Purpose: Great for diversity of workloads. webservers or code repository . Balance between Memory, compute and networking

Computer Optimized: Great for compute intensive task that requre high performance processor. Batch processing servers, Media Transcoding, High performance Web Servers, High Performance Computing (HPC) , Machine Learning, Dedicated Gaming Servers

Memory Optimized: Fast performance for workloads that process large data sets in memory. Relational/Non-Relational databases, Distributed cache services, Realtime processing.

Storage Optimized: High sequential read write access to large data sets on local storage. High frequency Online Transaction Processing(OLTP), SQL, NoSQL databases, Redis, Distributed File Systems, Data warehouse

Accelerated Computing, Instant features, Measuring Instance Performance.

Naming Convention: m5.2xLarge

m: instance class(general)
5: generation
2xLarge: size within the instance class

More at ec2instances.info

Security Groups

Fundamental of Network Security in AWS . Control how traffic is allowed into or out of EC2 instances (like as Firewall) . Only contains allow rules. can reference by IP or security group.

They regulate:
- Access to ports
- Authorize IP ranges
- Control of inbound network
- Control of outbound network

- can be attached to multiple EC2 instances
- Locked down to VPC/Region combination
- Does live outside EC2
- If traffic is blocked EC2 won't see it
- Recommend to create separate SG for SSH access
- All inbound traffic are blocked by default
- All outbound are authorized by default

If application is not accessible(Time out), then it's Security group issue

if Connection refused, it's application error or it's not launched

Reference other Security Group

You can add an existing group to another group

If there is more than one rule for a specific port, we apply the most 
permissive rule. For example, if you have a rule that allows access to TCP 
port 22 (SSH) from IP address 203.0.113.1 and another rule that allows access 
to TCP port 22 from everyone, everyone has access to TCP port 22.

Classic Ports:
22 :SSH (Secure Shell) - Log into a linux shell
21 :FTP (File Transfer Protocol) - Upload files into file share
22 :SFTP (Secure file transfer protocol) - upload file using SSH
80 :HTTP -  access unsecured websites
443 :HTTPS access secured websites
3389 :RDP Remote desktop protocol - Log into a windows instance

vs RDS Databases ports:
PostgreSQL: 5432
MySQL: 3306
Oracle RDS: 1521
MSSQL Server: 1433
MariaDB: 3306 (same as MySQL)
Aurora: 5432 (if PostgreSQL compatible) or 3306 (if MySQL compatible)

SSH : Command line interface utility

SSH allows to control the remote machine all using command line.

SSH can be used in Linux, Mac and Windows ≥10 , PUTTY can be used in windows < 10 or ≥ 10

EC2 Instance Connect: Connect with EC2 instance within browser without SSH keys. Works on Linux 2 AM1. Can do from AWS EC2>Instances> Instance connect dashboard

 ssh -i filename.pem ec2-user@13.233.108.192

EC2 Instance Roles : You can assign role in EC2 instance. Always provide role.

EC2 Instance Purchasing Option

1. On-Demand Instances: Short workload, predictable pricing, pay by second/hour.
2. Spot Instance: Can get up-to 90% discount compared to on-demand . Define max spot price and terminate or stop after reaching current spot price to max spot(2 minute notice period). Used for data analysis, Batch jobs, or workloads that are resilient to failures. But not for Critical jobs or database. You can configure one-time or persistent instance. For Persistent Spot Instance, You have to terminate Spot request and then EC2 instance.
3. Reserved Instance: 72% discount compared to on-Demand, Reserved for 1 & 3 years. Long workloads, You have convertible Reserved instance option. You don’t need control over the underlying hardware.
4. Dedicated Host: Book entire physical server, control instance placement. fully dedicated to your use. You need to meet licensing or compliance requirements.
5. Dedicated Instance: No other customer will share your hardware. You need hardware isolation but don’t require control over the physical server. You don’t have specific licensing requirements tied to physical hardware.
6. Capacity Reservation: Reserve specific capacity in a specific AZ for any duration.

Spot Fleet

Set of Spot Instances + (Optional) On-Demand Instances.

By default, AWS virtualizes servers, so you don’t see the underlying physical hardware details unless you use a specific purchasing option like Dedicated Hosts

VPC (Virtual Private Cloud)

Isolated Network in a Region. Your own cloud network. Based on Region

Subnet

Networks in a network, Your own IP ranges, Each subnet take subsets of VPC IP ranges. Based on AZ

Private IP vs Public IP(IPV4)

IPV4 allows 3.7 billion different address in the public space

[0–255]-[0–255]-[0–255]-[0–255]

Elastic IPs: EC2 instance changed public IP every time when re-started. To make fixed public IP, you need elastic IP (Public IPv4). You can have 5 elastic IP. Asked to AWS to make more then 5 elastic IP. Configure from Elastic IPs tab.

Placement Groups

A placement group is a logical grouping of interdependent instances to meet the needs of your workload.

Cluster Placement: Logical grouping of instances within single AZ. Low latency, Great Network (10 gbps bandwidth). If AZ fails, all instance fails. Big data job that need to complete fast, Application that need high network throughput. same type of instances

Partition: EC2 divides each group into logical segments called partition. No two partition share the same racks, can have up 7 partition on AZ. Each partition can have multiple instances. Can span over multiple AZ. HDFS, Cassandra, Kafka

Spread: Group of instances each placed in distinct racks, with each rack having it’s own network and power source. Can span over multiple AZ(7 instances per AZ). Rocommend for applications that have small number of critical instance that should be kept separate from each other. Mixing instance types.

Elastic Network Interfaces (ENI)

. Logical component in Virtual Private Cloud(VPC) that represents Virtual Network Card. We can create ENI independently and attach to EC2 for failover. Bound to specific AZ. You can redirect network from one instance to another from moving ENI to another instance. It gives EC2 instances access to the network .It has followings

. Primary Private IPV4 address and more secondary private IPV4s
. One Elastic IPV4 per private IPV4
. one public IPV4
. One or more security groups
. A MAC address
. You can create ENI independently and attach later
. Bound to specific AZ

New - Elastic Network Interfaces in the Virtual Private Cloud | Amazon Web Services

EC2 Stop=> That data on disk not damaged in the next start

EC2 Terminate: Root EBS volume also setup to be destroyed will be lost

EC2 Hibernate: InMemory state resumed, Booting faster(OS is not restarted/Stopped), inMemory RAM state is written in EBS, EBS Volume should encrypted . Used in long running process, Service that take time to initialize, Saving RAM state.

Instance RAM size must be less than 150gb, Hibernate no more then 60 days

EBS Volume( BLOCK Storage)

Elastic Block Store is network drive(not physical drive) that you can attach to your instance while they run to allow your instance to persist data even after their termination. Mounted one instance at a time. Bound to AZ.

Think as Network USB Stick, Free tier 30GB EBS storage SSD type or Magnetic per month. Can attach and detach from EC2 instance. But to move volume, you need to snapshot it.

By default, root EBS Volume will be deleted . You can preserve EBS volume

EBS Volume Types

6 types:

Gp2/Gp3 (SSD): General Purpose. balance workload and pricing. low latency. upto 16000 IOPS

iO1/iO2 (SSD): Highest performance, used in mission-critical low latency, High throughput (IOPS: I/O Operation Per Second) . great for database workloads, application need more than 16000 IOPS. virtual desktops,

64,000 is the maximum IOPS you can achieve when you’re using EBS io1 or io2 volume types.

256,000 is the maximum IOPS you can achieve when you’re using the EBS io2 Block Express volume type.

St1(HDD): Low cost HDD volumne designed for frequent access, throughput intensive workloads. big data, data warehouse, log processing, Max throughput 500MiB/s and IOPS 500

Sc1: Lowest cost HDD, designed for less frequently accessed workloads , max IOSP 250

EBS volume types characterized by Size | Throughput | IOPS

Only SSD can be used in boot volumes

Amazon EBS volume types

EBS Multi-Attach — io1/io2 family

Attach the same EBS volume to multiple EC2 instances in the same AZ. Achieve Higher application availability in clustered Linux application. Application must manage concurrent write operations. Upto 16 instances at a time. must use a file system that cluster aware(not XFS, EXT4 etc)

EBS Encryption

When you created encrypted EBS all data will be encrypted, All snapshots will be encrypted, All volume created from snapshot. Encryption has minimal impact. Use AES-256

way to encrypt unencrypted EBS volume: Create EBS snapshot of Volume, Encrypt snapshot using copy, and create volume from snapshot and attach volume to original instance.

EBS Snapshots

Backup of EBS volume at a point of time. Not necessary to detach volume to take snapshot but recommended. Can copy snapshots across AZ or Region.

Can take snapshots of DB over time

EBS Snapshots feature:

EBS Snapshot Archive: Move Snapshot to archive tierthat is 75% cheaper. take 24 to 72 hour to restore the archive

Recycle Bin Snapshot: COnfigure to retain deleted snapshot from 1 day to 1 year

Fast Snapshot Restore: No Latencty in first use

EC2 Instance Store

Physical storage for EC2 instance. EBS volumes are network drives with good but limited performance. If you need high performance hardware disk, you need EC2 Instance Store.

Better I/O performance, lose storage ater EC2 instance stopped. Good for buffer, cache, temporary content. Risk of data loss if hardware fails.

High IOPS up to millions

Amazon EFS -Elastic File System (File Storage for Linux)

Managed network file system (NFS) that can be mounted on many EC2. EFS works with EC2 with multi AZ. Highly available, expensive(3 x gp2), scalable, pay per use.

Use cases : Content Management, web serving, data sharing , wordpress,

Used NFS Protocol, use security group to control access to EFS, Compatible with Linux based AMI not windows.

EBS vs EFS

EBS: 
- one instance except multi-attach io2/io3
- are locked at AZ
- io increase in gp2 is disk size increases
- Root EBS get terminated default when EC2 terminated 
- Optimized for IOPS, latency-sensitive tasks

EFS: 
- mount 100s of ECs across different AZ
- EFS share website files
- only for linux instances
- higher price
- Optimized for throughput
- can leverage storage tiers for cost saving

AMI (Amazon Machine Image)

Customization of EC2 instance. Add own SW, configuration, OS, monitoring. Faster boot. Built on specific region and copy across region

1. Public AMI: Aws Provided ( eg. amazon linux 2 AMI)
2. Your own AMI: make and maintain Own
3. AWS marketplace: made by someone( potentially sells)

Scalability and High Availability

Scalable

handle huge loads

Vertical Scalability: increase size of instance. from t2.micro to t2.large. hardware limit.

Horizontal Scalability: Increasing number of instances.

High Availability

Running your application at least 2 data centers/AZ. survive data center loss

Load Balancing

Only Network Load Balancer provides both static DNS name and static IP. While, Application Load Balancer provides a static DNS name but it does NOT provide a static IP. The reason being that AWS wants your Elastic Load Balancer to be accessible using a static endpoint, even if the underlying infrastructure that AWS manages changes.

Server that forward traffic to multiple instances. ELB (Elastic Load Balancer)
Spread traffic across multipe downstream instances
Expose single point of access (DNS) to your apppication
Handle failure of downstream instances
Do regular health checkups for the instance 
PRovide SSL termination( HTTPS ) access to server
Separate public traffic from private traffic 
high availability across zones

Why uses ELB (Elastic Load Balancer)?

Managed load balancer, AWS guarantee that it is working. Aws take care update, upgrade and maintenance

Health Checks

Crucial for load balancer. They enable the load balancer to know if the instance it forwarded request are available to reply to request. /health route. if response is not 200, instance is not healthy

Classic Load Balancer (Old generation) — 2009 : HTTP, HTTPS, TCP, SSL

Application Load Balancer (New generation) — 2016: — Layer 7

Operate at request level. You can read the HTTP request, and check host, path, query and route accordingly.

HTTP,HTTPS, Web Socket. Load balancing to multiple HTTP applications. support routing tables. Balancing to Target Groups

Based on path (example.com/users and example.com/posts)
Based on Host: example.com and random.com
Based on query string: example.com/users?id=123 and example.com/users?id=345

Great fit for micro-services and container based applications
Dockker and Amazon ECS

Port mapping feature


Http based traffic route to target groups. 

Target Groups are : 
EC2 instances managed by Auto Scaling Group. 
ECS task (Elastic Container Service) 
Lamba functions (HTTP translated on JSON event)
IP addresses - must be private

ALB can route to multiple target groups and health checks are at the target group level. Application server can found ip of client at x-forwarded-for and port at x-forwarded-port

In ALB you cas use listener setting to handle Route/Path/Error etc filtering Traffic

Network Load Balancer — 2017 Layer 4 TCP, TLS, UDP (Transport layer)

Operates on connection level. Routes connection based on IP protocol data.Offer Ultra high performance, Low latency, and TLS offloading at scale. Can have static one IP per AZ /Elastic IP. Helpful for whitelisting Specific IP. Not included in AWS free tier
If you specify targets using an instance ID, traffic is routed to instances using the primary private IP address specified in the primary network interface for the instance. The load balancer rewrites the destination IP address from the data packet before forwarding it to the target instance.

If you specify targets using IP addresses, you can route traffic to an instance using any private IP address from one or more network interfaces. This enables multiple applications on an instance to use the same port. Note that each network interface can have its security group. The load balancer rewrites the destination IP address before forwarding it to the target.

*It can’t perform any check/modification on http layer

Target Groups:
EC2 Instances
UDP/ static IP address
Application Load Balancer
Health Check protocol HTTP/HTTPS/TCP. NLB supports HTTP health checks as well as TCP and HTTPS

Create NLB: 
1. Go to EC2 Service > Load Balancer > Create Network Load Balancer 
2. Choose Application Load Balancer / Network Load Balancer / Gateway Load Balancer
3. Internet Facing type as we accept request from internet
4. Mapping all AZ and each AZ has Subnet and assigned IPv4 from AWS (If you have Elastic IP, You can assign)
5. Add security Groups and allow inbound all http tcp protocal from anywhere
6. Listener and Routing: TCP port 80 select target group
7. Create target group > Instances> Protocol TCP on port 80 > 
8. include new Target group.
9. create load balancer

Gateway Load Balancer — 2020 : layer 3 (Network Layer) — IP protocol.

Used in front of virtual appliances such as Firewalls, IDS/IPS and deep packet inspection systems, payload manipulation. Listen for all packets on all ports. Exchange protocol with appliances using GENEVE protocol on port 6081. Virtual appliances inspect traffic and possiblly drop out back to GWLB.

Sticky Sessions

It is possible to implement stickiness so that same client is always redirected to same server. Works in ALB and NLB. the cookies used in stickiness has expiry date you can set.

1. Application based cookie:
a) Custom cookie: Generated by target. Can include any custom attribute required by application.name. DON’T use AWSALBAPP, AWSALB, AWSALBTG. They are reserved
b) Application cookie: Generated by LB. name: AWSALBAPP
2. Duration Based cookie: Cookie generated from LB. AWSALB, AWSELB

EC2> new Target group > Target Selection Configuration. > stickiness configure

Cross Zone Load Balancing

traffic distributed evenly across all EC2 instance regardless of AZ. Each of them get equal distribution.

Enabled by default in Application Load balancer. No charges

Disable by default in Network Load balancer. Charges apply

EC2 > Load Balancer > Attributes > Cross Zone Load Balancing
In Target group, we can inherit settings from load balancer or handle manually
in target selection configuration. 

SSL/TLS certificate

SSL : Secure Socket Layer — allows traffic between clients and load balancer to be encrypted in transit (in-flight encryption). Public SSL issued by Certificate Authorities. COMODO,GoDaddy..

TLS: Transport layer Security —

Load balancer uses X.509 certificate(SSL/TLS server certificate). You can manage certificate using ACS (Aws certificate Manager). you can create default and upload external certificate.

SSL — Server Name Indication (SNI)

SNI solves the problem of loading multiple SSL certificates onto one server(To serve multiple websites in single IP address)

Only works in NLB and ALB and CloudFront.

EC2> Load Balancer > LoadBalanCerA> Add Listener > 

ELB- Connection Draining- Deregistration Delay

Time to complete in-flight requests while the instance is de-registering or unhealthy. Stop sending new request to EC2 instances that is de-registering

Between 1 to 3600 sec (Default is 300s)

ASG- Auto Scaling Group

Scale out to match increased load/ Scale in to match decreased load. We have to set number of min , max and desired EC2 instances . Auto register new instance to load balancer. Re-create if one in unhealthy. ASG also work with ELB. Launch template is needed includes. A launch configuration is an instance configuration template that an Auto Scaling group uses to launch Amazon EC2 instances. It is not possible to modify a launch configuration once it is created.

AMI + Instance Type, EC2 User Data, EBS Volumes, Security Groups, SSH key pair , IAM roles for EC2 instances, Network + Subnet information, Load balancer information.
We can trigger from Alarms (Cloud Watch)

EC2 > Auto Scaling Group > Create Auto Scaling Group >

Auto Scaling Policies

Dynamic Scaling : Target Tracking ( be ASG CPU around 40 %), Step scaling ( CPU > 70% add new instance)

Scheduled Scaling: Based on known usage pattern. Increase capacity at 5pm on Fridays.

Predictive Scaling: Continuously forecast load and schedule scaling ahead

Metrics for ASG Scale

CPU Utilization, Request Count per Target, Average Network In /Out. Any custom metrics you pushed in cloud watch.

ASG Cool Downs

Period after Scaling activity (Default 300s). ASG will not allow to terminate/add additional instances to maintain stabilize.

EC@ > ASG > DemoASG > Automatic Scaling > 

Amazon RDS (Relational Database Service)

Use SQL query language. Postgres, MySql, MariaDB, Oracle, Microsoft SQL Server, IBM DB2, Aurora ( AWS Proprietary database). RDS is managed service. Point in time Store. Monitoring database, Multi AZ setup, Storage backed by EBS. can’t SSH into your instance.
However, RDS does not allow you to access the host OS of the database. For the given use-case, you need to use Amazon RDS Custom for Oracle as it allows you to access and customize your database server host and operating system, for example by applying special patches and changing the database software settings to support third-party applications that require privileged access.

Amazon RDS applies operating system updates by performing maintenance on the standby, then promoting the standby to primary and finally performing maintenance on the old primary, which becomes the new standby

Storage Auto Scaling

Help to increase storage in RDS DB instance. Auto modify storage if:

• Free storage is less then 10% of allocated storage.
• Low storage last at least 5 minutes
• 6 hours passed since last modifications

Useful for unpredictable workloads

Read Replicas

Help to scale your reads. Within AZ, across AZ or Across Region. Replication is async so eventually consistent. Application must update connection string to use read replica. No cost for replica within same Region.

RDS Multi AZ (Disaster Recovery)

Sync Replication. One DNS name — Auto app failover to standby. Increase availability. Failover incase AZ loss, loss of network, DB failure. No downtime to go from single AZ to Multi Az

RDS> Create Database 

AWS Aurora

Aurora is proprietary technology from AWS (Not open source). Postgres and MySql Supported. 5x time performance over MySQL and 3X times Postgres. Automatically grows (Scaling). can have up to 15 read replicas. Support cross region replica.

Writer endpoint point to master. Reader endpoint connected to replica (load balancing)

Aurora Replicas and Auto Scaling

When request traffic reached high, Replica Auto Scaling will add new replicas. Writer Instance/Reader Instance

Aurora — Custom Endpoint

Define a subset of Aurora Instances as Custom endpoint. Eg, Run analytical queries on specific replicas.

Aurora — Serverless

Automated database instantiation and auto scaling based on actual usage. Good for infrequent and unpredictable workloads. No capacity plan Required.

Global Aurora

1. Aurora Cross-Region Read Replicas: Useful for disaster recover. SImple to put in place.
2. Aurora Global Database: 1 Primary region(read/ write). Upto 5 secondary(read-only) regions, replication lag is less than 1 sec. Upto 16 read replicas per secondary region. helps for decreasing latency. Cross regional replication took less then 1 sec. For disaster recovery we have Recovery Time Objective is < 1min from another region.

Aurora Machine Learning

Fraud Detection, ads targeting, Sentiment analysis, Product recommendations. Supported services: Amazon SageMaker, Amazon Comprehend.

RDS Backup

Automated Backups: Daily full backup of database within backup window. Transactions logs backed up by RDS at every 5 min. 1 to 35 days of automated backup( Can not disable in Aurora).

Manual DB Snapshots: Manually triggered by user.

Trick: If you have stopped DB, you have to pay for storage. To reduce cost, take snapshot and restore it later.

Restore

Restoring RDS/Aurora Backup or create new database from snapshot. Restoring MySQL RDS from s3. For MySql Aurora, Percona XtraBackup is used.

Aurora Database Cloning

Create new Aurora DB cluster from existing one. Faster then restore and snapshot. use copy-on-write protocol. Initially new DB cluster uses same data, volume as original cluster. When update are madeto new DB cluster data, additional storage is allocated. very fast and cost-effective. Create staging database from production database without impacting prod data.

RDS Security

DB master and replicas encrypted using AWS KMS — must be defined in launch time. If master is not encrypted, replicas not encrypted. To encrypt unencrypted db, take snapshot and restore encrypted snapshot. AWS TLS root certificate use in client side. IAM roles to connect to DB instead Usr/pass

Oracle RDS database technology does NOT support IAM Database Authentication

RDS Proxy

Fully managed Proxy for RDS. Allows Apps to pool and share DB connections established with database. Improve DB efficiency by reducing stress to DB resources and minimize open connections . Serverless, auto scaling and Cross Multiple AZ. Failover time reduce by 66%. Enforce IAM auth, store cred in AWS Secrets manager. Never publicly accesible, Must accessed from VPC.

AWS Elastic Cache

Get managed Redis or MemCached. In-memory DB with high performance. Reduce load-off of database. Redis is Multi AZ, Read replicas, high availability, backup and restore. MemCached is sharding data, Not High availability.

Elastic Cache supports IAM Authentication for Redis.

Amazon Route 53

Fully managed and Authoritative (You can managed dns records) DNS. ALso a Domain Registrar. Check health of services. 100% availability. 53 is traditional DNS port.

Route 53 Record

How you want to route traffic for a domain. It contains domain/Subdomain name, Record type, Value, Routing policy, TTL.

1. A- Maps a hostname to IPv4
2. AAAA- Maps a hostname to IPv6
3. CNAME- Map hostname to another hostname
4. NS- Name server of hosted Zone

Top Level Domain: .com, .org
Second Level Domain: example.com
Sub Doamin: www.example.com
Fully qualified Domain Name: api.www.example.com

Browser Request --> Local DNS server --> Root DNS server (managed by ICANN) 
ROOT DNS It has top level domain IP. (IP of .com)
This IP now Goes to Top Level Domain DNS Server(Managed by IANA) (to get IP of example.com)
This IP now again redirected to Your Domain registrar (Like Route53) to know  the IP api.example.com
This Ip now go to your server


Route52> Registered Domain

Records TTL

TTL is mandatory on each DNS record except Alias Record. Amount of time DNS record cache in DNS resolver.

CNAM vs ALIAS

CNAME: point one hostname to another Hostname. Only for non-root domain. Second Level Domain + First Level Domain = Root domain

Alias: Point host name to AWS resource. Work for root and non-root domain. free of charge. Auto know the IP changes. Always of type A/AAAA (IPV4/IPV6). you can’t set TTL. Set Auto by Route53. Targets for Alias :

Elastic Load Balancer, CloudFront Distributions, API Gateway, S3 website, VPC,

You can’t set an ALIAS record for an EC2 DNS name.

Routing Policy: Simple

Route traffic to a single resource. If multiple value returned, use random IP value chose by client. Can’t be associated by Health Checks.

Routing Policy: weighted

Control the % of request that go to specific resources. Assign each record a relative weight. Health Checks, Load balancing between regions, new app version check. Assign record 0 to send none.

Routing Policy: Latency-Based

Redirect to resource that has least latency. Latency is based on traffic between users and AWS regions.

Health Checks

HTTP health check only for public resources. 15 global health checkers will check endpoint health. healthy/unhealthy threshold 3, interval — 30 sec. supported protocol HTTP,HTTPS,TCP. If >18% health checker report healthy consider healthy. 5120 bytes response.

Route 53 > Health Check 

Routing Policy-Failover

Create Route53 failover.example.com record with routing policy failover and failover type Primary/secondary. Incase of primary failed, traffic gors to seconday route

Routing Policy- GeoLocation

Route based on GeoLocation (COntinent, Country,State). Should create Default record for other. Website Localization, Restrict content distribution, Load balancing

Routing Policy- GeoProximity

Based on users location and AWS resources. Ability to shift more traffic to the resource based on defined bias. Bias(1–99) — more traffic to this resource, Bias(-1 — -99) — less traffic to resource. Resource can be AWS resource(regions) or Lat/Long. Must use Traffic flow policy.

Routing Policy- IP-Based

Based on client’s IP. Provide CIDRs (User-IP-To-Endpoint-mapping). Optimize performance, Reduce network cost.

Routing Policy- Multi value

use when you want to route traffic to multi resources. Upto 8 healthy records are returned.

Domain Registrar: use to register Domain Name by paying annual charge. Domain Registrar also usually provide DNS service to manage DNS record.

DNS service: use to create DNS record and routing policy. If you want to use different domain registrar(Go daddy) in AWS route53, Create Public Hosted Zone and Copy name servers and update them in domain registrar(Go-Daddy).

Amazon S3 (Simple Storage Service)

Main building blocks of AWS. Infinitely Scaling. Backup and storage, Disaster Recovery, Archive, to host application, images, media, big data analytics. Store object (file) in Bucket(directories)

Bucket Naming Convention

No UpperCase, NoUnderscore, 3–63 char long, Not an IP, Must start with lowerCase or number, not start with prefix xn- - . Not End with Suffix -s3alias

Objects (files) has key. key is full path s3://myfile-path/

Max size of file is 500GB. if big upload in multi-part.

Bucket Policy

User-Based: IAM policies — Which API should be allowed for specific person from IAM. Explicit DENY in an IAM Policy will take precedence over an S3 bucket policy.

Bucket policies in Amazon S3 can be used to add or deny permissions across some or all of the objects within a single bucket. Policies can be attached to users, groups, or Amazon S3 buckets, enabling centralized management of permissions. With bucket policies, you can grant users within your AWS Account or other AWS Accounts access to your Amazon S3 resources.

You can further restrict access to specific resources based on certain conditions. For example, you can restrict access based on request time (Date Condition), whether the request was sent using SSL (Boolean Conditions), a requester’s IP address (IP Address Condition), or based on the requester’s client application (String Conditions). To identify these conditions, you use policy keys

Resource-Based: Bucket-policies: Bucket wide rules, can be used for WWW user.

S3 Versioning

It is enabled in bucket level. Same key overwrite change the version.

S3-Replication

• must enable versioning on source and destination buckets.
• CRR (Cross Region Replication)
• SRR (Same Region Replication)
• Bucket can be in different AWS account
• COPYING is asynchronous . Give proper IAM permission,

S3 Replication allows you to replicate data from an S3 bucket to another in the same/different AWS Region. Replicate always from direct source.

After you enable replication, Only new objects are replicate. Replicate existing object using S3 Batch Replication

AWS>Bucket>Bucket-name>Management>Replication Rules

S3 Storage Class

The minimum storage duration is 30 days before you can transition objects from Amazon S3 Standard to Amazon S3 One Zone-IA.

1. Standard- General Purpose: Used for frequently accessed data, High Throughput, low latency, Sustain two concurrent facility failures, Used in Big data analytics, mobile gaming app, content distribution. Stores data across multiple Availability Zones (AZs)
2. Standard- Infrequent Access IA: Data that is accessed less frequently but requires rapid access when needed (e.g., backups, disaster recovery). Data is still stored across multiple AZs.
3. S3 One Zone- Infrequent Access IA: Infrequently accessed data that does not require the resilience of multiple AZs (e.g., secondary backups or easily reproducible data). Data is stored in a single AZ. Lower cost with higher risk (data is lost if the AZ fails).
4. S3 Intelligent-Tiering: Data with unknown or changing access patterns. Automatically moves objects between access tiers (frequent, infrequent, and archival) based on access patterns
5. S3 Glacier Instant Retrieval: Rarely accessed data with occasional retrieval needs and millisecond access time. Archiving and backup, Min storage duration 90 days. great for retrieval once a quarter.
6. S3 Glacier Flexible Retrieval (Formerly S3 Glacier): Archival data with rare retrieval requirement. Expedited: 1–5 minutes, sandard: 3–5 hours,Bulk: 5–12 hours. min 90 days storage duration.
7. S3 Glacier Deep Archive: Long-term archival storage. Standard: 12 hours, Bulk: 48 hours. Min storage duration 180 days.

S3 Lifecycle Rules

1. Transition Actions: Configure a object to transition to another storage class.
2. Expiration Actions: Configure object to expire after sometime. Delete old version, incomplete multi-part upload

S3 Requester Pays

In general, Bucket owner pays all the S3 storage and transfer bills associated with their bucket. With Requester pays bucket, the requester will pay the cost of the data transfer. Requester must be authenticated (AWs user)

S3 Event Notification

s3:objectCreaded, s3:objectReplicated, etc.. Can create may S3 events. We need Resource access policy to get Notification from S3 to SNS, SQS, Lamda function. (For multiple Send) You can send to AWS EventBridge service and this service can send other 18 services over AWS(Step functions, kinesis streams, firehose). You can create Events as many as you can.

usecase: create thumbnail after file upload.

S3 Batch Operation

Perform bulk operation on existing S3 objects. with a single request, Modify Object metadata and properties, Copy objects between s3 buckets, Encrypt un-crypted objects

S3 Baseline Performance

App can achieve at least 3500 PUT/COPY/POST/DELETE or 5500 READ/HEAD requests per second per prefix in bucket. prefix is all path after bucket bucket/folder1/sub1/file => /folder1/sub1/

Multi-Part Upload: recommend for >100mb files and must for >5GB file. can help parallel upload.

Use S3 Lifecycle policy to delete old/unfinished parts

S3 Transfer Acceleration: Transfer file to edge location and then forward to target region by Private network which is fast.

S3 Byte range Fetch: Parallel GETs by requesting specific byte ranges. Using the Range HTTP header in a GET Object request, you can fetch a byte-range from an object, transferring only the specified portion. issue a Byte Range Fetch for the first 250 bytes of files.

Amazon S3 Select is a new Amazon S3 capability designed to pull out only the data you need from an object, which can dramatically improve the performance and reduce the cost of applications that need to access data in Amazon S3. You cannot use Byte Range Fetch parameter with S3 Select to traverse the Amazon S3 bucket and get the first bytes of a file.

Storage Lens

used to understand, analyze, and optimize S3 storage across entire AWS.

AWS S3 Encryption

Server Side Encryption with AWS S3 managed Key (Enable by Default), when user upload file to S3, set header to “x-amz-server-side-encryption”:”AES256"

Server Side Encryption with AWS KMS(Key Management Service) managed Key, “x-amz-server-side-encryption”:”aws:kms". When upload it will call GenerateDataKey KMS API, and download will call KMS decrypt key.

With SSE-KMS, the encryption happens in AWS, and the encryption keys are managed by AWS but you have full control over the rotation policy of the encryption key. Encryption keys stored in AWS.

Server Side Encryption with Customer provided Key, Key is managed outside of AWS. Must use HTTPS, Key must provide in HTTP header for every request.

Client Side Encryption, Client must encrypt file before upload. USe client library Amazon S3 Client-Side Encryption Library.

Encryption on Flight (HTTPS)

DSSE-KMS is just “double encryption based on KMS”.

CORS(Cross origin Resource Sharing)

Origin=> Schema(Protocol) +host (Domain) + port
for https://www.example.com/api
Protocol=> https
host=> www.example.com
port=> 443 for https

website A request to Website B => Website B must enable CORS

the request won’t be fulfilled unless the other origin allowed for request , using CORS header eg. Access-Control-Allow-Origin

MFA -DELETE

Versioning should be enabled to use MFA Delete, Bucket owner able to disable/enable MFA. Currently can enabled from AWS CLI. MFA Delete forces users to use MFA codes before deleting S3 objects. It’s an extra level of security to prevent accidental deletions.

Root acount > My security Credentials > Create Access Key > 

AWS CLI =>
aws configure --profile root-mfa-demo
aws s3 ls
aws s3 ls --profile root-mfa-demo
aws s3api put-bucket-versioning --bucket <bucket-name> --versioning-configuration Status=Enabled,MFADelete=Enabled
--mfa "<arn-of-mfa-device> <mfa-code>" --profile <profile-name>
//<arn-of-mfa-device> found under Root acount > My security Credentials > MFA

S3 Access Logs

Any request made to s3, from account, denied, authorized all logged into another S3 bucket. can be analyze using analytics tools. Target logging bucket must be on same AWS region. Do not use you Logging Bucket to be monitored Bucket. It will create loop.

S3> Create Bucket for logging  
S3 > Go another Bucket which you want to track logs > Properties 
  > Server Access Logging > Destination to Logging Bucket > enabled

S2 Pre-Signed URLs

Pre signed URL using S3 Console, AWS SDK, AWS CLI.

URL Expirations: S3 console 1 min up to 12 hour. AWS CLI url default expiration 3600sec, max 168 hours. User given a pre signed URL inherit the permission of the user that generated the URL for PUT/GET

S3> Bucket > object  > Object URL is bydefault private
S3> Bucket > object > Object Actions > Share Presigned URL

S3 Glacier Vault Lock

WORM (Write Once Read More). Create Vault Lock Policy. Object never be deleted and edited. Must enable versioning to Vault lock policy.

Retention — Compliance Mode: Object version can’t be overwritten or deleted including root user. can’t changed and retention period can’t be shortened

Retention — Governance Mode: Most user can’t but some special user can overwrite or delete object versions.

Legal Hold: Protect object infinitely. can be removed by s3:putObjectLegalHold IAM permission

S3 Access Points

Simplify Security management for S3 Buckets. It has DNS name (Internet origin, VPC origin), access point policy like bucket policy. We can defined only access from VPC for that you need to create VPC Endpoint.

S3 Object Lambda

Use AWS Lamda function to change the object before it is retrieve by the application. Redacting PHI data.

AWS CloudFront

Content Delivery Network, Improve read performance, content is cached at the edge location. 216 point presence globally (Edge location). DDoS Protection, Integrate with shield. AWS web application firewall.

You cannot attach IAM roles to the Amazon CloudFront distribution. Here you need to use an OAI ( origin access identity). Create a special Amazon CloudFront user called an origin access identity (OAI) and associate it with your distribution.

AWS Accelerator — Global Accelerator

let’s deploy a server for global users, users from different region has to go through public network and different hops, and increase latency. We wish to go as fast as possible to minimize latency. Global Accelerator help this.

Unicast IP: hold 1 IP one server, AnyCast IP: All servers hold same IP, and client is routed to nearest one.

On Global Accelerator 2 Anycast IP will be assigned to your application. User will be redirected to nearest edge location and Leverage the AWS internal route network to reach to destination server.

CloudFront and Global Accelerator both uses Edge location. but Global Accelerator make all way to server (not cache). Accelerator is useful for non-http like Gaming, streaming and useful for http with static IP address.

Aws Snow Ball

Helps to trasfer large data from local server to AWS Cloud (physically use device). Snowball can’t directly import data to S3 Glacier, you must use Amazon S3 first and lifecycle policy to move to glacier. Snowball Edge comes with computing capabilities and allows you to pre-process the data while it’s being moved into Snowball.

Amazon FSX

3rd party high performance file system on AWS. Fully managed service.

1. FSX for windows: For windows, support Microsoft Distributed file system. Support SMB Protocol and Windows NTFS. Can be mounted on Linux instance. Scale up to 10s of GB, Millions of IOPS, Storage Option SSD, HDD, Can be access on-premises. Multi-AZ, data backed up daily. Use for workloads such as machine learning, high-performance computing (HPC), video processing, and financial modeling. Microsoft Active Directory (AD) integration. FSx for Windows does not allow you to present S3 objects as files and does not allow you to write changed data back to S3
2. FSX for Lustre: High Performance File System. Linux + Cluster, Distribute file system for large computing. Machine learning High Performance Computing. Scale 100s GBs, Millions IOPS, sub-ms latency. SSD/HDD, Can integrate with AWS S3. Can be on-premises. Optimize cost. The open-source Lustre file system is designed for applications that require fast storage — where you want your storage to keep up with your compute. FSx for Lustre integrates with Amazon S3, making it easy to process data sets with the Lustre file system. When linked to an S3 bucket, an FSx for Lustre file system transparently presents S3 objects as files and allows you to write changed data back to S3. FSx for Lustre provides the ability to both process the ‘hot data’ in a parallel and distributed fashion as well as easily store the ‘cold data’ on Amazon S3. Therefore this option is the BEST fit for the given problem statement.
3. FSX for NetApp ONTAP: Compatible for NFS/ SMB/iSCSI protocol, works with different OS. Autoscale storage. snapshot feature available. Point in time cloning. move data loading on NAS/ONTAP/AWS
4. FSX for OpenZFS: Compatible for NFS protocol, works with different OS. Upto millions IOPS and 0.5ms latency. snapshot feature available. Point in time cloning. move data loading on ZFS/AWS

FSX deployment Options

1. Scratch file system: Temporary storage, data not replicated. Faster, Short term processing.
2. Persistent File System: Long term storage, Data replicated within same AZ(copy data). Longterm processing, sensitive data.

AWS Storage Gateway

Bridge between on-premises and cloud data.

1. S3 File Gateway: Send data to AWS S3 (we can send glacier/Deep Archive using lifecycle only). Install S3 File gateway on-premises. Configured S3 access using NFS/SMB protocol. Most used data cached on file gateway. IAM role use by gateway to access S3.
2. FSX gateway: Send data to AWS FSx Windows File Server. and auto backed by S3 once in a while. local cache.
3. Volume Gateway: Send data to AWS S3 and then Backed by EBS snapshots, Cached, Entire dataset is on on-premises. Schedule backup to S3.
4. Tape Gateway: Send data to AWS S3 Using tape library, eject from backup app to glacier and deep archive

AWS Transfer Family

Fully managed service to transfer file in/out from S3 or Amazon EFS. supported protocol are FTP(File Transfer Protocol), FTPS(File Transfer Protocol over SSL), SFTP(Secure File Transfer Protocol). Multi AZ, scalable,

Data Sync Service

Use to move large data to and from

1. On-Premises/other cloud( EFS, FSx, S3 different storage)
2. File permission and meta data preserved

AWS Storage Gateway v/s AWS DataSync service. When to use which service?

Snowcone/ Snowball/Snowmobile to move large amount of data to cloud physically.

SQS

Amazon Standard SQS

Out of order messages, Max msg size 256kb, default retention 4 days upto 14 days. Unlimited throughput, can duplicate (Atleast once delivery)

Amazon FIFO Queue

Ordered messaging, Can’t duplicate. group ID use as key. 1 group ID can have one consumer. By default, FIFO queues support up to 3,000 messages per second with batching, or up to 300 messages per second (300 send, receive, or delete operations per second) without batching. Therefore, using batching you can meet a throughput requirement of upto 3,000 messages per second.

The name of a FIFO queue must end with the .fifo suffix. The suffix counts towards the 80-character queue name limit. To determine whether a queue is FIFO, you can check whether the queue name ends with the suffix.

You can’t convert an existing standard queue into a FIFO queue. To make the move, you must either create a new FIFO queue for your application or delete your existing standard queue and recreate it as a FIFO queue.

SQS queue with ASG

SQS scales automatically

Amazon SNS (Simple Notification Service)

What if you want to send message to multiple receivers ?

Note: S3 directly can’t send event to SNS. S3 can send event to SQS. Also SNS/SQS can’t send message directly to Kinesis Data streams. use EventBridge to send data to kinesis from SQS/SNS

Up to 12500000 subscriptions per topic (100000 topics limit in acc). Inflight encryption OR At rest Encryption OR Client side encryption. IAM policies to regulate access to SNS api. SNS access policies

How to Publish?

Topic Publish(using SDK): create topic, create a subscription, publish to a topic.

Direct Publish(for mobile APP SDK): create platform application, create a platform endpoint, publish to platform endpoint. Google GCM, Apple APNS,

Fan Out

SNS has topic, these topic subscribed by SQS. App publish notification through SNS topic, subscribers will receive notification/message

Push once on SNS, receive in all SQS queues that are subscribers. Make sure SQS Access policy allows SNS to write. Work on Cross region

Use Case:

For the same combination of Event type(Obj create) and prefix (/images), you can only have one S3 Event rule. If you want to same event to multiple SQS , use Fan-Out.

SNS — Message Filter

JSON policy used to filter message sent to SNS topic’s subscriptions. Type of SNS topics: Amazon Kinesis Data firehose, SQS,Lambda, Email, Email-JSON, HTTP,HTTPS, SMS

SQS, SNS and Lambda

Events are going to inserted on SQS, and Lambda will pull the Queue. incase of issue, it will put back message to Queue and try/retry to pull them. this can go to loop. so we can set up DLQ (Dead later Queue) after say 5 tries. and push that message into that queue.

Incase of SNS, Messages are going through it async to Lambda. Lambda has different retry behavior, incase fail, it will retry but internally. retry will be 3 times, If not successful it either discarded or we can set up DLQ (SQS QUEUE) from lambda function for later processing.

Kinesis

Make it easy to collect, process and analyze stream data in real time. Real time data like, Application logs, metrics, website clickstreams.

Kinesis Data Streams

Data is Record (Contains partition key and data). Capture, process and store data streams. Stream divide into N shards. Can scale Shards, Retention between 1 to 365 days. Ability to reprocess. Once inserted into kinesis, it can’t be deleted. Same partition key goes to same shard. Replay Capability. Real Time (~300ms). Need to write Consumer/Producer

Producer( Kinesis Agent, application,): Record (Contains partition key and data) pass through producer to kinesis data stream in 1mb/sec OR 1000 msg/sec per shard.

Consumer (Apps, Lamba, Kinesis Data firehose, Kinesis Data analytics): Pass Record (Contains partition key, sequence number, and data) to consumer in 2mb/sec per shard all consumers or per consumer

. In Provisioned Mode, you need to choose number shard, scale manually or Using API. 1MB/s in, 4MB/s out. 1000 records/sec

. On-Demand Mode: No need to provision, Default capacity 4mb/s in and 4000 record per sec). Auto scale based on last 30 days data.

Security: deployed in Region

Open CLI to send data to Kinesis data strea.

Aws Kinesis> Create Data stream > Define Data Stream Capacity > Create Data Stream

# remember stream name i.e test

## send data to Data Stream
Go to AWS CLI>
##Check version
aws --version 

1. Producer
##CLI V2
aws kinesis put-record --stream-name test 
--partition-key user1 --data "some data"
--cli-binary-format raw-in-base64-out

##CLI V1
aws kinesis put-record --stream-name test 
--partition-key user1 --data "some data"

2. Consumer
#describe the stream
aws kinesis describe-stream --stream-name test

#consume data
aws kinesis get-shard-iterator --stream-name test 
--shard-id shardID000
--shard-iterator-type TRIM_HORIZON

aws kinesis get-records --shard-iterator <>

Kinesis Data firehose (Delivery stream)

(Ingest, transform, load) Fully managed, Scalable, serverless service. Can send data to AWS destinations, S3 (can send failed data ), OpenSearch, 3rd party (Splunk, mongo, newrelic..) , Custom destinations(any http endpoint). Near real time transaction. No buffer (0) to upto 900sec. Buffer size min 1MB. data conversion/transform using lambda. Doesn’t have own data storage

Load data stream into AWS data store. Kinesis data streams also send data to Kinesis Data Firehose. Then data firehose send data to AWS S3,

Amazon Kinesis > Delivery Stream> Create stream
select Source type > select destination type> 
choose source > Transform setting >
Destination setting> 
Buffer size/interval

Kinesis Data Analytics

Analyze data stream into SQL or Apache Flink

Kinesis Video Streams

Capture, process and store Video streams.

Data Ordering in Kinesis

SQS vs SNS vs Kinesis

Amazon MQ is managed Message Broker used to use SNS, SQS in on-premises application without re-engineering. Amazon MQ is a managed message broker service for Apache ActiveMQ that makes it easy to set up and operate message brokers in the cloud. Message brokers allow different software systems–often using different programming languages, and on different platforms–to communicate and exchange information. If an organization is using messaging with existing applications and wants to move the messaging service to the cloud quickly and easily, AWS recommends Amazon MQ for such a use case.