AWS Solution Architect Associate — Part 2

First part Up to Application Decoupling here
AWS Solution Architect Associate — Part 1 | by Ramesh Pokhrel | Dec, 2024 | Medium

AWS ECS — Elastic Container Service

Amazon own container platform. Launch docker container on AWS == Launch ECS task on ECS cluster.

EFS volume can be shared between different EC2 instances and different ECS Tasks. It can be used as a persistent multi-AZ shared storage for your containers.

ECS — EC2 launch type

You must provision and maintain infrastructure (EC2 instances). Each EC2 instance must run ECS agent to register ECS cluster.

ECS — EC2 Farget type

You do not provision and maintain infrastructure. Serverless. Just create task definition. AWS run ECS task for you based on RAM, CPU you need.

ECS Cluster => EC2 = >ECS Agent => ECS Task

IAM Roles in ECS

1. EC2 Instance Profile (EC2 launch type only): Used by ECS agent to Make api call on ECS service, send container logs to cloud watch, pull docker image from ECR, Pull secret from Secrets manager.
2. ECS Task Role : ECS Task Role is the IAM Role used by the ECS task itself. Use when your container wants to call other AWS services like S3, SQS, etc.

#Create Cluster
AWS ECS > Create Cluster > 
Configure infrastructure (Farget/EC2 Instances/External Instances) 
  > Create Cluster
# it will also create EC2 instances/ASG/VPC

#Create Task Definitation
Task definition > Create Task Definition > Container information >port mapping > 

ECS Load Balancer

ALB supported and distributed traffic over clusters task. NLB recommended only for high throughput/High performance usecase.

ESC service — Auto Scaling

Auto increase/decreased number of ECS task. Use Aws Application Auto Scaling. Metrics are to scale = > ECS service average CPU utilization, Scale of RAM, ALB request count per target. EC2 AS task level is not same as EC2 AS instance level

Auto scaling EC2 instances: Accommodate ECS scaling by adding EC2 instances. Auto Scaling Group Scaling scale based on CPU utilization. ECS Cluster Capacity Provider Used to scale ECS task. Capacity Provider paired with ASG and Add EC2 only when you are missing resources (CPU/RAM..)

Amazon ECR — Elastic Container Registry

Store and manage docker images on AWS. Integrated with ECS and backed by S3, Private and public. Access controlled by IAM policy.

Amazon EKS — Elastic Kubernetes Service

way to manage Kubernetes cluster in AWS. alternative to ESC. open source. Kubernetes is cloud-agnostic

Amazon App Runner

Helps to run and deploy Web App to AWS without knowing anything.

AWS APP RUNNER > Source > Deploying setting > COnfigure Service (RAM, vCPU, Port)
> Auto Scaling configure >

AWS App2Container A2C

CLI tool for migrating and modernizing java and .NET web apps into docker container. Lift and Shift. Migrate legacy apps with no code changes.

AWS Lambda

AWS Lambda functions always operate from an AWS-owned VPC. By default, your function has the full ability to make network requests to any public internet address — this includes access to any of the public AWS APIs. For example, your function can interact with AWS DynamoDB APIs to PutItem or Query for records. You should only enable your functions for VPC access when you need to interact with a private resource located in a private subnet. An Amazon RDS instance is a good example.

Access S3 from AWS Lambda: Create an IAM role for the AWS Lambda function that grants access to the Amazon S3 bucket. Set the IAM role as the AWS Lambda function’s execution role. Make sure that the bucket policy also grants access to the AWS Lambda function’s execution role

Init > Invoke > Shutdown

Per region
  Execution:
    a. Memory Allocation: 128MB to 10GB (1MB increments)
    b. Maximum execution time 15 mins
    c. Env variables (4KB)
    d. Disk capacity in /tmp 512MB to 10GB
    e. Concurreny execution 1000 (can increase)
  
  Deployment:
    a. Lambda function deployment size in .zip 50MB
    b. Lambda function uncompressed size 250MB

Lambda SnapStart

Improve lambda function performance upto 10x at no extra cost for java 11 and above. When enabled , Function execute from pre-initialize state (no function initialization from scratch)

When you publish new version: Lamda initialize your function, Takes a snapshot of memory, and disk state of initialized function. Snapshot is cached for loa latency access.

Edge Function

Logic that you write on CloudFront distributions. Run close to user to minimize latency, execution time<1ms

Cloud Front Function: Written in JS, sub-ms startup, millions req/sec, execution time<5–10S, schange viewer req/res. Max memory 2mb. total packet size 10KB. To Cache Key Normalization, Transform request attributes, manipulate header information. URL redirects, rewrites, Validate JWT

Lambda@Edge: Written in NodeJs or Python, 1000 req/sec, change viewer req/res and origin req/res. Max memory 128MB to 10GB, Packet size 1MB-50MB. Network Access , File Access. Can execute 3rd party libraries

Lambda By Default

By default, Lambda function is launched outside VPC. To launched inside VPC, you must provide VPC ID, Subnet and Security Group. Lambda will create ENI in your subnet.

Invoking Lambda From RDS and AURORA

Invoke Lambda function from your DB instance. Must allow outbound traffic from DB instance to Lambda Function. DB instance must have required permission to invoke lambda function.

RDS Event notification will tell us about information of DB instance itself(create,deleted..). But don’t ave info of data . near real time events. send notification to SNS or subscribe to Events using EventBridge.

Dynamo DB

NoSQL, Scalable, High Availability, Multi AZ database. Max size of item is 400KB. Combination of Partition Key and Sort key make primary Key.

Provisioned Mode: Specify number of read and writes per second. Plan capacity beforehand, Pay for Reads Capacity Unit and Writes Capacity Unit. can add Auto Scaling for RCU and WCU

On-Demand Mode: Reads/Writes auto scale. Great for unpredictable work loads.

Dynamo DB >Create Table > Selct Partition key > Select Sort Key > configure > 

Dynamo DB Accelerator

memory cached for DynamoDB. 5min TTL for records.

Dynamo DB Stream Processing

React to real time changes in DB, Real time usage analysis, Invoke lambda, Real time usage analytics, cross region replication, Two types:
DynamoDB Stream: 24 hour retention, Limited consumer

Kinesis Data Stream: 1year retention, High number of consumers

DynamoDB also has Global Table help to make accessible in low latency in multi-region.

Dynamo DB has TTL attribute.

API Gateway

To prevent your API from being overwhelmed by too many requests, Amazon API Gateway throttles requests to your API using the token bucket algorithm, where a token counts for a request. Specifically, API Gateway sets a limit on a steady-state rate and a burst of request submissions against all APIs in your account. In the token bucket algorithm, the burst is the maximum bucket size

Edge Optimized Endpoint: For Global clients. API gateway still live in one region but accessible from global, Regional: Client with same region and Private: need VPC

Security in API Gateway

User Authentication with IAM Roles, Cognito to identify external users i.e mobile users, and Custo Authorize.

Custom Domain Name HTTPS

Amazon Cognito

Give users an identity to interact with our web and mobile application.

Cognito User Pool: Sign in Functionality for app users. Integrate API gateway and ALB.

Cognito Identity Pool: Provide temp AWS credential to user so they can AWS recourses directly. Integrated with Cognito user pools as identity provider. IAM policies that user granted will be defined in AWS Cognito.

Summary Of Databases

RDS

Details is on Part 1 https://medium.com/@kanxoramesh/aws-solution-architect-associate-9aec2845adee

1. Managed PostgreSQL/MYSQL/Oracle/SQL server/ DB2/MariaDB/Custom
2. Provisioned RDS instance Size and EBS volume type and size
3. Auto scaling capability for storage size
4. SUpport for Read replicas and Multi AZ
5.Security through IAM, Security Groups, KMS, SSL in transit 
6. Automated Backup with point in time restore feature (up to 35 days)
7. Manual DB snapshot for longer-term recovery
8. managed and scheduled maintenance with downtime
9. Support for IAM authentication , integration with Secrets manager. 
10. RDS custom to access and managed underlying Instance (Oracle & SQL server)
USe cases: Store relational databases(RDMS/OLTP), Perform SQL queries, Transactions.

Aurora

1. Aurora is proprietary technology from AWS (Not open source). 
2. Compatible API for Postgres and MySql, separation of storage and compute
3. Storage: data is stored in (default) 6 replicas, 3 AZ, High Availability, Self healing, Auto scaling
4. Compute: Cluster of DB instance accross Multi AZ, auto scaling and read replicas
5. Cluster: Custom endpoints for reader and writer DB instances.
Aurora Serverless: unpredictable workloads, no capacity planning
Aurora Global: Upto 16 read DB insatance in each region, < 1 storage replication
Aurora Machine Learning: Perform ML using SageMaker and Comprehend on Aurora
Aurora Database Cloning: New cluster form existing one, faster then restoring snapshot
Use Case: same as RDS but less maintenance/more flexiblity/more performance/more feature

ElastiCache

Managed Redis/Mem Cached (Similar offering as RDS but for caches )
In memory data store, sub-milli seconds latency
Select an elastiCache instance type
Support for clustering (Redis), MUlti AZ, Read Replicas (Sharding)
Security through IAM, Security Groups, KMS, Redis AUth
Backup/Snapshot/Point in time restore feature
Managed and schedule maintenance
Requires some code changes in application
Usecases: key/value store, Frequent read access/ 
  less writes/ Cache results forDB, Store Session Data, Can not use SQL

DynamoDB

1. AWS proprietary, managed serverless NoSQL database. ms latency
2. Capacity mode: Provisioned mode with optional Auto Scaling or On-Demand Capacity
3. Can replace ElastiCache to store key/value (session data ussing TTL)
4. High Available, Multi AZ by default, Read and Write are decoupled, Transaction capability
5. DAX (Dynamo DB accelerator) for read cache, microsecond latency
6. Security and Authentication from IAM
7. Event PRocessing: DynamoDB streams to integrate with AWS lambda or Kinesis Data Streams
8. Global table feature: active - active setup (anyone can read/write from any region)
9. Automated backup up to 35 days with PITR. or On-Dmand backups
10. Export to S3 without using RCP(Read capacity unit) and import backup without using WCP
Great for rapidly evolve Schema
Serverless/ if document size is small max 100kb/ distributed serverless cache

Amazon S3

Key/ Value Store for objects
Great for bigger objects/ not so great for small obj
serverless, scale infinitely/ max obj size in 5TB, versioning capability
Tiers: S3 Standard, S3 Infrequent Acess, S3 Intelligent, S3 Glacier + Lifecycle Policy
Features: Versioning/ Encryption/Replication/MFA-Delete/Access-Logs
Security: IAM security/Bucket Policies/ ACL/Accessp points/Object Lambda/CORS/Object/Vault lock
Encryption: SSE-S3,SSE-KMS,SSE-C,client-side,TLS in Transit, default encryption
Batch operations on object using S3 Batch, Listing files using S3 inventory
Batch operation help to encrypt, uncrypted objects, copy object files to another bucket
PErformance: Multi part Upload, S3 Transfer Acceleration (One region to another region)
, S3 Select only retrieve data you need out of S3
Auotomation: S3 Event Notifications, (SNS,SQS,Event Bridge)
Use Case: big objects, Static Files, Web Hosting

DocumentDB

1. Proprietary technology from AWS (Not open source). Same as Aurora but for MongoDB
2. NoSql, Similar Deployments as Aurora
3. MongoDB

Amazone Neptuen

Fully managed Graph Database
Graph Data set would be Social network
Highly Available across 3 AZ, with upto 15 read replicas, 
For highly connected datasets
Best for knowledge graph (Wikipedia), Fraud detection, Recommendation engines, social networking

Neptune Streams: Real time ordered sequence of every change.No duplicates, strict order
Stream data accessible by HTTP/S api

Amazon KeySpaces (For Apache Cassandra)

OpenSource, NoSQL distrbuted database. A managed apache cassandra compatible database service. Serverless, scalable managed by AWS. USe Cassandra Query Language. IoT, teme seris Data.

Amazon QLDB (Quantum Ledger Database)

Ledger is a book recording financial transaction, Used to review history of all changes made to your application data. Immutable no entry can be removed or modified. Cryptographically verifiable. Central Authorization

Amazon TimeStream

Time series database. Time series analytics function.

Data and Analytics

Amazon Athena

Serverless query service to analyzed data stored in Amazon S3. Uses standard SQL language to query the files. Support CSV, JSON, ORC,Avro and Parquet. $5 per TB of data scanned. Business Intelligence, Analytics, Reporting, Useful for Columnar format (Parquet format)

Analyze data in S3 using serverless SQL, use Athena.

Redshift

Based on PostgreSQL. Not OLTP (Online Transaction Processing), it’s OLAP(Online Analytical Processing). 10X better than other data warehouse.

Loading data to Redshift: Through Kinesis data firehose, copy command directly from S3, JDBC driver from EC2.

Amazon Redshift is a fully-managed petabyte-scale cloud-based data warehouse product designed for large scale data set storage and analysis.

Using Amazon Redshift Spectrum, you can efficiently query and retrieve structured and semistructured data from files in Amazon S3 without having to load the data into Amazon Redshift tables.

Amazon Redshift Spectrum resides on dedicated Amazon Redshift servers that are independent of your cluster. Redshift Spectrum pushes many compute-intensive tasks, such as predicate filtering and aggregation, down to the Redshift Spectrum layer. Thus, Amazon Redshift Spectrum queries use much less of your cluster’s processing capacity than other queries

OpenSearch

You can search any field. Managed cluster and serverless cluster. do not natively support SQl but enable by plugin. Ingest data from kinesis data firehose, IoT , Cloud Watch Logs.

Amazon EMR

Elastic MapReduce. Helps creating Hadoop Cluster (Big Data) to analyze and process large amount of data. The cluster can be made of 100 of EC2 instances.

Amazon QuickSight

Server-less machine learning-powered business intelligence service to create interactive Dashboards.

AWS Glue

Manage Extract, Transform and Load service. Serverless,

AWS Lake

Central place of data for analytics process. Fine grained access control. Build on top of AWS GLUE

Kinesis Data Analytics for SQL

Kinesis Data Analytics for Apache Flink

Read data from Kinesis data Stream and Amazon MSK. Use Flink (Java, Scala or SQL) to process streaming data. Flink doesn’t read from firehose, use for SQL instead.

Amazon Managed Streaming for Apache Kafka (Amazon MSK)

Alternative to Amazon Kinesis. Apache kafka on AWS.

Amazon Machine Learning

Amazon Rekognition

Image/video labeling, Text detection, face detection

Amazon Transcribe

Auto Convert speech to text.

Amzon Polly

text to speech

AWS Monitoring and Audit

CloudWatch : Watch what’s happening in real time. Think monitoring, alarms, and performance metrics.

CloudWatch Metrics

Provides metrics for every service in AWS. Metrics belongs to namespace. Dimension is an attribute of metric(Env, volume, instance ID). Up to 30 dimension per metric. Has timestamps. can create custom metrics.

You can stream these metrics outside like kinesis firehose, splunk,

CloudWatch Logs

Log Groups: usually Application name, Log Stream: application instance, log files/containers. Can setup expiration (1 day to 10 years). Logs can send to kinesis data streams, s3 , firehose, etc. You can get logs from different region or even different AWS accounts using subscription filter and subscription destination and send then different data analytics tools.

CloudWatch logs for EC2

Need to run cloud watch agent in EC2 to push logs to cloudwatch. Make EC2 has correct IAM permissions. CloudWatch Unified Agent can send system level metrics.

CloudWatch Alarm

State: OK, INSUFFICIENT_DATA,ALARM. Cloud watch is on single metric. multiple metrics alarm need composite alarm setup. Composite Alarm monitor the state of other Alarms. Reduce alarm noise

Event Bus: helps to route from source to another

Cloudwatch Container Insights: collect, aggregate, summarize logs/metrics from containers. ECS, EKS, Kubernetes platform on EC2, Fargate.

Cloudwatch Lambda Insights: collect logs from lamda function, serverless applications

CloudWatch Contributors Insight: Find Top-N and find out who or what is impacting system performance.

Cloudwatch Application Insights: Applications logs running on EC2. also for other AWS resources.

Amazon Event Bridge

Schedule: Cron Jobs , Event Pattern: React to service doing something.

Amazon CloudTrail

“who did what, when, and from where” (API Activity Tracking).Audit trail of actions taken by users, roles, or AWS services

AWS CloudTrail allows you to log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. It provides the event history of your AWS account activity, audit API calls made through the AWS Management Console, AWS SDKs, AWS CLI. So, the EC2 instance termination API call will appear here. You can use CloudTrail to detect unusual activity in your AWS accounts.

You can use the CloudTrail Console to view the last 90 days of recorded API activity. For events older than 90 days, use Athena to analyze CloudTrail logs stored in S3

Can be applied all region or single region. If resource is deleted, first check on Cloud trail.

In API call to delete some table in DynamoDB, CloudTrail log the API call, we can create event in EventBridge from cloudTrail and send Alert in Amazon SNS.

Aws Config

What has changed?” (Resource Configuration Tracking). “state” changes of resources. what changed, when it changed, and why it’s non-compliant.

AWS Config is a fully managed service that enables you to assess, audit, and evaluate the configurations of your AWS resources. It provides a detailed view of the configuration of AWS resources in your account and tracks their changes over time. AWS Config is particularly useful for governance, compliance, and operational auditing.

• Check if S3 buckets have public access disabled.
• Ensure EC2 instances are using specific types of instance sizes.
• Verify if encryption is enabled on RDS instances.

Cloud Watch vs Cloud Trail vs Config

Cloud Watch is Performance Monitoring (Metrics, CPU, networks, etc) and Dashboards. Events and alerting, Logs Aggregation and Analytics.

Cloud Trail Record Api calls made within Aws account by everyone, Global service, can define trails for specific resource. Who made changes

Config record configuration changes, get timeline of changes.

CloudTrail: "Who deleted an S3 bucket?" 
CloudWatch: "Is CPU usage > 80%?" 
Config: "Did the S3 bucket become public?"

AWS Organization

Global service, Allow to manage multiple AWS accounts. Main is management account, others are member account. We can define Service Control Policy (SCP) : IAM policies applied to specific Organizational Unit (OU) or Accounts to restrict users and roles.

IAM Roles vs Resource Based Policy

Use IAM Role Account OR Resource Policy,

• Role Policy → What actions can I do?
• Resource Policy → Who can access me?

In IAM policy, IAM Boundaries defined and supported for Users and Roles (Not group). Advanced feature to use a managed policy to set the maximum permissions an IAM entity can get.

Deny will take precedence. Not mentioned mention is also deny.

AWS Identity Center

One login for all your AWS account in AWS organizations, EC2 windows instance. Has Identity Provider.

KMS — Key Management Service

AWS manage encryption keys for us, Fully integrated with IAM authorization. Able to audit KMS Key usage using CloudTrail.

You can use the AWS Managed Service keys in KMS, therefore we don’t need to create our own KMS keys.

When you enable Automatic Rotation on your KMS Key, the backing key is rotated every 1 year.

KMS Keys Types

AWS Owned Keys, AWS managed Keys (Free), Customer Managed Keys

Region 1=> EBS volume encrypted with KMS key A 
         > EBS Snapshot encrypted with KMS key A
          
Region 2 copy=> EBS Snapshot KMS re-encrypt with KMS key B 
             => Create volume with encrytpion KMS key B

Key Policies

Control access to keys. You can’t control access without it. Default policy will allow root user, Custom will provide customization. Custom KMS key useful for cross-account access.

KMS Encrypt/Decrypt

Multi Region Keys

Identical KMS keys in different AWS regions that can be used interchangeably. Not recommended . Useful for global client side encryption, encryption on global DynamoDB, global Aurora

S3 replication KMS keys

Un-Encrypted objects and encrypted object with SSS-S3 are replicated by default. Obj encrypted with SSE-C (Customer Provided key) can be replicated. For obj encrypted with SSE-KMS, you need to enable the option (Update policy). You can use Multi Region Keys.

AMI sharing process Encrypted via KMS

Source account AMI is encrypted with KMS. To share to target acc, Must modify the image attribute to add a Launch Permission which correspond to specified target account. Must share KMS keys to target acc (via policy). IAM role/User in target key must have permission to DescribeKey/ReEncrypt, Decrypt.

SSM Parameter Store

Secure Storage for Configurations and Secrets. Version tracking, IAM security. We can have parameter policies to control SSM (TTL) sensitive data

AWS Secrets Manager

Capability to force rotation of secrets every x days. uses lambda to auto generate rotation key. Integrated with RDS. Encrypted with KMS. Can replicated to multi-region. Keep read replicas in sync with Primary Secret.

AWS Certificates Manager

Provisioned, managed and deploy TLS certificates. Provide inflight encryptions. Free for public TLS certificate. Auto renew. Integrated with ALB, CloudFront, API gateway. But not with EC2 instance.

Requesting Certificate:

1. List the domain names to include in certificate: Fully Qualified Domain name, WildCard Domain
2. Select Validation Method. DNS validation or Email Validation
3. It will took few hours
4. ACM auto renew ACM generated certificates before 60 days of expire.

Import Public certificate: No auto Renew. must import new before expiry. ACM send expiry events in event bridge daily before 45 days of expiry. AWS config can also check

Web Application Firewall (WAF)

Protect web app from web exploits. Layer 7 HTTP. Deploy on ALB, API Gateway, CloudFront, AppSync GraphQL API, Cognito User Pool.

AWS Shield — Protest from DDoS attack.

DDoS=> Distributed Denial of Service. Many request at the same time

If your organization has multiple AWS accounts, then you can subscribe multiple AWS Accounts to AWS Shield Advanced by individually enabling it on each account using the AWS Management Console or API. You will pay the monthly fee once as long as the AWS accounts are all under a single consolidated billing, and you own all the AWS accounts and resources in those accounts.

AWS Firewall Manager

Manage Firewall Rules in all accounts of AWs org.

WAF rules, Aws Shield Advanced, Security groups for EC2/VPC, ALB, ENI resource in VPC. Route53 DNS firewall

AWS Firewall Manager is a security management service that allows you to centrally configure and manage firewall rules across your accounts and applications in AWS Organizations. It is integrated with AWS Organizations so you can enable AWS WAF rules, AWS Shield Advanced protection, security groups, AWS Network Firewall rules, and Amazon Route 53 Resolver DNS Firewall rules.

WAF, Shield and Firewall manager are used together for comprehensive protection. For granular protection WAF alone is correct choice. If you want to use AWS WAF across accounts, accelerate configurations, Automate the protection of new resources, use firewall manager with WAF. Shield add Additional DDoS attacks over WAF. It provides dedicated support from Shield Response Team.

WAF & SHield > Create Web ACL (Access COntrol List)> CLoudFront Resource(Global) 
              or Regional

Amazon GuardDuty

Intelligent Threat discovery to protect your AWS account. Uses machine learning algorithms , Anomaly detection, 3rd party data. One click to enable. No need to install software. Input data includes: Cloudtrail Event Logs, VPC flow logs, DNS logs. Can setup EventBridge in case of findings. Can protect against CryptoCurrency attacks.

Amazon Inspector

Automated Security Assessments

1. EC2 Instances: Leveraging the AWS System Manager SSm agent, Analyze against unintended network accessibility. Analyze the running OS against known vulnerabilities.
2. For container Images push to Amazon ECR: Assessment of Container Images as they are pushed
3. For lambda functions: Identifies software vulnerabilities in function code and package dependencies. Assessing function when they are deploying

Amazon Macie

Fully managed data security and privacy service that uses machine learning and pattern matching to discover and protect your sensitive data in AWS. help to find PII (Personal Identifiable Information)

Networking

https://www.ipaddressguide.com/

VPC in AWS

can have multiple VPCs in AWS region(max 5 per region soft limit). Max CIDR (Classless Inter Domain Routing) per VPC is 5. For each CIDR min. size is /28 (16 IP address), max size is /16 (65536 IP address).

Subnet is sub range of IP (IPv4) within VPC. AWS reserve 5 IP address (First 4 and last 1). 10.0.16.0/20 => 4091, 10.0.32.0/20 => 4091, 10.0.0.0/24 => 251, 10.0.1.0/24 => 251

/28 means 16 IPs (=2^(32–28) = 2⁴), means only the last digit can change.

A private hosted zone is a container for records for a domain that you host in one or more Amazon virtual private clouds (VPCs). You create a hosted zone for a domain (such as example.com), and then you create records to tell Amazon Route 53 how you want traffic to be routed for that domain within and among your VPCs.

For each VPC that you want to associate with the Route 53 hosted zone, change the following VPC settings to true:

enableDnsHostnames

enableDnsSupport

IG Internet Gateway

Allows resources in VPC to connect with internet. Must be created separately from VPC. One VPC = One IG. IG on their own do not provide internet access, we must add route tables.

Internet User=> Internet Gateway=> Route table => AWS destination

1. Create VPC > Add IPv4 CIDR (10.0.0.0/16) (You can have upto 5 CIDR )
2. Tenancy to setup whether need dedicated hardware or default hardware. Dedicated EC2 instance will launch 
3. Withing VPC > create Subnet>select VPC > choose subnets with IPv4 CIDR block for subnet
4. Select AZ for each subnet
5. Go to VPC > Internet gateways >Create > Attach VPC
6. Create route table > assign VPC
6. In route table > Subnet Association > assign Subnets > 
7. Edit routes in route table > target as internet gateway

Bastion Host

EC2 instance in Public Subnet. Is a way to SSH to private EC2 instances.

NAT (Network Address Translation) Instance

Allows EC2 instance in Private subnet to access internet. NAT must be launched in public subnet. Must disable EC2 setting Source/Destination check. Must have elastic IP attached to it. Managed by you.

NAT Gateway

A NAT gateway is a Network Address Translation (NAT) service. You can use a NAT gateway so that instances in a private subnet can connect to services outside your VPC but external services cannot initiate a connection with those instances. It uses IPV4 only.

Highly available. NAT gateways in each Availability Zone are implemented with redundancy. Create a NAT gateway in each Availability Zone to ensure zone-independent architecture.

AWS managed, Higher availability, higher bandwidth (5GBPS), Higher scale (uptp 100 Gbps). Created in specific AZ, uses Elastic IP, Can’t be used by EC2 instance of same subnet. Require IGP (Internet Gateway) => Private Subnet > NAT GW > IGW. No security Groups and Bastion Host.

1. Create NAT gateway > Select Subnet > assign Elastic IP address 
2. VPC Route tables > Add destination from 0.0.0.0/0 to NAT gateway

NACL (Network Access Control List)

Like firewall which control traffic from and to subnets. One NACL per subnet. Default Subnets Accepts everything inbound/outbound with subnet it associated with. Do not modify default NACl, instead add custom one. Blocking IP from subnet level. Stateless

Rule have a number (1 - 32766), 1 is higher precedence. 
First rule match will drive the decision. The last rule is * and denise request incase no match
recommend rule indenx start from 100.

Security Group operate at instance label, where NACl operate in subnet level. 
security group allows rule only, NACL accept/deny rule
SG statefule, NACL stateless

Security > Network ACL> Create > Inbound/ Outbound rule (default will accept)

Ephemeral Port

Server respond back to client, for that client need to open a port which is Ephemeral port. Request has Ephemeral port.

VPC Peering

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Privately connect VPCs using AWS network, behave like they are on same network. Must not have overlap VPC CIDRs. You must update Route table for each VPC subnets. You can create VPCs connection on different AWs acc. Can refer same SG (works cross acc, Same region). Transitive Peering does not work for VPC peering connections

VPC must be established for each to connect eg,if we have VPC A -> VPC B -> VPC C
we can't connect VPC C to VPC A each other. We must have connection between VPC A to VPC C

VPC > Peering Connection >Create Peering Connetion > Peering Name >
 Select Local VPC > Select Another VPC > Create > It now Pending acceptance

After accept, You need to modify Route table. add route for both VPC route table.

VPC Endpoints

AWS private Link. Allows you to connect VPC to AWS services using private network instead public internet. In case of issues check Route tables and DNS Setting Resolution in VPC . Two Endpoint types.

Interface Endpoint: Provision an ENI (Private IP address) as an entry point. Must attach Security group. Support Most AWs services. Costly. Useful for site to site VPN. An Interface Endpoint is an Elastic Network Interface with a private IP address from the IP address range of your subnet that serves as an entry point for traffic destined to a supported service

Gateway Endpoint: Provision a gateway and must be used as target in route table. doesn’t use SG. Support both S3 and DynamoDB, free.A Gateway Endpoint is a gateway that you specify as a target for a route in your route table for traffic destined to a supported AWS service. The following AWS services are supported: Amazon S3 and Amazon DynamoDB.

Why need Gateway Endpoint?

You have a VPC with private subnets containing EC2 instances running a data processing application. The application needs to:
Fetch raw data stored in Amazon S3.
Upload the processed results back to another S3 bucket.

Problem Without Gateway Endpoint:
Instances in private subnets don’t have internet access directly.
To access S3:
You would need a NAT Gateway or NAT Instance, which incurs extra costs.
Traffic between your private subnets and S3 would traverse the public internet.

Solution Using Gateway Endpoint:
Configure a Gateway Endpoint for S3 in the VPC.
Update the route table for your private subnets to route traffic destined for S3 via the Gateway Endpoint.
Your EC2 instances can now directly access S3 buckets privately.

VPC Flow Logs

VPC Flow Logs is a VPC feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC.

VPC flow logs, Subnet flow logs, ENI flow logs, Help to monitor connectivity issues, Log data can go to S3, Firehose, Cloudwatch.

syntax: 
version account-id interface-id srcaddr dstaddr srcport dstport protocol packate bytes start end action

you can find flow logs inside VPC > Flow Logs > create flow logs 

AWS site to site VPN

Amazon VPC provides the facility to create an IPsec VPN connection (also known as AWS site-to-site VPN) between remote customer networks and their Amazon VPC over the internet.

Virtual Private Gateway (VPGW): AWS side of VPN connection

VPN connection: A secure connection between your on-premises equipment and your VPCs.

VPN tunnel: An encrypted link where data can pass from the customer network to or from AWS.

Customer Gateway (CGW): app on customer side. Use public Internet-Routable IP address for your Customer Gateway device. In case of private, use public IP address of NAT device.

To work properly, Enable Route Propagation for Virtual Private gateway in route table that is associated with your subnets.

If you need to ping EC2 instances from on-premises, make sure to add ICMP protocol on the inbound of Security group.

AWS VPN CloudHub provides secure communication between multiple VPNs. AWS VPN CloudHub allows you to securely communicate with multiple sites using AWS VPN. It operates on a simple hub-and-spoke model that you can use with or without a VPC.

VPN > Customer Gateways > 

Direct Connect (DX)

Provide dedicated direct physical connection from a remote network to VPC (use 3rd party direct connect hub). Access public and private resources on same connection. Direct Connect Gateway helps to Direct Connect to multiple VPCs in many different regions. Took more then 1 month to established new connection. No encryption, if need use IPSec . High Resiliency for critical workloads.

Hosted Direct Connect connection supports 50Mbps, 500Mbps, up to 10Gbps.

Site to Site VPN as backup for Direct Connect fails.

Transit Gateway

An AWS transit gateway is a network transit hub that you can use to interconnect your virtual private clouds (VPC) and on-premises networks. It has default route table that decide next hop based on destination IP of packet.

For having transitive peering between thousands of VPC and on-premises, start connection, Regional, Share cross-account . Route tables limit the route. Supports IP multicast. Can share Direct Connect to multiple accounts.

ECMP Equal Cost Multi Path Routing => Increase bandwidth in site-to-site VPN.

VPN to Virtual Private Gateway =>1 X VPC
VPN to Transit Gateway =>1 X Multi VPC

AWS VPC Traffic Mirroring: Inspect trafiic in VPC

IPV6

IPV4 => 4.3 billion address

IPV6 => 3.4 x 10~38 billion address, all IPV6 in AWs are public. x.x.x.x.x.x.x.x => 0000 to ffff (hexa decimal)

Egress only Network Gateway is same as NAT but for IPV6

Summary

Networking Cost per GB

AWS Network Firewall

Provide security for entire AWS VPC from layer 3 to layer 7

Disaster Recovery

RPO-> Recovery Point Objective : Data loss time before disaster occurs

RTO-> Recovery Time Objective : Data recover time from disaster.

Backup and Storage

Use AWS Storage Gateway, AWS Snowball, Snapshot etc. It has High RPO. S3 Backup,

Pilot Light

Critical Core is always run in AWS . low-cost, minimal state. They are just enough to keep things running in case of a failure. Lower RPO/RTO.

Warm Standby

Full system is running but at minimum size. costly.

MultiSite/Hot Site Approach

Very Low RTO, Costly, Full production is running in on-Premises and AWS.

DMS Database Migration Service

AWS DMS enables you to seamlessly migrate data from supported sources to relational databases, data warehouses, streaming platforms, and other data stores in AWS cloud.

Quickly migrate DB to AWS. Source database remain available during migration.

AWS SCT : Schema Conversion Tool

Data migrate from AWS RDS MySQL to Aurora MySQL (Same for PostgreSql)

Option 1: Take DB snapshot from RDS MySQL and restored as MySQL Aurora DB.

Option2: Create Aurora DB Replica from RDS MySQL and Promote to own DB Cluster. Can take time and cost

Data migrate from External MySQL to Aurora MySQL

option 1 : Create file backup on S3(Use Percona XtraBackup)and create Aurora DB from S3

option 2: Create Aurora MySQL DB and use mysqlutility to migrate MySQl to Aurora (aws_s3 aurora extension for Postgresql)

• Use DMS if both DB are up and running is another option

On-Premise Strategy with AWS

can use Amazon Linux 2 AMI as a VM and VM can import/export to AWS and migrate to AWS EC2

AWS Backup Vault Lock

Enforce WORM (Write Once Read Many) state for all the backups that you store in AWS backup vault. Backup can’t be deleted.

AWS Application Discovery Service

Plan Migration projects by gathering about on-premises data centers. Dependency Mapping, Checking system performance etc that need to migrate on-premise/virtual to AWS

AWS Application Migration Service (MGN)

Lift and Shift app to AWS from on-premises

Transferring large Data in AWS

Eg. Transfer 200TB data in the cloud. we have 100mbps internet connection

1. Over the internet/Site to Site VPN: Immediate to setup, will take around 185 days(200x1000x1000x8/100=185d
2. Over direct Connect 1 GBPS: Long for one time setup, will take 18.5d(200x1000x8
3. Snowball: Will take 2 to 3 snowball in parallel, Take about a week for end to end transfer. can be combine with DMS. AWS Snowball Edge Storage Optimized is the optimal choice if you need to securely and quickly transfer dozens of terabytes to petabytes of data to AWS. It provides up to 80 TB of usable HDD storage, 40 vCPUs, 1 TB of SATA SSD storage, and up to 40 Gb network connectivity to address large scale data transfer and pre-processing use cases. The data stored on AWS Snowball Edge device can be copied into Amazon S3 bucket and later transitioned into Amazon S3 Glacier via a lifecycle policy. You can’t directly copy data from AWS Snowball Edge devices into Amazon S3 Glacier.
4. For ongoing replication/transfer: Site to Site VPN or DX with DMS or DataSync.

AWS CloudFormation

Infrastructure as a Code. Declarative way to outline AWS infrastructure, for any resources. you simply provide code in right order. you can estimate the cost. Cloud on fly, you can changes the flow anytime.

is the de-facto service in AWS for infrastructure as code (IaC). It enables you to create and provision AWS infrastructure deployments predictably and repeatedly.

System Manager — SSM Session Manager

Allows you to start a secure shell on your EC2 and on-premises server. No SSh access, baston hosts, or SSH keys needed, No port 22 needed. Send log data to S3 or cloudwatch.

Create EC2 Instance > not used any keypair> diasble HTTP/SSh traffic >
 > Attach IAM role for SSM > AmazonSSMManagedInstanceCore > 

SSM > System Manager > Fleet Manager> All EC2 registered with SSM will appear here
SSM > System Manager>Node Management > Session Manager > Start Ssession

System Manager Run Command is use to execute document, or just a command. run command across multiple instances. No need for SSH, Command output can be shown in AWS console, S3, cloudWatch , Can be envoked with cloudBridge.

System Manager Patch Manager automates the process of patching managed instances. OS updates, app updates, security updates, on-demand or schedule.

Aws SES — Simple Email Service

use SMTP/APIs to send Email securely and globally . allows inbound/outbound. marketing Email, bulk Email, transactional email.

AWS PinPoint

Support Email, SMS, Push, Voice, In-App message. Inbound/Outbound marketing communication service. scales billions per day

Cost Explorer

Visualize, analyze and understand AWS costs and usage over time. Create custom reports. Choose optimal saving plan. Forecast upto 12 month

Cost Anomaly Detection use AI/ML to detect hidden cost usage , unusual cost and send report to you

AWS Outpost

Aws Server Wracks that offer same AWS infrastructure, services and tools to build your n-premises applications as like in server.

AWS AppFLow

Helps to securely transfer data from SaaS to AWS like Slack, Salesforce, SAP, etc. Destination might be AWS services like S3, Redshift

Instance Scheduler

Auto start/stop Aws services to reduce cost. Stop EC2 instace outside office hour. Support EC2, Auto scaling group, RDS instances. Schedule are managed in dynamoDB, Cross account and cross region support.

— — — — — — — — — — — — — — — — — — — — — — -

Event Processing in AWS

SNS, SQS, Lambda, EevntBridge, Fanout, CloudTrail event, API Gateway to Kinesis Data streams.

Caching Strategies

CloudFront (Edge Location) TTL, API Gateway also has caching strategy(Regional), EC2/Lambda Caching use Redis, MemCached, and DAX( Dynamo DB accelerator) which reduce burden to RDS database.

Blocking an IP Address

First would be NACL(Network Access Control List), we can add deny rule. Then EC2 Security Group can’t have Deny rule. if we have limit IPs, allow them. If our app is global, we don’t know all the IP address, then we could run optional firewall software in EC2 to block IP(CPU cost).

Another is add ALB with security group. and launch EC2 on Private IP and allowed only request from ALB security group. We can add WAF(Web Application Firewall) for complex filtering.

Another is add NLB with security group. and launch EC2 on Private IP and allowed only request from NLB security group.

Another is Block Client IP on CloudFront by GeoRestriction or can use WAF. ALB only accept Public IPs of CloudFront.

High Performance Computing (HPC)

For Data Management and Transfer:

1. AWS Direct Connect: Move GBs of data to the cloud, over a private secure network.
2. Snowball/ Snowmobile: Move PetaBytes of data to cloud through physical route.
3. AWS DataSync: Move large amount of data between on-premises and S3, EFS, FSx for windows

For Compute and Networking:

The aws S3 sync command uses the CopyObject APIs to copy objects between Amazon S3 buckets.

aws s3 sync s3://DOC-EXAMPLE-BUCKET-SOURCE s3://DOC-EXAMPLE-BUCKET-TARGET

Amazon S3 Batch Replication provides you a way to replicate objects that existed before a replication configuration was in place, objects that have previously been replicated, and objects that have failed replication. This is done through the use of a Batch Operations job.

Amazon S3 Transfer Acceleration (Amazon S3TA) is a bucket-level feature that enables fast, easy, and secure transfers of files over long distances between your client and an Amazon S3 bucket. You cannot use Transfer Acceleration to copy objects across Amazon S3 buckets in different Regions using Amazon S3 console

EC2 Instances

GPU,CPU optimized, Spot Instances, Spot Fleets, Auto Scaling. EC2 Placement Group for Cluster to improve network performance. EC2 Enhanced Networking with Elastic Network Adapter (ENA) up to 100 Gbps network speed. Elastic Fabric Adapter(EFA) is Improved ENA for HPC and works for Linux. Great for Tightly Coupled workloads. It used Message Passing Interface (MPI) to bypass Linux OS to provide lower latency.

Storage

1. Instance Attached storage: EBS (scale up-to 256000 IOPS with io2 block express), Instance Store(scale to millions of IOPS, linked to EC2 instance, low latency)
2. Network Storage: Amazon S3(Large blob, not file system), EFS( Scale IOPS based on total size, provisioned IOPS), FSx for Luxture(Linux, millions of IOPS, backed by S3)

Automation and Orchestration

1. AWS Batch: Support Multi-node parallel jobs, single job span to multiple instances.
2. Parallel Cluster: open source cluster management tool to deploy HPC on AWS. configure with text files, automate creation of VPC, subnet, cluster type and instance. Ability to enable EFA on the cluster.

How to make Highly Available EC2 Instance?

1. Attach Elastic IP to public EC2 instance.
2. Create Standby EC2 instance. If any issue found on Public Ec2 instance, implement cloudWatch monitor and create alarm based on metric. and send to lambda function. that function can start the API call to StandBy EC2 instance to attach Elastic IP Address, Then we effectly failed over new EC2 instance.
3. With Auto Scale Group: ASG in 2 AZ. For first AZ instance, We can set on user Data on EC2 instance, attach Elastic IP based on TAG. and incase of first EC2 fail, ASG will create replacement EC2 on second AZ with same user data and Elastic IP. we effectively fail over. EC2 instance role shoud have API call to attach Elastic IP address.
4. We will Attach EBS Volume on EC2 instance. ASG has lifecycle hooks, on termination hook, we can create script to take EBS Volume snapshot, we can took EBS Volume snapshot from first EC2 and attach this on second EC2 instance on launch hook.

AWS Well Architecture

Reference Architecture Examples and Best Practices

AWS Solutions Library | Amazon Web Services | AWS

AWS Trusted Advisor provides recommendations that help you follow AWS best practices. It evaluates your account by using checks. These checks identify ways to optimize your AWS infrastructure, improve security and performance, reduce costs, and monitor service quotas.

Bound to Global

IAM,

Bound To Region

VPC, ALB, NLB, ASG, RDS

Bound To AZ

EC2, ENI, EBS, Subnet,