examples/cdc-pipeline.md

GitHub

CDC Pipeline

Change Data Capture pipeline streaming database changes to analytics warehouse in near real-time.

Key Elements

Component Stencil
DynamoDB mxgraph.aws4.dynamodb
DynamoDB Stream mxgraph.aws4.dynamodb_stream
Aurora mxgraph.aws4.aurora
Kinesis Data Streams mxgraph.aws4.kinesis_data_streams
Lambda mxgraph.aws4.lambda_function
S3 mxgraph.aws4.s3
Redshift mxgraph.aws4.redshift

Example

DynamoDB and Aurora changes captured and replicated to Redshift for analytics:

@startuml
left to right direction

rectangle "Transactional Databases" {
  mxgraph.aws4.dynamodb "Orders\n(DynamoDB)" as ddb
  mxgraph.aws4.aurora "Users\n(Aurora)" as aurora
}

rectangle "Change Capture" {
  mxgraph.aws4.dynamodb_stream "DynamoDB\nStream" as ddbstream
  mxgraph.aws4.kinesis_data_streams "Kinesis\n(Aurora CDC)" as kds
}

rectangle "Transform" {
  mxgraph.aws4.lambda_function "CDC\nProcessor" as fn1
  mxgraph.aws4.lambda_function "Schema\nMapper" as fn2
}

rectangle "Analytics" {
  mxgraph.aws4.s3 "Staging\n(S3)" as s3
  mxgraph.aws4.redshift "Redshift\nWarehouse" as rs
}

ddb ..> ddbstream
aurora ..> kds : "binlog CDC"
ddbstream --> fn1
kds --> fn2
fn1 --> s3
fn2 --> s3
s3 --> rs : "COPY"
@enduml

Pattern Notes

  1. DynamoDB Streams: captures INSERT/UPDATE/DELETE with 24h retention — Lambda polls for new changes
  2. Aurora CDC: binlog-based change capture streamed to Kinesis for decoupled processing
  3. Dashed arrows: ..> for async CDC events from source databases to streams
  4. S3 staging: intermediate Parquet files enable idempotent Redshift COPY and replay on failure
Icon Stencil Use When
DMS mxgraph.aws4.database_migration_service Ongoing replication from source DBs
AppFlow mxgraph.aws4.appflow SaaS-to-lake CDC integration
EventBridge mxgraph.aws4.eventbridge Change event fanout routing
Firehose mxgraph.aws4.kinesis_data_firehose Buffered delivery to S3/Redshift
Glue mxgraph.aws4.glue Schema evolution handling
MSK Connect mxgraph.aws4.msk_amazon_msk_connect Debezium CDC connectors