examples/ml-feature-pipeline.md
GitHubML Feature Pipeline
Feature engineering pipeline for machine learning — from raw data to curated feature store.
Key Elements
| Component | Stencil |
|---|---|
| EMR | mxgraph.aws4.emr |
| EMR Engine | mxgraph.aws4.emr_engine |
| Glue | mxgraph.aws4.glue |
| Glue Data Catalog | mxgraph.aws4.glue_data_catalog |
| Athena | mxgraph.aws4.athena |
| S3 | mxgraph.aws4.s3 |
| DynamoDB | mxgraph.aws4.dynamodb |
| Lambda | mxgraph.aws4.lambda_function |
Example
Batch and real-time feature computation feeding an online/offline feature store:
@startuml
left to right direction
rectangle "Data Sources" {
mxgraph.aws4.s3 "Event\nLogs (S3)" as logs
mxgraph.aws4.rds "User\nDB (RDS)" as rds
mxgraph.aws4.kinesis_data_streams "Click\nStream" as clicks
}
rectangle "Batch Features" {
mxgraph.aws4.emr "EMR\nCluster" as emr
mxgraph.aws4.emr_engine "Spark\nJobs" as spark
mxgraph.aws4.glue "Glue\nETL" as glue
}
rectangle "Real-time Features" {
mxgraph.aws4.lambda_function "Feature\nLambda" as fn
}
rectangle "Feature Store" {
mxgraph.aws4.s3 "Offline Store\n(S3/Parquet)" as offline
mxgraph.aws4.dynamodb "Online Store\n(DynamoDB)" as online
mxgraph.aws4.glue_data_catalog "Feature\nCatalog" as catalog
}
logs --> emr
rds --> glue
emr --> spark
spark --> offline
glue --> offline
offline --> catalog
clicks ..> fn
fn --> online
@enduml
Pattern Notes
- Dual store: offline (S3 Parquet) for model training, online (DynamoDB) for low-latency inference
- EMR + Spark: heavy feature computation (aggregations, joins across billions of rows) on batch schedule
- Real-time features: Lambda computes sliding-window features from Kinesis and writes to DynamoDB
- Feature Catalog: Glue Data Catalog tracks feature schemas, versions, and lineage
Related Icons
| Icon | Stencil | Use When |
|---|---|---|
| SageMaker | mxgraph.aws4.sagemaker |
Model training & deployment |
| SageMaker Model | mxgraph.aws4.sagemaker_model |
Trained model artifact |
| Step Functions | mxgraph.aws4.step_functions |
Orchestrating feature pipelines |
| Athena | mxgraph.aws4.athena |
Ad-hoc feature validation SQL |
| Redshift | mxgraph.aws4.redshift |
Warehouse-based feature source |
| Glue DataBrew | mxgraph.aws4.glue_databrew |
Visual feature transformation |