Data engineering is roughly split into these areas:
1. Ingestion / Extraction — getting data out of source systems (APIs, databases, scrapers, event streams). Tools: Fivetran, Airbyte, custom Python EL scripts, Kafka, Debezium.
2. Storage — where the data lands. Data warehouses (Snowflake, BigQuery, Redshift), data lakes (S3, GCS), lakehouses (Databricks, Iceberg), OLAP stores (ClickHouse).
3. Transformation / Modeling — turning raw data into clean, queryable, business-meaningful tables. Tools: dbt, SQLMesh, Spark.
4. Orchestration — coordinating when and how all the above runs. Tools: Dagster, Airflow, Prefect, cron (basic). This is the “glue” layer that makes the whole stack run reliably without someone manually triggering things.
5. Quality & Observability — tests, monitoring, lineage, alerting. Tools: dbt tests, Great Expectations, Monte Carlo, Dagster’s asset checks.
6. Serving — getting modeled data to the consumers: BI tools, APIs, reverse-ETL, dashboards.