EV Intelligence Platform
End-to-End Azure Data Engineering

EV telemetry arrives from IoT sensors, fleet APIs, PostgreSQL operational DB, partner CSVs, and third-party XML feeds — all in different formats and at different frequencies.

Business teams can't trust vehicle usage reports because raw data has duplicates, late arrivals, schema drift, and no lineage tracking.

Analysts wait 24+ hours for daily reports. Charging station utilisation data is stale before decisions can be made on fleet routing and energy pricing.

DB passwords and API keys are hardcoded across notebooks and scripts. Every new joiner gets credentials in Slack — a security and audit nightmare.

Vehicle ID, make, model, battery capacity, registration, active flag — SCD Type 2 for historical tracking.

Station ID, location, charger type (AC/DC), max power, operator — geo-partitioned.

Full date spine and time grain — used for all joins in fact tables.

Session-level fact — vehicle, station, energy consumed, duration, cost, status. ~15M rows/month.

Aggregated hourly vehicle telemetry — speed, battery %, odometer, temperature. Pre-aggregated from raw 20M events/day.

Station-level cost per kWh by time window — joined with sessions for profitability reporting.

PySpark jobs, Delta Live Tables concept, Unity Catalog basics, job clusters, and workflow scheduling.

Real-time ingestion of EV IoT telemetry at 20M+ events/day into Bronze Delta table.

Tiered landing zones — hot tier for active ingestion, cool tier for historical Bronze files.

Parameterised pipelines, ForEach loops, copy activities, triggers, integration runtimes.

Gold layer serving — dedicated SQL pool, external tables, PolyBase, partitioning strategy.

All secrets — DB passwords, API keys, SAS tokens — fetched at runtime. No plaintext credentials anywhere.

Operational fleet + charging DB — we implement CDC and incremental extraction patterns on it.

ACID + schema enforcement + time travel + MERGE (upsert) — foundation of all three Medallion layers.

Pipeline run logs, failure alerts, SLA dashboards, and cost tracking — real production observability.

Service principals, RBAC for storage and Databricks, managed identities — zero-credential architecture.

Load raw → transform in-platform. Separation of concerns between ingestion and transformation layers.

Watermark-based and log-based CDC from PostgreSQL. Only process changed rows — not full dumps.

Track last-processed timestamp per source table. Re-runnable without reprocessing old data.

MERGE-based upserts on Delta tables. Every pipeline run produces identical output regardless of how many times it runs.

Re-process historical date ranges without breaking current data. Date-partition override approach.

Track vehicle and station attribute changes over time — effective_from / effective_to pattern.

Single parameterised reader handles Parquet, CSV, JSON, XML, ORC — format detected from metadata config.

Delta Lake schema mergeSchema + explicit schema checks. Alerting on unexpected column additions from upstream.

Event-time vs processing-time reconciliation. Reprocess affected partitions from Bronze when late events arrive.

Partitioning strategy, Z-ordering, file compaction, cluster sizing, cost vs latency trade-offs — all documented.

Malformed JSON / truncated Parquet from partner drops — quarantine to dead-letter folder, alert, continue pipeline.

PostgreSQL connection timeout mid-batch — retry logic with exponential backoff, checkpoint recovery.