Government of Canada: Greenfield Data Platform
Match each workload to the right execution pattern, then operate it as one governed platform.
Context
A greenfield data platform inside a Government of Canada digital service, covering the AWS data estate, its access model, and the reliability of the data delivered to one small operating team.
Constraint
A greenfield government data platform had to support fundamentally different ingestion patterns while remaining operable by one team: scheduled external APIs, mid-sized managed ETL workloads, and larger distributed processing. The responsibility included the AWS data estate, its access model, and the reliability of the data delivered to the team.
Each source uses the execution pattern that fits it, while shared orchestration, infrastructure controls, and curated outputs keep the estate operable.
System
Owned the end-to-end data engineering platform on AWS. Built scheduled Lambda integrations for external APIs, Glue jobs for mid-sized transformations, and PySpark workloads in managed containers for heavier processing, with Step Functions coordinating dependencies, retries, branching, and failure handling. Defined infrastructure, compute, orchestration, permissions, and networking in Terraform, and owned the managed Apache Superset environment that exposed governed datasets to analysts and operational users.
Outcome
Established a reusable, infrastructure-as-code data platform that turned disparate source systems into governed analytical products. The team could operate ingestion and transformation workloads appropriate to each source, while analysts worked from curated datasets rather than the raw lake.
Approach
- 01AuditMapped each source to an execution pattern and operating requirement, from external APIs to mid-sized ETL work and heavier distributed processing.
- 02ArchitectureDesigned and implemented an AWS processing framework with scheduled Lambdas for external APIs, Glue for mid-sized transforms, and PySpark in managed containers for higher-volume work.
- 03Orchestration & IaCUsed Step Functions to coordinate dependencies, retries, branching, and failure handling, while defining infrastructure, compute, orchestration, permissions, and networking in Terraform.
- 04Analytics enablementOwned the organization's managed Apache Superset environment, exposing curated, governed datasets to analysts and operational users through self-service dashboards instead of direct lake access.
- AI Opportunity & Roadmap Assessment - Decide what to build, evaluate or connect before committing implementation budget. I review your use case, existing systems and data readiness, then set out the options, dependencies and a practical next scope.
- Data Platform & Cloud Foundations - Build the data pipelines and cloud foundations behind your analytics and AI systems. I implement ingestion, transformation and usable data models, with validation, operational visibility and a handover your engineers can use.
- Fractional Technical Leadership - Senior AI and data engineering guidance alongside your team. I help review architecture, resolve technical decisions and shape implementation priorities. We agree a regular review cadence and the decisions or implementation work I will own.
- Architecture & Implementation - Work directly with a senior engineer to deliver a defined AI or data project. I take responsibility for architecture, hands-on implementation and technical handover, using acceptance criteria agreed with your team.
Discuss a similar initiative
30 minutes to map the decision, the data behind it, and the shortest credible path to a production outcome.