Government of Canada: Greenfield Data Platform
Match each workload to the right execution pattern, then operate it as one governed platform.
Context
A greenfield data platform inside a Government of Canada digital service, covering the AWS data estate, its access model, and the reliability of the data delivered to one small operating team.
Constraint
A greenfield government data platform had to support fundamentally different ingestion patterns while remaining operable by one team: scheduled external APIs, mid-sized managed ETL workloads, and larger distributed processing. The responsibility included the AWS data estate, its access model, and the reliability of the data delivered to the team.
Each source uses the execution pattern that fits it, while shared orchestration, infrastructure controls, and curated outputs keep the estate operable.
System
Owned the end-to-end data engineering platform on AWS. Built scheduled Lambda integrations for external APIs, Glue jobs for mid-sized transformations, and PySpark workloads in managed containers for heavier processing, with Step Functions coordinating dependencies, retries, branching, and failure handling. Defined infrastructure, compute, orchestration, permissions, and networking in Terraform, and owned the managed Apache Superset environment that exposed governed datasets to analysts and operational users.
Outcome
Established a reusable, infrastructure-as-code data platform that turned disparate source systems into governed analytical products. The team could operate ingestion and transformation workloads appropriate to each source, while analysts worked from curated datasets rather than the raw lake.
Approach
- 01AuditMapped each source to an execution pattern and operating requirement, from external APIs to mid-sized ETL work and heavier distributed processing.
- 02ArchitectureDesigned and implemented an AWS processing framework with scheduled Lambdas for external APIs, Glue for mid-sized transforms, and PySpark in managed containers for higher-volume work.
- 03Orchestration & IaCUsed Step Functions to coordinate dependencies, retries, branching, and failure handling, while defining infrastructure, compute, orchestration, permissions, and networking in Terraform.
- 04Analytics enablementOwned the organization's managed Apache Superset environment, exposing curated, governed datasets to analysts and operational users through self-service dashboards instead of direct lake access.
- AI Opportunity & Roadmap Assessment - A fixed-scope assessment that turns AI ambiguity into a decision-ready roadmap: where AI pays, in what order, and why, before you commit build budget.
- Data Platform & Cloud Foundations - Data your AI initiatives and analytics teams can trust: governed pipelines, a modeled warehouse and self-serve BI, built by someone who has shipped this in federal government and telecom environments.
- Fractional Technical Leadership - Principal-level AI and data leadership without the full-time hire: architecture review, technical direction between executives and engineering, and embedded hands-on delivery, grounded in years of building inside the Government of Canada.
Discuss a similar initiative
30 minutes to map the decision, the data behind it, and the shortest credible path to a production outcome.