Skip to main content

Data Foundations Service

Loading component author status...

The Data Foundations service component (component_code = "df") manages the central data transformation pipelines and data warehouse infrastructure for the AI Platform, hosting two key capabilities:

  • dbt: Scheduled Cloud Run jobs executing dbt models against BigQuery datasets.
  • Dataform: Google Cloud Dataform repository pipelines for automated SQL workflow orchestration.

Architecture​

  • GCP Projects: Dedicated projects per environment (development, staging, production) configured for BigQuery, Cloud Run, Artifact Registry, and Dataform.
  • CI/CD Repositories:
    • uk-dvt-gcloud-data-foundations-dbt-iac & uk-dvt-gcloud-data-foundations-dbt-codebase: Manage dbt transformation jobs and SQL models.
    • uk-dvt-gcloud-data-foundations-dataform-iac & uk-dvt-gcloud-data-foundations-dataform-codebase: Manage Dataform pipeline configurations and SQLX workflows.
  • Service Accounts: Runner service accounts configured with Workload Identity Federation for automated pipeline execution.