Demo deployment

The demo deployment offers a streamlined experience using a predefined sample-dataset. It provides an end-to-end workflow to deploy the Cortex Framework Dataform pipelines for orchestration and to initialize the corresponding BigQuery datasets with sample data.

Before you begin

Ensure you have completed the prerequisites outlined on the Prerequisites page, before continuing with the demo deployment.

Get the required IAM roles and permissions

Obtain the required Identity and Access Management (IAM) permissions to deploy Cortex Framework content within the target Google Cloud project.

Roles for target project

To get the permissions that you need to deploy Cortex Framework, ask your administrator to grant you the following IAM roles on your target project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Optional: Get the required IAM roles for a demo deployment using a service account

The following roles are required only in case of a demo-deployment with the optional parameter --service_account enabled. Providing service account during demo-deployment simplifies the post-deployment execution of the data pipelines, by the creation of workflows execution scheduling-configuration. These roles aren't required for deployments without the --service_account configuration.

To get the permissions that you need to deploy the demo of Cortex Framework, ask your administrator to grant you the Service Account Token Creator (roles/iam.serviceAccountTokenCreator) IAM role on your source and target project. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Deployment

Follow these steps to create a virtual Python environment, synchronize the dependencies, and trigger the data pipelines. Remember to use the Cortex Framework uv scripts to build and push your data assets to your Google Cloud project, transforming your local configurations into a live, scalable data architecture. For more information, see the uv installation in the Prerequisites section.

Execute the demo deployment

Run the following command to trigger the deployment. This process will perform the following actions:

  • Verify that all prerequisites are complete.
  • Load sample data into a BigQuery dataset to serve as the raw layer for demonstration purposes.
  • Build Dataform pipelines for processing the sample data through the Cortex Framework data layers.
  • Create the Dataform repository and workspace, then sync the compiled artifacts with the repository.
  • Create workflow scheduling and trigger the initial Dataform workflow execution.

For demo deployment without a service account, execute the following command:

uv run cortex-demo --project_id=PROJECT_ID

For demo deployment with a service account, execute the following command:

uv run cortex-demo --project_id=PROJECT_ID \
    --service_account="SA_DF_RUN@PROJECT_ID.iam.gserviceaccount.com"

For more information, see Supported BigQuery locations and Supported Dataform regions.

CLI demo demployment: you can optionally display a list of possible parameters by running the uv run cortex-demo -h command. For more information, see CLI Cortex Framework demo deployment.

Verification

Once the deployment completes:

  1. Open Dataform to inspect the new code created in the repository:

    1. Created Dataform repository.
    2. Created Dataform development workspace.
    3. Compiled and synchronised code.

  2. Execute Dataform actions:

    1. Open the Dataform workspace.
    2. Click Start execution.
    3. Click Execute actions.
    4. Click All actions.

    5. Click Start execution.

    6. Monitor the successful execution of all actions in your Dataform repository.

  3. Navigate to the BigQuery console to view the created datasets, and verify the schema and data. Created datasets:

    • cortex_demo_sap_ecc_raw: This dataset contains the raw data from the source system (in this example, SAP S/4HANA).
    • cortex_demo_sap_ecc_data_foundation: This dataset represents the data foundation layer, where raw data from the raw layer is CDC processed.
    • cortex_demo_data_product: This dataset contains the demo data products-refined, high-performance views or tables designed for end-user consumption.