Use the existing Airflow CaDeT deployer image

Oct 13, 2025 by magicalceylontours.com 39 views

Introduction

As a maintainer of CaDeT, it is essential to create a required workflow for EMDS to test in their account. This article outlines the steps to create a test (preprod) workflow for EMDS using an external Airflow role. The goal is to provide a seamless testing experience for EMDS, allowing them to replicate their requested test workflow without any issues.

Value / Purpose

Following the meeting with EMDS, we need to create an Airflow DAG that targets their requested test workflow for replication. EMDS will provide a branch to target that doesn't contain historic models for the purposes of testing. This workflow will enable EMDS to test their account without affecting the production environment.

Useful Contacts

The following contacts are useful for this project:

@matt-heery
@jhpyke

User Types

No specific user types are identified for this project.

Hypothesis

No hypothesis is proposed for this project.

Proposal

To create the test (preprod) workflow for EMDS using an external Airflow role, we will follow these steps:

Use the existing Airflow CaDeT deployer image: We will use the existing Airflow CaDeT deployer image to create a new release if required.
Populate the arguments: We will use the linked workflow to populate the arguments for the new DAG.
Target the same role as the existing Airflow job: We will ensure that the new DAG targets the same role as the existing Airflow job.
Replicate the secret value in their account: We may need to replicate the secret value in EMDS' account or specify that their role is able to access the SSH secret in our account to successfully clone the repo/branch.
Modify the secret/role: If required, we will modify the secret/role to allow the clone of the repo/branch.

Additional Information

No additional information is provided for this project.

Definition of Done

The following tasks are considered complete when the project is done:

DAG/Image Release exists for the EMDS workflow linked: A DAG/image release exists for the EMDS workflow linked.
EMDS are able to provide an assumable role for the workflow: EMDS are able to provide an assumable role for the workflow.
We are able to successfully clone the repo and deploy tables from a target branch in their account: We are able to successfully clone the repo and deploy tables from a target branch in EMDS' account.

Step-by-Step Guide to Creating the Test (Preprod) Workflow

Step 1: Create a New Airflow DAG

To create a new Airflow DAG, follow these steps:

Create a new file: Create a new file in the Airflow DAG directory with a name that reflects the purpose of the DAG.
Define the DAG: Define the DAG using the Airflow DAG syntax, including the DAG ID, description, and schedule.
Add tasks: Add tasks to the DAG using the Airflow task syntax, including the task ID, description, and dependencies.

Step 2: Populate the Arguments

To populate the arguments for the new DAG, follow these steps:

Use the linked workflow: Use the linked workflow to populate the arguments for the new DAG.
Update the DAG: Update the DAG to reflect the new arguments.

Step 3: Target the Same Role as the Existing Airflow Job

To target the same role as the existing Airflow job, follow these steps:

Identify the role: Identify the role used by the existing Airflow job.
Update the DAG: Update the DAG to target the same role.

Step 4: Replicate the Secret Value in Their Account

To replicate the secret value in EMDS' account, follow these steps:

Identify the secret: Identify the secret value required for the workflow.
Replicate the secret: Replicate the secret value in EMDS' account or specify that their role is able to access the SSH secret in our account.

Step 5: Modify the Secret/Role

To modify the secret/role, follow these steps:

Identify the secret/role: Identify the secret/role that needs to be modified.
Update the secret/role: Update the secret/role to allow the clone of the repo/branch.

Conclusion

Introduction

In our previous article, we outlined the steps to create a test (preprod) workflow for EMDS using an external Airflow role. In this article, we will answer some frequently asked questions (FAQs) related to this topic.

Q: What is the purpose of creating a test (preprod) workflow for EMDS?

A: The purpose of creating a test (preprod) workflow for EMDS is to provide a seamless testing experience for them, allowing them to replicate their requested test workflow without any issues.

Q: What is the difference between a test (preprod) workflow and a production workflow?

A: A test (preprod) workflow is a workflow that is used for testing and validation purposes, whereas a production workflow is a workflow that is used in a live production environment.

Q: How do I create a new Airflow DAG for the test (preprod) workflow?

A: To create a new Airflow DAG for the test (preprod) workflow, follow these steps:

Create a new file: Create a new file in the Airflow DAG directory with a name that reflects the purpose of the DAG.
Define the DAG: Define the DAG using the Airflow DAG syntax, including the DAG ID, description, and schedule.
Add tasks: Add tasks to the DAG using the Airflow task syntax, including the task ID, description, and dependencies.

Q: How do I populate the arguments for the new DAG?

A: To populate the arguments for the new DAG, follow these steps:

Use the linked workflow: Use the linked workflow to populate the arguments for the new DAG.
Update the DAG: Update the DAG to reflect the new arguments.

Q: How do I target the same role as the existing Airflow job?

A: To target the same role as the existing Airflow job, follow these steps:

Identify the role: Identify the role used by the existing Airflow job.
Update the DAG: Update the DAG to target the same role.

Q: How do I replicate the secret value in EMDS' account?

A: To replicate the secret value in EMDS' account, follow these steps:

Identify the secret: Identify the secret value required for the workflow.
Replicate the secret: Replicate the secret value in EMDS' account or specify that their role is able to access the SSH secret in our account.

Q: How do I modify the secret/role?

A: To modify the secret/role, follow these steps:

Identify the secret/role: Identify the secret/role that needs to be modified.
Update the secret/role: Update the secret/role to allow the clone of the repo/branch.

Q: What are the benefits of creating a test (preprod) workflow for EMDS?

A: The benefits of creating a test (preprod) workflow for EMDS include:

Improved testing experience: A test (preprod) workflow provides a seamless testing experience for EMDS, allowing them to replicate their requested test workflow without any issues.
Reduced risk: A test (preprod) workflow reduces the risk of errors or issues in the production environment.
Increased efficiency: A test (preprod) workflow increases efficiency by allowing EMDS to test their workflow in a controlled environment.

Conclusion

Creating a test (preprod) workflow for EMDS using an external Airflow role requires careful planning and execution. By following the steps outlined in this article and answering the FAQs, you can create a seamless testing experience for EMDS, allowing them to replicate their requested test workflow without any issues. Remember to update the DAG, populate the arguments, target the same role as the existing Airflow job, replicate the secret value in their account, and modify the secret/role as required.