F

Flyte

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Concerns About CICD Workflow Documentation

Summary

The user discusses their CICD process, which involves workflows, registered tasks, and sub-workflows, aiming to pre-register reusable tasks for various workflows. They express concerns about the current documentation on reference tasks, highlighting two main limitations: 1. The requirement to bind tasks to a specific version, which conflicts with their CICD approach of registering tasks first and allowing workflows to fetch the latest task versions.2. The need for a two-step process to package and register tasks, which complicates the fast registration of workflows that reference these tasks.The user questions whether they are misunderstanding the intended use of reference tasks or if the limitations are inherent to the system, as they seek to ensure consistent task version usage across multiple workflows.

Status
open
Tags
  • Workflows
  • Documentation Issue
  • CICD Process
  • flyte
  • Documentation
  • Developer
  • Question
  • CICD
Source
#ask-the-community
    n

    nuhaj

    11/13/2024

    I can take a look

    d

    david.espejo

    11/13/2024

    right. Do you think this is an area where you could contribute?

    n

    nuhaj

    11/12/2024

    especially the part about it fetching at runtime. No matter what you use, flyteRemote, reference tasks, launchplans, the version is coupled at registration of the workflow. If the task is modified in the future the registered workflow is still coupled with the older version of the task.

    n

    nuhaj

    11/12/2024

    yes!

    d

    david.espejo

    11/12/2024

    Hey <@U073L5Y9GJD> would something like what's described in this issue be helpful for you? https://github.com/flyteorg/flyte/issues/3656

    n

    nuhaj

    11/5/2024

    Hello, Our CICD consists of workflows, registered tasks and registered sub-workflows. The idea was to pre-register reusable tasks to be used by different workflows within a project. An example: a standardized QC task that can be referenced in many different workflows.

    The current https://docs.flyte.org/en/latest/user_guide/productionizing/reference_tasks.html#reference-tasks|documentation on registration seems to limit reference workflows to

    1. bind to a specific version
    2. Use the macro and package/register the task with the workflow at the same time. This seems to limit the power of reference tasks.

    • 1. knowing the version is not conducive to our CICD. The CICD registers the tasks first, then workflows. The workflows with reference tasks should be able to fetch the latest task versions. "<https://docs.flyte.org/en/latest/user_guide/concepts/main_concepts/tasks.html#tasks|Tasks are fully independent units of execution and first-class entities of Flyte>" • 2. the version macro requires the 2 step pkg + register instead of fast registration. if you want to pull in the latest reference task when you register your workflow you need to register the reference task using the same version you are about to register the workflow with. which sends us back to square one • 2. Limiting reference tasks to be packaged with workflows seems to negate the power of pre-registration and references tasks. Consider the scenario where a task is referenced in 2 different workflows and we want to ensure everyone is using the same task version. commands used pyflyte --pkgs workflows package --copy all -f flytectl register files --archive flyte-packag.tgz --project mypyoj --domain development --version test-0

    Are we missing something here or were reference tasks never meant to be used in the way we planned?