F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Managing Reusable Tasks in Flyte

Summary

The user is seeking advice on managing reusable tasks in Flyte, particularly regarding issues with registering multiple workflows that share tasks from the same git hash. They encounter a problem where only the first workflow registers successfully, while subsequent workflows fail due to task name and version conflicts. The user employs the command flytectl register --continueOnError and uses a versioning scheme of {branch_name} + {git_hash}. They provide a log example of the error faced when registering a second workflow and suggest a temporary solution of adding a timestamp to the version to prevent conflicts. Additionally, they inquire if Flyte might consider using a combination of workflow + function reference + version as a task key in the future, as relying solely on function references poses challenges for their users. They also discuss the need to update the version in the reference entity when the upstream entity is updated to avoid stale versions.

Status
resolved
Tags
    Source
    #ask-the-community