F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Guidance on Subworkflows and Error Handling

Summary

The user is seeking guidance from the Flyte community on two main issues regarding subworkflows. They want clarification on the differences between importing a workflow and using a reference workflow, including best practices, scenarios, and risks associated with reference workflows at scale. Additionally, they are inquiring about managing SubWorkflow failures at the node level in the parent workflow, aiming for graceful failure management while allowing dependent tasks to continue executing. The user highlights the need for flexibility in failure handling, especially when they lack control over subworkflows from different teams. They provide an example of a simulated failure in a subworkflow and discuss task dependencies, suggesting that some tasks should proceed even if others fail. They are also interested in the possibility of annotating task outputs as Optional for added flexibility and are looking for advice on error handling strategies, including failure thresholds and on-failure handlers. The user expresses gratitude for any assistance and reiterates their questions for clarity. They also mention the option of using @eager for arbitrary code execution and propose a call to discuss their use case further, asking about availability on Monday or Tuesday and the timezone of the respondent.

Status
resolved
Tags
    Source
    #ask-the-community