F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Error in Dynamic Workflow Creation

Summary

The user is attempting to create a data processing workflow using dynamic workflows and map_task with a custom data list. They are working on a main workflow called wf_transform_input_imgs that processes batches of CustomDataClass through a task named process_data within a dynamic workflow called map_data_proc_tasks, with the goal of merging results into a JSONL file. However, they encounter an error when registering the tasks and workflows, which indicates a conflict with an existing task structure. Despite restarting the sandbox cluster, the error persists, displaying an unclear long alphanumeric string. The error implies that a task with a different structure already exists, leading to the conflict. The user has not reinstalled any Python packages and continues to experience the same error with any dynamic workflow they create.

Status
open
Tags
    Source
    #ask-the-community