F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Azure Blob Storage Dynamic Workflow Issues

Summary

The user is facing issues with Azure Blob Storage while implementing dynamic workflows in Flyte. Standard tasks function correctly with Azure credentials, but dynamic workflows trigger additional Python tasks that lack these credentials, resulting in access failures. They seek advice on how to provide credentials to the flytekit container or prevent the initiation of extra Python tasks while keeping workflow dynamics intact. After adding environmental variables for Azure storage account details to the pod, access issues were resolved, but a new error related to the 'dispatch_execute' attribute in the 'PythonFunctionWorkflow' object has emerged, indicating an internal error unrelated to Azure. The user is looking for suggestions to address this new issue and mentions the complexity of their setup, which includes custom images in a private container registry. They are considering testing with Azure by creating a dummy public storage account and provide a definition of a dynamic workflow using the @workflow() and @dynamic decorators. They note that the problem arises only when both decorators are used together, while using @dynamic solely for subworkflows does not present issues.

Status
resolved
Tags
    Source
    #ask-the-community