Flyte

Thanks for the info :) I will take a look into it and get back to you if this doesn't work for our use case

<@U06V6CQTKL6> the long SAS has been an open issue for a long time because of some library complexities. have you tried https://azure.github.io/azure-workload-identity/docs/introduction.html|workload identity? feel free to reach out if you have any questions about implementation

Hi all <#C05315T4K5K|> :slightly_smiling_face: We want to use Azure Storage account blob SAS url to create a FlyteFile. The problem with this, is that the FlyteFile maps the whole file path + the sas in the url as the file name. This means that Flyte sees the url as a directory and not a file. Is there a workaround for this, or is this feature missing? Thank you in advance

<@UNR3C6Y4T>

we aren't running in prod and have a limited use case, but we have workload identity + azure blob store working for common flyte workflows

using workload identity and "remoteData is configured to set signedUrls:false " should be enough to bypass the issue

right <@U05QG8SE2LA> Is this also a limitation even if <@U06V6CQTKL6> used Workload Identity instead of SAS tokens for storage account access?

We might want to pull Yee into the conversation since he was looking at the python fixes. iirc, he had a local solution but was concerned about its implications for other use cases

as mentioned in the flyte golang pr, i don't think the lack of support for that endpoint is the fundamental problem here. i think the two flytekit (fsspec) bugs are actually to blame for the behavior <@U06V6CQTKL6> is seeing. https://github.com/flyteorg/flyte/issues/4701 https://github.com/flyteorg/flyte/issues/4700

sorry, i have been out of the flyte ecosystem for a while now. expected to get back in this week or next, so this is good timing. give me a little while to catch up though

Not sure, for now I'm deferring to <@U05QG8SE2LA> to validate what's missing to merge

Yes, it would :slightly_smiling_face: When do you think this could be available?

<@U06V6CQTKL6> would <https://github.com/flyteorg/flyte/pull/4629|this PR> cover what you intend to do?

It seems that it interprets the query from '/tmp/flytecdmj730u/local_flytekit/ec7c7207da04cd009680ed0636b3277e/a.txt?sp=r&st=2024-09-17T14:10:24Z&se=2024-11-01T23:10:24Z&spr=https&sv=2022-11-02&sr=b&sig=A2WINhWtCSfJNJ8sdqodQJrxKoNjz%2FGfmHUln1VlDf4%3D' , so just ?sp=r&st=2024-09-17T14:10:24Z&se=2024-11-01T23:10:24Z&spr=https&sv=2022-11-02&sr=b&sig=A2WINhWtCSfJNJ8sdqodQJrxKoNjz%2FGfmHUln1VlDf4%3D as part of the file name (perhaps because it doesn't end with the file extension) and flyte assumes it's a directory

Now I get it - yes if you are using http (signed url) it will use http protocol and that cannot download directories only files. But weirdly the path looks like a dir. this seems we should default to assuming a file and proceeding

Shared Access Signature - https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

Running a simple workflow like this: from flytekit import task, workflow, Resources from flytekit.types.file import FlyteFile import os

@task(_requests_=Resources(_cpu_="1", _mem_="1Gi"), _limits_=Resources(_cpu_="2", _mem_="1Gi")) def normal_task(_sas_: str) -> str: new_sas = FlyteFile.from_source(_sas_) with open(new_sas, "r") as f: text = f.read()

return text

@workflow def wf(_sas_: str) -> str: normal_output = normal_task(_sas_=_sas_)

return normal_output

We would need examples and pointers - I do not follow

Sorry what is sas?