Flyte

Hi everyone!

I am having some issues when trying to register a launchplan to Azure Blob Storage. When I execute pyflyte register ... I get the following errors: • AssertionError: Failed to upload file 00000 to https://<sa>.http://blob.core.windows.net/flyte/use-case/development/00000/tw56fjhrqbl9lzfbqqjg|blob.core.windows.net/flyte/use-case/development/00000/tw56fjhrqbl9lzfbqqjg reason An HTTP header that's mandatory for this request is not specified. https://github.com/flyteorg/flytekit/blob/v1.13.5/flytekit/remote/remote_fs.py#L85 • AssertionError: Failed to upload file 00000 to https://<sa>.http://blob.core.windows.net/flyte/use-case/development/00000/tw56fjhrqbl9lzfbqqjg|blob.core.windows.net/flyte/use-case/development/00000/tw56fjhrqbl9lzfbqqjg reason An HTTP header that's mandatory for this request is not specified. https://github.com/flyteorg/flytekit/blob/v1.13.5/flytekit/remote/remote_fs.py#L87 The workflow is very simple:

def output_result(output_data: pd.DataFrame) -> None:
    print(output_data)

...
@workflow
def wf(project: str = "testing") -> int:
    df_test = pd.DataFrame(
            {
                "test_id": [1234],
                "test_subject": ["Subject"],
                "test_content": ["Welcome!"],
            }
        )
    output_result(output_data=df_email)```
Any thoughts about how I could resolve or work around this? :slightly_smiling_face:

I can open an issue

could we at least identify the gaps on an issue_

Yeah that PR added the headers in the first required place but I think there are 2 other places that need similar treatment and one of them remains unmatched.

Thanks <@U05MFMUN6UC> I can confirm that adding those headers solved the issue. Seems like went on a wild goose chase as I must have logged the wrong set of headers (in my first message), because there the headers were present relating to <https://github.com/flyteorg/flytekit/pull/1784|this pull>. Because if I log the correct request headers resp.requests.headers I get:

{'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '3109'}```
The BlockBlob is not there anymore.

That does look familiar. I think its related to https://github.com/flyteorg/flytekit/blob/master/flytekit/remote/remote.py#L2458-L2461 not being used when it should be.

There is one place I'm aware of where this problem still exists. We've been using the following patch but we haven't got round to making a proper fix that we can upstream.

<@U06KZLKBV2Q> what type of identity are you using? is it Workload Identity or SAS token or? Definitely seems something is missing, but I wonder if maybe <@U05MFMUN6UC> or <@U05R10D3UNL> have seen a similar behavior?

Unfortunately the workaround is unstable.

With the workaround I am able to get 200 OKs from /flyteidl.service.AdminService/CreateTask :

[✔] Registration src.output.output_result_json type TASK successful with version DTbKVxvnhJ....```
And it registered properly. However, after about ~10 registrations. Then it started returning 502 Bad Gateways for `/flyteidl.service.AdminService/CreateWorkflow`:
```RPC Failed, with Status: StatusCode.UNAVAILABLE
	Details: Received http2 header with status: 502```

That said, I am not convinced that it is a solution to the problem. Because this just changes the dataframe (parquet) from being a FlyteFS to being an ABFS instead. Better yet would of course be to keep it being a FlyteFS and then upload it directly...

I looked into the problem a bit more and I noticed that by modifying https://github.com/flyteorg/flytekit/blob/2452c74fba00b21a4428d548fe4100966d779c72/flytekit/clis/sdk_in_container/register.py#L180,

...data_upload_location="<flyte://data>")
#to:
...data_upload_location="<abfs://data>")```
_almost_ solved the problem (I had to tweak my nginx ingress buffer size as well).

Thanks for your help!