F

Flyte

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Issue with Flyte function decorators

Summary

The user is facing an issue with using function decorators in Flyte to return a tuple of PyTorch classes, specifically encountering a pickling error with the DataLoader type due to ctypes objects containing pointers. They have shared their question in a Slack channel and noted that Flytekit uses type transformers for data handling between tasks. Since there is no pre-registered transformer for DataLoader, it defaults to pickle, causing the error. The user has been advised to create a custom transformer for DataLoader, as its implementation can vary widely based on data and distribution. An example for nn.module is provided in a GitHub link, and the user thanks two individuals for their helpful answers.

Status
resolved
Tags
    Source
    #ask-the-community
      h

      hassan.m.eng

      9/18/2024

      thanks <@UNW4VP36V> and <@UNZB4NW3S> both answers were helpful

      k

      kumare

      9/18/2024

      so <@U07MWGQ584D> this is an excellent question. So Flyte has to move data between tasks, flytekit provides a convenient way to handle any type of data. The type is used to invoke the right TypeTransformer . Incase Flytekit does not find a pre-registered transformer for a type, it resolves to using pickle.

      Your error says - that tye type DataLoader cannot be pickled. and this was because there is no inbuilt transformer.

      Solution you can implement a transformer for yourself. Reason to implement a custom one, is dataloaders can be completely data dependent, distributed etc. Hard to implement a common transformer.

      Example for nn.module https://github.com/flyteorg/flytekit/blob/fb55841f8660b2a31e99381dd06e42f8cd22758e/flytekit/extras/pytorch/native.py#L31

      h

      habuelfutuh

      9/18/2024

      Hey <@U07MWGQ584D>, I just posted the question on <#C06H1SFA19R|> and the answer pretty much sums up the current behavior: https://flyte-org.slack.com/archives/C06H1SFA19R/p1726676768973149?thread_ts=1726676767.636249&amp;cid=C06H1SFA19R

      h

      hassan.m.eng

      9/18/2024

      Hi when using flyte can function decorators handle pytorch classes as a return? for example im trying to return -&gt; tuple[pytorch class, pytrorch class] but I get error:

      'torch.utils.data.dataloader.DataLoader'&gt;.
Error Message: ctypes objects containing pointers cannot be pickled.```