Summary
The user is facing an issue with using function decorators in Flyte to return a tuple of PyTorch classes, specifically encountering a pickling error with the DataLoader type due to ctypes objects containing pointers. They have shared their question in a Slack channel and noted that Flytekit uses type transformers for data handling between tasks. Since there is no pre-registered transformer for DataLoader, it defaults to pickle, causing the error. The user has been advised to create a custom transformer for DataLoader, as its implementation can vary widely based on data and distribution. An example for nn.module
is provided in a GitHub link, and the user thanks two individuals for their helpful answers.
hassan.m.eng
thanks <@UNW4VP36V> and <@UNZB4NW3S> both answers were helpful
kumare
so <@U07MWGQ584D> this is an excellent question. So Flyte has to move data between tasks, flytekit provides a convenient way to handle any type
of data. The type
is used to invoke the right TypeTransformer
. Incase Flytekit does not find a pre-registered transformer for a type, it resolves to using pickle.
Your error says - that tye type DataLoader cannot be pickled. and this was because there is no inbuilt transformer.
Solution you can implement a transformer for yourself. Reason to implement a custom one, is dataloaders can be completely data dependent, distributed etc. Hard to implement a common transformer.
Example for nn.module
https://github.com/flyteorg/flytekit/blob/fb55841f8660b2a31e99381dd06e42f8cd22758e/flytekit/extras/pytorch/native.py#L31
habuelfutuh
Hey <@U07MWGQ584D>, I just posted the question on <#C06H1SFA19R|> and the answer pretty much sums up the current behavior: https://flyte-org.slack.com/archives/C06H1SFA19R/p1726676768973149?thread_ts=1726676767.636249&cid=C06H1SFA19R
hassan.m.eng
Hi when using flyte can function decorators handle pytorch classes as a return? for example im trying to return -> tuple[pytorch class, pytrorch class]
but I get error:
'torch.utils.data.dataloader.DataLoader'>.
Error Message: ctypes objects containing pointers cannot be pickled.```