F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Custom Serialization for FlyteFile

Summary

The user is seeking a method to serialize objects to JSON while ensuring that the _remote_source for FlyteFile is saved instead of just the local path. They currently use @dataclasses_json.to_json() for serialization, which works well except for FlyteFiles. The user suggests using .remote_source to set the path in FlyteFile, but faces challenges due to the nested structure of their objects. They are working on a PR to customize FlyteFile's serialization behavior by inheriting a Serialize Interface from mashumaro, but find the implementation complicated. The user notes that customizing FlyteFile's serialization is not straightforward without inheriting FlyteFile and writing custom code. They recommend iterating through all dataclass attributes to modify FlyteFile and highlight the differences in serialization and deserialization behavior between FlyteFile in flytekit and dataclass. They provide a code example for a custom serialization strategy and mention their PR addressing similar issues, expressing appreciation for the SerializableType abstraction for cleaner configurations. Ultimately, they opted for a different solution by storing longer strings in a file instead of serializing whole objects.

Status
resolved
Tags
    Source
    #ask-the-community