Summary
The user is using Flyte v1.11.0 and has implemented Intratask checkpoints, which work with normal tasks but not with mapped tasks. They reference documentation suggesting the use of ArrayNode for support and have imported map_task from flytekit.experimental, but still encounter an error regarding checkpointing availability. The user is questioning if version 1.11.0 supports checkpointing in mapped tasks. Additionally, they mention that the flyte.binary (v1.11.0) pod is restarting due to a fatal error: concurrent map writes, indicating possible issues with the array node. The user is using Python 3.11 and flytekit version 1.11.0 and plans to create a minimal example to share.
eric901201
And then I’ll ping maintianers
eric901201
Please provide example
eric901201
Ok
stephfen
That is a fair suggestion, unfortunately I have another issue with later flyte versions and the main reason I have not tried it. Will wait for that to be fixed (i.e. being able to define and run flyte within a jupyter notebook) and after that will try it.
What I mainly want to know if map_array is something that I should even try to use in v1.11.0 for intra task checkpoints or if I should just wait until I upgrade.
eric901201
I’m not familiar with map task, but I can try my best to help you
eric901201
Thank you
eric901201
And also try the latest flytekit
eric901201
I’ll help you ping someone who might know the answer after you try it
eric901201
Flyte
eric901201
Can you try the latest flute?
stephfen
Actually, flyte.binary (v1.11.0) pod is restarting due to an error fatal error: concurrent map writes so maybe the array node has some issues. (/flyteorg/build/flytepropeller/pkg/controller/nodes/array/execution_context.go:29 ). The python version is 3.11 and flytekit version is 1.11.0. I will try to create a minimal example and share.
eric901201
Can you provide your
stephfen
I currently am with Flyte v1.11.0 and started to use Intratask checkpoints. This works well when using normal tasks but fails when used inside mapped tasks. According to the docs (https://docs.flyte.org/en/v1.11.0/user_guide/advanced_composition/map_tasks.html) I would need to use ArrayNode in order to get support. So I imported the map_task from flytekit.experimental, but unfortunately, the same error Checkpointing is not available, please check the version of the platform. . Does v1.11.0 support checkpointing in mapped tasks?