Summary
The user reports encountering a "RuntimeExecutionError" during a map task, indicating that maximum retry attempts have been exhausted. This error is linked to plugin execution with Kubernetes pods, suggesting modifications to an object that need to be applied to its latest version. The user is using Flyte backend version 1.13.1 and is questioning if a missing finalizer or a bug related to object deletion is the cause, while considering an update to a later version. A fix has been released in version 1.13.1, and the user is advised to ensure their propeller version is at least 1.13.1. There are configuration adjustments that could help, but they are not expected to be necessary with the recent change.
rupsha
Update went smoothly.. AND the logs link for map tasks is finally working properly
paul043
yup - shouldn't need any code changes for map tasks for versions after 1.12. I think the latest is v1.13.11
rupsha
Ah.. I’m on 1.11.. switch to 1.12? Any code changes required with that?
paul043
oh wait, it looks like you're using legacy map tasks. That's why that https://github.com/flyteorg/flyte/pull/5673|change isn't resolving the issue as we didn't update the clearFinalizers method utilized by the <https://github.com/flyteorg/flyte/blob/5065f9b2f11307a7680de1e744881d351c726822/flyteplugins/go/tasks/plugins/array/k8s/subtask.go#L146|legacy map task's plugin>.
What version of flytekit are you running? In https://github.com/flyteorg/flytekit/releases/tag/v1.12.0|v1.12.0 array node map tasks became the default for map tasks. There shouldn't be a need for any changes to workflow code if/when you update flytekit to utilize the new map task implementation.
rupsha
Confirmed that propeller is on 1.13.1
paul043
We put in a https://github.com/flyteorg/flyte/pull/5673|~fix for this that was released as part of https://github.com/flyteorg/flyte/releases/tag/v1.13.1|1.13.1.
Just wanted to confirm that your propeller version is >= 1.13.1. There are a few configs you can try adjusting to alleviate this issue but those adjustments hopefully shouldn’t be needed with that change.
rupsha
umm.. so should we update to a later version?
kumare
It seems finalizer is missing or was this a bug that was fixed I think in object deletion
rupsha
Hi team.. we are occasionally seeing this error with a map task… rerun succeeds
Operation cannot be fulfilled on pods "f82e2be5d27cc00fc000-n7-0-dn0-0-2427": the object has been modified; please apply your changes to the latest version and try again```
Flyte backend: 1.13.1