Summary
The user is experiencing task failures with "terminated in the background, manually" errors during regular workflows. They have attempted various solutions from the Flyte community, including enabling inject-finalizer: true
, using on-demand instances, and setting interruptible = False
with retries = 3
. They seek further troubleshooting suggestions and are interested in whether others have faced similar issues. The user has identified that the workflows were being terminated by Karpenter and resolved the issue by adding the annotation {"karpenter.sh/do-not-disrupt": "true"}
to the task using PodTemplate
.