Summary
The user asks if a cleanup task can be performed upon termination via the user interface, mentioning that the failure node fn0 is skipped during UI termination. They observe that cleanup functions correctly when a node fails manually but not when using the UI's Abort function, indicating a need for cleanup in that scenario as well. The user also considers using agents for cleanup but finds the agents interface more complex regarding deployment and infrastructure, leading them to avoid it.
josh210
cc <@U077P82HG8H> <@U07JCU5EBTN> <@U07JLUBCNFN> This would make it a lot easier to control some of our spark costs
josh210
or help to test/develop it
josh210
honestly I think if you could just deploy an agent ephemerally as if it were a task (which is basically the proposal in the issue you linked) I'm sure we would use that feature
david.espejo
> the agents interface has a more complicated deploy/infra considerations so we've shied away from it I hear you, and your feedback is super appreciated. If there's any way we can help you, it'd be amazing
josh210
We need to cleanup some compute resources on termination. I think agents might also work but tbh the agents interface has a more complicated deploy/infra considerations so we've shied away from it
josh210
I think the first one covers our use case better
david.espejo
I think this one is closer: https://github.com/flyteorg/flyte/issues/3854
david.espejo
<@U07655DJTDM> is this similar to what's described by <https://github.com/flyteorg/flyte/issues/2810|this issue>?
josh210
yep, but we have things we need to teardown on Abort too!
architr
Ahh, so it doesn’t execute the cleanup task when killed via the UI based Abort :thinking_face:
josh210
I made a node fail by raising an exception in it, and it runs the cleanup as expected
josh210
the failure node is that fn0 , it's getting skipped when i terminate from the UI
architr
Perhaps failure node: https://docs.flyte.org/en/latest/user_guide/development_lifecycle/failure_node.html
josh210
Is there a way to make a cleanup task execute on termination from within the UI?