Flyte

<@U062Y21KSQG> the leader election in propeller is not really about consistency because the controller itself is stateless, it records execution state in etcd The mechanism is used to ensure that, while there may be multiple replicas in the propeller deployment, only one instance is active at a time and the other(s) remain "warm" in case the leader fails. Without leader election, K8s would still recreate the propeller pod in case of a failure but that could take a bit longer than just switching leaders. Also, the propeller replicas would be competing with each other, potentially trying to update the FlyteWorkflow CRD simultaneously.

<@U04H6UUE78B> thanks! Re: leader-election - what do I care? there's a database no?

> I didn't find where I should add memory/cpu to the pod You can uncomment and adjust deployment.resourcesto override default resources for the Pod:

https://github.com/flyteorg/flyte/blob/6c4f8dbfc6d23a0cd7bf81480856e9ae1dfa1b27/charts/flyte-binary/values.yaml#L235-L240

> Can I add replicas for the binary deployment? You can but there's no leader election mechanism enabled by default in single binary to handle properly multiple propeller instances so, for scaling out, flyte-core has these mechanisms available

<@U04H6UUE78B> thanks for getting back to me! I didn't find where I should add memory/cpu to the pod. I'm currently using the binary deployment and maybe with high load it's not good enough. Can I add replicas for the binary deployment?

<@U062Y21KSQG> running Flyte on OCI! I want to learn more :slightly_smiling_face:

Seems like you're hitting the max-workflow-retrieswhich is <https://docs.flyte.org/en/latest/deployment/configuration/generated/flytepropeller_config.html#max-workflow-retries-int|set to 10> by default. There must be a good reason the worker is running out of retries budget and I'd suggest using the <https://grafana.com/grafana/dashboards/21719-flyte-propeller-dashboard-via-prometheus/|Grafana dashboard> and look at patterns, especially during high load to understand better. Adding more resources to the Pod can help, but the next question would be: how much to add? Let us know if that helps

please give it some memory

also this explains why its dying

you did not configure it from the helm?

oddly, the flyte-binary pod has no resource requests/limts configured - how's that?

I didn't see anything specific with the resources memory was high, but CPU low

Also check resources etc

Are you using secrets, if not you can disable the webhook

Hi all, we have a self-hosted deployment of flyte-binary (on Orcale Cloud) and we started getting this error (only sometimes): Workflow[flyte-tasks:production:some-workflow] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: worker error(s) encountered: [0]: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": failed to call webhook: Post "<https://flyte-binary-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": EOF It's possible it happens mostly on higher loads. I have few questions:

1. Any way to increase or manage the retry attempts?
2. How can we understand why this actually failed? (the flyte-binary pod was overloaded?)
3. What does the webhook actually do?
4. If the answer to [3] is not something important - can we disable it? (I noticed propeller.disableWebhook configuration, or maybe the delete the relevant webhook resource?) Thanks!