Summary
The user is looking for a way to deploy the Flyte binary helm chart in a Kubernetes cluster with strict namespace isolation, facing issues due to Flyte's default behavior of creating namespaces and requiring ClusterRoles. They have tried using namespace-level Roles but still face errors related to custom resource definitions at the cluster scope. The user suggests that Flyte could function in a single namespace mode and proposes replacing ClusterRole with Role. They mention the need to modify the CLI for starting the propeller and provide a GitHub link to prevent Helm from creating the ClusterRole. Although they have not implemented these suggestions yet, they are considering creating a Role and RoleBinding for Flyte's service account and the CRD's API group.
david.espejo
<@U07FA02NECX> I tested the chart and can confirm the expected behavior with the CRD creation. Thank you! Merged :white_check_mark:
spillways
<@U04H6UUE78B> is there anything else needed for this PR or can you merge? :slightly_smiling_face:
spillways
I've made changes as per your suggestions - anything else needed to get this merged?
spillways
<@U04H6UUE78B> https://github.com/flyteorg/flyte/pull/5967
david.espejo
it should have that effect, yes Making it configurable for flyte-binary would be a great contribution. Let me know if you need help with that. Thank you!
spillways
Will setting
create-flyteworkflow-crd: false```
have a desired effect here? <https://github.com/flyteorg/flyte/blob/f7450303998094289cef2f8106c185eb8efc670a/charts/flyte-binary/templates/configmap.yaml#L43>
I could create PR for that
spillways
Yeah, I would prefer to use flyte-binary
, since it's easier and more suitable to our needs. I also saw that there is no option to install CRD as a part of Helm Chart in flyte-binary
:disappointed:
david.espejo
<@U07FA02NECX> with flyte-core
is possible to disable CRD creation by Helm so you could have more control over the process:
https://github.com/flyteorg/flyte/blob/13b3d82c94fb61047a49aa2093a2b17fb6a7c792/charts/flyte-core/templates/propeller/crds/flyteworkflow.yaml#L2
I think CRDs are always cluster-scoped but instances are, in the case of the flyteworkflow
CRD, namespaced https://github.com/flyteorg/flyte/blob/13b3d82c94fb61047a49aa2093a2b17fb6a7c792/charts/flyte-core/templates/propeller/crds/flyteworkflow.yaml#L19
spillways
<@U04H6UUE78B> this is exactly what I did - I have used rbac.create=false
and used my own Role
/ RoleBinding
instead.
> I've tried to work around the ClusteRole
by using namespace-level Role
instead
Once the Deployment starts in k8s, it wants to create the CRD at runtime and due to missing ClusterRole
- it fails.
> you will also have to change the cli for starting propeller
<@UNZB4NW3S> - which one? I'm using flyte-binary
helm chart
david.espejo
<@U07FA02NECX> according to the templates, if you set this to false
https://github.com/flyteorg/flyte/blob/13b3d82c94fb61047a49aa2093a2b17fb6a7c792/charts/flyte-binary/values.yaml#L368-L370
Helm won't create the ClusterRole
I haven't tried any of this, but then you could create a Role and RoleBinding to Flyte's service account and the CRD's API group
kumare
i like the idea of having a single namespace mode available
kumare
you will also have to change the cli for starting propeller
spillways
So replacement of ClusterRole by Role should work?
kumare
Flyte can work on one namespace
kumare
The crd I think needs a cluster role
spillways
Is it possible to have flyte deployment (I'm trying to deploy flyte-binary helm chart) scoped to a single / limited number of namespaces? We have K8s cluster that has strict isolation policies and Flyte's behaviour of creating namespaces + having the need to have some ClusterRole
s being present is a major blocker. I saw that for namespace's there's a
namespace_config:
namespace_mapping:
template: <my single NS>```
setting. I've tried to work around the `ClusteRole` by using namespace-level `Role` instead, but it still Flyte fails to start due to:
```cannot create resource \"customresourcedefinitions\" in API group \"<http://apiextensions.k8s.io|apiextensions.k8s.io>\" at the cluster scope```
Is there any way around that? :thinking_face: