F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Deploying Flyte with Namespace Isolation

Summary

The user is looking for a way to deploy the Flyte binary helm chart in a Kubernetes cluster with strict namespace isolation, facing issues due to Flyte's default behavior of creating namespaces and requiring ClusterRoles. They have tried using namespace-level Roles but still face errors related to custom resource definitions at the cluster scope. The user suggests that Flyte could function in a single namespace mode and proposes replacing ClusterRole with Role. They mention the need to modify the CLI for starting the propeller and provide a GitHub link to prevent Helm from creating the ClusterRole. Although they have not implemented these suggestions yet, they are considering creating a Role and RoleBinding for Flyte's service account and the CRD's API group.

Status
resolved
Tags
  • Helm Chart
  • flyte
  • Flyte
  • GitHub
  • Support Need
  • Developer
  • Deployment Issues
  • Product Help
  • Helm
  • Deployment
  • Namespace
Source
#flyte-deployment
    d

    david.espejo

    11/18/2024

    <@U07FA02NECX> I tested the chart and can confirm the expected behavior with the CRD creation. Thank you! Merged :white_check_mark:

    s

    spillways

    11/18/2024

    <@U04H6UUE78B> is there anything else needed for this PR or can you merge? :slightly_smiling_face:

    s

    spillways

    11/7/2024

    I've made changes as per your suggestions - anything else needed to get this merged?

    s

    spillways

    11/6/2024
    d

    david.espejo

    11/5/2024

    it should have that effect, yes Making it configurable for flyte-binary would be a great contribution. Let me know if you need help with that. Thank you!

    s

    spillways

    11/4/2024

    Will setting

          create-flyteworkflow-crd: false```
    have a desired effect here? <https://github.com/flyteorg/flyte/blob/f7450303998094289cef2f8106c185eb8efc670a/charts/flyte-binary/templates/configmap.yaml#L43>
    
    I could create PR for that
    
    s

    spillways

    11/4/2024

    Yeah, I would prefer to use flyte-binary, since it's easier and more suitable to our needs. I also saw that there is no option to install CRD as a part of Helm Chart in flyte-binary :disappointed:

    d

    david.espejo

    10/28/2024

    <@U07FA02NECX> with flyte-core is possible to disable CRD creation by Helm so you could have more control over the process: https://github.com/flyteorg/flyte/blob/13b3d82c94fb61047a49aa2093a2b17fb6a7c792/charts/flyte-core/templates/propeller/crds/flyteworkflow.yaml#L2

    I think CRDs are always cluster-scoped but instances are, in the case of the flyteworkflow CRD, namespaced https://github.com/flyteorg/flyte/blob/13b3d82c94fb61047a49aa2093a2b17fb6a7c792/charts/flyte-core/templates/propeller/crds/flyteworkflow.yaml#L19

    s

    spillways

    10/28/2024

    <@U04H6UUE78B> this is exactly what I did - I have used rbac.create=false and used my own Role / RoleBinding instead. > I've tried to work around the ClusteRole by using namespace-level Role instead

    Once the Deployment starts in k8s, it wants to create the CRD at runtime and due to missing ClusterRole - it fails.

    > you will also have to change the cli for starting propeller <@UNZB4NW3S> - which one? I'm using flyte-binary helm chart

    d

    david.espejo

    10/25/2024

    <@U07FA02NECX> according to the templates, if you set this to false https://github.com/flyteorg/flyte/blob/13b3d82c94fb61047a49aa2093a2b17fb6a7c792/charts/flyte-binary/values.yaml#L368-L370

    Helm won't create the ClusterRole

    I haven't tried any of this, but then you could create a Role and RoleBinding to Flyte's service account and the CRD's API group

    k

    kumare

    10/25/2024

    i like the idea of having a single namespace mode available

    k

    kumare

    10/25/2024

    you will also have to change the cli for starting propeller

    s

    spillways

    10/25/2024

    So replacement of ClusterRole by Role should work?

    k

    kumare

    10/25/2024

    Flyte can work on one namespace

    k

    kumare

    10/25/2024

    The crd I think needs a cluster role

    s

    spillways

    10/25/2024

    Is it possible to have flyte deployment (I'm trying to deploy flyte-binary helm chart) scoped to a single / limited number of namespaces? We have K8s cluster that has strict isolation policies and Flyte's behaviour of creating namespaces + having the need to have some ClusterRole s being present is a major blocker. I saw that for namespace's there's a

        namespace_config:
          namespace_mapping:
            template: &lt;my single NS&gt;```
    setting. I've tried to work around the `ClusteRole` by using namespace-level `Role` instead, but it still Flyte fails to start due to:
    ```cannot create resource \"customresourcedefinitions\" in API group \"<http://apiextensions.k8s.io|apiextensions.k8s.io>\" at the cluster scope```
    Is there any way around that? :thinking_face: