F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Issue with Flyte Custom Image Usage

Summary

The user is experiencing an issue with a Flyte workflow that is using a standard Docker image instead of their custom image. They are trying to understand why their custom image isn't being applied, despite providing a function for a custom pod specification. The user believes the base image cannot be overridden with a PodTemplate and thinks the custom image should be set directly on the task decorator. Additionally, their workflow is stuck in queue mode due to running in privileged mode, which conflicts with their GKE cluster policies that require privileged to be false. They are seeking help to find a Flyte base image that includes Docker installed inside it for use with a container image.

Status
open
Tags
    Source
    #ask-the-community
      a

      apprendidogcp

      9/27/2024

      then I can use container image

      a

      apprendidogcp

      9/27/2024

      can you please help me on that any flyte base image that contains docker installed inside it?

      a

      apprendidogcp

      9/27/2024

      yeah, that i tried when I am trying through task decorator either it might be pod spec or container image flyte workflow went into queue mode because i am running in privileged set to true, my gke cluster have some policies i.e., privileged set to false

      d

      david.espejo

      9/27/2024

      right, what I say is that (I'm not sure about this) you could still use podTemplate but override the image setting the container_image=your-image on the task decorator

      a

      apprendidogcp

      9/27/2024

      Here, I am using podspec

      d

      david.espejo

      9/27/2024

      I think the only thing you cannot override using a PodTemplate is the base image. What if you set your custom image on the task decorator instead?

      a

      apprendidogcp

      9/27/2024

      Hola I am running a workflow which should use custom image which specify but it using flyte default base image what is the reason can you please help this is my workflow

      from flytekit import task, workflow #from flytekitplugins.pod import PodTemplate from flytekit import workflow, task, PodTemplate from kubernetes.client import V1PodSpec, V1Container, V1SecurityContext, V1ResourceRequirements, V1VolumeMount, V1SecurityContext import subprocess from flytekitplugins.pod.task import Pod, PodFunctionTask from flytekit import task, workflow, dynamic from flytekit import PodTemplate, Resources import time import os def get_hardcoded_pod_spec() -> V1PodSpec: custom_image = "amorapprendido/flyte-gcld-dock-kube:v2" container = V1Container( name="custom-container", image=custom_image, ) pod_spec = V1PodSpec( containers=[container], restart_policy="OnFailure", security_context=V1SecurityContext(privileged=True) ) return pod_spec @task( task_config=Pod(pod_spec=get_hardcoded_pod_spec(), primary_container_name="custom-container") ) def hardcoded_pod_task() -> str: print("Hello from the task!") time.sleep(190) print("sleep complete") # Running the gsutil command and capturing the output # result = subprocess.run( # ['docker', 'images'], # check=True, # Ensures the command raises an error if it fails # capture_output=True, # Captures stdout and stderr # text=True # Returns output as a string # ) # Print the captured output # print("Subprocess output:", result.stdout.strip()) return "This is a hardcoded task running in a custom container!" @workflow def main_hardcoded_pod_workflow() -> str: return hardcoded_pod_task()