F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Python Task Execution Suggestion

Summary

The user suggests adding Python task execution on Databricks using a new Databricks agent with a different name, as it would lead to different behavior. They mention using Flytekit's entrypoint to run a Python function task on Databricks, but note that it is not necessary in this case. While combining everything is possible, the user expresses concern that the resulting code would be messy and lacks a clean, readable solution.

Status
resolved
Tags
    Source
    #flyte-agents
      k

      kumare

      10/6/2024

      You don’t have to just wanted to see if anyone else wants it - cc <@U06AMB69K38> ?

      e

      eric901201

      10/6/2024

      I can connect them, and ask them for getting some example

      e

      eric901201

      10/6/2024

      does any user want that?

      e

      eric901201

      10/6/2024

      yes I can do that

      k

      kumare

      10/6/2024

      Absolutely different agent - but the same deployment secrets etc

      e

      eric901201

      10/6/2024

      put them all together is doable, but the code will be super messy (or I didn't come out a solution to make code clean and readable for others

      e

      eric901201

      10/6/2024

      but in your case, we don't have to use flytekit entrypoint

      e

      eric901201

      10/6/2024

      since now we will use flytekit's entrypoint to run python function task on databricks

      e

      eric901201

      10/6/2024

      different agent name will have different behavior

      e

      eric901201

      10/6/2024

      with a different name in my opinion

      e

      eric901201

      10/6/2024

      I think in this case, we have to create a new databrcisk agent

      k

      kumare

      10/6/2024

      Shall we add python task execution on databricks through databricks agent?