F

Flyte

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Managing Multiple Workflows and Budgets

Summary

The user is seeking advice on managing multiple projects with separate budgets by limiting the number of simultaneously running workflows per project to avoid high costs. They note that maxParallelism only controls concurrent tasks within a workflow and does not address the total number of running workflows. The user is considering using task resource limits (CPU, GPU, memory) as a workaround but is open to better strategies or best practices. Additionally, they inquire about setting a global default timeout for tasks unless specified otherwise by the user.

Status
resolved
Tags
  • Workflow Management
  • Cost Control
  • flyte
  • User
  • Question
  • Support Request
  • Workflow Optimization
  • Resource Management
Source
#ask-the-community
    d

    david.espejo

    11/13/2024

    regarding the global timeout, I'm looking at this <https://docs.flyte.org/en/latest/deployment/configuration/generated/flyteadmin_config.html#workflow-active-deadline-config-duration|config parameter> and while I'm not sure it has the same effect as timeout in the task metadata, it's something we could try

    d

    david.espejo

    11/13/2024

    <@U06704L1RJ9> that's a very reasonable request.

    I think it's similar to what's described https://github.com/flyteorg/flyte/issues/5125|here? If that's the case please add a :+1::skin-tone-4: reaction to the issue, which is open for contributions btw

    As a workaround you could use ResourceQuotas that Flyte's ResourceManager can apply and enforce at the project-domain level.

    a

    adrian.salinero

    11/13/2024

    Hi everyone :wave:,

    We have multiple projects, each with its own budget, and we want to limit the number of workflows running simultaneously per project. This is to prevent users from unintentionally launching a large number of workflows, leading to unexpected high costs. The idea is to have a configurable cap for each project.

    As far as I know, maxParallelism only controls how many tasks within a single workflow can run concurrently, so it doesn’t solve our problem of limiting the total number of running workflows per project.

    I’m considering using task resource limits (CPU, GPU, memory) to control the amount of resources per task as a workaround. We could manually adjust these limits for each project. However, I’d love to hear if there are better strategies or best practices for this scenario.

    Also, is there a way to set a default timeout for every task globally, unless the user explicitly specifies a different timeout with @task(timeout=timedelta(hours=1))?

    Thanks in advance for any advice! :pray: