F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Assistance with Flyte Launch Plan Creation

Summary

The user is seeking assistance in programmatically creating a launch plan in Flyte using a code snippet. They are experiencing issues with their Python code not producing results and have had no success with pyflyte launchplan and pyflyte run --remote. The user is working on a GitHub pull request and is trying to create a launch plan with an invalid cron string to validate recent changes. They provided a code snippet for a workflow involving tasks for calculating mean, standard deviation, and standard scaling. Despite their efforts, the launch plan creation did not work as expected, and they are looking for insights or guidance, particularly regarding the raw_register method for registering a launch plan.

Status
resolved
Tags
  • Workflow Assistance
  • Launch Plan Help
  • flyte
  • Launch Plan
  • Flyte
  • Python
  • User
  • Launch Plan Issues
  • Support Need
  • Developer
  • Python Code
  • Question
  • Developer Help
  • GitHub Example
Source
#ask-the-community
    m

    michaellai901026

    11/7/2024

    I think lhs is only used for imported instances, which is not really ideal. One way to bypass this error is to move all wf & task definitions into another file and import them into your main module.

    k

    kumare

    11/7/2024

    thank you

    m

    michaellai901026

    11/7/2024

    Sure, I’ll give it a look when I have time

    k

    kumare

    11/7/2024

    it seems we broke the __main__ handling?

    k

    kumare

    11/7/2024

    cc <@U06SFQHEJ3H> can you please take a look, the above script fails with

    │ /Users/ketanumare/src/ketan-examples/extras/register_main.py:38 in &lt;module&gt;  │
    │                                                                              │
    │ ❱ 38 │   remote_standard_scale_workflow = remote.register_workflow(          │
    │                                                                              │
    │ /Users/ketanumare/src/flytekit/flytekit/remote/remote.py:939 in              │
    │ register_workflow                                                            │
    │                                                                              │
    │ ❱  939 │   │   │   _, _, _, module_file = extract_task_module(entity)        │
    │                                                                              │
    │ /Users/ketanumare/src/flytekit/flytekit/core/tracker.py:354 in               │
    │ extract_task_module                                                          │
    │                                                                              │
    │ ❱ 354 │   │   │   name = f.lhs                                               │
    │                                                                              │
    │ /Users/ketanumare/src/flytekit/flytekit/core/tracker.py:120 in lhs           │
    │                                                                              │
    │ ❱ 120 │   │   return self.find_lhs()                                         │
    │                                                                              │
    │ /Users/ketanumare/src/flytekit/flytekit/core/tracker.py:172 in find_lhs      │
    │                                                                              │
    │ ❱ 172 │   │   raise _system_exceptions.FlyteSystemException(f"Error looking  │
    ╰──────────────────────────────────────────────────────────────────────────────╯
    FlyteSystemException: SYSTEM:Unknown: error=Error looking for LHS in 
    register_main```
    
    k

    kumare

    11/7/2024

    i think this seems like a bug in the tracker

    k

    kumare

    11/7/2024

    aah i see its there

    k

    kumare

    11/7/2024

    yes will try to add the method once i get a chance

    p

    peterxcli

    11/6/2024

    Hi <@UNZB4NW3S> Thanks for answering:wave:

    I'm currently working on <https://github.com/flyteorg/flyte/pull/5951|this PR> and am trying to create a launch plan with an intentionally invalid cron string to validate the recent changes.

    I attempted the following setup:

    @task
    def mean(values: List[float]) -&gt; float:
        return sum(values) / len(values)
    
    @task
    def standard_deviation(values: List[float], mu: float) -&gt; float:
        variance = sum([(x - mu) ** 2 for x in values])
        return sqrt(variance)
    
    @task
    def standard_scale(values: List[float], mu: float, sigma: float) -&gt; List[float]:
        return [(x - mu) / sigma for x in values]
    
    @workflow
    def standard_scale_workflow(values: List[float]) -&gt; List[float]:
        mu = mean(values=values)
        sigma = standard_deviation(values=values, mu=mu)
        return standard_scale(values=values, mu=mu, sigma=sigma)
    
    if __name__ == "__main__":
        remote = FlyteRemote(config=Config.auto())
    
        standard_scale_launch_plan = LaunchPlan.get_or_create(
            workflow=standard_scale_workflow,
            name="standard_scale_lp_haha",
            schedule=CronSchedule(schedule="* * * * MON#1"),
        )
    
        remote_standard_scale_workflow: FlyteWorkflow = remote.register_workflow(
            standard_scale_workflow,
        )
    
        remote.register_launch_plan(standard_scale_launch_plan, version="v1", project="flytesnacks", domain="development")```
    Unfortunately, this approach didn’t work as expected, and I suspect there might be an issue with my process. I also explored the `raw_register` method you mentioned, but I couldn’t find much documentation on using it specifically to register a launch plan.
    
    If you have any insights or additional guidance, I would greatly appreciate it! Thank you very much.
    
    k

    kumare

    11/4/2024

    you can use raw_register

    k

    kumare

    11/4/2024

    you can , we should write a method in remote.py to do https://github.com/flyteorg/flytekit/blob/0662bc6546278154605f5f544782db0110bd463d/flytekit/remote/remote.py#L808-L820|this (its currently behind raw_register) sadly

    p

    peterxcli

    11/4/2024

    Anyone know how to create a launch plan programmatically in Flyte?

    I've been trying to run this code:

        name="my_launchplan",
        workflow=wf,
        schedule=CronSchedule(
            schedule="0 6 * * MON#1",  # UTC, see <https://docs.flyte.org/en/latest/concepts/schedules.html#cron-expression-table>
        )
    )```
    but running it directly with Python doesn't seem to do anything. I've tried `pyflyte launchplan` and `pyflyte run --remote`, but neither seems to support this use case.
    
    Any help would be much appreciated! :pray:
    
    (I'm running the example from here: <https://github.com/flyteorg/flytesnacks/blob/0ec8388759d34566a0ffc0c3c2d7443fd4a3a46f/examples/basics/basics/launch_plan.py>)