F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

High memory usage in map_task function

Summary

The user is testing the map_task function with different batch sizes and sleep durations, encountering high memory usage and out-of-memory errors on backend Pods related to gRPC calls. They request a review of the flyteadmin logs, suspecting an error due to object modifications under high concurrency. The user suggests disabling finalizers temporarily and asks to set inline.plugins.k8s.inject-finalizer to false for further testing. After this change, the task completed successfully, but the user still observes excessive memory pressure in flyte-binary, peaking at 19.5GiB for a 20,079-element map_task, and notes that memory usage remains high at 16.4GiB even 10 minutes after workflow completion.

Status
resolved
Tags
    Source
    #ask-the-community