S

SiftZendeskTest

High memory usage in map_task function

Summary

The user is testing the map_task function with different batch sizes and sleep durations, encountering high memory usage and out-of-memory errors on backend Pods related to gRPC calls. They request a review of the flyteadmin logs, suspecting an error due to object modifications under high concurrency. The user suggests disabling finalizers temporarily and asks to set inline.plugins.k8s.inject-finalizer to false for further testing. After this change, the task completed successfully, but the user still observes excessive memory pressure in flyte-binary, peaking at 19.5GiB for a 20,079-element map_task, and notes that memory usage remains high at 16.4GiB even 10 minutes after workflow completion.

Status
resolved
Tags
    Source
    #ask-the-community