F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Request Entity Too Large Error

Summary

The user is facing a 'Request Entity Too Large' error with a map task of cardinality 3000, which works until a status change occurs. They are looking for suggestions to resolve this issue and mention that increasing entity size is a gRPC setting. The user questions if this setting is part of the Flyte admin configuration, noting documentation that indicates a default size of 0, which they interpret as unbounded. They discuss the practice of keeping map tasks under 5k and mention that while they have tried increasing size limits, 10MB works fine. The user is confused about the large gRPC message, as it should only send a link to the data, and they provide details about the size limits, stating that with a 4MB limit, 3000 tasks would allow for around 1300 bytes per task. They also mention successfully running 5000 tasks with an increased gRPC limit.

Status
resolved
Tags
    Source
    #ask-the-community