Summary
The user noted a significant increase in memory usage of the flyte-binary pod from 500 MB to 2 GB under heavy load, resulting in an OOM Kill. They are unsure if this is due to internal caching, dataCatalog, or if all workflows are stored in memory. The user requests a feature to delete completed workflows from memory and suggests increasing resources to 4 CPUs and over 4 GB of RAM, questioning if 4 GB is adequate for approximately 10,000 daily pipeline triggers. They plan to explore the effects of cache control on memory scaling and stress the importance of monitoring, mentioning specific configuration parameters and performance optimization resources. Additionally, the user recommends more generous default settings for the flyte-binary, suspecting potential caching issues or a memory leak, as memory usage still rises to 8 GB before an OOM kill occurs, despite no workloads failing.