Flyte

The user reports a significant increase in memory usage of the flyte-binary pod from 500 MB to 2 GB under heavy load, resulting in an OOM Kill. They are unsure if this is due to internal caching, dataCatalog, or if all workflows are stored in memory. The user requests a feature to delete completed workflows from memory and suggests increasing resources to 4 CPUs and over 4 GB of RAM, questioning if 4 GB is sufficient for about 10,000 daily pipeline triggers. They plan to investigate cache control's impact on memory scaling and emphasize the importance of monitoring, mentioning specific configuration parameters and performance optimization resources. The user suspects caching issues or a memory leak, noting that memory usage can rise to 8 GB before an OOM kill occurs, despite no workloads failing. They observe system crashes every 2 days with memory around 8 GB, although it does not frequently crash under production workload. They consider moving to flyte-core for additional scaling mechanisms and mention a temporary solution while planning to build a POC of Flyte-core. Additionally, they note that Flyte 1.13.2 contains a fix for the memory issue in flyte-binary and suggest planning an update on Monday.