S

SiftZendeskTest

Integrating Python Profiler with Flyte Task

Summary

The user is integrating a sampling Python call stack profiler with a Flyte task by creating a new PodSpec in flytekit for a shared process namespace. They have configured the profiler to access the task's memory maps but are having difficulty identifying the correct process ID (PID) of the user task, as their method using pgrep -f pyflyte-execute is ineffective due to multiple processes. The user is looking for advice on the appropriate process name for obtaining the PID or considering a custom PythonFunctionTask that records the task's PID to a shared volume. Their goal is to identify hot paths in individual tasks using a sampling profiler and eventually render them as flamegraphs. They are currently using Austin to sample the task process frame stack and are exploring the possibility of attaching the profiler programmatically at the entry point, although they see limited advantages to this approach.

Status
resolved
Tags
    Source
    #ask-the-community