Summary
The user is seeking guidance on overriding task resources for a ShellTask and has encountered a warning about nested tasks when invoking a ShellTask from another task. They suspect the issue may be related to task configuration but cannot find relevant documentation. The user provides a code snippet for a ShellTask and a resource-requesting task, noting that calling a task from another task is generally unsupported, except in specific scenarios. They inquire about defining resources for a ShellTask and suggest using pod templates. The user also shares a solution code snippet for the ShellTask and mentions trying to use map_tasks with ShellTask, which resulted in an error indicating that ShellTasks require two default outputs, stdout and stderr. They question whether ShellTasks are simply running Python subprocesses and consider using a regular task that runs Python subprocesses instead of a ShellTask, asking about the advantages of ShellTasks.
nuhaj
How can you override task resources to a shellTask? If I call a shelltask from another task I get a warning on the pod about nested tasks I think it might be task_config but can't find documentation
shell_task = ShellTask(
name="shell_task",
debug=True,
script="""
{inputs.command} >> {outputs.stdout} 2>> {outputs.stderr}
""",
inputs=kwtypes(command=str),
output_locs=[
OutputLocation(var="stdout", var_type=FlyteFile, location="stdout.txt"),
OutputLocation(var="stderr", var_type=FlyteFile, location="stderr.txt"),
],
)
@task(
requests=Resources(cpu="4", mem="16Gi")
)
def call_shell_task(....```
nuhaj
I have tried to use map_tasks with ShellTask. I don't think its supported, the error I got was the output, ShellTasks require two default outputs, stdout and stderr.
Only tasks with a single output are supported in map tasks.```
Are Shell tasks just running python subprocess? <https://github.com/flyteorg/flytekit/blob/2dcbb90f7a331eb2d75a826053fc41f944ad4aa2/flytekit/extras/tasks/shell.py#L52>
I can use a regular task that runs python subprocess instead of ShellTask and map that task. What advantages do ShellTasks have ?
nuhaj
for anyone that makes it here via a search the solution is
name="shell_task",
debug=True,
script="""
{inputs.command} >> {outputs.stdout} 2>> {outputs.stderr}
""",
inputs=kwtypes(command=str),
output_locs=[
OutputLocation(var="stdout", var_type=FlyteFile, location="stdout.txt"),
OutputLocation(var="stderr", var_type=FlyteFile, location="stderr.txt"),
],
)```
ytong
no these should be just args to the ShellTask constructor, just like inputs… they’re hidden by kwargs so you have to follow up the stack. And yeah, docs could use a boost
nuhaj
I'm guessing thats by using task_config then. Its not well documented
ytong
along with pod template and pod template name
ytong
ShellTask should take requests/limits just like a function task.
nuhaj
how would I define resources for a shell task ? I think it might be possible with pod templates
ytong
the only time this is possible is in the dynamic case (because it becomes a new workflow) and in the eager case (wherein the parent task becomes its own orchestrator).
ytong
in general, you shouldn’t call a task from within another task. this is not supported behavior.