F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

ShellTask Implementation Issues

Summary

The user is attempting to use a ShellTask to download and decompress a file named date.json.gz into date.json. They express uncertainty about the implementation of the OutputLocation dataclass, which supports regex, and seek examples. They also want clarification on using arbitrary Python expressions within f-string braces. The user learns that only operations from the Format mini-language are allowed, which limits their approach. They decide to pass the filename without the .gz suffix and add it in their shell script instead. After trying a suggestion, they encounter a ValueError related to a missing variable in the inputs. Ultimately, they find a solution that allows them to stream data from the HTTP endpoint without needing the shell task.

Status
resolved
Tags
  • flyte
  • ValueError
  • File Processing
  • Question
  • Developer Help
  • ShellTask
  • date.json.gz
  • date.json
Source
#ask-the-community