F

Flyte enables you to build & deploy data & ML pipelines, hassle-free. The infinitely scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks. Explore and Join the Flyte Community!

Spark Task Failure on Flyte Setup

Summary

The user has set up Flyte on a Kubernetes cluster in Hetzner cloud and is experiencing issues with Spark tasks failing due to a NoCredentialsError when accessing an S3 bucket. The Spark driver pod is marked as "Failed," and there is a warning about a missing ConfigMap for Spark configuration. The user provides logs and details about the Spark driver pod and its environment, highlighting the execution problems. They also mention a recommendation to use a more suitable cluster for Spark, as a k3s single node may be inadequate. However, the user has resolved the issue by enabling the webhook for the Spark operator.

Status
resolved
Tags
    Source
    #ask-the-community