Summary
The user is developing a command line workflow on the union.ai platform that fails to run remotely, despite trying different image configurations and using a publicly accessible image. They have updated the ImageSpec with necessary libraries but still face the same error. The user finds the process of updating ImageSpec requirements cumbersome and is working on supporting custom base images securely on serverless. They provide a code snippet for their ImageSpec and describe the task registration and container image building process, but continue to encounter an authorization error when running remote.register_script. They are seeking updates on this issue.
thomas316
Can you share the execution url that failed for you?
alvaro.romo
hey everyone, any updates on this topic?
thomas316
Going to that branch is unexpected for registry=None. If the docker error was ImageNotFound, it should have went to: https://github.com/thomasjpfan/flytekit/blob/eb204594a5f8cd4a662d4bd44ab7e8d7dea22a98/flytekit/image_spec/image_spec.py#L115-L116
CC <@USU6W5ATA>
alvaro.romo
hey <@U0635LYB5PD> I have tried to generate the image as you said. When I register the tasks, the image is registered correctly. However, when I launch the workflow, the following error is displayed and the execution in http://union.ai|union.ai does not appear.
│ exist │
│ │
│ 133 │ │ except Exception as e: │
│ 134 │ │ │ tag = calculate_hash_from_image_spec(self) │
│ 135 │ │ │ # if docker engine is not running locally, use requests to check if the imag │
│ ❱ 136 │ │ │ if "localhost:" in self.registry: │
│ 137 │ │ │ │ container_registry = self.registry │
│ 138 │ │ │ elif self.registry and "/" in self.registry: │
│ 139 │ │ │ │ container_registry = self.registry.split("/")[0] │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ APIError = <class 'docker.errors.APIError'> │ │
│ │ docker = <module 'docker' from │ │
│ │ '/home/aromo/aromo-union/lib/python3.10/site-packages/docker/__init__.py'> │ │
│ │ ImageNotFound = <class 'docker.errors.ImageNotFound'> │ │
│ │ self = ImageSpec( │ │
│ │ │ name='classifiers_v2', │ │
│ │ │ python_version=None, │ │
│ │ │ builder=None, │ │
│ │ │ source_root=None, │ │
│ │ │ env=None, │ │
│ │ │ registry=None, │ │
│ │ │ packages=None, │ │
│ │ │ conda_packages=None, │ │
│ │ │ conda_channels=None, │ │
│ │ │ requirements='requirements_union.txt', │ │
│ │ │ apt_packages=None, │ │
│ │ │ cuda=None, │ │
│ │ │ cudnn=None, │ │
│ │ │ base_image=None, │ │
│ │ │ platform='linux/amd64', │ │
│ │ │ pip_index=None, │ │
│ │ │ pip_extra_index_url=None, │ │
│ │ │ registry_config=None, │ │
│ │ │ entrypoint=None, │ │
│ │ │ commands=None, │ │
│ │ │ tag_format=None │ │
│ │ ) │ │
│ │ tag = '3lqAJSS3fFLROiVbqpa1_Q' │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: argument of type 'NoneType' is not iterable```
thomas316
<@U07MNQQ72DT> In your ImageSpec, can you remove the configuration for registry or set it to the default value?
name="classifiers", ...
registry=None.
)```
On serverless, we'll update the image builder to raise an earlier error when `registry is not None` .
thomas316
Thank you for the details! I see where the bug is.
alvaro.romo
hey <@UPBBNMXD1>. Using this command automatically generates the image with the label.
union register clf --project default --domain development
When I launch the imperative workflow it automatically takes the label that I had generated when registering the tasks. The problem is that if I launch it with union command line it works.
union run --remote --copy-all run_imperative_wf.py outputs
But if I do it using UnionRemote as I showed you in the code I get the USER::admission error.
grantham
Hi Alvaro, I am happy to jump on a call to walk through your example.
I have used imperative (programmatic) workflows before, and they are quite powerful. I am really curious about your use case!
Also, while you could explicitly define all of your dependencies, I personally love using the requirements parameter of ImageSpec:
image = flytekit.ImageSpec(requirements="requirements.txt")```
Where you may define `requirements.txt` with a command like this:
```uv pip compile pyproject.toml > requirements.txt --python-platform linux```
This will freeze all of your local dependencies. This works _really_ well in handling complex version dependencies issues.
krogan
hey <@U07MNQQ72DT> where do you set classifiers:4V7kczm3iHLfhnAN69NKAA during registration in that case?
cc <@U0635LYB5PD>
kumare
Hmm this is weird are you using programmatic way of launching ? Cc <@UPBBNMXD1>
alvaro.romo
I am developing an imperative workflow that is dynamically built based on a json. I am using the http://union.ai|union.ai platform for testing. This part I have already developed and it runs correctly if I register the tasks and run the workflow through the command line.
However, when I try to run the workflow by connecting from the code with remote it fails with this error.
I have tried with various image configurations and also using a built and publicly accessible image but still I get the same error. What can I do to fix this error?
alvaro.romo
hey katrina, thanks for the answer. I already try update the ImageSpec with my requirements.
name="classifiers",
packages=[
"accelerate>=0.20.1",
"adlfs>=2023.4.0",
"ai2-tango>=1.3.0",
"azure-storage-blob>=12.17.0",
"datasets>=2.14.6",
"docker==6.1.3",
"einops>=0.6.1",
"mlflow>=2.12.1",
"nlpaug",
"onnx==1.15.0",
"onnxruntime==1.16.2",
"onnxruntime-gpu==1.16.2",
"plotnine",
"scikit-learn",
"sentence-transformers>=2.2.2",
"tabulate",
"torch>=2.0.1",
"einops>=0.6.1",
"accelerate",
"bitsandbytes",
"tritonclient[http]==2.39.0",
],
)
@task(
container_image=image, cache=True, cache_version="6",
)
def load_data_from_disk(dataset_path: str, ...):
...```
I register my tasks using `union register`, then a build_image execution is launched, building my container image. But now when I try to run remote.register_script I get same error -> `[1/1] currentAttempt done. Last Error: USER::admission webhook "<http://image-builder-webhook.union.ai|image-builder-webhook.union.ai>" denied the request: access to classifiers:Oh6eZo6iI15tuUomnYPN7w is not authorized.`
krogan
I know this isn't a great answers and I'm sorry if it's a bit more cumbersome, but please know we're exploring how to securely support custom base images on serverless!
krogan
as a workaround, you can you update the ImageSpec requirements with the exact libraries you need?
krogan
hey <@U07MNQQ72DT> right now public images are not supported in union serverless so unfortunately this is expected
kumare
Cc <@UNW4VP36V> <@UPBBNMXD1> can you help