Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

allow registering application package as predefined process #936

Open
jdries opened this issue Nov 12, 2024 · 5 comments
Open

allow registering application package as predefined process #936

jdries opened this issue Nov 12, 2024 · 5 comments
Assignees
@soxofaan

Comments

@jdries
Copy link
Contributor

jdries commented Nov 12, 2024 •
edited
Loading

This is step 1 in our application package support.
The subset I would like to support first, takes simple user defined parameters as input (e.g. bounding box, date range) and returns stac metadata as output.
So no data stage-in step is required on the openEO side for this scenario!

In the backend, we would set up a 'hard-coded' mapping of a specific process to an AP.
@soxofaan
Copy link
Member

also see https://github.com/eu-cdse/openeo-cdse-infra/issues/267

soxofaan added a commit to Open-EO/openeo-python-driver that referenced this issue Nov 18, 2024
@soxofaan
Add openeo_driver.config.load.exec_py_file
06f7670 
related to Open-EO/openeo-geopyspark-driver#936
soxofaan added a commit that referenced this issue Nov 19, 2024
@soxofaan
custom processes definition: add OPENEO_CUSTOM_PROCESSES env var support
cbb9357 
related to #936
@soxofaan
Copy link
Member

some more linked issues:

@jdries
Copy link
Contributor Author

jdries commented Nov 25, 2024

Extra complication: the application dependency on rwx volumes
I would want to check if we can avoid that for the most trivial cases of single step CWL.

soxofaan added a commit that referenced this issue Nov 29, 2024
@soxofaan
Issue #936 initial helper to build calrissian k8s job body
e64bb53 
based on eu-cdse/openeo-cdse-infra#267
soxofaan added a commit that referenced this issue Nov 29, 2024
@soxofaan
Issue #936 add launch_cwl_job_and_wait
eca7a61
soxofaan added a commit that referenced this issue Dec 2, 2024
@soxofaan
Issue #936 ad-hoc attempt to first run input staging job
1899538
soxofaan added a commit that referenced this issue Dec 12, 2024
@soxofaan
Issue #936 initial helper to build calrissian k8s job body
6d6184b 
based on eu-cdse/openeo-cdse-infra#267
soxofaan added a commit that referenced this issue Dec 12, 2024
@soxofaan
Issue #936 add launch_cwl_job_and_wait
87a4ab2
soxofaan added a commit that referenced this issue Dec 12, 2024
@soxofaan
Issue #936 ad-hoc attempt to first run input staging job
b55996b
soxofaan added a commit that referenced this issue Dec 12, 2024
@soxofaan
Issue #936 initial helper to build calrissian k8s job body
2dc97df 
based on eu-cdse/openeo-cdse-infra#267
soxofaan added a commit that referenced this issue Dec 12, 2024
@soxofaan
Issue #936 add launch_cwl_job_and_wait
71143fc
soxofaan added a commit that referenced this issue Dec 12, 2024
@soxofaan
Issue #936 ad-hoc attempt to first run input staging job
3d1806d
soxofaan added a commit that referenced this issue Dec 12, 2024
@soxofaan
Issue #936 encapsulate logic in CalrissianJobLauncher class
074be84 
for better code/value reuse
soxofaan added a commit that referenced this issue Dec 17, 2024
@soxofaan
Issue #936 create input CWL with b64 inline data
490b5df 
instead of tedious artifactory deploy
soxofaan added a commit that referenced this issue Dec 17, 2024
@soxofaan
Issue #936 CalrissianJobLauncher move volume info details to __init__
eafabd3
@soxofaan soxofaan mentioned this issue Dec 17, 2024
Open
soxofaan added a commit that referenced this issue Dec 18, 2024
@soxofaan
fixup! Issue #936 encapsulate logic in CalrissianJobLauncher class
94a5c72
soxofaan added a commit that referenced this issue Dec 18, 2024
@soxofaan
Issue #936 encapsulate CWL PoC in openEO process function
47c1d9b
soxofaan added a commit that referenced this issue Dec 18, 2024
@soxofaan
Issue #936 try to fetch CWL output from S3
630ce72
@soxofaan
Copy link
Member

after quite some learning curve climbing, I managed to run a first CWL based "hello world" example with an end-to-end openEO workflow:
Screenshot from 2024-12-18 11-27-36

soxofaan added a commit that referenced this issue Dec 18, 2024
@soxofaan
Issue #936 encapsulate run_cwl_workflow
bbb883a
@soxofaan soxofaan mentioned this issue Dec 18, 2024
Merged
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
fixup! Issue #936 encapsulate run_cwl_workflow
00d7351
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
fixup! Issue #936 create input CWL with b64 inline data
fbc7411
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
fixup! Issue #936 initial helper to build calrissian k8s job body
78a2b8d
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
fixup! Issue #936 encapsulate CWL PoC in openEO process function
6aed4d3
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
fixup! Issue #936 try to fetch CWL output from S3
419139b
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
Issue #936 fixup calrissian tests
fe5e963
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
Issue #936 improve naming (and uniqueness) in CalrissianJobLauncher
4ccc177
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
Issue #936 code/import style cleanups
29a3a8b
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
Issue #936 add changelog entry
965246f
soxofaan added a commit that referenced this issue Dec 19, 2024
@soxofaan
Issue #936 bump version and add changelog entry
65588fe
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
fixup! Issue #936 code/import style cleanups
475e09c
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 add test for CalrissianS3Result
18b31af
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 add tests about k8s interactions
3490af8
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 initial helper to build calrissian k8s job body
dfdc091 
based on eu-cdse/openeo-cdse-infra#267
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 add launch_cwl_job_and_wait
1091dd9
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 ad-hoc attempt to first run input staging job
31b5be3
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 encapsulate logic in CalrissianJobLauncher class
fb23a49 
for better code/value reuse
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 create input CWL with b64 inline data
441a485 
instead of tedious artifactory deploy
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 initial steps figuring out S3 URL of CWL output
21f67ba
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 CalrissianJobLauncher move volume info details to __init__
5f224e8
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 encapsulate CWL PoC in openEO process function
d7f47ec
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 try to fetch CWL output from S3
85f41d5
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 encapsulate run_cwl_workflow
f8dba82
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 bit of isort and code style cleanup
52fdc49
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 fixup calrissian tests
8776f02
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 improve naming (and uniqueness) in CalrissianJobLauncher
8e44197
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 code/import style cleanups
001328c
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 add test for CalrissianS3Result
f798385
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 add tests about k8s interactions
54beb92
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 bump version and add changelog entry
c6fab3a
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
fixup! Issue #936 add tests about k8s interactions
d2170c9
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 add tests about k8s interactions
d0246fc
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 drop temp adapter for openeo_driver<0.122
aaf5622
soxofaan added a commit that referenced this issue Dec 20, 2024
@soxofaan
Issue #936 bump version and add changelog entry
15ac6d6
@soxofaan
Copy link
Member

just merged #975 in 167e029 (just introduces some calrissian helpers and proof of concept process, should not really touch existing code paths)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@soxofaan soxofaan

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@soxofaan @jdries