Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prepare python3.11 docker image #23

Open
jdries opened this issue Aug 6, 2024 · 2 comments
Open

prepare python3.11 docker image #23

jdries opened this issue Aug 6, 2024 · 2 comments
Assignees
@jdries

Comments

@jdries
Copy link
Contributor

jdries commented Aug 6, 2024

No description provided.

@jdries jdries self-assigned this Aug 6, 2024
jdries added a commit that referenced this issue Aug 8, 2024
@jdries
back to using spark with hadoop
32e4b6a 
#23
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 8, 2024
@jdries
install scripts in version agnostic path
366e760 
Open-EO/openeo-geotrellis-kubernetes#23
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 9, 2024
@jdries
py extension is required for spark-submit to work
b217a39 
Open-EO/openeo-geotrellis-kubernetes#23
jdries added a commit that referenced this issue Aug 9, 2024
@jdries
set correct python version to avoid the use of system default 3.9
ea85430 
#23
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 9, 2024
@jdries
also upgrade numpy+pandas for higher python versions (getting import …
6de813c 
…issues)

Open-EO/openeo-geotrellis-kubernetes#23
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 9, 2024
@jdries
upgrade xarray, for numpy 2.x
8ad6464 
Open-EO/openeo-geotrellis-kubernetes#23
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 10, 2024
@jdries
defer pyspark.pandas to when we need it
9d2ddf7 
Open-EO/openeo-geotrellis-kubernetes#23
@jdries
Copy link
Contributor Author

jdries commented Aug 10, 2024 •
edited
Loading

numpy 2.x upgrade:

jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 12, 2024
@jdries
scipy is dependency for sentinel-3 reading
5275eca 
Open-EO/openeo-geotrellis-kubernetes#23
jdries added a commit that referenced this issue Aug 20, 2024
@jdries
use java-21, gcc-c++ for torch compile
be48a4e 
#23
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 20, 2024
@jdries
override python location, to avoid using env var from docker image
2a2c3cb 
Open-EO/openeo-geotrellis-kubernetes#23
jdries added a commit that referenced this issue Aug 20, 2024
@jdries
set PYSPARK_PYTHON to venv interpreter
e9c9e5c 
#23
jdries added a commit that referenced this issue Aug 20, 2024
@jdries
set PYSPARK_PYTHON to an interpreter that exists in this image
2010fe8 
#23
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 20, 2024
@jdries
update JAVA_HOME to point to new jdk
b66968e 
Open-EO/openeo-geotrellis-kubernetes#23
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 20, 2024
@jdries
pyspark_python was pointing to 3.9
ce1db54 
Open-EO/openeo-geotrellis-kubernetes#23
jdries added a commit that referenced this issue Aug 20, 2024
@jdries
point to generic python3
69036b7 
#23
@jdries
Copy link
Contributor Author

jdries commented Sep 19, 2024

Remaining issues are visible on the alma9 branch of openeo-geopyspark-driver, in the form of failing unit tests.

  • udf's on vector cubes-> will only be fixed in spark 4.x
  • jep runtime: jep.JepException: <class 'TypeError'>: cannot pickle 'DirectNDArray' object

So the image actually works for most cases, but not ready for prod.
I'm moving this one back into the backlog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jdries jdries

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jdries