Deltalake 0.4.3 (and 0.4.2) breaks our CI

[lucienfregosi]

Hi Again :)

We tried to upgrade our base code with deltalake 0.4.3 (also with 0.4.2) but we got an error during our CI process (when launching unit tests).

The error is
dags/utils/delta_lake_utils.py:4: in <module> from deltalake import DeltaTable, Schema ../../../virtualenv/python3.7.1/lib/python3.7/site-packages/deltalake/__init__.py:1: in <module> from .deltalake import RawDeltaTable, rust_core_version E ImportError: /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/deltalake/deltalake.abi3.so: undefined symbol: renameat2

Everything is working well with deltalake 0.4.1 do you have any idea about this issue ?

[houqp]

@lucienfregosi could you share your environment with us? including kernel and libc (or musl) version. We recently implemented concurrent multi-writer support for the filesystem backend using renameat2 system (#126). Here is the version requirements for this system call:

• kernel >= 3.15
• glibc >= 2.28

[houqp]

double checked the manylinux spec, looks like it does use a very old version of glibc: https://www.python.org/dev/peps/pep-0571/.

I am going to yank that release out of pypi until we lands a fallback for other linux kernel/userspace configurations.

[fvaleye]

@lucienfregosi could you share your environment with us? including kernel and libc (or musl) version. We recently implemented concurrent multi-writer support for the filesystem backend using renameat2 system (#126). Here is the version requirements for this system call:

• kernel >= 3.15
• glibc >= 2.28

The CI used Ubuntu 16.04 LTS distribution by default.
glibc version is 2.23

Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

We changed the distribution to Ubuntu 20.04.2 LTS and it is working fine!
glibc version is 2.31

Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

[houqp]

relevant discussion in the libc project on what's the minimal supported glibc version: rust-lang/libc#1412. Right now they require >= 2.26, but has already added features that would only work for 2.27 and 2.28. renameat2 is one of the feature that would only work for 2.28 even though we are not using the libc crate for syscall invocation at the moment.

[lucienfregosi]

All good with ubuntu 20.04 :)
We can close the issue I guess !

[jeisinge]

FYI - I'm seeing a similar issue on Databricks Runtime 7.6.

[houqp]

@jeisinge will be fixed with a new release after #148, hopefully by end of tomorrow.

[houqp]

FYI - I'm seeing a similar issue on Databricks Runtime 7.6.

As a workaround for now, please manually install a version lower than 0.4.2