Improve GraalVM Reachability Metadata and corresponding nativeTest related unit tests

[linghengqian]

Feature Request

For English only, other languages will not be accepted.

Please pay attention on issues you submitted, because we maybe need more details.
If no response anymore and we cannot make decision by current information, we will close it.

Please answer these questions before submitting your issue. Thanks!

Is your feature request related to a problem?

• Make ShardingSphere Proxy in GraalVM Native Image form available #21347.

Describe the feature you would like.

• In the current master branch, for the sharding, readwrite-splitting, mask, shadow and encrypt features in standalone mode and H2Database, there are already corresponding GraalVM Reachability Metadata and corresponding nativeTest-related unit tests located under GraalVM Native Image.

• The previous PR did not cover further situations. We need to add additional unit tests and corresponding GraalVM Reachability Metadata for the following functions. Some unit tests will require the GraalVM Reachability Metadata of testcontainers.

  • Modes except Local
    • apache/zookeeper
    • etcd-io/etcd
  • Distributed Transactions
    • atomikos/transactions-essentials
    • jbosstm/narayana
    • apache/incubator-seata
  • Databases except h2database
    • mysql:9.1.0-oraclelinux9
    • postgres:17.2-bookworm
    • enmotech/opengauss-lite:5.1.0
    • mcr.microsoft.com/mssql/server:2022-CU16-ubuntu-22.04
    • gvenzl/oracle-free and gvenzl/oracle-xe
    • apache/doris? Where is the docker image of runtime?
    • prestodb/presto
    • apache/hive:4.0.1
    • clickhouse/clickhouse-server:24.11.1.2557
  • Other features
    • DistSQL
    • Logging - qos-ch/logback

• In the current master branch, ShardingSphere temporarily hosts the GraalVM Reachability Metadata of some third-party libraries. These irrelevant json entries should be submitted to https://github.com/oracle/graalvm-reachability-metadata and then deleted from the ShardingSphere side. This sub-work is initially planned to be completed before the release of GraalVM CE for JDK 29.

  • ch.qos.logback:logback-core:1.2.12
  • org.apache.hive:hive-jdbc:4.0.1
  • com.alibaba:druid:1.2.20
  • com.atomikos:transactions:6.0.0
  • com.clickhouse:clickhouse-jdbc:0.6.3
  • com.github.ben-manes.caffeine:caffeine:2.9.3
    • Partially located at Add support for com.github.ben-manes.caffeine:caffeine:2.9.3 oracle/graalvm-reachability-metadata#388 .
  • com.github.docker-java:docker-java:3.4.0
  • com.google.protobuf:protobuf-java:3.21.12
    • Partially located at Add support for com.google.protobuf:protobuf-java-util:3.21.12 oracle/graalvm-reachability-metadata#167 .
  • com.h2database:h2:2.2.224
  • com.mysql:mysql-connector-j:8.3.0
  • com.typesafe:config:1.2.1
  • com.zaxxer:HikariCP:4.0.3
  • io.grpc:grpc-netty:1.65.1
  • io.netty:netty-transport-classes-epoll:4.1.112.Final
  • io.vertx:vertx-core:4.5.1
  • org.apache.calcite:calcite-core:1.35.0
  • org.apache.groovy:groovy:4.0.22
  • org.apache.hadoop:hadoop-common:3.3.6
  • org.apache.seata:seata-all:2.2.0
  • org.apache.zookeeper:zookeeper:3.9.0
  • org.apiguardian:apiguardian-api:1.1.2
  • org.jboss.logging:jboss-logging:3.2.1.Final
  • org.jboss.narayana.jta:jta:5.12.7.Final
  • org.junit.jupiter:junit-jupiter:5.11.1
  • org.slf4j:slf4j-api:1.7.36
  • org.springframework:spring-jcl:5.3.27
  • org.testcontainers:junit-jupiter:1.20.1
  • org.apache.shardingsphere.elasticjob:elasticjob-lite-core:3.0.4
    • this is addressed at Provide built-in GraalVM Reachability Metadata and introduce a separate nativeTest in CI shardingsphere-elasticjob#2258 .
    • Removed via Enable Safe Composition of Metadata for ShardingSphere Proxy Native #33179.
  • org.glassfish.jaxb:jaxb-runtime:2.3.1
    • Removed directly in Removes use of JAXB API via jackson-dataformat-xml #29384, no need to commit to GraalVM Reachability Metadata Repository side.

• The current issue involves several user-side experience issues.

  • Remove the requirement for --report-unsupported-elements-at-runtime of buildArgs in GraalVM Native Build Tools. This requires further breaking ShardingSphere's compatibility with previous versions under GraalVM Native Image.
  • Increased GraalVM CE version for handling NativeTest and building ShardingSphere Proxy Native from GraalVM CE 23.0.2 For JDK 17.0.9 to GraalVM CE 24.0.2 For JDK 22.0.2.
  • Completely prevent GraalVM Tracing Agent from collecting unreasonable GRM for unit tests related to org.apache.shardingsphere.test.natived.proxy.features.** #33206

[linghengqian]

• I'm working on this issue.

[linghengqian]

user-side experience:

• GraalVM CE 23.1.1 still fails to solve the problem that occurred at Javac-related exceptions are thrown when building Native Images with GraalVM 23.1.1 For JDK 21.0.1 oracle/graal#7500 in GraalVM CE 23.1.0 . I have refactored a small unit test at https://github.com/linghengqian/shardingsphere-graalvm-v2311-native-test .
  • Under investigation at Update the infra-expr-espresso module related GraalVM CE version to 23.1.1 #28609 . This involves breaking changes to the GraalVM Truffle framework.
  • update in 2023.12.16: Maybe [GR-50854] Fix clinit simulation for arraycopy with null values. oracle/graal#8039 .
  • update in 2024.01.18: Fixed by Updates the minimum GraalVM CE version for compiling ShardingSphere's Native Image products to JDK 21.0.2 #29764 .
  • update in 2024.06.07: Fixed by Bump the minimum GraalVM CE version required to compile ShardingSphere's GraalVM Native Image artifacts to JDK22 #31630 .
• Due to the impact of [GR-48612] Enable --strict-image-heap by default. oracle/graal#7474 in GraalVM CE 24.0.0 for JDK 22, I believe that to remove --report-unsupported-elements-at-runtime, then we must set a batch of --initialize-at-run-time=xxx.xxx.xxx in buildArgs like Spring Framework.
  • update in 2024.07.18: Maybe [GR-43837] Enable --report-unsupported-elements-at-runtime by default and deprecate the option. oracle/graal#8815 .

[linghengqian]

Databases other than H2database:

• Oracle JDBC Driver com.oracle.database.jdbc:ojdbc8:23.3.0.23.09 cannot be used under GraalVM Native Image and will not be considered. Reference Fatal error when running nativeTest with ojdbc 23.3.0.23.09 oracle/graal#7979 .
  • Wait https://issues.apache.org/jira/browse/LEGAL-663 .
• For Presto JDBC Driver com.facebook.presto:presto-jdbc:0.287, we need to wait for further clarification from [Feature]: Presto Module uses the outdated Docker Image of ghcr.io/trinodb/presto and needs to be updated to prestodb/presto testcontainers/testcontainers-java#8657 , its testcontainers-java module is quite suspicious .
• For Doris, as affected by [Feature] Runtime docker image doris#9426 , we may have to use a third-party Docker Image from selectdb. Refer to https://hub.docker.com/r/selectdb/doris.be-ubuntu and https://hub.docker.com/r/selectdb/doris.fe-ubuntu .
  • For https://github.com/apache/doris, we lack a real usable Runtime Image.
  • Wait New Apache Doris bitnami/containers#69243 .
• For HiveServer2, we need to wait for further clarification from
  • https://issues.apache.org/jira/browse/HIVE-28295
  • https://issues.apache.org/jira/browse/HIVE-28308
  • https://issues.apache.org/jira/browse/HIVE-28316
  • https://issues.apache.org/jira/browse/HIVE-28317
  • https://issues.apache.org/jira/browse/HIVE-28322
  • https://issues.apache.org/jira/browse/HIVE-28417
  • https://issues.apache.org/jira/browse/HIVE-28418
  • https://issues.apache.org/jira/browse/HIVE-28420
  • https://issues.apache.org/jira/browse/HIVE-28423
  • https://issues.apache.org/jira/browse/HIVE-28424
  • https://issues.apache.org/jira/browse/HIVE-28429
  • https://issues.apache.org/jira/browse/HIVE-28430
  • https://issues.apache.org/jira/browse/HIVE-28432
  • https://issues.apache.org/jira/browse/HIVE-28437
  • https://issues.apache.org/jira/browse/HIVE-28444
  • https://issues.apache.org/jira/browse/HIVE-28445
  • [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar spark#30250
  • Hive Connection Error with ZooKeeper dbeaver/dbeaver#22777
  • Add GraalVM native compilation support logging-log4j2#1539
  • grpc-netty-shaded cannot be successfully compiled under GraalVM grpc/grpc-java#10601
  • [QUESTION] Should we collect GraalVM Reachability Metadata for shaded packages? oracle/graalvm-reachability-metadata#377
  • HIVE-28315: Missing classes while using hive jdbc standalone jar hive#5313
  • HIVE-28191. Upgrade Hadoop Version to 3.4.0. hive#5187
  • When bump from 2.23.0 to 2.23.1, the status of the .properties configuration file no longer works logging-log4j2#2794
  • Fix handling of onMatch and onMismatch attributes in the properties configuration format logging-log4j2#2791
  • HIVE-28417: Bump Log4j2 to 2.24.1 to facilitate compilation of GraalVM Native Image hive#5375
  • HIVE-28656: Improve HPL/SQL tests hive#5381
  • Core: Drop support for Java 8 iceberg#10518
  • Remove Hadoop 2 iceberg#10940
  • ORC-1512: Drop Java 8/11 and make Java 17 by default orc#1627

[linghengqian]

Modes except Local:

• Etcd integration is affected by a bug in vertx-grpc. Refer to https://github.com/linghengqian/shardingsphere-master-b-etcd-test .

  • Fixed by Refactor EtcdRepository watch consume #29738 .

• Consul integration is blocked by:

  • Allow ConsulRawClient of cluster-mode-repository-consul to be configured on ports other than 8500 #29419
  • Cannot invoke "java.util.Collection.size()" because "storageNodes" is null #29509
  • Replaces com.ecwid.consul:consul-api with io.vertx:vertx-consul-client in shardingsphere-plugin-mode-cluster-repository-consul shardingsphere-plugin#33