Require Java 11+ to *build* Guava but continue to support Java 8 at runtime

[cpovirk]

I'd mentioned this in #3990 (comment).

It would be nice to be able to refer in source code to newer types (e.g., CRC32C, VarHandle, and @ReservedStackAccess)—with, of course, appropriate runtime guards. Of course, that would be incompatible with one of the possible approaches to #3990, which is to use --release.

(Using Java 11 might also still dodge bugs in older versions of javac.)

We'd probably accept a PR for this unless some of our users want to tell us that they build Guava from source themselves with javac 8 (or 9 or 10), in which case the decision would get more complex.

[cpovirk]

Of course, that would be incompatible with one of the possible approaches to #3990, which is to use --release.

Well, not completely: We could use --release 8 but then perform a second compilation with (e.g.) --release 11 to put a second set of classes into a multi-release jar. Even though multi-release jars don't do one of the things we might like, I think they could work well here. We could even imagine conditionally using APIs like CRC32C internally by providing an exception-throwing implementation for Java 8 in addition to a "real" implementation for newer versions and then making sure to perform appropriate checks before using the implementation class.

[ben-manes]

@ReservedStackAccess is only available to privileged code during bootstrap and be ignored otherwise, unless -XX:-RestrictReservedStack is specified. Similarly for @Contended and its flag -XX:-RestrictContended. That unfortunately makes it useless for library code.

It may be reasonable to use Atomic*FieldUpdaters instead of Unsafe. When introduced in Java 5 they were known to have performance issues. That may be mostly resolved according to Faster Atomic*FieldUpdaters for Everyone. You might consider running a fresh set of benchmarks on those usages to see if the impact is acceptable.

I would be inclined to instead make jdk11 the minimum version. Those who are performance sensitive should not be on jdk8 and those users are likely to only need critical bug fixes to be backported. Android annoyingly only supports a subset of jdk11, but likely enough for your needs. You'll certainly have people complain but external are suffering regardless and internal are posing a security risk, so you might not have as much actionable push back if pursued.

I recall some tools were not multi-release jar friendly (e.g. if naively creating a fatjar) and since they are not widely used, I'd consider that to likely be a higher risk of unexpectedly breaking someone in a surprising way than adopting jdk11.

[cpovirk]

Thanks, I'm not sure whether I ever knew that about the annotations. I'll have to dig up some past threads and leave a link from them to here.

I'll also read your link about the field updaters. That's good news for someday getting off Unsafe.

As for requiring a new Java version, that is above my pay grade :) Thankfully, even our final straggler internally was moved off Java 8 a while back, but we'll likely need to maintain compatibility as long as enough open-source Google and non-Google libraries do.

Sadly, if we try to make only bug fixes to a Java 8 backport, I'd expect the same kind of version conflicts that users are used to from the days when we often removed APIs: If one of your deps updates to Guava 32.4.0, then it might use APIs that aren't available to you.

It's good to know that we should look into the fat-jar situation if we try mrjars. I think we had some trouble with that kind of thing internally, if I'm remembering right. Hopefully it's been sorted out both internally and externally, but it's worth remembering that there are risks to everything (just in case the recent CVE-Windows experience didn't remind me enough :)). And at the moment, the benefits of using Java 9+ APIs are looking small (especially after what you said about the annotations), so we will have less motivation to go with a multi-release jar (though it's still a possibility).

--

One other note that I should have put in my original post:

Also related to toolchains: #5457 That's where I'd previously mentioned google/error-prone#3895 (comment), which is the key to testing Java 8 compatibility even when building with a newer version.

[cpovirk]

My guess at the full set of related commits in Error Prone is:

• google/error-prone@047e1e3
• google/error-prone@020a541
• google/error-prone@c2b71f9

[cpovirk]

We might also eventually see new versions of Maven plugins drop support for Java 8, as noted in uber/NullAway#778.

In fact, even today, we have special build setup for Error Prone, which requires Java 11+ [edit: for now].

[cpovirk]

If any of our annotation dependencies start being built for Java 9+, then we would start to need to build with a newer javac in all cases. I speculate that the annotations targeting Java 9+ might still be OK to references from a Guava that targets Java 8, but we'd want to verify that—especially since Java 8 has a bug in its handling of type-use annotations that it can't resolve.

[cpovirk]

An interesting question from an internal discussion:

I wonder if multi-release annotation processor jars work.

I would like to think so, but that sounds like a good thing for us to check if we ever look seriously into publishing a multi-release jar.

[cpovirk]

(As I just noted in #3990 (comment), even a build-time requirement could be inconvenient for users if they build Guava themselves: In this case, it could be inconvenient if they have continued to build with an older JDK version. So, for example, if an organization both builds with JDK 8 and runs with JDK 8, then they'd no longer be able to build Guava if its build required JDK 11, even if newer compilers could produce a version of Guava that runs on JDK 8.)

[cpovirk]

I am in the process of trying to conditionally use a Java 9+ API again as part of #6634 (comment) :\

[cpovirk]

On the toolchain front: I heard a while back that JDiff breaks at JDK ~13, so we wouldn't necessarily want to build with newer versions, at least for our Javadoc generation. Or we could finally migrate to japicmp, which seems to be the replacement that everyone recommends.

[edit: But I didn't see any obvious problems when #7087 changed Javadoc+JDiff to use Java 21. I suppose it's possible that JDiff failed silently. Or maybe what I heard is that JDiff breaks on JDK 13 bytecode or something?]

[edit: Or did it fail after all?]

[cpovirk]

This would also be convenient (but not strictly necessary) if we want to start generating a module-info.class (#2970): That requires JDK9+ at build time (perhaps configured like this, incidentally). If we don't have JDK9+, then our options are to submit a precompiled module-info.class to the repo or to skip the module-info compilation when run under JDK8 (and then set up our release configuration to require JDK9+ (which in practice means JDK11+, which is fine, since that's what we'd be using, anyway)).

[cpovirk]

I just noticed that we could simplify our build configuration slightly if we could require 17+:

guava/pom.xml

Lines 339 to 345 in 280b5d2

	Passes JDK 11-12-specific `no-module-directories` flag to Javadoc tool,
	which is required to make symbol search work correctly in the generated
	pages.
	This flag does not exist on 9-10 and 13+ (https://bugs.openjdk.java.net/browse/JDK-8215582).
	Consider removing it once our release and test scripts are migrated to a recent JDK (17+).

That alone is clearly not a reason to do so, but it's a reminder that sometimes upgrades solve problems. And while upgrading to JDK 13 or so would break JDiff, it would also let us use the nicer format for code snippets in our Javadoc. I think that, in particular, we could start lines with "@Override," etc. without having that interpreted as a Javadoc tag.

[cpovirk]

More toolchain thoughts:

• I wonder if running our tests n times in one job will end up slower than building+running our tests n times in m jobs. We'd have to test to see.
• Building under JDK21+ actually doesn't work right now (Set up CI for newer JDK versions #6830): Not only is there an Animal Sniffer error, but that error indicates that the built binary legitimately wouldn't work under older versions. While we could avoid releasing with JDK21+, we'd also need to avoid having our CI build with JDK21+ if it's going to then run our tests with that binary against older versions.

[cpovirk]

We're hitting a javac8 issue again in #7080:

[INFO] --- compiler:3.8.1:testCompile (default-testCompile) @ guava-tests ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 643 source files to /home/runner/work/guava/guava/guava-tests/target/test-classes
[INFO] -------------------------------------------------------------
Warning:  COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
Warning:  /home/runner/work/guava/guava/guava-tests/test/com/google/common/hash/MacHashFunctionTest.java:[34,23] ProviderList is internal proprietary API and may be removed in a future release
Warning:  /home/runner/work/guava/guava/guava-tests/test/com/google/common/hash/MacHashFunctionTest.java:[35,23] Providers is internal proprietary API and may be removed in a future release
Warning:  /home/runner/work/guava/guava/guava-tests/test/com/google/common/hash/MacHashFunctionTest.java:[84,4] ProviderList is internal proprietary API and may be removed in a future release
Warning:  /home/runner/work/guava/guava/guava-tests/test/com/google/common/hash/MacHashFunctionTest.java:[84,29] Providers is internal proprietary API and may be removed in a future release
Warning:  /home/runner/work/guava/guava/guava-tests/test/com/google/common/hash/MacHashFunctionTest.java:[85,30] ProviderList is internal proprietary API and may be removed in a future release
Warning:  /home/runner/work/guava/guava/guava-tests/test/com/google/common/hash/MacHashFunctionTest.java:[85,4] Providers is internal proprietary API and may be removed in a future release
[INFO] 6 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
Error:  COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
Error:  An exception has occurred in the compiler (1.8.0_402). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com/) after checking the Bug Database (http://bugs.java.com/) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.lang.AssertionError: annotation tree hasn't been attributed yet: @Nullable()
	at com.sun.tools.javac.util.Assert.error(Assert.java:133)
	at com.sun.tools.javac.util.Assert.checkNonNull(Assert.java:118)
	at com.sun.tools.javac.comp.Check.validateTypeAnnotation(Check.java:2748)
	at com.sun.tools.javac.comp.Attr$TypeAnnotationsValidator.visitAnnotation(Attr.java:4484)
...

(https://bugs.openjdk.org/browse/JDK-8144101)

Also, I think I saw that new versions of Maven will soon require Java 17. That may require us to finally do the toolchain setup if we want to test under Java 8.

[cpovirk]

• Our use of javac8 is going to complicate module-info support at least a little (by forcing us to make it conditional). [edit: Oops, I'd already covered this above.]
• I may have mentioned this already (too lazy to check), but javac does phase out support for older versions over time, as noted in feat: add jpms definition for annotations error-prone#4311. Eventually, Java 8 may become hard to target under new versions.
  • To be clear: If we need to continue to support Java 8, then the solution to that problem is going to be to someday not build with the newest javac versions—or at least to build with them only for testing (with the target version set higher) and then be careful not to use the result for a release. As always, what we really should be doing is building with some newish version of javac and then testing under other versions by using toolchains. All that may differ here is that we might not want to use the absolute newest javac if that version lacks support for -target 8.

[cpovirk]

Builds with Java 11 are fine, but if we go to even newer versions (under the theory that we should build with the newest version available), it appears that we'll have problems with Javadoc. Maybe we can continue generating Javadoc snapshots with java 11, even as we perform the main build with a newer version (especially since we target Java 8 and restrict ourselves to Java 8 APIs)? (Or maybe we can, you know, make it work with new versions :) But I fear that the problem is JDiff, so it might not be easy to fix.) We might see problems with that approach if we try to use Java 12+ APIs internally, but I don't know that we'll do that anytime soon. And even if we do, we should be providing Java-8-compatible sources alongside the Java 12+ sources, so I'd hope that we can "just" point Javadoc to those sources.

[ben-manes]

Not sure if it helps, but I was pinged a few years ago by @lvc showing off his tool (example). Looks like an inactive project and jdiff is too, but maybe could be revived easier?

Oh, and mentioned in #2963

[cpovirk]

Oh, good point, thanks.

I occasionally hear of a new such tool, and I dump it into a list I have in an internal issue about JDiff. It looks like the tool that I've seen the most energy around is japicmp (used by AdoptJDK and recommended by Kotlin).

[cpovirk]

We hit one interesting quirk when building with JDK 21+ and running tests under JDK 8 this week: jspecify/jspecify#517.

That happened specifically with JUnit 5, but it could happen in other code that performs reflection. I haven't tested Guava with the 21-8 mix to see what happens here, so I'm just mentioning the problem now as something that we'd want to remember in case we encounter it when we try toolchains for Guava or our other projects.

(We still won't be able to release under JDK 21+ or even 13+, thanks to JDiff.)

[cpovirk]

Another note on toolchains: https://openjdk.org/jeps/471 is coming, so eventually we'll find it difficult to compile against the newest JDK and continue to refer to Unsafe.

[cpovirk]

Related to my previous post: We may end up wanting this in order to conditionally use VarHandle.

[cpovirk]

It looks like we'd probably find it convenient for our tests to be able to use Cleaner in #7546 (and presumably skip those tests when running under Java 8).

[cpovirk]

Actually, wait, I guess I arguably fixed this in #7333, which automatically downloads JDK 23 and builds with that? However, we haven't started actually using Java 9+ APIs in our build. If we do start, we might always hear from users who build Guava themselves with JDK 8 outside of Maven, but it's possible that no one does that.

Maybe I'll try it in a followup to #7546 and then, if that goes well, look at using VarHandle in the future?

[cpovirk]

I'm going to try that.

Another way for trouble to arise is if users perform Animal-Sniffer-like compatibility checking of Guava in their own builds. Maybe that won't come up, either, though....

[cpovirk]

One potential issue for users is if their build tools check dependency completeness for Guava. If so, those tools will likely complain when Guava refers to APIs that aren't present in the JDK that they're using. My hope would be that the errors are mostly suppressible—and that they mostly don't come up in the first place because enough people are using recent enough JDK versions. (I could also "hope" that many build tools don't perform such checks in the first place :))

One interesting twist is that the ideal workarounds for using APIs conditionally without referencing them directly, MethodHandle and VarHandle, themselves can confuse dependency-checking tools. We've seen this with Animal Sniffer, we've seen it with Bazel (though I'm going to expand Bazel's existing support for those classes, at which point we should be good there), and we've seen it with at least one custom internal tool (which likewise had existing partial support).

That said, I don't think we heard from anyone when we started using MethodHandle for CRC32, so that is some kind of evidence, albeit evidence that is compatible with "Tools understand MethodHandle but not VarHandle," in which case users could see issues as I work to migrate from Unsafe to VarHandle.