Java Automatic Dependencies (Jadep)

Jadep is a Bazel BUILD file generator for Java projects. It adds BUILD dependencies that a Java file needs, aiming for <1s response times.

Jadep is intended to manage BUILD files for your own code in the current Bazel workspace (as opposed to BUILD files for third-party libraries).

Jadep is not an official Google product.

Contents

• Java Automatic Dependencies (Jadep)
  • Usage
  • Detailed Example: Migrating a Java project to Bazel
  • Building / Installation
  • How does it Work?
    • Detailed Flow
    • Extracting Class Names
    • Resolver: File System
    • Resolver: JDK / Android SDK
    • Reading BUILD files
  • Extending / Hacking / Future Ideas
  • Bugs
  • Contributing

Usage

~/bin/jadep path/to/File.java

Detailed Example: Migrating a Java project to Bazel

https://github.com/cgrushko/text/blob/master/migrating-gjf-to-bazel.md

Building / Installation

The following will build Jadep and its persistent server, and will copy them to ~/bin/ and ~/jadep/.

# Jadep
mkdir -p ~/bin
mkdir -p ~/jadep

bazel build -c opt //cmd/jadep

jadep=( bazel-bin/cmd/jadep/*/jadep ) # work around https://github.com/bazelbuild/rules_go/issues/1239
cp "${jadep[0]}" ~/bin/

# PackageLoader server
bazel build -c opt --nocheck_visibility //java/com/google/devtools/javatools/jade/pkgloader:GrpcLocalServer_deploy.jar

cp bazel-bin/java/com/google/devtools/javatools/jade/pkgloader/GrpcLocalServer_deploy.jar ~/jadep/
cp scripts/pkgloader_server.sh ~/jadep/

# JDK symbols [Jadep can run without these]
bazel build //:jdk_android_builtin_class_names

cp bazel-genfiles/jdk_android_builtin_class_names.txt ~/jadep/

How does it Work?

After parsing a Java file, Jadep extracts the class names it references.

It then tries to resolve each class name to BUILD rules that provide it, by employing a set of strategies ("resolvers") in sequence.

Once a set of possible BUILD rules is found, it is filtered down according to visibility, tags and so on.

The following subsections detail different parts of Jadep.

Detailed Flow

1. Connect to the PackageLoader server (GrpcLocalServer)

2. Jadep parses Java files to learn which fully-qualified names (FQNs) are referenced. This requires knowing which classes are defined in the same file (e.g., another inner class or a template type name) which is done by computing "jump-to-definition" information and then discarding all class names not defined in the same file.

   Implemented in https://github.com/bazelbuild/tools_jvm_autodeps/blob/master/lang/java/parser/parser.go

3. The FQNs are passed to a sequence of "resolvers". A "resolver" returns BUILD rule candidates that can be used to satisfy a dependency on an FQN. Once a resolver returns a candidate for an FQN (i.e., it resolves it), the FQN is not passed on to additional resolvers. This is done to (a) improve performance and (b) allow ordering resolvers by accuracy to improve its quality.

   The resolver interface is defined in https://github.com/bazelbuild/tools_jvm_autodeps/blob/2d9ab49baf4b1866abe0b4d670dd356ada30fbb4/jadeplib/jadeplib.go#L51

   More details in the Resolver sections, below.

4. Candidates are filtered by visibility, tags, etc. Visibility sometimes requires interpreting multiple BUILD files, and care was taken to interpret as many as possible in parallel.

   Code: https://github.com/bazelbuild/tools_jvm_autodeps/blob/master/filter/filter.go

5. Finally, Jadep asks the user which rule to add.

Extracting Class Names

Jadep parses a Java file to obtain an AST, then partially resolves it: each symbol is mapped to its place of definition. For example, a call to a method maps to the method's definition.

Jadep then walks the AST and finds all

1. symbols that must be class names based on the Java 8 grammar
2. symbols that can be class names, and aren't defined anywhere in the same Java file

Unqualified class names are assumed to be in the same package as the Java file.

This technique gives pretty good results, but the semantics of Java make it impossible to be 100% correct. For example, a subclass has access to all the (visible) inner classes of its superclass, without having to explicitly import them. Jadep doesn't follow inheritance chains because it means reading arbitrary files, so it doesn't know which symbols are inherited.

Resolver: File System

Java source files are typically organized in the file system according to their package and class name, and this resolver utilizes this structure to find BUILD rules.

It is based on the convention that a class named com.foo.Bar will be defined in a file named <content root>/com/foo/Bar.java.

The <content root> is by default either one of {src/main/java, src/test/java}.

The resolver derives a set of file names from the set of content roots and a transformation of the class names it's looking for, and searches for BUILD rules that have these files in their srcs attributes.

The resolver also handles java_library.exports attributes and alias() rules so long as they're in the same Bazel package as the composed file name.

Resolver: JDK / Android SDK

JDK class names (e.g. java.util.List) do not need any BUILD dependencies to build, so this resolver simply maps these classes to nothing, ensuring that Jadep won't add anything for them.

Bazel Android rules don't need dependencies for Android SDK classes, so this resolver also handles these classes.

Reading BUILD files

Since Jadep interacts with existing Bazel rules (e.g., when filtering by visibility) it needs to read BUILD files.

We use Bazel's Skylark interpreter rather than Buildozer, because the latter is unable to interpret macros.

Since the Skylark interpreter is written in Java, a persistent local gRPC server is used to avoid repeatedly paying startup costs.

Extending / Hacking / Future Ideas

• The dictresolver.go is a resolver that uses a plain-text class -> BUILD mapping encoded in CSV, and can be used as an example for how to write a performant resolver.
• A Maven Central resolver would be useful - it would search class names in Maven Central and add their coordinates to a bazel-deps configuration.
• Kythe could be used to generate an index that Jadep uses.

Bugs

1. Jadep doesn't yet handle external repositories. The bazel.Label data structure is unaware of them, as is GrpcLocalServer.

Contributing

See CONTRIBUTING.md