Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

First step to pipelining support - enable reading Java symbols from TASTy #19074

Merged
merged 4 commits into from
Nov 28, 2023
Merged

First step to pipelining support - enable reading Java symbols from TASTy #19074

merged 4 commits into from
Nov 28, 2023

Conversation

bishabosha
Copy link
Member

@bishabosha bishabosha commented Nov 24, 2023 •
edited
Loading

Release Notes

Road to Pipelined Builds

We made the next concrete preparation for introducing pipelined Scala 3 builds. Now TASTy can store outline signatures, and additionally the signatures of Java source files. This is the only TASTy breaking change required to introduce pipelining, which means that once it is ready, pipelined build support will be able to release in an upcoming patch version of Scala Next.

  • outline signatures (enabled with the OUTLINEattr TASTy attribute) only store what is necessary for separate compilation (i.e. method bodies can be elided). This will enable in the future the possibility of a faster type checking phase because a lot of work is no longer necessary to produce this outline TASTy. Elided expressions are represented by the new ELIDED tree in TASTy.
  • Java signatures (enabled with the JAVAattr TASTy attribute) can be produced faster than waiting for class files from javac, which will be necessary to enable pipelined builds that include Java sources.

Full Summary

This PR contains the minimal forward incompatible changes of pipelining - reading Java and outline symbols from TASTy.
The other changes to implement pipelining have no impact on TASTy or the standard library - so can come in a patch release.

To test reading TASTy produced from Java defined classes, we add two private flags -Yjava-tasty and -Yjava-tasty-output, which are not expected to be used by any build tool. The tests allow us to write just the java signatures to TASTy, package them in a jar, and then read the Java TASTy from the classpath.

  • Keep Java compilation units up to Pickler phase if -Yjava-tasty is set. Skip phases for Java when not needed.
  • Add JAVAattr and OUTLINEattr TASTy attributes, ELIDED tree tag. ELIDED trees are pickled as rhs of java term definitions. ELIDED trees can only be unpickled if OUTLINEattr is present.
  • Java units will set the JAVAattr TASTy attribute. As currently we outline parse Java files we also set the OUTLINEattr.
  • In the future we might expand OUTLINEattr to include outline Scala typing.
  • OUTLINEattr and JAVAattr do not need any special flags to read from the classpath, however to read outline tasty in the -from-tasty mode (so tasty-inspector and scaladoc are included) we do require an explicit -Yallow-outline-from-tasty, as method bodies are required for full functionality.
  • write java tasty files to a special jar, set with -Yjava-tasty-output
    this option is for testing purposes only.

Fix sealedDescendants method for Java Enums.

  • Rename JavaEnumTrait flags to JavaEnum (reflecting the actual flags set)
  • test java enum in SealedDescendantsTest

fixes #15908

@bishabosha bishabosha added the needs-minor-release This PR cannot be merged until the next minor release label Nov 24, 2023
@bishabosha
Copy link
Member Author

@nicolasstucki this PR contains some changes to handling of TASTy attributes so I request your review here

@bishabosha bishabosha mentioned this pull request Nov 24, 2023
Merged
@bishabosha
Copy link
Member Author

bishabosha commented Nov 24, 2023 •
edited
Loading

one open question here is not requiring any flag by default to read OUTLINEattr or JAVAattr - this is critical for usability with the usePipelining := true option in sbt. If project B depends on project A, but project B needs to disable pipelining, then Zinc will not add the -Ypickle-java flag, but the outline tasty from project A will still be on the classpath - so this would be extra friction to require the user enable some flag to read java pickles

@nicolasstucki nicolasstucki mentioned this pull request Nov 27, 2023
Merged
nicolasstucki added a commit that referenced this pull request Nov 27, 2023
@nicolasstucki
Refactor TASTy Attributes (#19091)
e7f8a2c 
We generalize the internal encoding of `Attributes` to be a list of
tags. Then we add add helper methods to have simpler ways to interact
with this abstraction using booleans. This implies that the
pickling/unpickling can be agnostic of the semantics of each tag.
Therefore reducing the number of places that need to be updated when we
add a new tag.

Useful for #19074, #19033, and #18948.
nicolasstucki added a commit to dotty-staging/dotty that referenced this pull request Nov 27, 2023
@nicolasstucki
Refactor TASTy Attributes (scala#19091)
3dbfd7d 
We generalize the internal encoding of `Attributes` to be a list of
tags. Then we add add helper methods to have simpler ways to interact
with this abstraction using booleans. This implies that the
pickling/unpickling can be agnostic of the semantics of each tag.
Therefore reducing the number of places that need to be updated when we
add a new tag.

Useful for scala#19074, scala#19033, and scala#18948.
@bishabosha bishabosha force-pushed the topic/java-tasty branch 2 times, most recently from 2994ebb to 7913851 Compare November 27, 2023 14:15
@bishabosha bishabosha assigned nicolasstucki and unassigned sjrd Nov 27, 2023
nicolasstucki
nicolasstucki approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@nicolasstucki nicolasstucki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CompilationUnitInfo changes LGTM

@bishabosha
fix sealedStrictDescendants for Java enum
40cf267 
- Rename JavaEnumTrait flags to JavaEnum (the actual flags set)
- test java enum in SealedDescendantsTest
@bishabosha
add CompilationUnitInfo to the CompilationUnit
7fa6565
@bishabosha
Support the -Yjava-tasty flag.
3e8e563 
- Keep Java compilation units up to Pickler phase if -Yjava-tasty.
  Skip phases for Java when not needed.
- Add JAVAattr and OUTLINEattr TASTy attributes, ELIDED tree tag.
  ELIDED trees are pickled as rhs of java term definitions.
  ELIDED trees can only be unpickled if OUTLINEattr is present.
  For now OUTLINEattr implies JAVAattr. In the future we might
  expand OUTLINEattr to include outline Scala typing.
- write java tasty files to a special jar, set with -Yjava-tasty-output
  this option is for testing purposes only.
@bishabosha
also test java tasty at runtime
9b12e4a
@bishabosha bishabosha enabled auto-merge November 28, 2023 12:29
@bishabosha bishabosha mentioned this pull request Nov 28, 2023
Closed
@bishabosha bishabosha added the release-notes Should be mentioned in the release notes label Nov 28, 2023
@bishabosha bishabosha merged commit f1e440e into scala:main Nov 28, 2023
17 checks passed
@bishabosha bishabosha deleted the topic/java-tasty branch November 28, 2023 14:03
@Kordyjan Kordyjan added this to the 3.4.0 milestone Dec 20, 2023
bishabosha added a commit that referenced this pull request Apr 4, 2024
@bishabosha
Add support for Pipelined builds (#18880)
a7f00e2 
This includes support for a single pass pipelined build, compatible with
sbt's `ThisBuild/usePipelining`,
- adds `-Ypickle-java` and `-Ypickle-write` flags, expected by Zinc when
pipelining is enabled in sbt.
- when `-Ypickle-write <directory|jar>` is set, then write tasty from
pickler to that output, (building upon
#19074 support for Java signatures
in TASTy files).
- call `apiPhaseCompleted` and `dependencyPhaseCompleted` callbacks,
which will activate early downstream compilation
- calls `generatedNonLocalClass` callbacks early, which enables Zinc to
run the incremental algorithm before starting downstream compilation
(including checking for macro definitions).

generally this can be reviewed commit-by-commit, as they each do an
isolated feature.

As well as many tests in the `sbt-test/pipelining` directory, this has
also been tested locally on `akka/akka-http`, `apache/incubator-pekko`,
`lichess-org/lila`, `scalacenter/scaladex`, `typelevel/fs2`,
`typelevel/http4s`, `typelevel/cats`, `slick/slick`.

This PR sets the ground work for an optional 2-pass compile (reusing the
`OUTLINEattr`), which should use a faster frontend (skipping rhs when
possible) before producing tasty signatures

fixes #19743
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nicolasstucki nicolasstucki nicolasstucki approved these changes

@sjrd sjrd sjrd approved these changes

Assignees

@nicolasstucki nicolasstucki

Labels
needs-minor-release This PR cannot be merged until the next minor release release-notes Should be mentioned in the release notes
Projects
None yet
Milestone
3.4.0
Development

Successfully merging this pull request may close these issues.

Illegal flagset combination for Java Enums in sealedStrictDescendants
4 participants
@bishabosha @sjrd @nicolasstucki @Kordyjan