Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIP 46 - read classpath from file, remove lib directory in distribution #20631

Merged
merged 3 commits into from
Jun 28, 2024
Merged

SIP 46 - read classpath from file, remove lib directory in distribution #20631

merged 3 commits into from
Jun 28, 2024

Conversation

bishabosha
Copy link
Member

@bishabosha bishabosha commented Jun 19, 2024 •
edited
Loading

This should be backported to the 3.5.0 line

write classpath to file, and do not write the lib directory, this saves duplicating the jars in the distribution.

In the distribution, we write /etc/scala.classpath, /etc/with_compiler.classpath, and /etc/scaladoc.classpath. each of these are formatted such that each line is a relative path from the /maven2 directory in the distribution.

To produce these files, I use the output of the same cs fetch command that is used to populate the maven2 directory - so the correspondence is 1:1. This is a much better solution than the previous manual resolution of classpath from within the bash/batch scripts - as the classpath libraries actually change a lot between versions.

The contents look like the following:

scala.classpath (compiler classpath used by scalac)

org/scala-lang/scala3-interfaces/3.5.1-RC1-bin-SNAPSHOT/scala3-interfaces-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/scala3-compiler_3/3.5.1-RC1-bin-SNAPSHOT/scala3-compiler_3-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/scala3-library_3/3.5.1-RC1-bin-SNAPSHOT/scala3-library_3-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/tasty-core_3/3.5.1-RC1-bin-SNAPSHOT/tasty-core_3-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/modules/scala-asm/9.6.0-scala-1/scala-asm-9.6.0-scala-1.jar
org/scala-sbt/compiler-interface/1.9.6/compiler-interface-1.9.6.jar
org/jline/jline-reader/3.25.1/jline-reader-3.25.1.jar
org/jline/jline-terminal/3.25.1/jline-terminal-3.25.1.jar
org/jline/jline-terminal-jna/3.25.1/jline-terminal-jna-3.25.1.jar
org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar
org/scala-sbt/util-interface/1.9.8/util-interface-1.9.8.jar
org/jline/jline-native/3.25.1/jline-native-3.25.1.jar
net/java/dev/jna/jna/5.14.0/jna-5.14.0.jar

with_compiler.classpath (concatenated to scala.classpath to run the scalac command)

org/scala-lang/scala3-staging_3/3.5.1-RC1-bin-SNAPSHOT/scala3-staging_3-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/scala3-tasty-inspector_3/3.5.1-RC1-bin-SNAPSHOT/scala3-tasty-inspector_3-3.5.1-RC1-bin-SNAPSHOT.jar

and scaladoc.classpath

org/scala-lang/scala3-interfaces/3.5.1-RC1-bin-SNAPSHOT/scala3-interfaces-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/scala3-compiler_3/3.5.1-RC1-bin-SNAPSHOT/scala3-compiler_3-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/scala3-library_3/3.5.1-RC1-bin-SNAPSHOT/scala3-library_3-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/tasty-core_3/3.5.1-RC1-bin-SNAPSHOT/tasty-core_3-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/modules/scala-asm/9.6.0-scala-1/scala-asm-9.6.0-scala-1.jar
org/scala-sbt/compiler-interface/1.9.6/compiler-interface-1.9.6.jar
org/jline/jline-reader/3.25.1/jline-reader-3.25.1.jar
org/jline/jline-terminal/3.25.1/jline-terminal-3.25.1.jar
org/jline/jline-terminal-jna/3.25.1/jline-terminal-jna-3.25.1.jar
org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar
org/scala-sbt/util-interface/1.9.8/util-interface-1.9.8.jar
org/jline/jline-native/3.25.1/jline-native-3.25.1.jar
net/java/dev/jna/jna/5.14.0/jna-5.14.0.jar
org/scala-lang/scala3-tasty-inspector_3/3.5.1-RC1-bin-SNAPSHOT/scala3-tasty-inspector_3-3.5.1-RC1-bin-SNAPSHOT.jar
org/scala-lang/scaladoc_3/3.5.1-RC1-bin-SNAPSHOT/scaladoc_3-3.5.1-RC1-bin-SNAPSHOT.jar
com/vladsch/flexmark/flexmark/0.62.2/flexmark-0.62.2.jar
com/vladsch/flexmark/flexmark-util-ast/0.62.2/flexmark-util-ast-0.62.2.jar
com/vladsch/flexmark/flexmark-util-data/0.62.2/flexmark-util-data-0.62.2.jar
com/vladsch/flexmark/flexmark-util-html/0.62.2/flexmark-util-html-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-anchorlink/0.62.2/flexmark-ext-anchorlink-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-autolink/0.62.2/flexmark-ext-autolink-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-emoji/0.62.2/flexmark-ext-emoji-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-gfm-strikethrough/0.62.2/flexmark-ext-gfm-strikethrough-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-gfm-tasklist/0.62.2/flexmark-ext-gfm-tasklist-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-wikilink/0.62.2/flexmark-ext-wikilink-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-tables/0.62.2/flexmark-ext-tables-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-yaml-front-matter/0.62.2/flexmark-ext-yaml-front-matter-0.62.2.jar
nl/big-o/liqp/0.8.2/liqp-0.8.2.jar
org/jsoup/jsoup/1.17.2/jsoup-1.17.2.jar
com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.15.1/jackson-dataformat-yaml-2.15.1.jar
com/vladsch/flexmark/flexmark-util-builder/0.62.2/flexmark-util-builder-0.62.2.jar
com/vladsch/flexmark/flexmark-util-collection/0.62.2/flexmark-util-collection-0.62.2.jar
com/vladsch/flexmark/flexmark-util-dependency/0.62.2/flexmark-util-dependency-0.62.2.jar
com/vladsch/flexmark/flexmark-util-format/0.62.2/flexmark-util-format-0.62.2.jar
com/vladsch/flexmark/flexmark-util-misc/0.62.2/flexmark-util-misc-0.62.2.jar
com/vladsch/flexmark/flexmark-util-sequence/0.62.2/flexmark-util-sequence-0.62.2.jar
com/vladsch/flexmark/flexmark-util-visitor/0.62.2/flexmark-util-visitor-0.62.2.jar
org/jetbrains/annotations/15.0/annotations-15.0.jar
com/vladsch/flexmark/flexmark-util/0.62.2/flexmark-util-0.62.2.jar
org/nibor/autolink/autolink/0.6.0/autolink-0.6.0.jar
com/vladsch/flexmark/flexmark-jira-converter/0.62.2/flexmark-jira-converter-0.62.2.jar
org/antlr/antlr4-runtime/4.7.2/antlr4-runtime-4.7.2.jar
com/fasterxml/jackson/core/jackson-annotations/2.15.1/jackson-annotations-2.15.1.jar
com/fasterxml/jackson/core/jackson-core/2.15.1/jackson-core-2.15.1.jar
com/fasterxml/jackson/core/jackson-databind/2.15.1/jackson-databind-2.15.1.jar
com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.12.1/jackson-datatype-jsr310-2.12.1.jar
ua/co/k/strftime4j/1.0.5/strftime4j-1.0.5.jar
org/yaml/snakeyaml/2.0/snakeyaml-2.0.jar
com/vladsch/flexmark/flexmark-util-options/0.62.2/flexmark-util-options-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-ins/0.62.2/flexmark-ext-ins-0.62.2.jar
com/vladsch/flexmark/flexmark-ext-superscript/0.62.2/flexmark-ext-superscript-0.62.2.jar

the bash/batch scripts then read these files, concatenating each line to the absolute path of the maven2 directory, and joining with the systems path separator (and replacing / with \ in batch)

fixes #20711

@bishabosha bishabosha requested a review from hamzaremmal June 19, 2024 15:42
@bishabosha bishabosha changed the title replace pack command, do not produce lib directory SIP 46 - do not produce lib directory in distribution Jun 19, 2024
@bishabosha

This comment was marked as outdated.

Sign in to view
@bishabosha
Copy link
Member Author

bishabosha commented Jun 20, 2024 •
edited
Loading

so it would seem that on the dist-linux-x86_64 platform that only the external maven dependencies are being written to the classpath file, not the local libraries 🤔 - see the output in https://github.com/scala/scala3/actions/runs/9600565037/job/26477095041

Edit: so this was because on linux it seems the ivy2local was being prioritised by cs fetch, so now we don't use any default repository.

@bishabosha bishabosha force-pushed the scala-command-remove-lib branch from e55b9b6 to 9d41c7b Compare June 21, 2024 11:39
@bishabosha bishabosha mentioned this pull request Jun 21, 2024
Closed
@bishabosha bishabosha force-pushed the scala-command-remove-lib branch from 93e22c0 to bf67032 Compare June 21, 2024 13:24
@bishabosha bishabosha changed the title SIP 46 - do not produce lib directory in distribution SIP 46 - read classpath from file, remove lib directory in distribution Jun 21, 2024
@bishabosha bishabosha added the area:runner Issues tied to the scala runner command. label Jun 21, 2024
@bishabosha bishabosha requested a review from Gedochao June 21, 2024 14:14
@bishabosha bishabosha added needs-minor-release This PR cannot be merged until the next minor release backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. labels Jun 21, 2024
@bishabosha
Copy link
Member Author

bishabosha commented Jun 21, 2024 •
edited
Loading

Also mentioning @unkarjedy here to see if this classpath idea is sufficient for IntelliJ as mentioned in issue #20413 - the scala3-staging and scala3-tasty-inspector dependencies are separated from the ordinary scala classpath.

@bishabosha bishabosha removed the needs-minor-release This PR cannot be merged until the next minor release label Jun 26, 2024
@bishabosha
Copy link
Member Author

bishabosha commented Jun 26, 2024 •
edited
Loading

it was decided we could do a this in a patch release

@Gedochao Gedochao removed the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label Jun 26, 2024
hamzaremmal
hamzaremmal approved these changes Jun 28, 2024
View reviewed changes
dist/bin/scalac
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why GitHub says that this file is empty...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be investigated: There is a change of permissions in this file and in /dist/bin/common but the CI can still execute the script, not sure why

Copy link
Member Author

@bishabosha bishabosha Jun 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the permissions yeah. Also common is only ran from within other bash scripts with source so it isn't ever directly executed

@hamzaremmal hamzaremmal added the backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. label Jun 28, 2024
@bishabosha bishabosha merged commit f880d6e into scala:main Jun 28, 2024
24 checks passed
@bishabosha bishabosha deleted the scala-command-remove-lib branch June 28, 2024 14:30
@WojciechMazur WojciechMazur added this to the 3.5.0 milestone Jul 1, 2024
@WojciechMazur WojciechMazur mentioned this pull request Jul 1, 2024
Merged
@WojciechMazur WojciechMazur added backport:accepted This PR needs to be backported, once it's been backported replace this tag by "backport:done" and removed backport:nominated If we agree to backport this PR, replace this tag with "backport:accepted", otherwise delete it. labels Jul 1, 2024
WojciechMazur added a commit that referenced this pull request Jul 2, 2024
@WojciechMazur
Backport "SIP 46 - read classpath from file, remove lib directory in …
e2674a8 
…distribution" to 3.5.0 (#20908)

Backports #20631 to 3.5.0-RC3
@WojciechMazur WojciechMazur added the backport:done This PR was successfully backported. label Jul 2, 2024
@WojciechMazur WojciechMazur removed the backport:accepted This PR needs to be backported, once it's been backported replace this tag by "backport:done" label Jul 2, 2024
@hamzaremmal hamzaremmal mentioned this pull request Jul 4, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hamzaremmal hamzaremmal hamzaremmal approved these changes

@Gedochao Gedochao Awaiting requested review from Gedochao

Assignees

@hamzaremmal hamzaremmal

Labels
area:runner Issues tied to the scala runner command. backport:done This PR was successfully backported.
Projects
None yet
Milestone
3.5.0
Development

Successfully merging this pull request may close these issues.

SIP 46 - avoid duplication of jars in binary distribution
4 participants
@bishabosha @hamzaremmal @Gedochao @WojciechMazur