Scala 3 #1105

i10416 · 2023-03-04T03:40:02Z

address compiler warnings/errors
setup kind-projector migration
implement common derive macros
Cuber and Cuber derivation
Roller and derive macros
resolve conflict
write ArbitraryCaseClassMacro in Scala 3 and pass tests

NOTE:

changes for compatibility between 2.11, 2.12, 2.13 and 3.x

Source files compatible across all Scala version above are located in algebird-core/src/main/scala directory.
The most of changes in scala dir are to add explicit apply (()) and to resolve ambiguous values.

Some files in scala-2.11 and scala-2.12+ are almost the same but there are a few differences.

*s from kind-projector are replaced by _ in 2.12+.
Type wildcards _ are replaced by ? in 2.12+.
There are a few code that require explicit type annotation or unsafe conversion (asInstanceOf[T])

(1) and (2) are necessary mainly because Scala 2.11 does not support "-Xsource:3" and "-P:kind-projector:underscore-placeholders" which are necessary for smooth kind-projector migration.

These duplicated files can be deleted after dropping Scala 2.11. For more details, see https://docs.scala-lang.org/scala3/guides/migration/plugin-kind-projector.html

Scala 3 macros

Macros in algebird-core are ported to Scala 3, but there are still room for peformance and type safety improvement.
Scala 3 macros in algebird-core are extracted to https://github.com/i10416/algebird-macros for debug and test purpose.

address compiler warnings/errors and setup kind-projector migration

CLAassistant · 2023-03-04T03:40:09Z

All committers have signed the CLA.

johnynek · 2023-03-04T04:57:26Z

Thanks for doing this!

I'll look at it in the next days.

It's quite large obviously so it will take some time.

codecov-commenter · 2023-03-04T05:02:26Z

Codecov Report

Patch coverage: 80.94% and project coverage change: -0.03 ⚠️

Comparison is base (3cefba6) 89.60% compared to head (b4f663c) 89.58%.

❗ Current head b4f663c differs from pull request most recent head 1d4bf3d. Consider uploading reports for the commit 1d4bf3d to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1105      +/-   ##
===========================================
- Coverage    89.60%   89.58%   -0.03%     
===========================================
  Files          119      124       +5     
  Lines        10016    10050      +34     
  Branches       695      572     -123     
===========================================
+ Hits          8975     9003      +28     
- Misses        1041     1047       +6

Impacted Files	Coverage Δ
...d-core/src/main/scala-2.12/monad/EitherMonad.scala	`0.00% <0.00%> (ø)`
.../scala-2/com/twitter/algebird/macros/package.scala	`88.88% <ø> (ø)`
.../src/main/scala/com/twitter/algebird/Batched.scala	`67.69% <0.00%> (ø)`
...main/scala/com/twitter/algebird/HashingTrick.scala	`0.00% <0.00%> (ø)`
...main/scala/com/twitter/algebird/MomentsGroup.scala	`81.57% <0.00%> (ø)`
...src/main/scala/com/twitter/algebird/Preparer.scala	`32.14% <0.00%> (ø)`
...rc/main/scala/com/twitter/algebird/SketchMap.scala	`92.45% <ø> (ø)`
...m/twitter/algebird/matrix/SparseColumnMatrix.scala	`30.76% <0.00%> (ø)`
...ore/src/main/scala-2.12/monad/StateWithError.scala	`48.38% <48.38%> (ø)`
...gebird-core/src/main/scala-2.12/monad/Reader.scala	`50.00% <50.00%> (ø)`
... and 57 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

regadas · 2023-03-04T07:40:39Z

This is great! I can help out with the review.

Some source files are moved or copy-pasted to reduce compile errors in Scala 3. They will be removed after tweaking build in future commits.

i10416 · 2023-04-03T15:02:40Z

algebird-core/src/main/scala-3/com/twitter/algebird/compat.scala

+private[algebird] trait CompatDecayedVector {
+  // This is the default monoid that never thresholds.
+  // If you want to set a specific accuracy you need to implicitly override this
+  implicit def monoid[F, C[_]](implicit vs: VectorSpace[F, C], metric: Metric[C[F]]):Monoid[DecayedVector[C]] =


This won't compile in Scala 3 as Scala 3 requires explicit type annotation for implicit def.

Error says

Cannot find VectorSpace type class for Container: C
where: C is a type in method monoid with bounds <: [_] =>> Any
and Ring: Double

i10416 · 2023-04-07T16:59:10Z

Now, algebirdCore 3.2.2 compiles in d123047 ~~though tests do not compile yet~~

copy changes from twitter#1108 to make it easier to resolve conflict

use Math.random() to avoid auto application of `()` while keeping compatibility between 2.11 and 2.13

i10416 · 2023-04-09T16:22:50Z

Now, algebird-core and algebird-test compile in Scala 2.11, 2.12, 2.13 and 3 at 7042270

and all tests but CollectionSpecification pass. CollectionSpecification gets StackOverflow probably because of difference of implicit resolution between Scala 2.x and Scala 3.

I'm going to reduce copy-paste files in algebird-core using traits as in https://github.com/twitter/algebird/pull/1105/files#diff-1ae6af35211d6d6fea648cfe842339cd2d4d08933aed665a3ec78e5aee3c12f1.

i10416 · 2023-04-09T16:28:23Z

algebird-test/src/test/scala/com/twitter/algebird/CollectionSpecification.scala

@@ -85,22 +85,23 @@ class CollectionSpecification extends CheckProperties {
    monoidLaws[Set[Int]]
  }

-  implicit def mapArb[K: Arbitrary, V: Arbitrary: Monoid] = Arbitrary { // scalafix:ok
+  implicit def mapArb[K: Arbitrary, V: Arbitrary: Monoid]:Arbitrary[Map[K, V]] = Arbitrary { // scalafix:ok


Strangely, explicit type annotation Arbitrary[Map[K, V]] here makes tests stack overflow.

i10416 · 2023-04-09T18:05:06Z

I didn't come up with nice solution for them, so I will keep them for now.

i10416 · 2023-04-09T18:09:54Z

@regadas Hi, could you run CI and look through changes? For now, algebird-core and algebird-test compile in Scala 3 andall tests in algebird-core and all tests but CollectionSpecification in algebird-test pass on my local machine.

i10416 · 2023-04-13T16:35:09Z

Sorry for forgetting fmt and check before asking for the review.

scalameta says "pattern must be a value" for macro entrypoint, but it compiles and is proper format.

i10416 · 2023-04-13T16:49:20Z

This is a false-positive error message at macro entrypoint. Possibly bugs in scalameta?

i10416 · 2023-04-13T16:50:41Z

I found the same issue in scalameta. scalameta/scalameta#3063

Workaround: skip compiling java api for Scala 3 because 1. it seems Scala 3 generates Monoid$ in later stage than Scala 2.x, which prevents javaapi from compiling due to missing symbol. 2. we cannot use `compileOrder := CompileOrder.ScalaThenJava` as algebird-core must compile CassandraMurmurHash.java BEFORE Scala, but at the same time algebird-core must compile javaapi AFTER Scala.

i10416 · 2023-04-13T18:53:37Z

@regadas Sorry for the previous request, it is now ready for review (this time, for sure)

i10416 · 2023-04-14T10:58:11Z

minimized reproduction of d51aa1b
scala/scala3#17255

https://github.com/i10416/reproduce-missing-symbol-error-in-scala-3

i10416 · 2023-06-04T06:41:10Z

@regadas Could you review changes when you have a time?

prepare for scala 3

1846662

address compiler warnings/errors and setup kind-projector migration

prepare: move scala 2.x macros to dedicated directory

1d4bf3d

i10416 changed the title ~~prepare for scala 3~~ scala 3 Mar 4, 2023

i10416 added 9 commits April 3, 2023 04:05

prapare: move or copy source files to compile in Scala 3

dfe6cf1

Some source files are moved or copy-pasted to reduce compile errors in Scala 3. They will be removed after tweaking build in future commits.

feat: implement derive macros for common typeclasses

11e2345

tweak to compile

dd316f5

add Cuber and Cuber derivation

d725b47

explicitly abort on Sum types

cbfac0f

update: tweak build to reduce copy-paste

9991f1c

remove changes in 2.11

727a430

remove changes in 2.11

ec1643c

make kind projector symbol consistent

4500887

i10416 commented Apr 3, 2023

View reviewed changes

i10416 changed the title ~~scala 3~~ Scala 3 Apr 3, 2023

feat: add roller macro and refine Abstract member type

9f3722b

workaround: comment out lines to compile

d123047

i10416 force-pushed the scala-3 branch from c7834ec to d123047 Compare April 7, 2023 17:01

i10416 added 8 commits April 8, 2023 02:23

tweak: prepare for compiling algebird-tests

8863340

copy changes from twitter#1108

ea9e5b3

copy changes from twitter#1108 to make it easier to resolve conflict

address compiler warnings

12bab8e

prefer explicit apply

48bbd70

address compiler warnings

e8335bd

update dependencies

caa51f0

resolve conflict

aae4864

cross compile algebird core and test against 2.11,12,13

88526ad

i10416 added 6 commits April 8, 2023 04:16

address some compiler warnings

a25138a

fix: use Math.random()

331415c

use Math.random() to avoid auto application of `()` while keeping compatibility between 2.11 and 2.13

prepare for cross build

5ec33b9

chore: fmt

c3b84ae

feat: implement ArbitraryCaseClassMacro

6784882

update: address compiler errors and warnings

7042270

i10416 marked this pull request as ready for review April 9, 2023 16:23

i10416 commented Apr 9, 2023

View reviewed changes

i10416 added 5 commits April 10, 2023 01:31

ci: add 3.2.2 to test matrix

9bd13ba

improve: reduce copy-pastes

09c7a79

improve: reduce copy paste in tests

0f44cc1

fix compiler error due to missing collection compat import

5274d3f

hack: skip CollectionSpecification in Scala 3

2f1f2f7

i10416 mentioned this pull request Apr 9, 2023

Try to cross-compile Scio-core for Scala 3 spotify/scio#4638

Draft

i10416 added 2 commits April 14, 2023 01:26

fix: address format errors

f462fd5

fix: address scalafix errors

72f68f1

fix: workaround false-positive format error

b3c6195

scalameta says "pattern must be a value" for macro entrypoint, but it compiles and is proper format.

update Scala to 3.3.0

d5defd6

i10416 force-pushed the scala-3 branch from 7ab1c82 to d5defd6 Compare June 4, 2023 06:39

SethTisue mentioned this pull request Apr 29, 2024

Scala 3 support? #1099

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scala 3 #1105

Scala 3 #1105

i10416 commented Mar 4, 2023 •

edited

Loading

CLAassistant commented Mar 4, 2023 •

edited

Loading

johnynek commented Mar 4, 2023

codecov-commenter commented Mar 4, 2023

regadas commented Mar 4, 2023

i10416 Apr 3, 2023 •

edited

Loading

i10416 commented Apr 7, 2023 •

edited

Loading

i10416 commented Apr 9, 2023 •

edited

Loading

i10416 Apr 9, 2023 •

edited

Loading

i10416 commented Apr 9, 2023

i10416 commented Apr 9, 2023 •

edited

Loading

i10416 commented Apr 13, 2023 •

edited

Loading

i10416 commented Apr 13, 2023 •

edited

Loading

i10416 commented Apr 13, 2023

i10416 commented Apr 13, 2023

i10416 commented Apr 14, 2023 •

edited

Loading

i10416 commented Jun 4, 2023

Scala 3 #1105

Are you sure you want to change the base?

Scala 3 #1105

Conversation

i10416 commented Mar 4, 2023 • edited Loading

changes for compatibility between 2.11, 2.12, 2.13 and 3.x

Scala 3 macros

CLAassistant commented Mar 4, 2023 • edited Loading

johnynek commented Mar 4, 2023

codecov-commenter commented Mar 4, 2023

Codecov Report

regadas commented Mar 4, 2023

i10416 Apr 3, 2023 • edited Loading

Choose a reason for hiding this comment

i10416 commented Apr 7, 2023 • edited Loading

i10416 commented Apr 9, 2023 • edited Loading

i10416 Apr 9, 2023 • edited Loading

Choose a reason for hiding this comment

i10416 commented Apr 9, 2023

i10416 commented Apr 9, 2023 • edited Loading

i10416 commented Apr 13, 2023 • edited Loading

i10416 commented Apr 13, 2023 • edited Loading

i10416 commented Apr 13, 2023

i10416 commented Apr 13, 2023

i10416 commented Apr 14, 2023 • edited Loading

i10416 commented Jun 4, 2023

i10416 commented Mar 4, 2023 •

edited

Loading

CLAassistant commented Mar 4, 2023 •

edited

Loading

i10416 Apr 3, 2023 •

edited

Loading

i10416 commented Apr 7, 2023 •

edited

Loading

i10416 commented Apr 9, 2023 •

edited

Loading

i10416 Apr 9, 2023 •

edited

Loading

i10416 commented Apr 9, 2023 •

edited

Loading

i10416 commented Apr 13, 2023 •

edited

Loading

i10416 commented Apr 13, 2023 •

edited

Loading

i10416 commented Apr 14, 2023 •

edited

Loading