Merge hazel into rules haskell #733

Profpatsch · 2019-03-07T13:41:18Z

This merges all commits from https://github.com/DACH-NY/hazel/commits/da-master into rules_haskell.
So far, only the mechanical work has been done (moving files and merging the branches).

There are a few unsolved questions before we can merge:

Should we keep the hazel WORKSPACE file, and if yes, which parts
How to manage attribution and licenses
The fork is from Digital Asset, while the original work and “official upstream” is by FormationAI. We chose the fork because it is compatible with current rules_haskell.

cc @mboes @judah @thumphries @aherrmann @gdeest @nmattia

It is able to download and build packages automatically from Hackage. So far it compiles a small but nontrivial number of packages; see `.circleci/config.yaml` for examples. See the Readme for more details on use.

Also support for: - `*.lhs` and `*.hsc` files - Main files with ".." in their name - Better normalisation in general - Duplicate dependencies and files

There's more potential cleanup, but this seemed like a good milestone. Modifies the previous version of Hazel to use `cabal2build`'s approach: Have a Haskell file that that reifies the `.cabal` structure to `*.bzl`, but otherwise write all logic directly as a Bazel macro. (The previous version had a more awkward round tripp between Haskell and Bazel.) Supports: - libraries and executables - boot files - preprocessors (happy/alex) - data files - `./configure` scripts - `Paths_*` modules In particular, builds a few nontrivial packages: - `aeson` - `language-c` (codegen from `happy`/`alex`) - `lens` (dependencies use `Paths_` modules) - `network` (`./configure` script) See `third_party/cabal2bazel/Readme.md` for a list of changes from the original `cabal2bazel` sources. Note: this PR manually checks in the data-files that Happy and Alex use and which are generated automatically by their custom `Setup.hs` scripts. Figuring out a better solution is TODO; see `templates/Readme.md` for more details.

Integrate cabal2bazel.

Something changed in its CI environment and now several things don't work anymore. - Pin the docker image tag and the nixpkgs revision. (Not sure if that was actually the cause of the breakage; things didn't change afterwards.) - "Brace expansions" aren't working. Take the opportunity to move our list of "golden" packages to a separate file. - The network package doesn't build anymore. Temporarily remove it from that list.

It turns out they're generated automatically by GHC since version 8.0, as long as we pass the right package name and version. That's now possible as of #246. (So bump `rules_haskell` accordingly.) Also add network back since the new `rules_haskell` handles `cc_library` `defines` attributes better.

Make it more clear which parts are about setting up a new project, versus writing rules that use the autogenerated repositories.

* Allow to provide extra libs to auto-generated packages * Simplify the solution according to Judah's feedback

That package has extra-src-files with `.` in their path, which Bazel doesn't allow: ``` extra-source-files: ./cbits/rdrand.c, ./cbits/rdrand.h, README.md ```

A package may list them twice with different version ranges. Fr example, in `http-client`: ``` build-depends: ... , network >= 2.4 if flag(network-uri) build-depends: network >= 2.6, network-uri >= 2.6 else build-depends: network < 2.6 ```

The package expects to be able to import a header from the `unix` package's `install-includes`, which are part of the GHC distribution. Error: ``` ERROR: /private/var/tmp/_bazel_judah.jacobson/0dd0c61b72eb0880ee09f3b1c25cfb68/external/haskell_unix_compat/BUILD:14:1: C++ compilation of rule '@haskell_unix_compat//:unix-compat-cbits' failed (Exit 1) In file included from external/haskell_unix_compat/cbits/HsUnixCompat.c:1: external/haskell_unix_compat/include/HsUnixCompat.h:1:10: fatal error: 'HsUnixConfig.h' file not found ^~~~~~~~~~~~~~~~ ``` This solution is a little hacky because it hard-codes the logic specifically for `unix` in `BUILD.ghc` and `cabal_package.bzl`. It should be possible to add more install-includes in the future, though at first glance I couldn't find any other packages that would need it. Honestly, long-term we should probably replace `BUILD.ghc` with a more principled repository rule. But hopefully this change will give us some more mileage out of the current approach.

The current logic was broken in a previous refactor that removed the trailing `"-lib"`. Add `pretty-show` to the CI tests. It turns out that we can just remove the special case altogether, since it's fine to refer to a target in the same BUILD file by its absolute path.

- unix-time declares one of the `./configure` script's outputs in `extra-tmp-files` - old-time's C library expects `__GLASGOW_HASKELL__` to be set.

By default, GHC loads the user database at `~/.ghc/.../package.conf.d`. This makes the build less hermetic. For example, my laptop has `Cabal-2.2` installed in the user DB, even though we're using a GHC with an older version of Cabal, and this broke the build because cabal2build doesn't work yet with that version. I'll make another PR to get cabal2bazel building with newer versions of Cabal.

* Expose the issue with conflicting module names * Hide “other-modules” automatically

* Allow to fetch GitHub repos via SSH * Try to use newer nixpkgs

Also adds a rule for ghc-paths. I left out the `docdir` parameter since (a) I wasn't sure how to get it, and (b) it's only really useful for `haddock`, but we already use the one distributed with GHC.

The `native.new_http_archive` rule has been deprecated as of bazel-0.12.

The current rules_haskell setup is heavily Nix-based (including the Bazel binary) and its usability relies on having Cachix or some other custom Nix cache. - Switch to an Ubuntu-based container - Add and use cc_configure.bzl. - Update the versions of nixpkgs for consistency - Use the FormationAI/rules_haskell fork, temporarily

Uses the functionality from #337. Hazel now generates repository rules for "core" GHC packages like `base` and `bytestring`. Thus, rules that uses Hazel shouldn't need to specify the `prebuilt_dependencies` attribute anymore. Also change all references to "prebuilt" to "core packages", to match the terminology in stack build plans.

hazel/.circleci/config.yml

hazel/.gitignore

hazel/LICENSE

hazel/README.md

hazel/test/BUILD

hazel/third_party/cabal2bazel/LICENSE

hazel/third_party/cabal2bazel/Readme.md

hazel/tools/bazel.rc

nmattia · 2019-03-25T16:55:27Z

hazel/tools/ghc.bzl

@@ -0,0 +1,28 @@
+load("@bazel_tools//tools/cpp:lib_cc_configure.bzl", "get_cpu_value")
+
+default_ghc_workspaces = {


What is @ghc? Are these good defaults, are they correct?

These defaults expect rules_haskell users to define a @ghc workspace. I think we should rather use the default names for bindists.

nmattia · 2019-03-25T16:56:09Z

hazel/workspace.bzl

+        build_file = "@ai_formation_hazel//third_party/haskell:BUILD.text-metrics",
+    )
+
+    hazel_custom_package_github(


This won't work because of the new CI names. Also embedFile should be fixed once and for all.

…to merge-hazel-repo

hazel/repositories.bzl

gdeest

I am fine with merging this now, but we should really look into CI run time on Darwin…

Profpatsch · 2019-03-27T11:07:20Z

Can you squash all WIP commits? That makes it easier to review.

Profpatsch · 2019-03-27T14:39:05Z

Looks good! Let’s merge once CI is green.

judah and others added 30 commits April 13, 2018 13:52

Initial commit

41a6f9b

An initial version of Hazel.

30f152a

It is able to download and build packages automatically from Hackage. So far it compiles a small but nontrivial number of packages; see `.circleci/config.yaml` for examples. See the Readme for more details on use.

Refactor, and add support for executables.

b4abbcb

Also support for: - `*.lhs` and `*.hsc` files - Main files with ".." in their name - Better normalisation in general - Duplicate dependencies and files

Raw import of cabal2bazel.

55b3431

Merge pull request #3 from FormationAI/cabal2bazel

c816cde

Integrate cabal2bazel.

Clarify the sections of the README. (#9)

26b6615

Make it more clear which parts are about setting up a new project, versus writing rules that use the autogenerated repositories.

Allow defining packages manually (#10)

e3b69ed

Allow to provide extra libs to auto-generated packages (#11)

7f6044e

* Allow to provide extra libs to auto-generated packages * Simplify the solution according to Judah's feedback

Fix Hazel build of entropy. (#12)

47fcc69

That package has extra-src-files with `.` in their path, which Bazel doesn't allow: ``` extra-source-files: ./cbits/rdrand.c, ./cbits/rdrand.h, README.md ```

Dedup package dependency names. (#8)

f60c13b

A package may list them twice with different version ranges. Fr example, in `http-client`: ``` build-depends: ... , network >= 2.4 if flag(network-uri) build-depends: network >= 2.6, network-uri >= 2.6 else build-depends: network < 2.6 ```

Allow to pull packages from “megarepos” (#15)

04e27de

Fix the old-time and unix-time packages. (#16)

3408659

- unix-time declares one of the `./configure` script's outputs in `extra-tmp-files` - old-time's C library expects `__GLASGOW_HASKELL__` to be set.

Provide headers of external libraries to cbits (#19)

b39c6a2

Add c2hs support (#20)

9c8914c

Add ‘doctest’ to the toolchain (#23)

c8b7fe1

Test support for extra source files (#22)

a59cc1d

Hide other-modules to avoid module conflicts (#25)

0505bb8

* Expose the issue with conflicting module names * Hide “other-modules” automatically

Allow to fetch GitHub repos via SSH (#24)

22a0ce3

* Allow to fetch GitHub repos via SSH * Try to use newer nixpkgs

Use the doctest binary that Hazel itself builds. (#28)

ce16efa

Also adds a rule for ghc-paths. I left out the `docdir` parameter since (a) I wasn't sure how to get it, and (b) it's only really useful for `haddock`, but we already use the one distributed with GHC.

Add a status badge to the Hazel README. (#29)

b84eedd

Use the Skylark http_archive. (#30)

e425d23

The `native.new_http_archive` rule has been deprecated as of bazel-0.12.

Specify locales in the toolchain (#31)

f2d6128

Adding non-unix Hazel tests to Windows CI

c8535a2

gdeest force-pushed the merge-hazel-repo branch from 5cb848e to c8535a2 Compare March 25, 2019 16:16

Gaël Deest and others added 3 commits March 25, 2019 17:29

Adding win32 to core packages

30fb0cf

Generate Paths_ in subdir

6eeaac9

More readble CI names

21522d1

nmattia suggested changes Mar 25, 2019

View reviewed changes

Gaël Deest added 7 commits March 26, 2019 08:45

Fixing win32 version.

c44753f

Merge branch 'merge-hazel-repo' of github.com:/tweag/rules_haskell in…

264fc6d

…to merge-hazel-repo

Maybe

1d71f8f

In progress

b3fbeba

Commeting out htaglib

86a84c1

Commeting out failing tests

92a72d1

Commenting out zlib

7d6bfd6

nmattia suggested changes Mar 26, 2019

View reviewed changes

hazel/repositories.bzl Outdated Show resolved Hide resolved

Source haskell_zlib from hazel

7c18ac7

nmattia mentioned this pull request Mar 26, 2019

Provide cache for CI #781

Closed

gdeest approved these changes Mar 27, 2019

View reviewed changes

nmattia approved these changes Mar 27, 2019

View reviewed changes

nmattia changed the title ~~WIP: merge hazel into rules haskell~~ Merge hazel into rules haskell Mar 27, 2019

Nicolas Mattia added 4 commits March 27, 2019 13:30

Drop hazel/repositories.bzl

1527c37

Allow patching github repos

2dd7fc9

Package wai-app-static patch

e312cb9

More windows tests for hazel

dfff93e

nmattia force-pushed the merge-hazel-repo branch from e61102f to dfff93e Compare March 27, 2019 12:33

WORKSPACE: move hazel subrepository to top, document

13080d0

nmattia merged commit 75e5b9e into master Mar 27, 2019

nmattia deleted the merge-hazel-repo branch March 27, 2019 15:22

Profpatsch mentioned this pull request Apr 3, 2019

Add rules_haskell to regression-checked bazel projects #754

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge hazel into rules haskell #733

Merge hazel into rules haskell #733

Profpatsch commented Mar 7, 2019 •

edited by nmattia

Loading

nmattia Mar 25, 2019

gdeest Mar 26, 2019

nmattia Mar 25, 2019

gdeest left a comment

Profpatsch commented Mar 27, 2019

Profpatsch commented Mar 27, 2019

		@@ -0,0 +1,28 @@
		load("@bazel_tools//tools/cpp:lib_cc_configure.bzl", "get_cpu_value")

		default_ghc_workspaces = {

Merge hazel into rules haskell #733

Merge hazel into rules haskell #733

Conversation

Profpatsch commented Mar 7, 2019 • edited by nmattia Loading

nmattia Mar 25, 2019

Choose a reason for hiding this comment

gdeest Mar 26, 2019

Choose a reason for hiding this comment

nmattia Mar 25, 2019

Choose a reason for hiding this comment

gdeest left a comment

Choose a reason for hiding this comment

Profpatsch commented Mar 27, 2019

Profpatsch commented Mar 27, 2019

Profpatsch commented Mar 7, 2019 •

edited by nmattia

Loading