Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] Move R Windows CI to GitHub Actions #3168

Merged
merged 4 commits into from
Jun 22, 2020

Conversation

jameslamb
Copy link
Collaborator

This PR addresses #3065 (review), moving all Windows R CI jobs to GitHub Actions.

Why this is important

This should reduce the total runtime of CI for this project. GitHub Actions allows 20 concurrent builds...AppVeyor (which we currently use) allows 1 😬

While testing on my fork, I found that thanks to this concurrency, all 12 jobs for R run in 12-13 minutes!!!

image

As of this PR, we will have four R Windows jobs all in GitHub Actions:

  • R 4.0 + MSYS2 toolchain
  • R 4.0 + Visual Studio 2019
  • R 3.6 + MINGW toolchain
  • R 3.6 + Visual Studio 2017

You can see the full details of software versions for the GitHub Actions environments here.

What I Changed

We didn't move R jobs into GitHub Actions in #3119 because of the issues with Powershell jobs on GitHub Actions (https://github.community/t/powershell-steps-fail-nondeterministically/115496) / Powershell itself (PowerShell/PowerShell#12823).

R, Rscript, and initexmf all produce harmless messages to stderr, which could trigger the issues linked above. I tried many approaches to fix this, including:

  • redirecting to $null
  • setting $ErrorActionPreference = "SilentlyContinue"
  • using -ErrorAction flag
  • piping to Out-Null with |
  • piping to Out-String with |
  • redirecting to a file

None of these approaches worked, but the one in this PR did: calling these commands with R and using R's built-in redirection, sink().

@jameslamb jameslamb requested a review from StrikerRUS June 16, 2020 03:27
@jameslamb jameslamb requested a review from Laurae2 as a code owner June 16, 2020 03:27
Copy link
Collaborator

@StrikerRUS StrikerRUS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jameslamb Wow! Surprised how fast you've got it work! You are truly magician!

Is everything OK with return code from Run-R-Code-Redirect-Stderr?

Just some minor comments:

.ci/test_r_package_windows.ps1 Outdated Show resolved Hide resolved
.ci/test_r_package_windows.ps1 Show resolved Hide resolved
.ci/test_r_package_windows.ps1 Outdated Show resolved Hide resolved
.github/workflows/main.yml Show resolved Hide resolved
@StrikerRUS
Copy link
Collaborator

@jameslamb Very strange error at master r-package (windows-2016, MSVC, R 3.6):

2020-06-22T20:39:00.9756578Z Downloading R and Rtools
2020-06-22T20:39:01.0197729Z Downloading https://cloud.r-project.org/bin/windows/base/old/3.6.3/R-3.6.3-win.exe
2020-06-22T20:39:07.7544683Z Downloading https://cloud.r-project.org/bin/windows/Rtools/Rtools35.exe
2020-06-22T20:39:17.8466524Z Installing R
2020-06-22T20:39:54.6747608Z Done installing R
2020-06-22T20:39:54.6748751Z Installing Rtools
2020-06-22T20:40:39.7710780Z Done installing Rtools
2020-06-22T20:40:39.7712023Z Installing dependencies
2020-06-22T20:40:43.0911257Z package 'callr' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.1132167Z package 'prettyunits' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.1380186Z package 'backports' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.1600585Z package 'assertthat' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.1874920Z package 'glue' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.2174329Z package 'fansi' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.2429247Z package 'desc' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.2662949Z package 'pkgbuild' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.2903885Z package 'rprojroot' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.3226199Z package 'rstudioapi' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.3549368Z package 'ps' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.3881218Z package 'cli' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.4292255Z package 'crayon' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.4636846Z package 'digest' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.4783234Z package 'ellipsis' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.5258321Z package 'evaluate' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.5506307Z package 'magrittr' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.5764769Z package 'pkgload' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.5961975Z package 'praise' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.6386148Z package 'rlang' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.6659431Z package 'withr' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.7596015Z package 'data.table' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.8169609Z package 'jsonlite' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.1616615Z package 'Matrix' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.1981826Z package 'processx' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.2213825Z package 'R6' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.2953141Z package 'testthat' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.2992830Z 
2020-06-22T20:40:44.2993625Z The downloaded binary packages are in
2020-06-22T20:40:44.2993977Z 	C:\Users\runneradmin\AppData\Local\Temp\RtmpgNi5C6\downloaded_packages
2020-06-22T20:40:44.3519903Z Building R package
2020-06-22T20:40:58.0415255Z ----- build and install logs -----
2020-06-22T20:40:58.0506063Z * checking for file 'D:/a/LightGBM/LightGBM/lightgbm_r/DESCRIPTION' ... OK
2020-06-22T20:40:58.0506283Z 
2020-06-22T20:40:58.0506432Z * preparing 'lightgbm':
2020-06-22T20:40:58.0506821Z 
2020-06-22T20:40:58.0506993Z * checking DESCRIPTION meta-information ... OK
2020-06-22T20:40:58.0507164Z 
2020-06-22T20:40:58.0507639Z * cleaning src
2020-06-22T20:40:58.0507745Z 
2020-06-22T20:40:58.0507902Z * checking for LF line-endings in source and make files and shell scripts
2020-06-22T20:40:58.0507981Z 
2020-06-22T20:40:58.0508163Z * checking for empty or unneeded directories
2020-06-22T20:40:58.0508228Z 
2020-06-22T20:40:58.0508377Z WARNING: directory 'lightgbm/src/compute/test' is empty
2020-06-22T20:40:58.0508510Z 
2020-06-22T20:40:58.0508660Z * looking to see if a 'data/datalist' file should be added
2020-06-22T20:40:58.0508731Z 
2020-06-22T20:40:58.0508877Z * building 'lightgbm_2.3.2.tar.gz'
2020-06-22T20:40:58.0508941Z 
2020-06-22T20:40:58.0509033Z 
2020-06-22T20:40:58.0509084Z 
2020-06-22T20:40:58.0509306Z * installing to library 'd:/a/LightGBM/LightGBM/RLibrary'
2020-06-22T20:40:58.0509418Z 
2020-06-22T20:40:58.0509520Z * installing *source* package 'lightgbm' ...
2020-06-22T20:40:58.0509627Z 
2020-06-22T20:40:58.0509767Z ** using staged installation
2020-06-22T20:40:58.0509827Z 
2020-06-22T20:40:58.0509973Z ** libs
2020-06-22T20:40:58.0510027Z 
2020-06-22T20:40:58.0510218Z       1 [main] make (856) d:\a\LightGBM\LightGBM\RLibrary\Rtools\bin\make.exe: *** fatal error - cygheap base mismatch detected - 0xF35410/0xE55410.
2020-06-22T20:40:58.0510430Z This problem is probably due to using incompatible versions of the cygwin DLL.
2020-06-22T20:40:58.0510610Z Search for cygwin1.dll using the Windows Start->Find/Search facility
2020-06-22T20:40:58.0512177Z and delete all but the most recent version.  The most recent version *should*
2020-06-22T20:40:58.0512409Z reside in x:\cygwin\bin, where 'x' is the drive on which you have
2020-06-22T20:40:58.0512585Z installed the cygwin distribution.  Rebooting is also suggested if you
2020-06-22T20:40:58.0512745Z are unable to find another cygwin DLL.
2020-06-22T20:40:58.0512948Z       1 [main] make 3824 fork: child -1 - forked process 856 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
2020-06-22T20:40:58.0513149Z make: d:/a/LightGBM/LightGBM/RLibrary/R/share/make/winshlib.mk:13: fork: Resource temporarily unavailable
2020-06-22T20:40:58.0513364Z       1 [main] make (140) d:\a\LightGBM\LightGBM\RLibrary\Rtools\bin\make.exe: *** fatal error - cygheap base mismatch detected - 0xF35410/0xF15410.
2020-06-22T20:40:58.0513595Z This problem is probably due to using incompatible versions of the cygwin DLL.
2020-06-22T20:40:58.0513771Z Search for cygwin1.dll using the Windows Start->Find/Search facility
2020-06-22T20:40:58.0513951Z and delete all but the most recent version.  The most recent version *should*
2020-06-22T20:40:58.0514247Z reside in x:\cygwin\bin, where 'x' is the drive on which you have
2020-06-22T20:40:58.0514422Z installed the cygwin distribution.  Rebooting is also suggested if you
2020-06-22T20:40:58.0514587Z are unable to find another cygwin DLL.
2020-06-22T20:40:58.0514783Z   34110 [main] make 3824 fork: child -1 - forked process 140 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
2020-06-22T20:40:58.0514984Z make: d:/a/LightGBM/LightGBM/RLibrary/R/share/make/winshlib.mk:13: fork: Resource temporarily unavailable
2020-06-22T20:40:58.0515203Z       1 [main] make (520) d:\a\LightGBM\LightGBM\RLibrary\Rtools\bin\make.exe: *** fatal error - cygheap base mismatch detected - 0xF35410/0x1085410.
2020-06-22T20:40:58.0515407Z This problem is probably due to using incompatible versions of the cygwin DLL.
2020-06-22T20:40:58.0515540Z Search for cygwin1.dll using the Windows Start->Find/Search facility
2020-06-22T20:40:58.0515718Z and delete all but the most recent version.  The most recent version *should*
2020-06-22T20:40:58.0515899Z reside in x:\cygwin\bin, where 'x' is the drive on which you have
2020-06-22T20:40:58.0516074Z installed the cygwin distribution.  Rebooting is also suggested if you
2020-06-22T20:40:58.0516232Z are unable to find another cygwin DLL.
2020-06-22T20:40:58.0516423Z   66502 [main] make 3824 fork: child -1 - forked process 520 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
2020-06-22T20:40:58.0516623Z make: d:/a/LightGBM/LightGBM/RLibrary/R/share/make/winshlib.mk:13: fork: Resource temporarily unavailable
2020-06-22T20:40:58.0516836Z       1 [main] make (2348) d:\a\LightGBM\LightGBM\RLibrary\Rtools\bin\make.exe: *** fatal error - cygheap base mismatch detected - 0xF35410/0xF15410.
2020-06-22T20:40:58.0517033Z This problem is probably due to using incompatible versions of the cygwin DLL.
2020-06-22T20:40:58.0517205Z Search for cygwin1.dll using the Windows Start->Find/Search facility
2020-06-22T20:40:58.0517341Z and delete all but the most recent version.  The most recent version *should*
2020-06-22T20:40:58.0517515Z reside in x:\cygwin\bin, where 'x' is the drive on which you have
2020-06-22T20:40:58.0517684Z installed the cygwin distribution.  Rebooting is also suggested if you
2020-06-22T20:40:58.0517841Z are unable to find another cygwin DLL.
2020-06-22T20:40:58.0518030Z   98870 [main] make 3824 fork: child -1 - forked process 2348 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
2020-06-22T20:40:58.0518203Z make: fork: Resource temporarily unavailable
2020-06-22T20:40:58.0518357Z ERROR: compilation failed for package 'lightgbm'
2020-06-22T20:40:58.0518427Z 
2020-06-22T20:40:58.0518578Z * removing 'd:/a/LightGBM/LightGBM/RLibrary/lightgbm'
2020-06-22T20:40:58.0518683Z 
2020-06-22T20:40:58.0518894Z ----- end of build and install logs -----
2020-06-22T20:40:58.0610797Z The wrong compiler was used. Check the build logs.
2020-06-22T20:40:58.1127926Z ##[error]Process completed with exit code -1.

@jameslamb
Copy link
Collaborator Author

😱 I have never seen that one!

@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants