Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 0.24 - March 2019 #6968

Closed
laurentlb opened this issue Dec 19, 2018 · 101 comments
Closed

Release 0.24 - March 2019 #6968

laurentlb opened this issue Dec 19, 2018 · 101 comments
Assignees
Labels

Comments

@laurentlb
Copy link
Contributor

Target RC date: March 1st

@katre katre assigned katre and unassigned philwo Jan 17, 2019
@laurentlb laurentlb pinned this issue Feb 28, 2019
@katre
Copy link
Member

katre commented Mar 1, 2019

Looks like we have a good baseline for the release in 235e76b. Test run is here: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/831. @meteorcloudy has investigated the two failing projects and they are not release-related.

@meteorcloudy
Copy link
Member

@oquenchil 235e76b won't contain your flip of --incompatible_disable_legacy_cc_provider

@katre
Copy link
Member

katre commented Mar 1, 2019

The bazel-at-head-plus-downstream tests that include the flag flip for
incompatible_disable_legacy_cc_provider show many failures of downstream projects related to the missing "cc" provider (see https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/834). I think this feature needs more time to migrate downstream projects.

@meteorcloudy
Copy link
Member

@katre Yes, @oquenchil and I just agreed to rollback the flag flip

@katre
Copy link
Member

katre commented Mar 1, 2019

Creating rc1:

scripts/release/release.sh create 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86

Tests are running now.

@katre
Copy link
Member

katre commented Mar 1, 2019

Release is blocked by #7605: I cannot actually get the release binaries to build.

@meteorcloudy
Copy link
Member

meteorcloudy commented Mar 4, 2019

@katre Thanks for adding migration-0.24 for the incompatible flags didn't make into 0.24.0.
But we should not remove migration-0.22 and migration-0.23 for them, because if we are using an old version of Bazel on CI we won't be able to fetch them for testing. (For example, we currently still uses 0.22.0 because 0.23.0 has a bug and 0.23.1 is not yet released)
https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/22#98b39372-548a-43be-827f-b096f70ab30e only got 7 incompatible flags, but there are more that should be tested.

I'm adding the migration labels back for 0.22 and 0.23

@philwo
Copy link
Member

philwo commented Mar 4, 2019

Thank you @meteorcloudy!

@katre
Copy link
Member

katre commented Mar 4, 2019

@meteorcloudy My mistake. We should update the docs at https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md and clarify that (I think I see where I went wrong, so expect a PR from me today).

@katre
Copy link
Member

katre commented Mar 4, 2019

Creating rc2:

scripts/release/release.sh create --force_rc=2 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86 badd82e4c5cda7b3232481e1c63a5550ac898cd8

@vbatts
Copy link

vbatts commented Mar 28, 2019

@hlopko fedora 30 is soon to release, and the current "rawhide" will be fedora 31 about 6 months later. I bet there will be a fix before then

@werkt
Copy link
Contributor

werkt commented Mar 28, 2019

I'd like to suggest that a 0.24.1 be released which incorporates changes from #7860 - it will not apply cleanly, as it comes after 9813c58, but it represents a serious bug in 24 tracked in #7856 that we'd much rather not wait for a cut of 25 for a fix of.

@katre
Copy link
Member

katre commented Mar 28, 2019

@werkt Is that issue a release blocker? Asking @buchgr to help triage.

@werkt
Copy link
Contributor

werkt commented Mar 28, 2019

I'll defer to @buchgr on that, but we have been broken since 0.22 with this particular issue (0.23 introduced the break), are still broken on 0.24, won't be fixed until 0.25 without intervention, and have only had a chance to properly diagnose it now (we were late to adopt 0.23 as well).

@katre
Copy link
Member

katre commented Mar 28, 2019

Makes sense for a patch release. I want to wait for the result on #7874 before I cut it, however.

@hlopko
Copy link
Member

hlopko commented Mar 29, 2019

Oh wait gcc-9 is not yet released :) Ok I say #7874 doesn't warrant a patch release.

@vbatts
Copy link

vbatts commented Mar 29, 2019 via email

@katre
Copy link
Member

katre commented Mar 29, 2019

SInce #7874 isn't critical, I will cut a patch release today that includes 80bff99 (to address #7856).

@katre
Copy link
Member

katre commented Mar 29, 2019

I tried to create 0.24.1rc1:

./scripts/release/release.sh create --force_rc=1 0.24.1 f092ec388a4b58788285b928c86f0f81561d4be4 80bff99811356ea5ec2a66f0217df75a56923069

However, I cannot merge this cherry-pick: the code has changed too far in between and I have no idea what should actually be fixed.

@werkt and @buchgr Please give me directions (or, better yet, a commit I can apply to 0.24.0) on how to resolve this. Or wait for the 0.25.0 RC to be delivered early next week.

@werkt
Copy link
Contributor

werkt commented Mar 29, 2019

Working one up based on release-0.24.0 now

@werkt
Copy link
Contributor

werkt commented Mar 29, 2019

https://github.com/werkt/bazel/tree/retry-exception-fix-0.24 contains a single commit that addresses the issue and applies to the base commit of 0.24

@katre
Copy link
Member

katre commented Mar 29, 2019

Okay, second attempt at 0.24.1rc1:

$ git branch -D release-0.24.1
$ git remote add werkt [email protected]:werkt/bazel.git
$ git fetch werkt
$ ./scripts/release/release.sh create --force_rc=1 0.24.1 f092ec388a4b58788285b928c86f0f81561d4be4 c56c489119e6587975964c44ceb9e429ad950736

Waiting on tests and binaries now.

@katre
Copy link
Member

katre commented Mar 29, 2019

0.24.1rc1 is now out: https://releases.bazel.build/0.24.1/rc1/index.html
@werkt Please test this.

@werkt
Copy link
Contributor

werkt commented Mar 29, 2019

Tested, this correctly generates an error that would result in a fallback, excellent!

@werkt
Copy link
Contributor

werkt commented Apr 1, 2019

This release candidate looks acceptable from my end, is there any update on this?

@katre
Copy link
Member

katre commented Apr 2, 2019

I am releasing it today (two business days have passed with no new release blockers).

@katre
Copy link
Member

katre commented Apr 2, 2019

0.24.1 is now released (https://github.com/bazelbuild/bazel/releases/tag/0.24.1).

@petemounce, @vbatts, and @excitoon: can you update the packages you each maintain? Thanks!

@vbatts
Copy link

vbatts commented Apr 2, 2019 via email

@petemounce
Copy link
Contributor

Pushed.

@menny
Copy link
Contributor

menny commented Apr 3, 2019

Bazel crash immediately with error:

/var/lib/spotify/buildagent/teamcity/work/70b3e9bc32b57ee2/bin/bazel: relocation error: /var/lib/spotify/buildagent/teamcity/work/70b3e9bc32b57ee2/bin/bazel: symbol _ZTVNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEEE, version GLIBCXX_3.4.21 not defined in file libstdc++.so.6 with link time reference

On Linux gew1-buildagent-ssd-0933-d.gew1.spotify.net 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

0.23.1 works fine.

@katre
Copy link
Member

katre commented Apr 3, 2019

What version of glibcxx are you using?

@hlopko, is this something that is easy to fix?

@hlopko
Copy link
Member

hlopko commented Apr 4, 2019

@menny, do you use your own C++ toolchain? Did you migrate for #6861, including enabling features that used to be enabled by default in the past:

  • dependency_file
  • random_seed
  • module_maps
  • module_map_home_cwd
  • header_module_compile
  • include_paths,
  • pic
  • preprocessor_define

@hlopko
Copy link
Member

hlopko commented Apr 4, 2019

Oh ignore me, I misuderstood the issue.

@katre
Copy link
Member

katre commented Apr 5, 2019

@menny @hlopko Please file this as a separate issue to ensure it isn't missed for 0.25.

@hlopko
Copy link
Member

hlopko commented Apr 8, 2019

Hi @menny - is this still an issue? Can you repro with a clean install of Ubuntu 14.04 (docker will be fine)? Would you mind creating a new issue and pinging me there so we can investigate properly? Thanks!

@menny
Copy link
Contributor

menny commented Apr 8, 2019

@hlopko, It's still an issue on some of our CI pools (the ones that have ubuntu 14).
I'll try to reproduce it on a Docker container.

@katre
Copy link
Member

katre commented Apr 9, 2019

I am closing this issue since 0.24.1 is fully released. Once the followup issue about GLIBCXX_3.4.21 is triaged and we determine if a further patch to 0.24 is required, I will re-open.

@katre katre closed this as completed Apr 9, 2019
@katre katre unpinned this issue Apr 9, 2019
@menny
Copy link
Contributor

menny commented Apr 9, 2019

@hlopko I've decided to upgrade the old CI machine to Ubuntu 18.
The crash is not an issue any longer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests