Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Swap Java SDK container to use eclipse-temurin as the base instead of OpenJDK #25402

Merged
merged 1 commit into from
Feb 16, 2023
Merged

Swap Java SDK container to use eclipse-temurin as the base instead of OpenJDK #25402

merged 1 commit into from
Feb 16, 2023

Conversation

lukecwik
Copy link
Member

@lukecwik lukecwik commented Feb 9, 2023

As discussed on https://lists.apache.org/thread/h4m02o0br5g5t3o3m8r4lkl431vvzhsg

Fixes #25371

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

Run Java Spark PortableValidatesRunner Batch

@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

R: @apilloud

@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

Run Java Flink PortableValidatesRunner Batch

apilloud
apilloud approved these changes Feb 9, 2023
View reviewed changes
Copy link
Member

@apilloud apilloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

Run Java Flink PortableValidatesRunner Batch

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2023

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2023

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@kennknowles
Copy link
Member

Is flink docker PVR real signal? Is it signal we care about?

@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

Run Java Flink PortableValidatesRunner Batch

@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

No to my knowledge since it passed here https://ci-beam.apache.org/job/beam_PreCommit_Java_PVR_Flink_Docker_Commit/2457/ on my WIP PR #25408 that added additional docker logging to try to get more error details.

I'll try running again and merge if green.

@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

Run Java_PVR_Flink_Docker PreCommit

1 similar comment
@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

Run Java_PVR_Flink_Docker PreCommit

@lukecwik
Copy link
Member Author

lukecwik commented Feb 9, 2023

Run Java Flink PortableValidatesRunner Batch

3 similar comments
@lukecwik
Copy link
Member Author

Run Java Flink PortableValidatesRunner Batch

@lukecwik
Copy link
Member Author

Run Java Flink PortableValidatesRunner Batch

@lukecwik
Copy link
Member Author

Run Java Flink PortableValidatesRunner Batch

@lukecwik
Copy link
Member Author

Run Java_PVR_Flink_Docker PreCommit

1 similar comment
@lukecwik
Copy link
Member Author

Run Java_PVR_Flink_Docker PreCommit

@lukecwik
Copy link
Member Author

Run Java Flink PortableValidatesRunner Batch

@lukecwik
Copy link
Member Author

Run Java_PVR_Flink_Docker PreCommit

@lukecwik
Copy link
Member Author

lukecwik commented Feb 13, 2023 •
edited
Loading

Currently failing due to license pulling falling over

ERROR:root:Invalid url for zstd-jni-1.5.2-5: https://opensource.org/licenses/BSD-2-Clause. Retrying...

Looks like it might have moved to https://opensource.org/license/bsd-license-php/ or zstd-jni changed licenses

@lukecwik
Copy link
Member Author

Run Java_PVR_Flink_Docker PreCommit

@lukecwik
Copy link
Member Author

Turns out that boot.go is crashing:

[CHAIN MapPartition (MapPartition at [4]PAssert$0/GroupGlobally/{Create.Values, Flatten.PCollections, Window.Into(), WithKeys}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)#0] ERROR org.apache.beam.runners.fnexecution.environment.DockerEnv
ironmentFactory - Docker container f6bb1afde12f48ad8d8179ef5fe67df2979eb73ac90cbda9a910bb48d85de742 logs:
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f8430b84a7c m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: g 0: unknown pc 0x7f8430b84a7c
stack: frame={sp:0x7fffac802010, fp:0x0} stack=[0x7fffac003590,0x7fffac8025a0)
0x00007fffac801f10:  0x0000000000000000  0x00007fffac8022f0 
0x00007fffac801f20:  0x00007fffac8023e0  0x00000000027db2e0 
0x00007fffac801f30:  0x0000000000000178  0x00007f8430d34cee 
0x00007fffac801f40:  0x0000000000000000  0x00007f8430b89c9b 
0x00007fffac801f50:  0x0000000000000000  0x00007fffac8023e0 
0x00007fffac801f60:  0x00007f8430d08723  0x00007f8430d08723 
0x00007fffac801f70:  0x000000ffffffffff  0x0000000000000000 
0x00007fffac801f80:  0x000000ffffffffff  0x0000000000000000 
0x00007fffac801f90:  0x415353454d5f434c  0x505f434c00534547 
0x00007fffac801fa0:  0x0000000000000000  0x0000000000000000 
0x00007fffac801fb0:  0x0000000000000000  0x0000000000000000 
0x00007fffac801fc0:  0x0000000000000000  0x0000000000000000 
0x00007fffac801fd0:  0x0000000000000000  0x0000000000000000 
0x00007fffac801fe0:  0x6e75720000000000  0x6f67632f656d6974 
0x00007fffac801ff0:  0x0000000000000000  0x0000000000000000 
0x00007fffac802000:  0x00000000009710db  0x00007f8430b84a6e 
0x00007fffac802010: <0x0000000000ddf64a  0x0000000000000000 
0x00007fffac802020:  0x000000000096f77e  0x0000000000000004 
0x00007fffac802030:  0x0000000000ddf64c  0x0000000000000000 
0x00007fffac802040:  0x000000000097027a  0x0000000000000005 
0x00007fffac802050:  0x0000000000000000  0x0000000000000000 
0x00007fffac802060:  0x0000000000000000  0x0000000000000000 
0x00007fffac802070:  0x0000000000000000  0x0000000000000000 
0x00007fffac802080:  0x0000000000000000  0x0000000000000000 
0x00007fffac802090:  0x0000000000000000  0x5c5d1ab52c477200 
0x00007fffac8020a0:  0x00007f8430aeb740  0x0000000000000006 
0x00007fffac8020b0:  0x00000000027db2e0  0x0000000000000178 
0x00007fffac8020c0:  0x0000000000daed00  0x00007f8430b30476 
0x00007fffac8020d0:  0x00007f8430d08e90  0x00007f8430b167f3 
0x00007fffac8020e0:  0x0000000000000020  0x0000000000000000 
0x00007fffac8020f0:  0x0000000000000000  0x0000000000000000 
0x00007fffac802100:  0x0000000000000000  0x0000000000000000 
runtime: g 0: unknown pc 0x7f8430b84a7c
stack: frame={sp:0x7fffac802010, fp:0x0} stack=[0x7fffac003590,0x7fffac8025a0)
0x00007fffac801f10:  0x0000000000000000  0x00007fffac8022f0 
0x00007fffac801f20:  0x00007fffac8023e0  0x00000000027db2e0 
0x00007fffac801f30:  0x0000000000000178  0x00007f8430d34cee 
0x00007fffac801f40:  0x0000000000000000  0x00007f8430b89c9b 
0x00007fffac801f50:  0x0000000000000000  0x00007fffac8023e0 
0x00007fffac801f60:  0x00007f8430d08723  0x00007f8430d08723 
0x00007fffac801f70:  0x000000ffffffffff  0x0000000000000000 
0x00007fffac801f80:  0x000000ffffffffff  0x0000000000000000 
0x00007fffac801f90:  0x415353454d5f434c  0x505f434c00534547 
0x00007fffac801fa0:  0x0000000000000000  0x0000000000000000 
0x00007fffac801fb0:  0x0000000000000000  0x0000000000000000 
0x00007fffac801fc0:  0x0000000000000000  0x0000000000000000 
0x00007fffac801fd0:  0x0000000000000000  0x0000000000000000 
0x00007fffac801fe0:  0x6e75720000000000  0x6f67632f656d6974 
0x00007fffac801ff0:  0x0000000000000000  0x0000000000000000 
0x00007fffac802000:  0x00000000009710db  0x00007f8430b84a6e 
0x00007fffac802010: <0x0000000000ddf64a  0x0000000000000000 
0x00007fffac802020:  0x000000000096f77e  0x0000000000000004 
0x00007fffac802030:  0x0000000000ddf64c  0x0000000000000000 
0x00007fffac802040:  0x000000000097027a  0x0000000000000005 
0x00007fffac802050:  0x0000000000000000  0x0000000000000000 
0x00007fffac802060:  0x0000000000000000  0x0000000000000000 
0x00007fffac802070:  0x0000000000000000  0x0000000000000000 
0x00007fffac802080:  0x0000000000000000  0x0000000000000000 
0x00007fffac802090:  0x0000000000000000  0x5c5d1ab52c477200 
0x00007fffac8020a0:  0x00007f8430aeb740  0x0000000000000006 
0x00007fffac8020b0:  0x00000000027db2e0  0x0000000000000178 
0x00007fffac8020c0:  0x0000000000daed00  0x00007f8430b30476 
0x00007fffac8020d0:  0x00007f8430d08e90  0x00007f8430b167f3 
0x00007fffac8020e0:  0x0000000000000020  0x0000000000000000 
0x00007fffac8020f0:  0x0000000000000000  0x0000000000000000 
0x00007fffac802100:  0x0000000000000000  0x0000000000000000 

goroutine 1 [running]:
runtime.systemstack_switch()
        /home/lcwik/sdk/go1.19.3/src/runtime/asm_amd64.s:459 fp=0xc000074780 sp=0xc000074778 pc=0x465480
runtime.main()
        /home/lcwik/sdk/go1.19.3/src/runtime/proc.go:170 +0x6d fp=0xc0000747e0 sp=0xc000074780 pc=0x439b0d
runtime.goexit()
        /home/lcwik/sdk/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000747e8 sp=0xc0000747e0 pc=0x4676a1

rax    0x0
rbx    0x7f8430aeb740
rcx    0x7f8430b84a7c
rdx    0x6
rdi    0x1
rsi    0x1
rbp    0x1
rsp    0x7fffac802010
r8     0x7fffac8020e0
r9     0x7fffffff
r10    0x8
r11    0x246
r12    0x6
r13    0x16
r14    0xdaed00
r15    0x7f840977a35b
rip    0x7f8430b84a7c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

It looks like the issue is an out of date docker installation which explains why the tests pass locally but fail on Jenkins.
Jenkins is using Docker version 18.09.4, build d14af54 while when testing locally I'm using Docker version 20.10.21, build baeda1f

Filed https://issues.apache.org/jira/browse/INFRA-24205 to upgrade Docker on Beam's Jenkins hosts to 20.10.21+

@kennknowles
Copy link
Member

Could this impact users?

@lukecwik
Copy link
Member Author

Could this impact users?

Yes, users will need a new enough version of Docker. Even the swap to Ubuntu Focal will need a new enough version of Docker as well. I updated the dev@ thread with these details, lets keep the discussion going on there.

@lukecwik
Copy link
Member Author

Run Java Examples on Dataflow Runner V2

@lukecwik
Copy link
Member Author

Run Java_PVR_Flink_Docker PreCommit

@lukecwik
Copy link
Member Author

Docker has been upgraded on the Jenkins hosts

@lukecwik lukecwik merged commit 684917f into apache:master Feb 16, 2023
@Abacn Abacn mentioned this pull request Oct 6, 2023
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@apilloud apilloud apilloud approved these changes

Assignees
No one assigned
Labels
Next Action: Reviewers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Task]: Migrate to supported JDK8 and JDK11 containers
3 participants
@lukecwik @kennknowles @apilloud