Publishing k8s-enabled assembly fails #109
Comments
|
You shouldn't run go offline for maven given how it's versioning works.
…On Tue, 21 Feb 2017 at 18:45, Andrew Ash ***@***.***> wrote:
See https://circleci.com/gh/palantir/spark/406 and
https://circleci.com/gh/palantir/spark/407
It looks like the Spark Project Assembly task pulls SNAPSHOT jars from a
Maven repo rather than locally -- and because upstream doesn't have a
org.apache.spark:spark-kubernetes_2.11:jar:2.2.0-SNAPSHOT artifact the
assembly fails.
...
[INFO] Spark Project Mesos ................................ SUCCESS [ 0.408 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 8.447 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 0.594 s]
[INFO] Spark Project Assembly ............................. FAILURE [ 0.460 s]
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Integration for Kafka 0.8 .................... SKIPPED
[INFO] Spark Kinesis Integration .......................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED
[INFO] Spark Project Kinesis Assembly ..................... SKIPPED
[INFO] Spark Project Java 8 Tests ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23.474 s
[INFO] Finished at: 2017-02-21T08:45:27+00:00
[INFO] Final Memory: 57M/995M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-assembly_2.11: Could not resolve dependencies for project org.apache.spark:spark-assembly_2.11:pom:2.2.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-kubernetes_2.11:jar:2.2.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-assembly_2.11: Could not resolve dependencies for project org.apache.spark:spark-assembly_2.11:pom:2.2.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-kubernetes_2.11:jar:2.2.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:245)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project org.apache.spark:spark-assembly_2.11:pom:2.2.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-kubernetes_2.11:jar:2.2.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:211)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
... 23 more
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact org.apache.spark:spark-kubernetes_2.11:jar:2.2.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:205)
... 24 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact org.apache.spark:spark-kubernetes_2.11:jar:2.2.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
... 25 more
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.spark:spark-kubernetes_2.11:jar:2.2.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:39)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:355)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:581)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:249)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:520)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:421)
... 27 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-assembly_2.11
./build/mvn -DskipTests -Phadoop-2.7 -Pkinesis-asl -Pkubernetes -Pmesos -Pyarn -Phive-thriftserver -Phive -Psparkr dependency:go-offline returned exit code 1
cc @mccheah <https://github.com/mccheah>
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#109>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAfQVA0l-H7vUJ5JBsFQr3Wfk5MsNLelks5reyLWgaJpZM4MHpC_>
.
|
|
Should we simply remove this line? |
|
Yes, I say we do that. Though watch out since this is default for circle
maven projects so will run it regardless. Need to put in a dummy there.
Maven cannot resolve inter project dependencies within same build
dynamically so this is doesn't work for complex projects at all.
…On Tue, 21 Feb 2017 at 19:49, mccheah ***@***.***> wrote:
Should we simply remove this line?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#109 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAfQVNZU9TJuJ5XNDFHNKp2IzPgCDyJlks5rezHHgaJpZM4MHpC_>
.
|
|
Maybe we can just |
mccheah
pushed a commit
that referenced
this issue
May 1, 2018
## What changes were proposed in this pull request? Union of map and other compatible column result in unresolved operator 'Union; exception Reproduction `spark-sql>select map(1,2), 'str' union all select map(1,2,3,null), 1` Output: ``` Error in query: unresolved operator 'Union;; 'Union :- Project [map(1, 2) AS map(1, 2)#106, str AS str#107] : +- OneRowRelation$ +- Project [map(1, cast(2 as int), 3, cast(null as int)) AS map(1, CAST(2 AS INT), 3, CAST(NULL AS INT))#109, 1 AS 1#108] +- OneRowRelation$ ``` So, we should cast part of columns to be compatible when appropriate. ## How was this patch tested? Added a test (query union of map and other columns) to SQLQueryTestSuite's union.sql. Author: liutang123 <[email protected]> Closes apache#21100 from liutang123/SPARK-24012.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
See https://circleci.com/gh/palantir/spark/406 and https://circleci.com/gh/palantir/spark/407
It looks like the
Spark Project Assemblytask pulls SNAPSHOT jars from a Maven repo rather than locally -- and because upstream doesn't have aorg.apache.spark:spark-kubernetes_2.11:jar:2.2.0-SNAPSHOTartifact the assembly fails.cc @mccheah
The text was updated successfully, but these errors were encountered: