Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List process container images in preview mode #4069

Merged
merged 71 commits into from
Aug 24, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
cb555cb
Dump container image for each process during preview run
bentsherman Jun 30, 2023
e1640fb
Support any directives, save report to JSON file
bentsherman Jul 5, 2023
a84511a
Add -preview-report CLI option to enable preview report
bentsherman Jul 6, 2023
2a85e7a
Merge branch 'master' into 3340-preview-container-images
bentsherman Jul 27, 2023
181c8b3
Apply suggestions from review
bentsherman Jul 27, 2023
aef325e
Merge branch 'master' into 3340-preview-container-images
pditommaso Jul 30, 2023
b330461
Merge branch '3340-preview-container-images' of github.com:nextflow-i…
pditommaso Aug 2, 2023
05c8c58
Merge branch 'master' into 3340-preview-container-images
pditommaso Aug 2, 2023
0db293f
Rename to PreviewContainersObserver [ci fast]
pditommaso Aug 2, 2023
938c7f3
Fix minor type error [ci skip]
pditommaso Aug 2, 2023
6f72b30
Revert unrelated changes
bentsherman Aug 2, 2023
d8a704f
Format JSON as array of objects
bentsherman Aug 2, 2023
631f38b
Pushing missing log [ci skip]
pditommaso Aug 2, 2023
4f1642a
Replace file output with stdout, suppress console output
bentsherman Aug 4, 2023
050874b
Update Wave and Fusion docs (#4149)
bentsherman Aug 3, 2023
4838bfb
Fix strict mode docs (#4150)
bentsherman Aug 3, 2023
6f65591
Fix bug with K8s resource labels (#4147) [ci fast]
bentsherman Aug 3, 2023
b009ecf
Add `-value` option to `config` command (#4142)
afishman Aug 3, 2023
f738ec0
Fix Wave disable flag
pditommaso Aug 4, 2023
7be17d3
Restore Tower CacheManager for backward compatibility
pditommaso Aug 5, 2023
36d6de3
Increase Wave client max attempts
pditommaso Aug 5, 2023
dd7d922
Bump [email protected]
pditommaso Aug 5, 2023
3555e00
Bump [email protected]
pditommaso Aug 5, 2023
167de54
Bump [email protected]
pditommaso Aug 5, 2023
ac2c473
Bump [email protected]
pditommaso Aug 5, 2023
2954436
Update changelog
pditommaso Aug 5, 2023
a391a75
[release 23.08.0-edge] Update timestamp and build number [ci fast]
pditommaso Aug 5, 2023
822184f
Enable cloud cache based on environment variable (#4160)
bentsherman Aug 8, 2023
2079efc
Minor changes [ci fast]
pditommaso Aug 9, 2023
8f0fa99
Fix typos in source code comments (#4173) [ci fast]
mribeirodantas Aug 10, 2023
8482c35
Add 429 http status code to Wave retriable errors
pditommaso Aug 10, 2023
d5ec1f0
Update changelog
pditommaso Aug 11, 2023
bd026e5
Improve Conda build error report [ci fast]
pditommaso Aug 13, 2023
da6ceba
Fix Process hangs when using flatten and finish errorStrategy
pditommaso Aug 13, 2023
1eb5293
Fix Execution should fail if report or timeline file already exists
pditommaso Aug 13, 2023
901ceb7
Fix env output when changing task workdir
pditommaso Aug 13, 2023
41385fa
Improve handling of name-only container env variables
pditommaso Aug 13, 2023
3a3fbf0
Document error about trailing backslash with space (#4180)
bentsherman Aug 13, 2023
ce01de2
Update AWS instructions for creating a custom AMI [ci skip] (#4174)
manuelesimi Aug 13, 2023
df20ac4
Remove lock file from cloudcache (#4167)
bentsherman Aug 13, 2023
f04f62c
Fix checkpoint thread termination (#4166)
pditommaso Aug 13, 2023
333ea7e
Update tip about modifying maps (#4153) [ci skip]
bentsherman Aug 14, 2023
2b13235
Apply K8s Pod metadata to Job (#4057) [ci fast]
bentsherman Aug 14, 2023
694cff0
Remove dockerize launcher classpath file (#4191) [ci fast]
bentsherman Aug 15, 2023
cec558c
Fix `workflow.container` map resolution (#4190)
bentsherman Aug 15, 2023
062cb1c
Escape semicolons in paths (#4193)
bentsherman Aug 15, 2023
7a26e4a
Use root user in Wave container based on micromamba (#4038) [ci fast]
marcodelapierre Aug 15, 2023
109f3b1
FIx nested InvocationTargetException (#4192) [ci fast]
bentsherman Aug 15, 2023
8b2c276
Fix if-guard on log.trace in trask processor inner class [ci fast]
pditommaso Aug 16, 2023
42ea6c8
Add resource labels support for Azure Batch (#4178)
bentsherman Aug 17, 2023
eb38aea
Bump [email protected]
pditommaso Aug 17, 2023
8223dbe
Bump [email protected]
pditommaso Aug 17, 2023
a70ec26
Bump [email protected]
pditommaso Aug 17, 2023
24a695d
Bump [email protected]
pditommaso Aug 17, 2023
ac72bc9
Update changelog [ci fast]
pditommaso Aug 17, 2023
d7c4fc1
[release 23.08.1-edge] Update timestamp and build number [ci fast]
pditommaso Aug 17, 2023
d2ac7d6
Fix security deps in nf-azure plugin
pditommaso Aug 17, 2023
513e0d2
Add support for Wave native build for singularity
pditommaso Aug 20, 2023
8ee9cb2
Bump groovy 3.0.19
pditommaso Aug 23, 2023
e69ec67
Move containers preview to its own command
pditommaso Aug 23, 2023
006623d
Merge branch 'master' into 3340-preview-container-images
pditommaso Aug 23, 2023
d549a82
apply suggestions from review, minor edits
bentsherman Aug 23, 2023
544235b
Add tests [ci fast]
pditommaso Aug 23, 2023
0481202
Fix & more
pditommaso Aug 23, 2023
44614b4
Just a blank [ci skip]
pditommaso Aug 23, 2023
a6c12b2
remove extraneous tests
bentsherman Aug 23, 2023
314f348
Add tests [ci fast]
pditommaso Aug 24, 2023
7ec45eb
Add prompt confirmation [ci fast]
pditommaso Aug 24, 2023
8f0eb45
Restructure json result
pditommaso Aug 24, 2023
15659cf
Add Wave dry-run mode + inspect -concretize
pditommaso Aug 24, 2023
dfdeeb4
update docs [ci fast]
bentsherman Aug 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -1108,6 +1108,24 @@ The following settings are available:

Read the {ref}`container-podman` page to learn more about how to use Podman containers with Nextflow.

(config-preview)=

### Scope `preview`

:::{versionadded} 23.07.0-edge
:::

The `preview` scope controls the preview report that is generated when running a pipeline in preview mode (i.e. using the `-preview` command line option).

`preview.directives`
: The list of process directives to preview (default: `['container', 'cpus', 'memory', 'time']`).

`preview.file`
: Preview report file name. Must be a JSON file (default: `preview-<timestamp>.json`).

`preview.overwrite`
: When `true` overwrites any existing preview report with the same name.

(config-process)=

### Scope `process`
Expand Down
2 changes: 2 additions & 0 deletions modules/nextflow/src/main/groovy/nextflow/Session.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ import nextflow.script.ScriptRunner
import nextflow.script.WorkflowMetadata
import nextflow.spack.SpackConfig
import nextflow.trace.AnsiLogObserver
import nextflow.trace.PreviewReportWriter
import nextflow.trace.TraceObserver
import nextflow.trace.TraceObserverFactory
import nextflow.trace.TraceRecord
Expand Down Expand Up @@ -468,6 +469,7 @@ class Session implements ISession {
CH.broadcast()

if( preview ) {
PreviewReportWriter.create(this).render()
terminated = true
}
else {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,15 @@ class TaskProcessor {

boolean hasErrors() { errorCount>0 }

/**
* Get a preview task config.
*/
TaskConfig getPreviewConfig() {
def config = config.createTaskConfig()
config.context = new TaskContext(this)
return config
}

protected void checkWarn(String msg, Map opts=null) {
if( NF.isStrictMode() )
throw new ProcessUnrecoverableException(msg)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
/*
* Copyright 2013-2023, Seqera Labs
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package nextflow.trace

import java.nio.file.Path

import groovy.json.JsonBuilder
import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import nextflow.Session
import nextflow.dag.DAG
import nextflow.exception.AbortOperationException
import nextflow.file.FileHelper
import nextflow.util.Duration
import nextflow.util.MemoryUnit
/**
* Render the preview report when running a pipeline
* in preview mode.
*
* @author Ben Sherman <[email protected]>
*/
@Slf4j
@CompileStatic
class PreviewReportWriter implements TraceObserver {

static private final List<String> DEF_DIRECTIVES = ['container', 'cpus', 'memory', 'time']

static private final String DEF_FILE_NAME = "preview-${TraceHelper.launchTimestampFmt()}.json"

private DAG dag

private List<String> directives

private Path file

private boolean overwrite

static PreviewReportWriter create(Session session) {
final directives = session.config.navigate('preview.directives', DEF_DIRECTIVES) as List
def file = session.config.navigate('preview.file', DEF_FILE_NAME)
file = (file as Path).complete()
final overwrite = session.config.navigate('preview.overwrite', false) as boolean

return new PreviewReportWriter(session.dag, directives, file, overwrite)
}
bentsherman marked this conversation as resolved.
Show resolved Hide resolved

PreviewReportWriter( DAG dag, List<String> directives, Path file, boolean overwrite ) {
this.dag = dag
this.directives = directives
this.file = file
this.overwrite = overwrite

// check file existance
final attrs = FileHelper.readAttributes(file)
if( attrs ) {
if( overwrite && (attrs.isDirectory() || !file.delete()) )
throw new AbortOperationException("Unable to overwrite existing preview file: ${file.toUriString()}")
else if( !overwrite )
throw new AbortOperationException("Preview file already exists: ${file.toUriString()} -- enable `preview.overwrite` in your config file to overwrite existing preview files")
}
}

void render() {
// get preview data
final jsonData = [:]

for( def vertex : dag.vertices ) {
// skip nodes that are not processes
final process = vertex.process
if( !process )
continue

// get preview task config
final taskConfig = process.getPreviewConfig()

// get preview for each directive
def config = [:]

for( def directive : directives ) {
// try to resolve directive value
def value = null
try {
value = taskConfig.get(directive)
}
catch( Exception e ) {
log.warn1 "Unable to preview directive `${directive}` for process `${process.name}`: ${e}"
}

// convert custom types to string values
if( value instanceof Duration || value instanceof MemoryUnit )
value = value.toString()

// add directive value if it was resolved
if( value != null )
config[directive] = value
}

jsonData[process.name] = config
}

// write preview data to file
file.text = new JsonBuilder(jsonData).toString()
}

}