diff --git a/docs/amazons3.md b/docs/amazons3.md index 146dcce370..6da7ac4b0d 100644 --- a/docs/amazons3.md +++ b/docs/amazons3.md @@ -16,7 +16,7 @@ s3://my-bucket/data/sequences.fa The usual file operations can be applied to a path handle with the above notation. For example, the content of an S3 file can be printed as follows: -```groovy +```nextflow println file('s3://my-bucket/data/sequences.fa').text ``` diff --git a/docs/aws.md b/docs/aws.md index 5c48cb56af..5c703ada9c 100644 --- a/docs/aws.md +++ b/docs/aws.md @@ -252,7 +252,7 @@ Container options may be passed in long form (e.g `--option value`) or short for Few examples: -```groovy +```nextflow containerOptions '--tmpfs /run:rw,noexec,nosuid,size=128 --tmpfs /app:ro,size=64' containerOptions '-e MYVAR1 --env MYVAR2=foo2 --env MYVAR3=foo3 --memory-swap 3240000 --memory-swappiness 20 --shm-size 16000000' diff --git a/docs/azure.md b/docs/azure.md index df63c5e816..6087c18ec2 100644 --- a/docs/azure.md +++ b/docs/azure.md @@ -163,7 +163,7 @@ By default, the `cpus` and `memory` directives are used to find the smallest mac To specify multiple Azure machine families, use a comma separated list with glob (`*`) values in the `machineType` directive. For example, the following will select any machine size from D or E v5 machines, with additional data disk, denoted by the `d` suffix: -```config +```groovy process.machineType = "Standard_D*d_v5,Standard_E*d_v5" ``` diff --git a/docs/cache-and-resume.md b/docs/cache-and-resume.md index 2149d5fe46..e0bf78cca1 100644 --- a/docs/cache-and-resume.md +++ b/docs/cache-and-resume.md @@ -112,9 +112,9 @@ While Nextflow tries to make it easy to write safe concurrent code, it is still Consider the following example: -```groovy -Channel.of(1,2,3) | map { it -> X=it; X+=2 } | view { "ch1 = $it" } -Channel.of(1,2,3) | map { it -> X=it; X*=2 } | view { "ch2 = $it" } +```nextflow +Channel.of(1,2,3) | map { v -> X=v; X+=2 } | view { v -> "ch1 = $v" } +Channel.of(1,2,3) | map { v -> X=v; X*=2 } | view { v -> "ch2 = $v" } ``` The problem here is that `X` is declared in each `map` closure without the `def` keyword (or other type qualifier). Using the `def` keyword makes the variable local to the enclosing scope; omitting the `def` keyword makes the variable global to the entire script. @@ -123,12 +123,12 @@ Because `X` is global, and operators are executed concurrently, there is a *race The solution is to not use a global variable where a local variable is enough (or in this simple example, avoid the variable altogether): -```groovy +```nextflow // local variable -Channel.of(1,2,3) | map { it -> def X=it; X+=2 } | view { "ch1 = $it" } +Channel.of(1,2,3) | map { v -> def X=v; X+=2 } | view { v -> "ch1 = $v" } // no variable -Channel.of(1,2,3) | map { it -> it * 2 } | view { "ch2 = $it" } +Channel.of(1,2,3) | map { v -> v * 2 } | view { v -> "ch2 = $v" } ``` (cache-nondeterministic-inputs)= @@ -137,7 +137,7 @@ Channel.of(1,2,3) | map { it -> it * 2 } | view { "ch2 = $it" } Sometimes a process needs to merge inputs from different sources. Consider the following example: -```groovy +```nextflow workflow { ch_foo = Channel.of( ['1', '1.foo'], ['2', '2.foo'] ) ch_bar = Channel.of( ['2', '2.bar'], ['1', '1.bar'] ) @@ -158,7 +158,7 @@ It is tempting to assume that the process inputs will be matched by `id` like th The solution is to explicitly join the two channels before the process invocation: -```groovy +```nextflow workflow { ch_foo = Channel.of( ['1', '1.foo'], ['2', '2.foo'] ) ch_bar = Channel.of( ['2', '2.bar'], ['1', '1.bar'] ) diff --git a/docs/channel.md b/docs/channel.md index 11177015db..7c5c6404ac 100644 --- a/docs/channel.md +++ b/docs/channel.md @@ -37,7 +37,7 @@ channels if it is invoked with all value channels, including simple values which For example: -```groovy +```nextflow process foo { input: val x @@ -67,7 +67,7 @@ Channel factories are functions that can create channels. For example, the `Channel.of()` factory can be used to create a channel from an arbitrary list of arguments: -```groovy +```nextflow Channel.of(1, 2, 3).view() ``` diff --git a/docs/conda.md b/docs/conda.md index d3ea9ee56c..20f667ecdb 100644 --- a/docs/conda.md +++ b/docs/conda.md @@ -45,7 +45,7 @@ Alternatively, it can be specified by setting the variable `NXF_CONDA_ENABLED=tr Conda package names can specified using the `conda` directive. Multiple package names can be specified by separating them with a blank space. For example: -```groovy +```nextflow process foo { conda 'bwa samtools multiqc' @@ -94,7 +94,7 @@ Read the Conda documentation for more details about how to create [environment f The path of an environment file can be specified using the `conda` directive: -```groovy +```nextflow process foo { conda '/some/path/my-env.yaml' @@ -124,7 +124,7 @@ Like before, the extension matters. Make sure the dependencies file has a `.txt` If you already have a local Conda environment, you can use it in your workflow specifying the installation directory of such environment by using the `conda` directive: -```groovy +```nextflow process foo { conda '/path/to/an/existing/env/directory' diff --git a/docs/conf.py b/docs/conf.py index 7c556dd275..f7f02e97d7 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -336,3 +336,77 @@ # Allow duplicate toc entries. #epub_tocdup = True + +# see also: https://github.com/pygments/pygments/blob/master/pygments/lexers/jvm.py +from pygments.lexer import RegexLexer +from sphinx.highlighting import lexers + +class NextflowLexer(RegexLexer): + """ + For Nextflow source code. + """ + + import re + from pygments.lexer import bygroups, using, this, default + from pygments.token import Comment, Operator, Keyword, Name, String, Number, Whitespace + from pygments.util import shebang_matches + + name = 'Nextflow' + url = 'https://nextflow.io/' + aliases = ['nextflow', 'nf'] + filenames = ['*.nf'] + mimetypes = ['text/x-nextflow'] + # version_added = '1.5' + + flags = re.MULTILINE | re.DOTALL + + tokens = { + 'root': [ + # Nextflow allows a file to start with a shebang + (r'#!(.*?)$', Comment.Preproc, 'base'), + default('base'), + ], + 'base': [ + (r'[^\S\n]+', Whitespace), + (r'(//.*?)(\n)', bygroups(Comment.Single, Whitespace)), + (r'/\*.*?\*/', Comment.Multiline), + # keywords: go before method names to avoid lexing "throw new XYZ" + # as a method signature + (r'(assert|catch|else|' + r'if|instanceof|new|return|throw|try|in|as)\b', + Keyword), + # method names + (r'^(\s*(?:[a-zA-Z_][\w.\[\]]*\s+)+?)' # return arguments + r'(' + r'[a-zA-Z_]\w*' # method name + r'|"(?:\\\\|\\[^\\]|[^"\\])*"' # or double-quoted method name + r"|'(?:\\\\|\\[^\\]|[^'\\])*'" # or single-quoted method name + r')' + r'(\s*)(\()', # signature start + bygroups(using(this), Name.Function, Whitespace, Operator)), + (r'@[a-zA-Z_][\w.]*', Name.Decorator), + (r'(def|enum|include|from|output|process|workflow)\b', Keyword.Declaration), + (r'(boolean|byte|char|double|float|int|long|short|void)\b', + Keyword.Type), + (r'(true|false|null)\b', Keyword.Constant), + (r'""".*?"""', String.Double), + (r"'''.*?'''", String.Single), + (r'"(\\\\|\\[^\\]|[^"\\])*"', String.Double), + (r"'(\\\\|\\[^\\]|[^'\\])*'", String.Single), + (r'/(\\\\|\\[^\\]|[^/\\])*/', String), + (r"'\\.'|'[^\\]'|'\\u[0-9a-fA-F]{4}'", String.Char), + (r'(\.)([a-zA-Z_]\w*)', bygroups(Operator, Name.Attribute)), + (r'[a-zA-Z_]\w*:', Name.Label), + (r'[a-zA-Z_$]\w*', Name), + (r'[~^*!%&\[\](){}<>|+=:;,./?-]', Operator), + (r'[0-9][0-9]*\.[0-9]+([eE][0-9]+)?[fd]?', Number.Float), + (r'0x[0-9a-fA-F]+', Number.Hex), + (r'[0-9]+L?', Number.Integer), + (r'\n', Whitespace) + ], + } + + def analyse_text(text): + return shebang_matches(text, r'nextflow') + +lexers['nextflow'] = NextflowLexer() diff --git a/docs/config.md b/docs/config.md index 36b8c5f33a..ef1ac5a97c 100644 --- a/docs/config.md +++ b/docs/config.md @@ -20,6 +20,8 @@ When more than one of these options for specifying configurations are used, they You can use the `-C ` option to use a single configuration file and ignore all other files. ::: +(config-syntax)= + ## Syntax The Nextflow configuration syntax is based on the Nextflow script syntax. It is designed for setting configuration options in a declarative manner while also allowing for dynamic expressions where appropriate. diff --git a/docs/developer/nextflow.ast.md b/docs/developer/nextflow.ast.md index 5048ca6a8d..0fe680e344 100644 --- a/docs/developer/nextflow.ast.md +++ b/docs/developer/nextflow.ast.md @@ -25,7 +25,7 @@ You can see the effect of Nextflow's AST transforms by using the Nextflow consol Here is the example from {ref}`your-first-script`: -```groovy +```nextflow params.str = 'Hello world!' process splitLetters { diff --git a/docs/developer/plugins.md b/docs/developer/plugins.md index 336fd449ae..fd0e44d641 100644 --- a/docs/developer/plugins.md +++ b/docs/developer/plugins.md @@ -138,7 +138,7 @@ class MyExecutor extends Executor implements ExtensionPoint { You can then use this executor in your pipeline: -```groovy +```nextflow process foo { executor 'my-executor' } @@ -177,7 +177,7 @@ class MyExtension extends PluginExtensionPoint { You can then use this function in your pipeline: -```groovy +```nextflow include { reverseString } from 'plugin/my-plugin' channel.of( reverseString('hi') ) @@ -185,7 +185,7 @@ channel.of( reverseString('hi') ) You can also use an alias: -```groovy +```nextflow include { reverseString as anotherReverseMethod } from 'plugin/my-plugin' ``` @@ -226,7 +226,7 @@ class MyExtension extends PluginExtensionPoint { You can then use them in your pipeline: -```groovy +```nextflow include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb' def sql = 'select * from FOO' diff --git a/docs/dsl1.md b/docs/dsl1.md index 17e7f68f73..e7503fda91 100644 --- a/docs/dsl1.md +++ b/docs/dsl1.md @@ -6,7 +6,7 @@ In Nextflow version `22.03.0-edge`, DSL2 became the default DSL version. In vers In Nextflow versions prior to `22.03.0-edge`, you must enable DSL2 explicitly in order to use it. You can either set the feature flag in your pipeline script: -```groovy +```nextflow nextflow.enable.dsl=2 ``` @@ -20,7 +20,7 @@ export NXF_DEFAULT_DSL=2 In DSL1, a process definition is also the process invocation. Process inputs and outputs are connected to channels using `from` and `into`. Here is the {ref}`your-first-script` example written in DSL1: -```groovy +```nextflow nextflow.enable.dsl=1 params.str = 'Hello world!' @@ -54,7 +54,7 @@ To migrate this code to DSL2, you need to move all of your channel logic through Refer to the {ref}`workflow-page` page to learn how to define a workflow. The DSL2 version of the above script is duplicated here for your convenience: ```{literalinclude} snippets/your-first-script.nf -:language: groovy +:language: nextflow ``` ## Channel forking @@ -65,7 +65,7 @@ In DSL2, channels are automatically forked when connecting two or more consumers For example, this would not work in DSL1 but is not a problem in DSL2: -```groovy +```nextflow Channel .from('Hello','Hola','Ciao') .set{ cheers } @@ -105,7 +105,7 @@ DSL2 scripts cannot exceed 64 KB in size. Large DSL1 scripts may need to be spli For example: - ```groovy + ```nextflow process foo { input: tuple X, 'some-file.sam' @@ -121,7 +121,7 @@ DSL2 scripts cannot exceed 64 KB in size. Large DSL1 scripts may need to be spli Use: - ```groovy + ```nextflow process foo { input: tuple val(X), path('some-file.sam') @@ -164,7 +164,7 @@ An early preview of DSL2 was available in 2020. Note that some of that early DSL For example: - ```groovy + ```nextflow include './some/library' include bar from './other/library' @@ -176,7 +176,7 @@ An early preview of DSL2 was available in 2020. Note that some of that early DSL Should be replaced with: - ```groovy + ```nextflow include { foo } from './some/library' include { bar } from './other/library' diff --git a/docs/flux.md b/docs/flux.md index 081e823685..8d72ce4971 100644 --- a/docs/flux.md +++ b/docs/flux.md @@ -36,9 +36,9 @@ For additional Flux settings, see the {ref}`flux-executor` section. Here is an example pipeline that we will use: -```groovy +```nextflow workflow { - breakfast = Channel.from '🥞️', '🥑️', '🥧️', '🍵️', '🍞️' + breakfast = Channel.of '🥞️', '🥑️', '🥧️', '🍵️', '🍞️' haveMeal(breakfast) } diff --git a/docs/google.md b/docs/google.md index b2a0bdfa98..b187b35685 100644 --- a/docs/google.md +++ b/docs/google.md @@ -99,7 +99,7 @@ location and that fits the requested resources. If `memory` is not specified, 1 The `machineType` directive can be used to request a specific VM instance type. It can be any predefined Google Compute Platform [machine type](https://cloud.google.com/compute/docs/machine-types) or [custom machine type](https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type). -```groovy +```nextflow process myTask { cpus 8 memory '40 GB' @@ -124,7 +124,7 @@ process anotherTask { The `machineType` directive can also be a comma-separated list of patterns. The pattern can contain a `*` to match any number of characters and `?` to match any single character. Examples of valid patterns: `c2-*`, `m?-standard*`, `n*`. -```groovy +```nextflow process myTask { cpus 8 memory '20 GB' @@ -142,7 +142,7 @@ process myTask { The `machineType` directive can also be an [Instance Template](https://cloud.google.com/compute/docs/instance-templates), specified as `template://`. For example: -```groovy +```nextflow process myTask { cpus 8 memory '20 GB' @@ -172,7 +172,7 @@ The `disk` directive can be used to set the boot disk size or provision a disk f Examples: -```groovy +```nextflow // set the boot disk size disk 100.GB @@ -335,7 +335,7 @@ The process `machineType` directive may optionally be used to specify a predefin Examples: -```groovy +```nextflow process custom_resources_task { cpus 8 memory '40 GB' diff --git a/docs/index.md b/docs/index.md index 4b35977527..f1585124d4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -78,6 +78,7 @@ module notifications secrets sharing +vscode dsl1 ``` diff --git a/docs/metrics.md b/docs/metrics.md index 45850e5f03..8ccb43bbaa 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -15,7 +15,7 @@ Let's illustrate how this plot behaves with several examples. In the first example, let's consider the simple use case in which a process performs one task of pure computation using one CPU. Then, you expect the `Raw Usage` tab to report 100%. If the task is distributed over, 2, 3, 4, `etc.` CPUs, then the `Raw Usage` will be 200%, 300%, 400%, `etc.` respectively. The `% Allocated` tab just rescales the raw value usage with respect to the number of CPUs set with the `cpus` directive (if not set with the directive, the number of CPUs is set to 1, thus showing the same values as in the `Raw Usage` tab). Using the program [stress](https://people.seas.harvard.edu/~apw/stress/) as follows would report 100% in the `Raw Usage` tab and 50% in the `% Allocated` tab since the process asked twice the number of CPUs needed by the process: -```groovy +```nextflow #!/usr/bin/env nextflow process CpuUsageEx1 { @@ -29,7 +29,7 @@ process CpuUsageEx1 { In the second example, some time will be spent performing pure computation and some time just waiting. Using the program [stress](https://people.seas.harvard.edu/~apw/stress/) and `sleep` as follows would report 75% in the `Raw Usage` tab: -```groovy +```nextflow #!/usr/bin/env nextflow process CpuUsageEx2 { @@ -51,7 +51,7 @@ $$ The third example is similar to the second one except that the pure computation stage is performed in a single step forked on 2 CPUs: -```groovy +```nextflow #!/usr/bin/env nextflow process CpuUsageEx3 { @@ -226,7 +226,7 @@ int main(int argc, char **argv) { The first and second programs are executed in `foo` and `bar` processes respectively as follows: -```groovy +```nextflow #!/usr/bin/env nextflow process foo { @@ -276,7 +276,7 @@ The plot has two tabs the job duration (a.k.a. elapsed real time, real time or w The plot has two tabs showing how many data were read and/or written each process. For example, the following processes read and write 1GB and 256MB of data respectively: -```groovy +```nextflow #!/usr/bin/env nextflow process io_read_write_1G { diff --git a/docs/module.md b/docs/module.md index 0fca91dcaa..c9dae998ff 100644 --- a/docs/module.md +++ b/docs/module.md @@ -14,7 +14,7 @@ You can include any definition from a module into a Nextflow script using the `i For example: -```groovy +```nextflow include { foo } from './some/module' workflow { @@ -49,7 +49,7 @@ some When defined as a directory, the module must be included by specifying the module directory path: -```groovy +```nextflow include { foo } from './some/module' ``` @@ -59,7 +59,7 @@ Module directories allow the use of module scoped binaries scripts. See [Module A Nextflow script can include any number of modules, and an `include` statement can import any number of definitions from a module. Multiple definitions can be included from the same module by using the syntax shown below: -```groovy +```nextflow include { foo; bar } from './some/module' workflow { @@ -75,7 +75,7 @@ workflow { When including definition from a module, it's possible to specify an *alias* with the `as` keyword. Aliasing allows you to avoid module name clashes, by assigning them different names in the including context. For example: -```groovy +```nextflow include { foo } from './some/module' include { foo as bar } from './other/module' @@ -87,7 +87,7 @@ workflow { You can also include the same definition multiple times under different names: -```groovy +```nextflow include { foo; foo as bar } from './some/module' workflow { @@ -106,7 +106,7 @@ As a best practice, parameters should be used in the entry workflow and passed t A module can define parameters using the same syntax as a Nextflow workflow script: -```groovy +```nextflow params.foo = 'Hello' params.bar = 'world!' @@ -117,7 +117,7 @@ def sayHello() { When including a module, the module will first use parameters from the including context. For example: -```groovy +```nextflow params.foo = 'Hola' params.bar = 'Mundo' @@ -144,7 +144,7 @@ It is best to define all pipeline parameters *before* any `include` statements. The `addParams` option can be used to pass parameters to the module without adding them to the including scope. -```groovy +```nextflow params.foo = 'Hola' params.bar = 'Mundo' @@ -163,7 +163,7 @@ Ciao Mundo Alternatively, the `params` option can be used to pass parameters to module without adding them to the including scope, *and* without inheriting any parameters from the including scope. -```groovy +```nextflow params.foo = 'Hola' params.bar = 'Mundo' @@ -255,7 +255,7 @@ Modules can define binary scripts that are locally scoped to the processes defin To enable this feature, set the following flag in your pipeline script or configuration file: -```groovy +```nextflow nextflow.enable.moduleBinaries = true ``` diff --git a/docs/notifications.md b/docs/notifications.md index 6df39db2d2..9f58f56fd4 100644 --- a/docs/notifications.md +++ b/docs/notifications.md @@ -16,7 +16,7 @@ Due to the asynchronous nature of Nextflow the termination of a script does not The `onComplete` event handler is invoked by the framework when the workflow execution is completed. It allows one to access the workflow termination status and other useful information. For example: -```groovy +```nextflow workflow.onComplete { println "Pipeline completed at: $workflow.complete" println "Execution status: ${ workflow.success ? 'OK' : 'failed' }" @@ -29,7 +29,7 @@ workflow.onComplete { The `onError` event handler is invoked by Nextflow when a runtime or process error caused the pipeline execution to stop. For example: -```groovy +```nextflow workflow.onError { println "Error: Pipeline execution stopped with the following message: ${workflow.errorMessage}" } @@ -49,7 +49,7 @@ The built-in function `sendMail` allows you to send a mail message from a workfl The mail attributes are specified as named parameters or an equivalent map. For example: -```groovy +```nextflow sendMail( to: 'you@gmail.com', subject: 'Catch up', @@ -60,7 +60,7 @@ sendMail( which is equivalent to: -```groovy +```nextflow mail = [ to: 'you@gmail.com', subject: 'Catch up', @@ -113,7 +113,7 @@ The following parameters can be specified: Another version of `sendMail` allows a more idiomatic syntax: -```groovy +```nextflow sendMail { to 'you@gmail.com' from 'me@gmail.com' @@ -161,7 +161,7 @@ Moreover for each attachment it's possible to specify any of the following optio For example: -```groovy +```nextflow sendMail { to 'you@dot.com' attach '/some/file.txt', fileName: 'manuscript.txt' @@ -215,7 +215,7 @@ ses:SendRawEmail You can use the `sendMail` function with a {ref}`workflow completion handler ` to notify the completion of a workflow completion. For example: -```groovy +```nextflow workflow.onComplete { def msg = """\ diff --git a/docs/overview.md b/docs/overview.md index 2bfb585202..a393c823de 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -20,7 +20,7 @@ Any process can define one or more channels as *input* and *output*. The interac A Nextflow script looks like this: -```groovy +```nextflow // Script parameters params.query = "/some/data/sample.fa" params.db = "/some/path/pdb" diff --git a/docs/process.md b/docs/process.md index 5e0f68984e..d84112d99a 100644 --- a/docs/process.md +++ b/docs/process.md @@ -6,7 +6,7 @@ In Nextflow, a **process** is a specialized function for executing scripts in a Here is an example process definition: -```groovy +```nextflow process sayHello { output: path 'hello.txt' @@ -32,7 +32,7 @@ The script string is executed as a [Bash]( blast_result @@ -56,7 +56,7 @@ When you need to access a system environment variable in your script, you have t If you don't need to access any Nextflow variables, you can define your script block with single-quotes: -```groovy +```nextflow process printPath { ''' echo The path is: $PATH @@ -66,7 +66,7 @@ process printPath { Otherwise, you can define your script with double-quotes and escape the system environment variables by prefixing them with a back-slash `\` character, as shown in the following example: -```groovy +```nextflow process doOtherThings { """ blastp -db \$DB -query query.fa -outfmt 6 > blast_result @@ -92,7 +92,7 @@ A pipeline may be composed of processes that execute very different tasks. With To use a language other than Bash, simply start your process script with the corresponding [shebang](). For example: -```groovy +```nextflow process perlTask { """ #!/usr/bin/perl @@ -127,7 +127,7 @@ The `script` block is like a function that returns a string. This means that you If-else statements based on task inputs can be used to produce a different script. For example: -```groovy +```nextflow mode = 'tcoffee' process align { @@ -165,7 +165,7 @@ Process scripts can be externalised to **template** files, which can be reused a A template is simply a shell script file that Nextflow is able to execute by using the `template` function as shown below: -```groovy +```nextflow process templateExample { input: val STR @@ -206,7 +206,7 @@ The `shell` block is a string expression that defines the script that is execute This way, it is possible to use both Nextflow and Bash variables in the same script without having to escape the latter, which makes process scripts easier to read and maintain. For example: -```groovy +```nextflow process myTask { input: val str @@ -238,7 +238,7 @@ The `exec` block executes the given code without launching a job. For example: -```groovy +```nextflow process simpleSum { input: val x @@ -271,7 +271,7 @@ A native process is very similar to a {ref}`function `. However You can define a command *stub*, which replaces the actual process command when the `-stub-run` or `-stub` command-line option is enabled: -```groovy +```nextflow process INDEX { input: path transcriptome @@ -330,7 +330,7 @@ See {ref}`process reference ` for the full list of inp The `val` qualifier accepts any data type. It can be accessed in the process script by using the specified input name, as shown in the following example: -```groovy +```nextflow process basicExample { input: val x @@ -359,7 +359,7 @@ While channels do emit items in the order that they are received, *processes* do :::{note} When the process declares exactly one input, the pipe `|` operator can be used to provide inputs to the process, instead of passing it as a parameter. Both methods have identical semantics: -```groovy +```nextflow process basicExample { input: val x @@ -379,7 +379,7 @@ workflow { The `path` qualifier allows you to provide input files to the process execution context. Nextflow will stage the files into the process execution directory, and they can be accessed in the script by using the specified input name. For example: -```groovy +```nextflow process blastThemAll { input: path query_file @@ -399,21 +399,21 @@ It's worth noting that in the above example, the name of the file in the file-sy There may be cases where your task needs to use a file whose name is fixed, i.e. it does not have to change along with the actual provided file. In this case, you can specify a fixed name with the `name` attribute in the input file parameter definition, as shown in the following example: -```groovy +```nextflow input: path query_file, name: 'query.fa' ``` or, using a shorter syntax: -```groovy +```nextflow input: path 'query.fa' ``` The previous example can be re-written as shown below: -```groovy +```nextflow process blastThemAll { input: path 'query.fa' @@ -435,7 +435,7 @@ This feature allows you to execute the process command multiple times without wo Channel factories like `Channel.fromPath` produce file objects, but a `path` input can also accept a string literal path. The string value should be an absolute path, i.e. it must be prefixed with a `/` character or a supported URI protocol (`file://`, `http://`, `s3://`, etc), and it cannot contain special characters (`\n`, etc). -```groovy +```nextflow process foo { input: path x @@ -457,7 +457,7 @@ By default, `path` inputs will accept any number of files and stage them accordi For example: -```groovy +```nextflow input: path('one.txt', arity: '1') // exactly one file is expected path('pair_*.txt', arity: '2') // exactly two files are expected @@ -469,7 +469,7 @@ When a task is executed, Nextflow will check whether the received files for each :::{note} Process `path` inputs have nearly the same interface as described in {ref}`stdlib-types-path`, with one difference which is relevant when files are staged into a subdirectory. Given the following input: -```groovy +```nextflow path x, name: 'my-dir/*' ``` @@ -482,7 +482,7 @@ A `path` input can also accept a collection of files instead of a single value. When the input has a fixed file name and a collection of files is received by the process, the file name will be appended with a numerical suffix representing its ordinal position in the list. For example: -```groovy +```nextflow process blastThemAll { input: path 'seq' @@ -521,7 +521,7 @@ The target input file name may contain the `*` and `?` wildcards, which can be u The following example shows how a wildcard can be used in the input file definition: -```groovy +```nextflow process blastThemAll { input: path 'seq?.fa' @@ -543,7 +543,7 @@ Rewriting input file names according to a named pattern is an extra feature and When the input file name is specified by using the `name` option or a string literal, you can also use other input values as variables in the file name string. For example: -```groovy +```nextflow process simpleCount { input: val x @@ -567,7 +567,7 @@ In most cases, you won't need to use dynamic file names, because each task is ex The `env` qualifier allows you to define an environment variable in the process execution context based on the input value. For example: -```groovy +```nextflow process printEnv { input: env 'HELLO' @@ -593,7 +593,7 @@ hola world! The `stdin` qualifier allows you to forward the input value to the [standard input](http://en.wikipedia.org/wiki/Standard_streams#Standard_input_.28stdin.29) of the process script. For example: -```groovy +```nextflow process printAll { input: stdin @@ -625,7 +625,7 @@ hello The `tuple` qualifier allows you to group multiple values into a single input definition. It can be useful when a channel emits tuples of values that need to be handled separately. Each element in the tuple is associated with a corresponding element in the `tuple` definition. For example: -```groovy +```nextflow process tupleExample { input: tuple val(x), path('input.txt') @@ -649,7 +649,7 @@ A `tuple` definition may contain any of the following qualifiers, as previously The `each` qualifier allows you to repeat the execution of a process for each item in a collection, each time a new value is received. For example: -```groovy +```nextflow process alignSequences { input: path seq @@ -672,7 +672,7 @@ In the above example, each time a file of sequences is emitted from the `sequenc Input repeaters can be applied to files as well. For example: -```groovy +```nextflow process alignSequences { input: path seq @@ -715,7 +715,7 @@ As a result, channel values are consumed sequentially and any empty channel will For example: -```groovy +```nextflow process foo { input: val x @@ -745,7 +745,7 @@ A different semantic is applied when using a {ref}`value channel ` for the full l The `val` qualifier allows you to output any Nextflow variable defined in the process. A common use case is to output a variable that was defined in the `input` block, as shown in the following example: -```groovy +```nextflow process foo { input: each x @@ -833,7 +833,7 @@ workflow { The output value can be a value literal, an input variable, any other Nextflow variable in the process scope, or a value expression. For example: -```groovy +```nextflow process foo { input: path infile @@ -864,7 +864,7 @@ workflow { The `path` qualifier allows you to output one or more files produced by the process. For example: -```groovy +```nextflow process randomNum { output: path 'result.txt' @@ -891,7 +891,7 @@ By default, `path` outputs will accept any number of matching files from the tas For example: -```groovy +```nextflow output: path('one.txt', arity: '1') // exactly one file is expected path('pair_*.txt', arity: '2') // exactly two files are expected @@ -904,7 +904,7 @@ When a task completes, Nextflow will check whether the produced files for each p When an output file name contains a `*` or `?` wildcard character, it is interpreted as a [glob][glob] path matcher. This allows you to capture multiple files into a list and emit the list as a single value. For example: -```groovy +```nextflow process splitLetters { output: path 'chunk_*' @@ -947,7 +947,7 @@ Read more about glob syntax at the following link [What is a glob?][glob] When an output file name needs to be expressed dynamically, it is possible to define it using a dynamic string which references variables in the `input` block or in the script global context. For example: -```groovy +```nextflow process align { input: val species @@ -983,7 +983,7 @@ To sum up, the use of output files with static names over dynamic ones is prefer The `env` qualifier allows you to output a variable defined in the process execution environment: ```{literalinclude} snippets/process-out-env.nf -:language: groovy +:language: nextflow ``` (process-stdout)= @@ -993,7 +993,7 @@ The `env` qualifier allows you to output a variable defined in the process execu The `stdout` qualifier allows you to output the `stdout` of the executed process: ```{literalinclude} snippets/process-stdout.nf -:language: groovy +:language: nextflow ``` (process-out-eval)= @@ -1006,7 +1006,7 @@ The `stdout` qualifier allows you to output the `stdout` of the executed process The `eval` qualifier allows you to capture the standard output of an arbitrary command evaluated the task shell interpreter context: ```{literalinclude} snippets/process-out-eval.nf -:language: groovy +:language: nextflow ``` Only one-line Bash commands are supported. You can use a semi-colon `;` to specify multiple Bash commands on a single line, and many interpreters can execute arbitrary code on the command line, e.g. `python -c 'print("Hello world!")'`. @@ -1019,7 +1019,7 @@ If the command fails, the task will also fail. In Bash, you can append `|| true` The `tuple` qualifier allows you to output multiple values in a single channel. It is useful when you need to associate outputs with metadata, for example: -```groovy +```nextflow process blast { input: val species @@ -1035,7 +1035,7 @@ process blast { } workflow { - ch_species = Channel.from('human', 'cow', 'horse') + ch_species = Channel.of('human', 'cow', 'horse') ch_query = Channel.fromPath('*.fa') blast(ch_species, ch_query) @@ -1051,7 +1051,7 @@ While parentheses for input and output qualifiers are generally optional, they a Here's an example with a single path output (parentheses optional): -```groovy +```nextflow process foo { output: path 'result.txt', hidden: true @@ -1064,7 +1064,7 @@ process foo { And here's an example with a tuple output (parentheses required): -```groovy +```nextflow process foo { output: tuple path('last_result.txt'), path('result.txt', hidden: true) @@ -1083,7 +1083,7 @@ process foo { The `emit` option can be used on a process output to define a name for the corresponding output channel, which can be used to access the channel by name from the process output. For example: -```groovy +```nextflow process FOO { output: path 'hello.txt', emit: hello @@ -1107,7 +1107,7 @@ See {ref}`workflow-process-invocation` for more details. Normally, if a specified output is not produced by the task, the task will fail. Setting `optional: true` will cause the task to not fail, and instead emit nothing to the given output channel. -```groovy +```nextflow output: path("output.txt"), optional: true ``` @@ -1130,7 +1130,7 @@ The `when` block allows you to define a condition that must be satisfied in orde It can be useful to enable/disable the process execution depending on the state of various inputs and parameters. For example: -```groovy +```nextflow process find { input: path proteins @@ -1176,7 +1176,7 @@ Software dependencies: The `task` object also contains the values of all process directives for the given task, which allows you to access these settings at runtime. For examples: -```groovy +```nextflow process foo { script: """ @@ -1195,7 +1195,7 @@ A directive can be assigned *dynamically*, during the process execution, so that To be defined dynamically, the directive's value needs to be expressed using a {ref}`closure `. For example: -```groovy +```nextflow process foo { executor 'sge' queue { entries > 100 ? 'long' : 'short' } @@ -1221,7 +1221,7 @@ All directives can be assigned a dynamic value except the following: :::{tip} Assigning a string value with one or more variables is always resolved in a dynamic manner, and therefore is equivalent to the above syntax. For example, the above directive can also be written as: -```groovy +```nextflow queue "${ entries > 100 ? 'long' : 'short' }" ``` @@ -1236,7 +1236,7 @@ It's a very common scenario that different instances of the same process may hav The [Dynamic directives](#dynamic-directives) evaluation feature can be used to modify the amount of computing resources requested in case of a process failure and try to re-execute it using a higher limit. For example: -```groovy +```nextflow process foo { memory { 2.GB * task.attempt } time { 1.hour * task.attempt } @@ -1261,7 +1261,7 @@ The directive {ref}`process-maxretries` set the maximum number of time the same Task resource requests can be updated relative to the {ref}`trace record ` metrics of the previous task attempt. The metrics can be accessed through the `task.previousTrace` variable. For example: -```groovy +```nextflow process foo { memory { task.attempt > 1 ? task.previousTrace.memory * 2 : (1.GB) } errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' } @@ -1278,7 +1278,7 @@ In the above example, the {ref}`process-memory` is set according to previous tra There are cases in which the required execution resources may be temporary unavailable e.g. network congestion. In these cases immediately re-executing the task will likely result in the identical error. A retry with an exponential backoff delay can better recover these error conditions: -```groovy +```nextflow process foo { errorStrategy { sleep(Math.pow(2, task.attempt) * 200 as long); return 'retry' } maxRetries 5 diff --git a/docs/reference/channel.md b/docs/reference/channel.md index cfad21283c..fa1cb16825 100644 --- a/docs/reference/channel.md +++ b/docs/reference/channel.md @@ -20,7 +20,7 @@ Use [channel.of](#of) or [channel.fromList](#fromlist) instead. The `channel.from` method allows you to create a channel emitting any sequence of values that are specified as the method argument, for example: -```groovy +```nextflow ch = channel.from( 1, 3, 5, 7 ) ch.subscribe { v -> println "value: $v" } ``` @@ -36,7 +36,7 @@ value: 7 The following example shows how to create a channel from a *range* of numbers or strings: -```groovy +```nextflow zeroToNine = channel.from( 0..9 ) strings = channel.from( 'A'..'Z' ) ``` @@ -47,14 +47,14 @@ When the `channel.from` argument is an object implementing the (Java) [Collectio Thus the following two declarations produce an identical result even though in the first case the items are specified as multiple arguments while in the second case as a single list object argument: -```groovy +```nextflow channel.from( 1, 3, 5, 7, 9 ) channel.from( [1, 3, 5, 7, 9] ) ``` But when more than one argument is provided, they are always managed as *single* emissions. Thus, the following example creates a channel emitting three entries each of which is a list containing two elements: -```groovy +```nextflow channel.from( [1, 2], [5,6], [7,9] ) ``` @@ -67,7 +67,7 @@ channel.from( [1, 2], [5,6], [7,9] ) The `channel.fromList` method allows you to create a channel emitting the values provided as a list of elements, for example: -```groovy +```nextflow channel .fromList( ['a', 'b', 'c', 'd'] ) .view { v -> "value: $v" } @@ -91,7 +91,7 @@ See also: [channel.of](#of) factory method. You can create a channel emitting one or more file paths by using the `channel.fromPath` method and specifying a path string as an argument. For example: -```groovy +```nextflow myFileChannel = channel.fromPath( '/data/some/bigfile.txt' ) ``` @@ -105,7 +105,7 @@ object for the specified file. Whenever the `channel.fromPath` argument contains a `*` or `?` wildcard character it is interpreted as a [glob][glob] path matcher. For example: -```groovy +```nextflow myFileChannel = channel.fromPath( '/data/big/*.txt' ) ``` @@ -117,7 +117,7 @@ Two asterisks, i.e. `**`, works like `*` but crosses directory boundaries. This For example: -```groovy +```nextflow files = channel.fromPath( 'data/**.fa' ) moreFiles = channel.fromPath( 'data/**/*.fa' ) pairFiles = channel.fromPath( 'data/file_{1,2}.fq' ) @@ -131,13 +131,13 @@ As in Linux Bash, the `*` wildcard does not catch hidden files (i.e. files whose Multiple paths or glob patterns can be specified using a list: -```groovy +```nextflow channel.fromPath( ['/some/path/*.fq', '/other/path/*.fastq'] ) ``` In order to include hidden files, you need to start your pattern with a period character or specify the `hidden: true` option. For example: -```groovy +```nextflow expl1 = channel.fromPath( '/path/.*' ) expl2 = channel.fromPath( '/path/.*.fa' ) expl3 = channel.fromPath( '/path/*', hidden: true ) @@ -149,7 +149,7 @@ By default a [glob][glob] pattern only looks for regular file paths that match t You can use the `type` option specifying the value `file`, `dir` or `any` in order to define what kind of paths you want. For example: -```groovy +```nextflow myFileChannel = channel.fromPath( '/path/*b', type: 'dir' ) myFileChannel = channel.fromPath( '/path/a*', type: 'any' ) ``` @@ -186,7 +186,7 @@ Available options: The `channel.fromFilePairs` method creates a channel emitting the file pairs matching a [glob][glob] pattern provided by the user. The matching files are emitted as tuples in which the first element is the grouping key of the matching pair and the second element is the list of files (sorted in lexicographical order). For example: -```groovy +```nextflow channel .fromFilePairs('/my/data/SRR*_{1,2}.fastq') .view() @@ -209,13 +209,13 @@ The glob pattern must contain at least one `*` wildcard character. Multiple glob patterns can be specified using a list: -```groovy +```nextflow channel.fromFilePairs( ['/some/data/SRR*_{1,2}.fastq', '/other/data/QFF*_{1,2}.fastq'] ) ``` Alternatively, it is possible to implement a custom file pair grouping strategy providing a closure which, given the current file as parameter, returns the grouping key. For example: -```groovy +```nextflow channel .fromFilePairs('/some/data/*', size: -1) { file -> file.extension } .view { ext, files -> "Files with the extension $ext are $files" } @@ -253,7 +253,7 @@ Available options: The `channel.fromSRA` method queries the [NCBI SRA](https://www.ncbi.nlm.nih.gov/sra) database and returns a channel emitting the FASTQ files matching the specified criteria i.e project or accession number(s). For example: -```groovy +```nextflow channel .fromSRA('SRP043510') .view() @@ -273,7 +273,7 @@ It returns: Multiple accession IDs can be specified using a list object: -```groovy +```nextflow ids = ['ERR908507', 'ERR908506', 'ERR908505'] channel .fromSRA(ids) @@ -295,7 +295,7 @@ This method uses the NCBI [ESearch](https://www.ncbi.nlm.nih.gov/books/NBK25499/ To access the ESearch API, you must provide your [NCBI API keys](https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities) through one of the following ways: - The `apiKey` option: - ```groovy + ```nextflow channel.fromSRA(ids, apiKey:'0123456789abcdef') ``` @@ -333,7 +333,7 @@ Available retry policy properties: The following code snippet shows an example for using the `Channel.fromSRA` factory method with a custom `retryPolicy`. - ```groovy + ```nextflow channel.fromSRA(ids, retryPolicy: [delay: '250ms', maxAttempts: 5]) ``` @@ -343,7 +343,7 @@ The following code snippet shows an example for using the `Channel.fromSRA` fact The `interval` method emits an incrementing index (starting from zero) at a periodic interval. For example: -```groovy +```nextflow Channel.interval('1s').view() ``` @@ -351,7 +351,7 @@ The above snippet will emit 0, 1, 2, and so on, every second, forever. You can u An optional closure can be used to transform the index. Additionally, returning `Channel.STOP` will close the channel. For example: -```groovy +```nextflow ch = Channel.interval('1s') { i -> i == 10 ? Channel.STOP : i } @@ -367,7 +367,7 @@ ch.view() The `channel.of` method allows you to create a channel that emits the arguments provided to it, for example: -```groovy +```nextflow ch = channel.of( 1, 3, 5, 7 ) ch.view { v -> "value: $v" } ``` @@ -384,7 +384,7 @@ value: 7 Ranges of values are expanded accordingly: -```groovy +```nextflow channel .of(1..23, 'X', 'Y') .view() @@ -424,7 +424,7 @@ You can think of it as a channel that is shared across many different processes A process output can be assigned to a topic using the `topic` option on an output, for example: -```groovy +```nextflow process foo { output: val('foo'), topic: my_topic @@ -439,7 +439,7 @@ process bar { The `channel.topic` method allows referencing the topic channel with the specified name, which can be used as a process input or operator composition as any other Nextflow channel: -```groovy +```nextflow channel.topic('my-topic').view() ``` @@ -458,7 +458,7 @@ See also: {ref}`process-additional-options` for process outputs. The `channel.value` method is used to create a value channel. An optional (not `null`) argument can be specified to bind the channel to a specific value. For example: -```groovy +```nextflow expl1 = channel.value() expl2 = channel.value( 'Hello there' ) expl3 = channel.value( [1,2,3,4,5] ) @@ -477,7 +477,7 @@ The condition on files to watch can be specified by using `*` or `?` wildcard ch For example: -```groovy +```nextflow channel .watchPath( '/path/*.fa' ) .subscribe { fa -> println "Fasta file: $fa" } @@ -492,7 +492,7 @@ argument that specifies what event(s) to watch. The supported events are: You can specify more than one of these events by using a comma separated string as shown below: -```groovy +```nextflow channel .watchPath( '/path/*.fa', 'create,modify' ) .subscribe { fa -> println "File created or modified: $fa" } diff --git a/docs/reference/operator.md b/docs/reference/operator.md index db1aa17dc8..3426e925c2 100644 --- a/docs/reference/operator.md +++ b/docs/reference/operator.md @@ -16,7 +16,7 @@ The `branch` operator forwards each item from a source channel to one of multipl The selection criteria is a {ref}`closure ` that defines, for each output channel, a unique label followed by a boolean expression. When an item is received, it is routed to the first output channel whose expression evaluates to `true`. For example: ```{literalinclude} ../snippets/branch.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/branch.out @@ -30,7 +30,7 @@ The above output may be printed in any order since the two `view` operations are A fallback condition can be specified using `true` as the last branch condition: ```{literalinclude} ../snippets/branch-with-fallback.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/branch-with-fallback.out @@ -40,7 +40,7 @@ A fallback condition can be specified using `true` as the last branch condition: The value emitted to each branch can be customized with an expression statement (or statements) after the branch condition: ```{literalinclude} ../snippets/branch-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/branch-with-mapper.out @@ -54,7 +54,7 @@ When the `return` keyword is omitted, the value of the last expression statement The `branchCriteria()` method can be used to create a branch criteria as a variable that can be passed as an argument to any number of `branch` operations, as shown below: ```{literalinclude} ../snippets/branch-criteria.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/branch-criteria.out @@ -74,7 +74,7 @@ This operator has multiple variants: : Emits each subset when `closingCondition` is satisfied. The closing condition can be a literal value, a {ref}`regular expression `, a type qualifier (i.e. Java class), or a boolean predicate. For example: ```{literalinclude} ../snippets/buffer-with-closing.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/buffer-with-closing.out @@ -86,7 +86,7 @@ This operator has multiple variants: : Creates a new subset when `openingCondition` is satisfied and emits the subset when is `closingCondition` is satisfied. The opening and closing conditions can each be a literal value, a {ref}`regular expression `, a type qualifier (i.e. Java class), or a boolean predicate. For example: ```{literalinclude} ../snippets/buffer-with-opening-closing.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/buffer-with-opening-closing.out @@ -98,7 +98,7 @@ This operator has multiple variants: : Emits a new subset for every `n` items. Remaining items are discarded. For example: ```{literalinclude} ../snippets/buffer-with-size.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/buffer-with-size.out @@ -108,7 +108,7 @@ This operator has multiple variants: The `remainder` option can be used to emit any remaining items as a partial subset: ```{literalinclude} ../snippets/buffer-with-size-remainder.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/buffer-with-size-remainder.out @@ -120,7 +120,7 @@ This operator has multiple variants: : Emits a new subset for every `n` items, skipping `m` items before collecting each subset. For example: ```{literalinclude} ../snippets/buffer-with-size-skip.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/buffer-with-size-skip.out @@ -144,7 +144,7 @@ This operator has multiple variants: : Collects items into groups of `size` items: ```{literalinclude} ../snippets/collate.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/collate.out @@ -154,7 +154,7 @@ This operator has multiple variants: By default, any remaining items are emitted as a partial group. You can specify `false` as the second parameter to discard them instead: ```{literalinclude} ../snippets/collate-with-no-remainder.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/collate-with-no-remainder.out @@ -170,7 +170,7 @@ This operator has multiple variants: : Collects items into groups of `size` items using a *sliding window* that moves by `step` items at a time: ```{literalinclude} ../snippets/collate-with-step.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/collate-with-step.out @@ -190,7 +190,7 @@ See also: [buffer](#buffer) The `collect` operator collects all items from a source channel into a list and emits it as a single item: ```{literalinclude} ../snippets/collect.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/collect.out @@ -200,7 +200,7 @@ The `collect` operator collects all items from a source channel into a list and An optional {ref}`closure ` can be used to transform each item before it is collected: ```{literalinclude} ../snippets/collect-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/collect-with-mapper.out @@ -230,7 +230,7 @@ This operator has multiple variants: : Collects the items and saves them to a single file specified by the `name` option: ```{literalinclude} ../snippets/collectfile.nf - :language: groovy + :language: nextflow ``` `collectFile( closure, [options] )` @@ -238,7 +238,7 @@ This operator has multiple variants: : Collects the items into groups and saves each group to a file, using a grouping criteria. The grouping criteria is a {ref}`closure ` that maps each item to a pair, where the first element is the file name for the group and the second element is the content to be appended to that file. For example: ```{literalinclude} ../snippets/collectfile-closure.nf - :language: groovy + :language: nextflow ``` ```{literalinclude} ../snippets/collectfile-closure.out @@ -249,7 +249,7 @@ This operator has multiple variants: The following example shows how to use a closure to collect and sort all sequences in a FASTA file from shortest to longest: -```groovy +```nextflow Channel .fromPath('/data/sequences.fa') .splitFasta( record: [id: true, sequence: true] ) @@ -312,7 +312,7 @@ The `combine` operator produces the combinations (i.e. cross product, "Cartesian For example: ```{literalinclude} ../snippets/combine.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/combine.out @@ -322,7 +322,7 @@ For example: The `by` option can be used to combine items that share a matching key. The value should be the zero-based index of the tuple, or a list of indices. For example: ```{literalinclude} ../snippets/combine-by.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/combine-by.out @@ -354,7 +354,7 @@ In other words, given *N* channels, the items from the *i+1*-th channel are emit For example: ```{literalinclude} ../snippets/concat.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/concat.out @@ -372,7 +372,7 @@ See also: [mix](#mix) The `count` operator computes the total number of items in a source channel and emits it: ```{literalinclude} ../snippets/count.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/count.out @@ -382,7 +382,7 @@ The `count` operator computes the total number of items in a source channel and An optional filter can be provided to select which items to count. The selection criteria can be a literal value, a {ref}`regular expression `, a type qualifier (i.e. Java class), or a boolean predicate. For example: ```{literalinclude} ../snippets/count-with-filter-number.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/count-with-filter-number.out @@ -390,7 +390,7 @@ An optional filter can be provided to select which items to count. The selection ``` ```{literalinclude} ../snippets/count-with-filter-regex.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/count-with-filter-regex.out @@ -398,7 +398,7 @@ An optional filter can be provided to select which items to count. The selection ``` ```{literalinclude} ../snippets/count-with-filter-closure.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/count-with-filter-closure.out @@ -448,7 +448,7 @@ The `cross` operator emits every pairwise combination of two channels for which By default, the key is defined as the first entry in a list or map, or the value itself for any other data type. For example: ```{literalinclude} ../snippets/cross.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/cross.out @@ -458,7 +458,7 @@ By default, the key is defined as the first entry in a list or map, or the value An optional closure can be used to define the matching key for each item: ```{literalinclude} ../snippets/cross-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/cross-with-mapper.out @@ -479,7 +479,7 @@ See also: [combine](#combine) The `distinct` operator forwards a source channel with *consecutively* repeated items removed, such that each emitted item is different from the preceding one: ```{literalinclude} ../snippets/distinct.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/distinct.out @@ -489,7 +489,7 @@ The `distinct` operator forwards a source channel with *consecutively* repeated An optional {ref}`closure ` can be used to transform each value before it is evaluated for distinct-ness: ```{literalinclude} ../snippets/distinct-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/distinct-with-mapper.out @@ -509,7 +509,7 @@ The `dump` operator prints each item in a source channel when the pipeline is ex The `tag` option can be used to select which channels to dump: ```{literalinclude} ../snippets/dump.nf -:language: groovy +:language: nextflow ``` Then, you can run your pipeline with `-dump-channels foo` or `-dump-channels bar` to dump the content of either channel. Multiple tag names can be specified as a comma-separated list. @@ -535,7 +535,7 @@ The `filter` operator emits the items from a source channel that satisfy a condi The following example filters a channel with a regular expression that only matches strings beginning with `a`: ```{literalinclude} ../snippets/filter-regex.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/filter-regex.out @@ -545,7 +545,7 @@ The following example filters a channel with a regular expression that only matc The following example filters a channel with the `Number` type qualifier so that only numbers are emitted: ```{literalinclude} ../snippets/filter-type.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/filter-type.out @@ -555,7 +555,7 @@ The following example filters a channel with the `Number` type qualifier so that The following example filters a channel using a boolean predicate, which is a {ref}`closure ` that returns a boolean value. In this case, the predicate is used to select only odd numbers: ```{literalinclude} ../snippets/filter-closure.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/filter-closure.out @@ -571,7 +571,7 @@ The following example filters a channel using a boolean predicate, which is a {r The `first` operator emits the first item in a source channel, or the first item that matches a condition. The condition can be a {ref}`regular expression`, a type qualifier (i.e. Java class), or a boolean predicate. For example: ```{literalinclude} ../snippets/first.nf -:language: groovy +:language: nextflow ``` (operator-flatmap)= @@ -585,7 +585,7 @@ The `flatMap` operator applies a *mapping function* to each item from a source c When the mapping function returns a list, each element in the list is emitted separately: ```{literalinclude} ../snippets/flatmap-list.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/flatmap-list.out @@ -595,7 +595,7 @@ When the mapping function returns a list, each element in the list is emitted se When the mapping function returns a map, each key-value pair in the map is emitted separately: ```{literalinclude} ../snippets/flatmap-map.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/flatmap-map.out @@ -611,7 +611,7 @@ When the mapping function returns a map, each key-value pair in the map is emitt The `flatten` operator flattens each item from a source channel that is a list or other collection, such that each element in each collection is emitted separately: ```{literalinclude} ../snippets/flatten.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/flatten.out @@ -635,7 +635,7 @@ To be more precise, the operator transforms a sequence of tuples like *(K, V, W, For example: ```{literalinclude} ../snippets/grouptuple.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/grouptuple.out @@ -645,7 +645,7 @@ For example: By default, the first element of each tuple is used as the grouping key. The `by` option can be used to specify a different index, or list of indices. For example, to group by the second element of each tuple: ```{literalinclude} ../snippets/grouptuple-by.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/grouptuple-by.out @@ -655,7 +655,7 @@ By default, the first element of each tuple is used as the grouping key. The `by By default, if you don't specify a size, the `groupTuple` operator will not emit any groups until *all* inputs have been received. If possible, you should always try to specify the number of expected elements in each group using the `size` option, so that each group can be emitted as soon as it's ready. In cases where the size of each group varies based on the grouping key, you can use the built-in `groupKey()` function, which allows you to define a different expected size for each group: ```{literalinclude} ../snippets/grouptuple-groupkey.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/grouptuple-groupkey.out @@ -691,7 +691,7 @@ Available options: The `ifEmpty` operator emits a source channel, or a default value if the source channel is *empty* (doesn't emit any value): ```{literalinclude} ../snippets/ifempty-1.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/ifempty-1.out @@ -699,7 +699,7 @@ The `ifEmpty` operator emits a source channel, or a default value if the source ``` ```{literalinclude} ../snippets/ifempty-2.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/ifempty-2.out @@ -723,7 +723,7 @@ To be more precise, the operator transforms a sequence of tuples like *(K, V1, V For example: ```{literalinclude} ../snippets/join.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/join.out @@ -735,7 +735,7 @@ By default, the first element of each item is used as the key. The `by` option c By default, unmatched items are discarded. The `remainder` option can be used to emit them at the end: ```{literalinclude} ../snippets/join-with-remainder.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/join-with-remainder.out @@ -767,7 +767,7 @@ See also: [combine](#combine), [cross](#cross) The `last` operator emits the last item from a source channel: ```{literalinclude} ../snippets/last.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/last.out @@ -783,7 +783,7 @@ The `last` operator emits the last item from a source channel: The `map` operator applies a *mapping function* to each item from a source channel: ```{literalinclude} ../snippets/map.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/map.out @@ -799,7 +799,7 @@ The `map` operator applies a *mapping function* to each item from a source chann The `max` operator emits the item with the greatest value from a source channel: ```{literalinclude} ../snippets/max.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/max.out @@ -811,7 +811,7 @@ An optional {ref}`closure ` can be used to control how the items The following examples show how to find the longest string in a channel: ```{literalinclude} ../snippets/max-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/max-with-mapper.out @@ -819,7 +819,7 @@ The following examples show how to find the longest string in a channel: ``` ```{literalinclude} ../snippets/max-with-comparator.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/max-with-comparator.out @@ -835,7 +835,7 @@ The following examples show how to find the longest string in a channel: The `merge` operator joins the items from two or more channels into a new channel: ```{literalinclude} ../snippets/merge.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/merge.out @@ -845,7 +845,7 @@ The `merge` operator joins the items from two or more channels into a new channe An optional closure can be used to control how two items are merged: ```{literalinclude} ../snippets/merge-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/merge-with-mapper.out @@ -873,7 +873,7 @@ You should always use a matching key (e.g. sample ID) to merge multiple channels The `min` operator emits the item with the lowest value from a source channel: ```{literalinclude} ../snippets/min.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/min.out @@ -885,7 +885,7 @@ An optional {ref}`closure ` can be used to control how the items The following examples show how to find the shortest string in a channel: ```{literalinclude} ../snippets/min-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/min-with-mapper.out @@ -893,7 +893,7 @@ The following examples show how to find the shortest string in a channel: ``` ```{literalinclude} ../snippets/min-with-comparator.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/min-with-comparator.out @@ -909,7 +909,7 @@ The following examples show how to find the shortest string in a channel: The `mix` operator emits the items from two or more source channels into a single output channel: ```{literalinclude} ../snippets/mix.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/mix.out @@ -945,7 +945,7 @@ The multi-map criteria is a {ref}`closure ` that defines, for ea For example: ```{literalinclude} ../snippets/multimap.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/multimap.out @@ -955,7 +955,7 @@ For example: Multiple labels can share the same mapping expression using the following shorthand: ```{literalinclude} ../snippets/multimap-shared.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/multimap-shared.out @@ -967,7 +967,7 @@ The above example creates two channels as before, but now they both receive the You can use the `multiMapCriteria()` method to create a multi-map criteria as a variable that can be passed as an argument to any number of `multiMap` operations, as shown below: ```{literalinclude} ../snippets/multimap-criteria.nf -:language: groovy +:language: nextflow ``` :::{note} @@ -983,7 +983,7 @@ If you use `multiMap` to split a tuple or map into multiple channels, it is reco The `randomSample` operator emits a randomly-selected subset of items from a source channel: ```{literalinclude} ../snippets/random-sample.nf -:language: groovy +:language: nextflow ``` The above snippet will print 10 randomly-selected numbers between 1 and 100 (without replacement). @@ -991,7 +991,7 @@ The above snippet will print 10 randomly-selected numbers between 1 and 100 (wit An optional second parameter can be used to set the initial *seed* for the random number generator, which ensures that the `randomSample` operator produces the same pseudo-random sequence across runs: ```{literalinclude} ../snippets/random-sample-with-seed.nf -:language: groovy +:language: nextflow ``` The above example will print 10 randomly-selected numbers between 1 and 100 (without replacement). Each subsequent script execution will produce the same sequence. @@ -1007,7 +1007,7 @@ The `reduce` operator applies an *accumulator function* sequentially to each ite For example: ```{literalinclude} ../snippets/reduce.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/reduce.out @@ -1017,7 +1017,7 @@ For example: By default, the first item is used as the initial accumulated value. You can optionally specify a different initial value as shown below: ```{literalinclude} ../snippets/reduce-with-initial-value.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/reduce-with-initial-value.out @@ -1032,13 +1032,13 @@ By default, the first item is used as the initial accumulated value. You can opt The `set` operator assigns a source channel to a variable, whose name is specified as a closure parameter: -```groovy +```nextflow Channel.of(10, 20, 30).set { my_channel } ``` Using `set` is semantically equivalent to assigning a variable: -```groovy +```nextflow my_channel = Channel.of(10, 20, 30) ``` @@ -1055,7 +1055,7 @@ The `splitCsv` operator parses and splits [CSV-formatted](http://en.wikipedia.or For example: ```{literalinclude} ../snippets/splitcsv.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/splitcsv.out @@ -1067,7 +1067,7 @@ The above example shows hows CSV text is parsed and split into individual rows, When the CSV begins with a header line defining the column names, and the `header` option is `true`, each row is returned as a map instead: ```{literalinclude} ../snippets/splitcsv-with-header.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/splitcsv-with-header.out @@ -1077,7 +1077,7 @@ When the CSV begins with a header line defining the column names, and the `heade The `header` option can also just be a list of columns: ```{literalinclude} ../snippets/splitcsv-with-columns.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/splitcsv-with-columns.out @@ -1126,7 +1126,7 @@ The `splitFasta` operator splits [FASTA-formatted](http://en.wikipedia.org/wiki/ The `by` option can be used to group sequences into chunks of a given size. The following example shows how to read a FASTA file and split it into chunks of 10 sequences each: -```groovy +```nextflow Channel .fromPath('misc/sample.fa') .splitFasta( by: 10 ) @@ -1139,7 +1139,7 @@ Chunks are stored in memory by default. When splitting large files, specify `fil The `record` option can be used to split FASTA content into *records* instead of text chunks. Each record is a map that allows you to access the FASTA sequence data with ease. For example: -```groovy +```nextflow Channel .fromPath('misc/sample.fa') .splitFasta( record: [id: true, seqString: true] ) @@ -1198,7 +1198,7 @@ The `splitFasta` operator splits [FASTQ formatted](http://en.wikipedia.org/wiki/ The `by` option can be used to group sequences into chunks of a given size. The following example shows how to read a FASTQ file and split it into chunks of 10 sequences each: -```groovy +```nextflow Channel .fromPath('misc/sample.fastq') .splitFastq( by: 10 ) @@ -1211,7 +1211,7 @@ Chunks are stored in memory by default. When splitting large files, specify `fil The `record` option can be used to split FASTQ content into *records* instead of text chunks. Each record is a map that allows you to access the FASTQ sequence data with ease. For example: -```groovy +```nextflow Channel .fromPath('misc/sample.fastq') .splitFastq( record: true ) @@ -1220,7 +1220,7 @@ Channel The `pe` option can be used to split paired-end FASTQ files. The source channel must emit tuples containing the file pairs. For example: -```groovy +```nextflow Channel .fromFilePairs('/my/data/SRR*_{1,2}.fastq', flat: true) .splitFastq(by: 100_000, pe: true, file: true) @@ -1282,7 +1282,7 @@ The `splitJson` operator splits [JSON formatted](https://en.wikipedia.org/wiki/J If the source item is a JSON array, each element of the array will be emitted: ```{literalinclude} ../snippets/splitjson-array.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/splitjson-array.out @@ -1292,7 +1292,7 @@ If the source item is a JSON array, each element of the array will be emitted: If the source item is a JSON object, each key-value pair will be emitted as a map with the properties `key` and `value`: ```{literalinclude} ../snippets/splitjson-object.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/splitjson-object.out @@ -1302,7 +1302,7 @@ If the source item is a JSON object, each key-value pair will be emitted as a ma The `path` option can be used to query a section of the JSON document to parse and split: ```{literalinclude} ../snippets/splitjson-with-path.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/splitjson-with-path.out @@ -1327,7 +1327,7 @@ See also: [countJson](#countjson) The `splitText` operator splits multi-line text content from a source channel into chunks of *N* lines: -```groovy +```nextflow Channel .fromPath('/some/path/*.txt') .splitText() @@ -1338,7 +1338,7 @@ The above example loads a collection of text files, splits the content of each f The `by` option can be used to emit chunks of *N* lines: -```groovy +```nextflow Channel .fromPath('/some/path/*.txt') .splitText( by: 10 ) @@ -1350,7 +1350,7 @@ Channel An optional {ref}`closure ` can be used to transform each text chunk produced by the operator. The following example shows how to split text files into chunks of 10 lines and transform them to uppercase letters: -```groovy +```nextflow Channel .fromPath('/some/path/*.txt') .splitText( by: 10 ) { v -> v.toUpperCase() } @@ -1398,7 +1398,7 @@ See also: [countLines](#countlines) The `subscribe` operator invokes a custom function for each item from a source channel: ```{literalinclude} ../snippets/subscribe.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/subscribe.out @@ -1408,7 +1408,7 @@ The `subscribe` operator invokes a custom function for each item from a source c The `subscribe` operator supports multiple types of event handlers: ```{literalinclude} ../snippets/subscribe-with-on-complete.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/subscribe-with-on-complete.out @@ -1439,7 +1439,7 @@ Available options: The `sum` operator emits the sum of all items in a source channel: ```{literalinclude} ../snippets/sum.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/sum.out @@ -1449,7 +1449,7 @@ The `sum` operator emits the sum of all items in a source channel: An optional {ref}`closure ` can be used to transform each item before it is added to the sum: ```{literalinclude} ../snippets/sum-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/sum-with-mapper.out @@ -1465,7 +1465,7 @@ An optional {ref}`closure ` can be used to transform each item b The `take` operator takes the first *N* items from a source channel: ```{literalinclude} ../snippets/take.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/take.out @@ -1485,7 +1485,7 @@ See also: [until](#until) The `tap` operator assigns a source channel to a variable, and emits the source channel. It is a useful way to extract intermediate output channels from a chain of operators. For example: ```{literalinclude} ../snippets/tap.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/tap.out @@ -1501,7 +1501,7 @@ See also: [set](#set) The `toInteger` operator converts string values from a source channel to integer values: ```{literalinclude} ../snippets/tointeger.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/tointeger.out @@ -1511,7 +1511,7 @@ The `toInteger` operator converts string values from a source channel to integer :::{note} `toInteger` is equivalent to: -```groovy +```nextflow map { v -> v as Integer } ``` ::: @@ -1527,7 +1527,7 @@ You can also use `toLong`, `toFloat`, and `toDouble` to convert to other numeric The `toList` operator collects all the items from a source channel into a list and emits the list as a single item: ```{literalinclude} ../snippets/tolist.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/tolist.out @@ -1542,7 +1542,7 @@ There are two main differences between `toList` and `collect`: In other words, `toList` is equivalent to: -```groovy +```nextflow collect(flat: false).ifEmpty([]) ``` ::: @@ -1556,7 +1556,7 @@ See also: [collect](#collect) The `toSortedList` operator collects all the items from a source channel into a sorted list and emits the list as a single item: ```{literalinclude} ../snippets/tosortedlist.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/tosortedlist.out @@ -1566,7 +1566,7 @@ The `toSortedList` operator collects all the items from a source channel into a An optional closure can be used to control how items are compared when sorting. For example, to sort tuples by their second element in descending order: ```{literalinclude} ../snippets/tosortedlist-with-comparator.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/tosortedlist-with-comparator.out @@ -1576,7 +1576,7 @@ An optional closure can be used to control how items are compared when sorting. :::{note} `toSortedList` is equivalent to: -```groovy +```nextflow collect(flat: false, sort: true).ifEmpty([]) ``` ::: @@ -1594,7 +1594,7 @@ To be more precise, the operator transforms a sequence of tuples like *(K, list( For example: ```{literalinclude} ../snippets/transpose-1.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/transpose-1.out @@ -1604,7 +1604,7 @@ For example: If each source item has more than two elements, these will be flattened by the first element in the item, and a new item will be emitted only when it is complete: ```{literalinclude} ../snippets/transpose-2.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/transpose-2.out @@ -1614,7 +1614,7 @@ If each source item has more than two elements, these will be flattened by the f The `remainder` option can be used to emit any incomplete items: ```{literalinclude} ../snippets/transpose-2-with-remainder.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/transpose-2-with-remainder.out @@ -1640,7 +1640,7 @@ See also: [groupTuple](#grouptuple) The `unique` operator emits the unique items from a source channel: ```{literalinclude} ../snippets/unique.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/unique.out @@ -1650,7 +1650,7 @@ The `unique` operator emits the unique items from a source channel: An optional {ref}`closure ` can be used to transform each item before it is evaluated for uniqueness: ```{literalinclude} ../snippets/unique-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/unique-with-mapper.out @@ -1672,7 +1672,7 @@ See also: [distinct](#distinct) The `until` operator emits each item from a source channel until a stopping condition is satisfied: ```{literalinclude} ../snippets/until.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/until.out @@ -1690,7 +1690,7 @@ See also: [take](#take) The `view` operator prints each item from a source channel to standard output: ```{literalinclude} ../snippets/view.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/view.out @@ -1700,7 +1700,7 @@ The `view` operator prints each item from a source channel to standard output: An optional closure can be used to transform each item before it is printed: ```{literalinclude} ../snippets/view-with-mapper.nf -:language: groovy +:language: nextflow ``` ```{literalinclude} ../snippets/view-with-mapper.out diff --git a/docs/reference/process.md b/docs/reference/process.md index d64c5f0c59..646bebbdf4 100644 --- a/docs/reference/process.md +++ b/docs/reference/process.md @@ -215,7 +215,7 @@ The following options are available for all process outputs: The `accelerator` directive allows you to request hardware accelerators (e.g. GPUs) for the task execution. For example: -```groovy +```nextflow process foo { accelerator 4, type: 'nvidia-tesla-k80' @@ -257,7 +257,7 @@ When combined with the {ref}`container directive `, the `afte The `arch` directive allows you to define the CPU architecture to build the software in use by the process' task. For example: -```groovy +```nextflow process cpu_task { spack 'blast-plus@2.13.0' arch 'linux/x86_64', target: 'cascadelake' @@ -299,7 +299,7 @@ A job array is a collection of jobs with the same resource requirements and the The directive should be specified with a given array size, along with an executor that supports job arrays. For example: -```groovy +```nextflow process cpu_task { executor 'slurm' array 100 @@ -350,7 +350,7 @@ The `beforeScript` directive allows you to execute a custom (Bash) snippet *befo For example: -```groovy +```nextflow process foo { beforeScript 'source /cluster/bin/setup' @@ -370,7 +370,7 @@ The `cache` directive allows you to store the process results to a local cache. The cache is enabled by default, but you can disable it for a specific process by setting the `cache` directive to `false`. For example: -```groovy +```nextflow process noCacheThis { cache false @@ -402,7 +402,7 @@ The `clusterOptions` directive allows the usage of any native configuration opti The cluster options can be a string: -```groovy +```nextflow process foo { clusterOptions '-x 1 -y 2' // ... @@ -415,7 +415,7 @@ Prior to this version, grid executors that require each option to be on a separa The cluster options can also be a string list: -```groovy +```nextflow process foo { clusterOptions '-x 1', '-y 2', '--flag' // ... @@ -440,7 +440,7 @@ The `conda` directive allows for the definition of the process dependencies usin Nextflow automatically sets up an environment for the given package names listed by in the `conda` directive. For example: -```groovy +```nextflow process foo { conda 'bwa=0.7.15' @@ -464,7 +464,7 @@ It requires the Docker daemon to be running in machine where the pipeline is exe For example: -```groovy +```nextflow process runThisInDocker { container 'dockerbox:tag' @@ -490,7 +490,7 @@ This directive is ignored for processes that are {ref}`executed natively - """ + """ + blastp -query + """ } ``` @@ -978,7 +977,7 @@ You can repeat the `module` directive for each module you need to load. Alternat The `penv` directive allows you to define the parallel environment to be used when submitting a parallel task to the {ref}`SGE ` resource manager. For example: -```groovy +```nextflow process big_job { cpus 4 penv 'smp' @@ -1002,7 +1001,7 @@ The `pod` directive allows the definition of pod specific settings, such as envi For example: -```groovy +```nextflow process your_task { pod env: 'FOO', value: 'bar' @@ -1196,7 +1195,7 @@ The following options are available: The `publishDir` directive allows you to publish the process output files to a specified folder. For example: -```groovy +```nextflow process foo { publishDir '/data/chunks' @@ -1221,7 +1220,7 @@ The `publishDir` directive can be specified more than once in order to publish o By default files are published to the target folder creating a *symbolic link* for each process output that links the file produced into the process working directory. This behavior can be modified using the `mode` option, for example: -```groovy +```nextflow process foo { publishDir '/data/chunks', mode: 'copy', overwrite: false @@ -1295,7 +1294,7 @@ Available options: The `queue` directive allows you to set the `queue` where jobs are scheduled when using a grid based executor in your pipeline. For example: -```groovy +```nextflow process grid_job { queue 'long' executor 'sge' @@ -1309,7 +1308,7 @@ process grid_job { :::{tip} Some grid executors support multiple queue names as a comma-separated list: -```groovy +```nextflow queue 'short,long,cn-el6' ``` @@ -1329,7 +1328,7 @@ This directive is only used by certain executors. Refer to the {ref}`executor-pa The `resourceLabels` directive allows you to specify custom name-value pairs that Nextflow applies to the computing resource used to carry out the process execution. Resource labels can be specified using the syntax shown below: -```groovy +```nextflow process my_task { resourceLabels region: 'some-region', user: 'some-username' @@ -1366,7 +1365,7 @@ See also: [label](#label) The `resourceLimits` directive allows you to specify environment-specific limits for task resource requests. Resource limits can be specified in a process as follows: -```groovy +```nextflow process my_task { resourceLimits cpus: 24, memory: 768.GB, time: 72.h @@ -1379,7 +1378,7 @@ process my_task { Or in the Nextflow configuration: -```groovy +```nextflow process { resourceLimits = [ cpus: 24, memory: 768.GB, time: 72.h ] } @@ -1404,7 +1403,7 @@ This is useful when your pipeline is launched by using a grid executor, because In its basic form simply specify `true` at the directive value, as shown below: -```groovy +```nextflow process simpleTask { scratch true @@ -1446,7 +1445,7 @@ The following values are supported: The `shell` directive allows you to define a custom shell command for process scripts. By default, script blocks are executed with `/bin/bash -ue`. -```groovy +```nextflow process doMoreThings { shell '/bin/bash', '-euo', 'pipefail' @@ -1470,7 +1469,7 @@ The `spack` directive allows for the definition of the process dependencies usin Nextflow automatically sets up an environment for the given package names listed by in the `spack` directive. For example: -```groovy +```nextflow process foo { spack 'bwa@0.7.15' @@ -1542,7 +1541,7 @@ In more detail, it affects the process execution in two main ways: The following example shows how to use the `storeDir` directive to create a directory containing a BLAST database for each species specified by an input parameter: -```groovy +```nextflow process formatBlastDatabases { storeDir '/db/genomes' @@ -1574,7 +1573,7 @@ The use of AWS S3 paths is supported, however it requires the installation of th The `tag` directive allows you to associate each process execution with a custom label, so that it will be easier to identify them in the log file or in the trace execution report. For example: -```groovy +```nextflow process foo { tag "$code" @@ -1607,7 +1606,7 @@ See also {ref}`Trace execution report ` The `time` directive allows you to define how long a process is allowed to run. For example: -```groovy +```nextflow process big_job { time '1h' diff --git a/docs/reference/stdlib.md b/docs/reference/stdlib.md index af5acd262e..8c09f1ac96 100644 --- a/docs/reference/stdlib.md +++ b/docs/reference/stdlib.md @@ -46,7 +46,7 @@ The following constants are globally available in a Nextflow script: For example: - ```groovy + ```nextflow if( !nextflow.version.matches('>=23.10') ) { error "This workflow requires Nextflow version 23.10 or greater -- You are running version $nextflow.version" } @@ -293,7 +293,7 @@ You can create a duration by adding a time unit suffix to an integer, e.g. `1.h` You can also create a duration with `Duration.of()`: -```groovy +```nextflow // integer value (milliseconds) oneSecond = Duration.of(1000) @@ -306,7 +306,7 @@ complexDuration = Duration.of('1day 6hours 3minutes 30seconds') Durations can be compared like numbers, and they support basic arithmetic operations: -```groovy +```nextflow a = 1.h b = 2.h @@ -359,7 +359,7 @@ Technically speaking, a kilobyte is equal to 1000 bytes, whereas 1024 bytes is c You can also create a memory unit with `MemoryUnit.of()`: -```groovy +```nextflow // integer value (bytes) oneKilobyte = MemoryUnit.of(1024) @@ -369,7 +369,7 @@ oneGigabyte = MemoryUnit.of('1 GB') Memory units can be compared like numbers, and they support basic arithmetic operations: -```groovy +```nextflow a = 1.GB b = 2.GB @@ -451,7 +451,7 @@ The following methods are useful for getting attributes of a file: `toUriString()` : Gets the file path along with the protocol scheme: - ```groovy + ```nextflow def ref = file('s3://some-bucket/foo.txt') assert ref.toString() == '/some-bucket/foo.txt' @@ -533,19 +533,19 @@ The following methods are available for manipulating files and directories in a : *When copying a file to another file:* if the target file already exists, it will be replaced. - ```groovy + ```nextflow file('/some/path/my_file.txt').copyTo('/another/path/new_file.txt') ``` : *When copying a file to a directory:* the file will be copied into the directory, replacing any file with the same name. - ```groovy + ```nextflow file('/some/path/my_file.txt').copyTo('/another/path') ``` : *When copying a directory to another directory:* if the target directory already exists, the source directory will be copied into the target directory, replacing any sub-directory with the same name. If the target path does not exist, it will be created automatically. - ```groovy + ```nextflow file('/any/dir_a').moveTo('/any/dir_b') ``` @@ -558,7 +558,7 @@ The following methods are available for manipulating files and directories in a `delete()` : Deletes the file or directory at the given path, returning `true` if the operation succeeds, and `false` otherwise: - ```groovy + ```nextflow myFile = file('some/file.txt') result = myFile.delete() println result ? "OK" : "Cannot delete: $myFile" @@ -569,7 +569,7 @@ The following methods are available for manipulating files and directories in a `deleteDir()` : Deletes a directory and all of its contents. - ```groovy + ```nextflow file('any/path').deleteDir() ``` @@ -585,7 +585,7 @@ The following methods are available for manipulating files and directories in a `mkdir()` : Creates a directory at the given path, returning `true` if the directory is created successfully, and `false` otherwise: - ```groovy + ```nextflow myDir = file('any/path') result = myDir.mkdir() println result ? "OK" : "Cannot create directory: $myDir" @@ -596,14 +596,14 @@ The following methods are available for manipulating files and directories in a `mkdirs()` : Creates a directory at the given path, including any nonexistent parent directories: - ```groovy + ```nextflow file('any/path').mkdirs() ``` `mklink( linkName, [options] )` : Creates a *filesystem link* to a given path: - ```groovy + ```nextflow myFile = file('/some/path/file.txt') myFile.mklink('/user/name/link-to-file.txt') ``` @@ -622,21 +622,21 @@ The following methods are available for manipulating files and directories in a `renameTo( target )` : Rename a file or directory: - ```groovy + ```nextflow file('my_file.txt').renameTo('new_file_name.txt') ``` `setPermissions( permissions )` : Sets a file's permissions using the [symbolic notation](http://en.wikipedia.org/wiki/File_system_permissions#Symbolic_notation): - ```groovy + ```nextflow myFile.setPermissions('rwxr-xr-x') ``` `setPermissions( owner, group, other )` : Sets a file's permissions using the [numeric notation](http://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation), i.e. as three digits representing the **owner**, **group**, and **other** permissions: - ```groovy + ```nextflow myFile.setPermissions(7,5,5) ``` diff --git a/docs/reference/syntax.md b/docs/reference/syntax.md index 1d136f60ec..bb7cd098dd 100644 --- a/docs/reference/syntax.md +++ b/docs/reference/syntax.md @@ -8,13 +8,13 @@ This page provides a comprehensive description of the Nextflow language. A line comment starts with `//` and includes the rest of the line. -```groovy +```nextflow println 'Hello world!' // line comment ``` A block comment starts with `/*` and includes all subsequent characters up to the first `*/`. -```groovy +```nextflow /* * block comment */ @@ -39,13 +39,13 @@ Script declarations are in turn composed of statements and expressions. If there are no top-level declarations, a script may contain one or more [statements](#statements), in which case the entire script is treated as an entry workflow. For example: -```groovy +```nextflow println 'Hello world!' ``` Is equivalent to: -```groovy +```nextflow workflow { println 'Hello world!' } @@ -67,7 +67,7 @@ The first line of a script can be a [shebang](https://en.wikipedia.org/wiki/Sheb A feature flag declaration is an assignment. The target should be a valid {ref}`feature flag ` and the source should be a literal (i.e. number, string, boolean): -```groovy +```nextflow nextflow.preview.topic = true ``` @@ -75,7 +75,7 @@ nextflow.preview.topic = true An include declaration consists of an *include source* and one or more *include clauses*: -```groovy +```nextflow include { foo as bar } from './some/module' ``` @@ -83,7 +83,7 @@ The include source should be a string literal and should refer to either a local Include clauses can be separated by semi-colons or newlines: -```groovy +```nextflow // semi-colons include { foo ; bar as baz } from './some/module' @@ -96,7 +96,7 @@ include { Include clauses can also be specified as separate includes: -```groovy +```nextflow include { foo } from './some/module' include { bar as baz } from './some/module' ``` @@ -111,7 +111,7 @@ The following definitions can be included: A parameter declaration is an assignment. The target should be a pipeline parameter and the source should be an expression: -```groovy +```nextflow params.message = 'Hello world!' ``` @@ -125,7 +125,7 @@ A workflow can be a *named workflow* or an *entry workflow*. A *named workflow* consists of a name and a body, and may consist of a *take*, *main*, *emit*, and *publish* section: -```groovy +```nextflow workflow greet { take: greetings @@ -151,7 +151,7 @@ workflow greet { An *entry workflow* has no name and may consist of a *main* and *publish* section: -```groovy +```nextflow workflow { main: greetings = Channel.of('Bonjour', 'Ciao', 'Hello', 'Hola') @@ -179,7 +179,7 @@ Entry workflow definitions are ignored when a script is included as a module. Th A process consists of a name and a body. The process body consists of one or more [statements](#statements). A minimal process definition must return a string: -```groovy +```nextflow process sayHello { """ echo 'Hello world!' @@ -189,7 +189,7 @@ process sayHello { A process may define additional sections for *directives*, *inputs*, *outputs*, *script*, *shell*, *exec*, and *stub*: -```groovy +```nextflow process greet { // directives errorStrategy 'retry' @@ -224,7 +224,7 @@ Each section may contain one or more statements. For directives, inputs, and out The script section can be substituted with a shell or exec section: -```groovy +```nextflow process greetShell { input: val greeting @@ -258,7 +258,7 @@ See {ref}`process-page` for more information on the semantics of each process se A function consists of a name, parameter list, and a body: -```groovy +```nextflow def greet(greeting, name) { println "${greeting}, ${name}!" } @@ -268,7 +268,7 @@ The function body consists of one or more [statements](#statements). The last st The [return statement](#return) can be used to explicitly return from a function: -```groovy +```nextflow // return with no value def greet(greeting, name) { if( !greeting || !name ) @@ -288,7 +288,7 @@ def fib(x) { An enum type declaration consists of a name and a body. The body consists of a comma-separated list of identifiers: -```groovy +```nextflow enum Day { MONDAY, TUESDAY, @@ -310,7 +310,7 @@ Enum types cannot be included across modules at this time. The output block consists of one or more *target blocks*. A target block consists of a *target name* and one or more *target directives* for configuring the corresponding publish target: -```groovy +```nextflow output { 'fastq' { path 'samples' @@ -327,7 +327,7 @@ Only one output block may be defined in a script. See {ref}`workflow-output-def` Statements can be separated by either a newline or a semi-colon: -```groovy +```nextflow // newline println 'Hello!' println 'Hello again!' @@ -340,13 +340,13 @@ println 'Hello!' ; println 'Hello again!' Variables can be declared with the `def` keyword: -```groovy +```nextflow def x = 42 ``` Multiple variables can be declared in a single statement if the initializer is a [list literal](#list) with the same number of elements and declared variables: -```groovy +```nextflow def (x, y) = [ 1, 2 ] ``` @@ -360,7 +360,7 @@ Process input variables exist for the entire process body. Variables declared in Variables declared in an if or else branch exist only within that branch: -```groovy +```nextflow if( true ) def x = 'foo' println x // error: `x` is undefined @@ -374,7 +374,7 @@ println x A variable cannot be declared with the same name as another variable in the same scope or an enclosing scope: -```groovy +```nextflow def clash(x) { def x // error: `x` is already declared if( true ) @@ -386,7 +386,7 @@ def clash(x) { An assignment statement consists of a *target* expression and a *source* expression separated by an equals sign: -```groovy +```nextflow v = 42 list[0] = 'first' map.key = 'value' @@ -396,7 +396,7 @@ The target expression must be a [variable](#variable), [index](#binary-expressio Multiple variables can be assigned in a single statement as long as the source expression is a [list literal](#list) with the same number of elements and assigned variables: -```groovy +```nextflow (x, y) = [ 1, 2 ] ``` @@ -410,7 +410,7 @@ In general, the only expressions that can have any effect as expression statemen An assert statement consists of the `assert` keyword followed by a boolean expression, with an optional error message separated by a colon: -```groovy +```nextflow assert 2 + 2 == 4 : 'The math broke!' ``` @@ -420,7 +420,7 @@ If the condition is false, an error will be raised with the given error message. An if/else statement consists of an *if branch* and an optional *else branch*. Each branch consists of a boolean expression in parentheses, followed by either a single statement or a *block statement* (one or more statements in curly braces). For example: -```groovy +```nextflow def x = Math.random() if( x < 0.5 ) { println 'You lost.' @@ -434,7 +434,7 @@ If the condition is true, the if branch will be executed, otherwise the else bra If/else statements can be chained any number of times by making the else branch another if/else statement: -```groovy +```nextflow def grade = 89 if( grade >= 90 ) println 'You get an A!' @@ -450,7 +450,7 @@ else A more verbose way to write the same code is: -```groovy +```nextflow def grade = 89 if( grade >= 90 ) { println 'You get an A!' @@ -479,7 +479,7 @@ else { A return statement consists of the `return` keyword with an optional expression: -```groovy +```nextflow def add(a, b) { return a + b } @@ -495,7 +495,7 @@ Return statements can only be used in functions and closures. In the case of a n If a function or closure has multiple return statements (including implicit returns), all of the return statements should either return a value or return nothing. If a function or closure does return a value, it should do so for every conditional branch. -```groovy +```nextflow def isEven1(n) { if( n % 2 == 1 ) return // error: return value is required here @@ -517,13 +517,13 @@ If the last statement is not a return or expression statement (implicit return), A throw statement consists of the `throw` keyword followed by an expression that returns an error type: -```groovy +```nextflow throw new Exception('something failed!') ``` :::{note} In general, the appropriate way to raise an error is to use the {ref}`error ` function: -```groovy +```nextflow error 'something failed!' ``` ::: @@ -532,7 +532,7 @@ error 'something failed!' A try/catch statement consists of a *try block* followed by any number of *catch clauses*: -```groovy +```nextflow def text = null try { text = file('foo.txt').text @@ -554,7 +554,7 @@ Every expression has a *type*, which may be resolved at compile-time or run-time A variable expression is a reference to a variable or other named value: -```groovy +```nextflow def x = 42 x @@ -565,7 +565,7 @@ x A number literal can be an integer or floating-point number, and can be positive or negative. Integers can specified in binary with `0b`, octal with `0`, or hexadecimal with `0x`. Floating-point numbers can use scientific notation with the `e` or `E` prefix. Underscores can be used as thousands separators to make long numbers more readable. -```groovy +```nextflow // integer 42 -1 @@ -584,7 +584,7 @@ A number literal can be an integer or floating-point number, and can be positive A boolean literal can be `true` or `false`: -```groovy +```nextflow assert true != false assert !true == false assert true == !false @@ -594,7 +594,7 @@ assert true == !false The null literal is specified as `null`. It can be used to represent an "empty" value: -```groovy +```nextflow def x = null x = 42 ``` @@ -607,14 +607,14 @@ Using a null value in certain expressions (e.g. the object of a property express A string literal consists of arbitrary text enclosed by single or double quotes: -```groovy +```nextflow println "I said 'hello'" println 'I said "hello" again!' ``` A triple-quoted string can span multiple lines: -```groovy +```nextflow println ''' Hello, How are you today? @@ -628,13 +628,13 @@ println """ A *slashy string* is enclosed by slashes instead of quotes: -```groovy +```nextflow /no escape!/ ``` Slashy strings can also span multiple lines: -```groovy +```nextflow / Patterns in the code, Symbols dance to match and find, @@ -650,7 +650,7 @@ A slashy string cannot be empty because it would become a line comment. Double-quoted strings can be interpolated using the `${}` placeholder with an expression: -```groovy +```nextflow def names = ['Thing 1', 'Thing 2'] println "Hello, ${names.join(' and ')}!" // -> Hello, Thing 1 and Thing 2! @@ -658,7 +658,7 @@ println "Hello, ${names.join(' and ')}!" If the expression is a name or simple property expression (one or more identifiers separated by dots), the curly braces can be omitted: -```groovy +```nextflow def name = [first: '', last: ''] println "Hello, ${name.first} ${name.last}!" // -> Hello, ! @@ -666,7 +666,7 @@ println "Hello, ${name.first} ${name.last}!" Multi-line double-quoted strings can also be interpolated: -```groovy +```nextflow """ blastp \ -in $input \ @@ -678,7 +678,7 @@ blastp \ Single-quoted strings are not interpolated: -```groovy +```nextflow println 'Hello, ${names.join(" and ")}!' // -> Hello, ${names.join(" and ")}! ``` @@ -687,7 +687,7 @@ println 'Hello, ${names.join(" and ")}!' A list literal consists of a comma-separated list of zero or more expressions, enclosed in square brackets: -```groovy +```nextflow [1, 2, 3] ``` @@ -695,19 +695,19 @@ A list literal consists of a comma-separated list of zero or more expressions, e A map literal consists of a comma-separated list of one or more *map entries*, enclosed in square brackets. Each map entry consists of a *key expression* and *value expression* separated by a colon: -```groovy +```nextflow [foo: 1, bar: 2, baz: 3] ``` An empty map is specified with a single colon to distinguish it from an empty list: -```groovy +```nextflow [:] ``` Both the key and value can be any expression. Identifier keys are treated as string literals (i.e. the quotes can be omitted). A variable can be used as a key by enclosing it in parentheses: -```groovy +```nextflow def x = 'foo' [(x): 1] // -> ['foo': 1] @@ -717,7 +717,7 @@ def x = 'foo' A closure, also known as an anonymous function, consists of a parameter list followed by zero or more statements, enclosed in curly braces: -```groovy +```nextflow { a, b -> a + b } ``` @@ -725,7 +725,7 @@ The above closure takes two arguments and returns their sum. The closure body is identical to that of a [function](#function). Statements should be separated by newlines or semi-colons, and the last statement is implicitly treated as a [return statement](#return): -```groovy +```nextflow { v -> println 'Hello!' println "We're in a closure!" @@ -736,7 +736,7 @@ The closure body is identical to that of a [function](#function). Statements sho Closures can access variables outside of their scope: -```groovy +```nextflow def factor = 2 println [1, 2, 3].collect { v -> factor * v } // -> [2, 4, 6] @@ -744,7 +744,7 @@ println [1, 2, 3].collect { v -> factor * v } Closures can declare local variables that exist only for the lifetime of each closure invocation: -```groovy +```nextflow def result = 0 [1, 2, 3].each { v -> def squared = v * v @@ -761,7 +761,7 @@ See {ref}`standard library ` and {ref}`operator ` fo An index expression consists of a *left expression* and a *right expression*, with the right expression enclosed in square brackets: -```groovy +```nextflow myList[0] ``` @@ -769,7 +769,7 @@ myList[0] A property expression consists of an *object expression* and a *property*, separated by a dot: -```groovy +```nextflow file.text ``` @@ -779,25 +779,25 @@ The property must be an identifier or string literal. A function call consists of a name and argument list: -```groovy +```nextflow printf('Hello %s!\n', 'World') ``` A *method call* consists of an *object expression* and a function call separated by a dot: -```groovy +```nextflow myList.size() ``` The argument list may contain any number of *positional arguments* and *named arguments*: -```groovy +```nextflow file('hello.txt', checkIfExists: true) ``` The named arguments are collected into a map and provided as the first positional argument to the function. The above function call can be rewritten as: -```groovy +```nextflow file([checkIfExists: true], 'hello.txt') ``` @@ -805,7 +805,7 @@ The argument name must be an identifier or string literal. The parentheses can be omitted when the function call is also an [expression statement](#expression-statement) and there is at least one argument: -```groovy +```nextflow // positional args printf 'Hello %s!\n', 'World' @@ -815,7 +815,7 @@ file 'hello.txt', checkIfExists: true If the last argument is a closure, it can be specified outside of the parentheses: -```groovy +```nextflow // closure arg with additional args [1, 2, 3].inject('result:') { acc, v -> acc + ' ' + v } @@ -830,13 +830,13 @@ If the last argument is a closure, it can be specified outside of the parenthese A constructor call consists of the `new` keyword followed by a *type name* and an argument list enclosed in parentheses: -```groovy +```nextflow new java.util.Date() ``` If the type is implicitly available in the script, the *fully-qualified type name* can be elided to the *simple type name*: -```groovy +```nextflow new Date() ``` @@ -846,7 +846,7 @@ See {ref}`stdlib-default-imports` for the set of types which are implicitly avai A unary expression consists of a *unary operator* followed by an expression: -```groovy +```nextflow !(2 + 2 == 4) ``` @@ -861,7 +861,7 @@ The following unary operators are available: A binary expression consists of a *left expression* and a *right expression* separated by a *binary operator*: -```groovy +```nextflow 2 + 2 ``` @@ -903,7 +903,7 @@ The following binary operators are available: A ternary expression consists of a *test expression*, a *true expression*, and a *false expression*, separated by a question mark and a colon: -```groovy +```nextflow println x % 2 == 0 ? 'x is even!' : 'x is odd!' ``` @@ -913,7 +913,7 @@ If the test expression is true, the true expression is evaluated, otherwise the Any expression can be enclosed in parentheses: -```groovy +```nextflow 1 + 2 * 3 // -> 1 + 6 -> 7 diff --git a/docs/script.md b/docs/script.md index 8ea70050a6..eec9320ffe 100644 --- a/docs/script.md +++ b/docs/script.md @@ -18,7 +18,7 @@ Nextflow scripts have a maximum size of 64 KiB. To avoid this limit for large pi You can use the `println` function to print to the console: -```groovy +```nextflow println 'Hello, World!' ``` @@ -27,7 +27,7 @@ println 'Hello, World!' Variables are declared using the `def` keyword: -```groovy +```nextflow def num = 1 println num @@ -52,19 +52,19 @@ Variables can also be declared without `def` in some cases. However, this practi Lists are defined using square brackets: -```groovy +```nextflow def myList = [1776, -1, 33, 99, 0, 928734928763] ``` You can access a given item in the list with square-bracket notation (indexes start at 0): -```groovy +```nextflow println myList[0] ``` In order to get the length of the list use the `size` method: -```groovy +```nextflow println myList.size() ``` @@ -74,7 +74,7 @@ Refer to the [Java](https://docs.oracle.com/en/java/javase/11/docs/api/java.base Maps are used to store *associative arrays* (also known as *dictionaries*). They are unordered collections of heterogeneous, named data: -```groovy +```nextflow def scores = ["Brett": 100, "Pete": "Did not finish", "Andrew": 86.87934] ``` @@ -82,21 +82,21 @@ Note that each of the values stored in the map can be of a different type. `Bret We can access the values in a map in two main ways: -```groovy +```nextflow println scores["Pete"] println scores.Pete ``` To add data to or modify a map, the syntax is similar to adding values to list: -```groovy +```nextflow scores["Pete"] = 3 scores["Cedric"] = 120 ``` You can also use the `+` operator to add two maps together: -```groovy +```nextflow def new_scores = scores + ["Pete": 3, "Cedric": 120] ``` @@ -112,7 +112,7 @@ See the [Java](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java One of the most important features of any programming language is the ability to execute different code under different conditions. The simplest way to do this is to use the `if` construct: -```groovy +```nextflow def x = Math.random() if( x < 0.5 ) { println "You lost." @@ -126,14 +126,14 @@ else { Strings can be defined by enclosing text in single or double quotes (`'` or `"` characters): -```groovy +```nextflow println "he said 'cheese' once" println 'he said "cheese!" again' ``` Strings can be concatenated with `+`: -```groovy +```nextflow def a = "world" print "hello " + a + "\n" ``` @@ -146,7 +146,7 @@ There is an important difference between single-quoted and double-quoted strings In practice, double-quoted strings can contain the value of an arbitrary variable by prefixing its name with the `$` character, or the value of any expression by using the `${expression}` syntax, similar to Bash/shell scripts: -```groovy +```nextflow def foxtype = 'quick' def foxcolor = ['b', 'r', 'o', 'w', 'n'] println "The $foxtype ${foxcolor.join()} fox" @@ -166,7 +166,7 @@ $x + $y A block of text that span multiple lines can be defined by delimiting it with triple single or double quotes: -```groovy +```nextflow def text = """ hello there James how are you today? @@ -179,7 +179,7 @@ Like before, multi-line strings inside double quotes support variable interpolat As in Bash/shell scripts, terminating a line in a multi-line string with a `\` character prevents a newline character from separating that line from the one that follows: -```groovy +```nextflow def myLongCmdline = """ blastp \ -in $input_query \ @@ -211,21 +211,21 @@ Regular expressions are available via the `~/pattern/` syntax and the `=~` and ` Use `=~` to check whether a given pattern occurs anywhere in a string: -```groovy +```nextflow assert 'foo' =~ /foo/ // return TRUE assert 'foobar' =~ /foo/ // return TRUE ``` Use `==~` to check whether a string matches a given regular expression pattern exactly. -```groovy +```nextflow assert 'foo' ==~ /foo/ // return TRUE assert 'foobar' ==~ /foo/ // return FALSE ``` The `~` operator creates a [Pattern](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html) from the given string, while the `=~` operator creates a [Matcher](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Matcher.html): -```groovy +```nextflow x = ~/abc/ println x.class // prints java.util.regex.Pattern @@ -241,7 +241,7 @@ See the linked Java documentation for the available operations for these classes To replace pattern occurrences in a given string, use the `replaceFirst` and `replaceAll` methods: -```groovy +```nextflow def x = "colour".replaceFirst(/ou/, "o") println x // prints: color @@ -253,7 +253,7 @@ println y To remove part of a string, simply replace it with a blank string: -```groovy +```nextflow def z = 'Hello World!'.replaceFirst(/(?i)\s+Wo\w+/, '') println z // prints: Hello! @@ -265,7 +265,7 @@ You can match a pattern that includes groups. First create a matcher object with Here's how it works: -```groovy +```nextflow def programVersion = '2.7.3-beta' def m = programVersion =~ /(\d+)\.(\d+)\.(\d+)-?(.+)/ @@ -278,7 +278,7 @@ assert m[0][4] == 'beta' Applying some syntactic sugar, you can do the same in just one line of code: -```groovy +```nextflow def programVersion = '2.7.3-beta' def (full, major, minor, patch, flavor) = (programVersion =~ /(\d+)\.(\d+)\.(\d+)-?(.+)/)[0] @@ -297,7 +297,7 @@ A closure is a function that can be used like a regular value. Typically, closur For example: -```groovy +```nextflow def square = { v -> v * v } ``` @@ -305,7 +305,7 @@ The above example defines a closure, which takes one parameter named `v` and ret `square` can now be called like a function: -```groovy +```nextflow println square(9) ``` @@ -313,25 +313,25 @@ The above example prints `81`. The main use case for a closure is as an argument to a higher-order function: -```groovy +```nextflow [ 1, 2, 3, 4 ].collect(square) ``` The `collect` method of a list applies a mapping function to each value in the list and produces a new list. The above example produces: -```groovy +```nextflow [ 1, 4, 9, 16 ] ``` The example can be expressed more concisely as: -```groovy +```nextflow [ 1, 2, 3, 4 ].collect { v -> v * v } ``` Another example is the `each` method of a map, which takes a closure with two arguments corresponding to the key and value of each map entry: -```groovy +```nextflow [ "Yue" : "Wu", "Mark" : "Williams", "Sudha" : "Kumari" ].each { key, value -> println "$key = $value" } @@ -347,7 +347,7 @@ Sudha = Kumari Closures can access variables outside of their scope: -```groovy +```nextflow def counts = ["China": 1, "India": 2, "USA": 3] def result = 0 @@ -360,7 +360,7 @@ println result A closure can also declare local variables that exist only for the lifetime of each closure invocation: -```groovy +```nextflow def result = 0 myMap.keySet().each { v -> def count = myMap[v] @@ -370,7 +370,7 @@ myMap.keySet().each { v -> While the `each` method is a convenient way to iterate through a collection and build up some result, a more idiomatic way to do this is to use the `inject` method: -```groovy +```nextflow def result = counts.values().inject { sum, v -> sum + v } ``` @@ -379,7 +379,7 @@ This way, the closure is fully "self-contained" because it doesn't access or mut :::{note} When a closure takes a single parameter, the parameter can be omitted, in which case the implicit `it` parameter will be used: -```groovy +```nextflow [1, 2, 3].each { println it } ``` ::: @@ -392,7 +392,7 @@ In practice, however, Nextflow scripts are composed of *workflows*, *processes*, To transition a code snippet into a proper workflow script, simply wrap it in a `workflow` block: -```groovy +```nextflow workflow { println 'Hello!' } @@ -402,7 +402,7 @@ This block is called the *entry workflow*. It serves as the entrypoint when the You can also break up code into functions, for example: -```groovy +```nextflow def sayHello() { println 'Hello!' } diff --git a/docs/secrets.md b/docs/secrets.md index e75368271f..393992b18d 100644 --- a/docs/secrets.md +++ b/docs/secrets.md @@ -51,7 +51,7 @@ Secrets **cannot** be assigned to pipeline parameters. Secrets can be access by pipeline processes by using the `secret` directive. For example: -```groovy +```nextflow process someJob { secret 'MY_ACCESS_KEY' secret 'MY_SECRET_KEY' @@ -79,7 +79,7 @@ This feature is only available when using the local or grid executors (Slurm, Gr Secrets can be accessed in the pipeline script using the `secrets` variable. For example: -```groovy +```nextflow workflow.onComplete { println("The secret is: ${secrets.MY_SECRET}") } diff --git a/docs/sharing.md b/docs/sharing.md index 021d0d768c..61a840e9f8 100644 --- a/docs/sharing.md +++ b/docs/sharing.md @@ -142,7 +142,7 @@ When a pipeline requires some small data that rarely changes, it may be easier t The following example references the file `dataset/sequences.fa` in the pipeline repository: -```groovy +```nextflow sequences = file("$projectDir/dataset/sequences.fa") sequences.splitFasta { println it diff --git a/docs/spack.md b/docs/spack.md index 636b0952f8..f721fb6971 100644 --- a/docs/spack.md +++ b/docs/spack.md @@ -45,7 +45,7 @@ Alternatively, it can be specified by setting the variable `NXF_SPACK_ENABLED=tr Spack package names can specified using the `spack` directive. Multiple package names can be specified by separating them with a blank space. For example: -```groovy +```nextflow process foo { spack 'bwa samtools py-multiqc' @@ -89,7 +89,7 @@ Read the Spack documentation for more details about how to create [environment f The path of an environment file can be specified using the `spack` directive: -```groovy +```nextflow process foo { spack '/some/path/my-env.yaml' @@ -107,7 +107,7 @@ The environment file name **must** have a `.yaml` extension or else it won't be If you already have a local Spack environment, you can use it in your workflow specifying the installation directory of such environment by using the `spack` directive: -```groovy +```nextflow process foo { spack '/path/to/an/existing/env/directory' diff --git a/docs/vscode.md b/docs/vscode.md new file mode 100644 index 0000000000..f132368289 --- /dev/null +++ b/docs/vscode.md @@ -0,0 +1,528 @@ + +# VS Code integration + +The [Nextflow VS Code extension](https://marketplace.visualstudio.com/items?itemName=nextflow.nextflow) provides IDE support for Nextflow pipelines. + +## Features + +### Syntax highlighting + +The VS Code extension highlights Nextflow scripts and config files for better readability. + +### Diagnostics + +The extension highlights source code in red for errors and yellow for warnings. + +To view all diagnostics for the workspace, open the **Problems** tab. Here, you can search for diagnostics by diagnostic message, filename, and so on. + +### Hover hints + +When you hover over certain source code elements, such as variable names and function calls, the extension provides a tooltip with related information, such as the definition and/or documentation for the element. + +### Code navigation + +The **Outline** section in the Explorer panel lists top-level definitions when you view a script. Include declarations in scripts and config files act as links, and ctrl-clicking them opens the corresponding script or config file. + +To view the definition of a symbol (e.g., a workflow, process, function, or variable), right-click the symbol and select **Go to Definition**. Ctrl-click the symbol to view the definition. Ctrl-click the definition to show all references. + +### Code completion + +The extension suggests auto-completions for variable names, function names, config settings, and other symbols as you type. The extension also provides several snippets for common script declarations, such as processes and workflows. + +### Formatting + +The extension can format your scripts and config files based on a standard set of formatting rules. Rules can be customized using the **Nextflow > Formatting** [extension settings](#settings). + +Use the **Format Document** command in the command palette to format the current file. + +### Renaming symbols + +The extension can rename all references of a symbol (e.g., a workflow, process, function, or variable) throughout the code. + +To rename a symbol, right-click the symbol, select **Rename Symbol**, and enter a new name. + +### Parameter schema + +If a `nextflow_schema.json` file exists in the same directory as a script with an entry workflow, the extension uses the schema to provide validation, hover hints, and code completion for params in the entry workflow. + +### DAG preview for workflows + +The extension can generate a workflow DAG that includes the workflow inputs, outputs, and any processes or workflows that are called by the selected workflow. The workflow DAG is displayed in a new panel to the side. + +To preview the DAG of a workflow, select the **Preview DAG** CodeLens above the workflow definition. + +:::{note} +The **Preview DAG** CodeLens is only available when the script does not contain any errors. +::: + +## Syntax guide + +The language server parses scripts and config files according to the {ref}`Nextflow language specification `. It enforces a stricter syntax compared to the Nextflow CLI. As a result, the language server is able to perform more extensive error checking and provide more specific error messages. However, unlike the Nextflow CLI which allows all Groovy syntax, the Nextflow language specification is not a superset of Groovy. You may need to adjust your code to adhere to the strict syntax, especially if you use more advanced Groovy syntax. + +This section describes some of the most common unsupported features and how to address them. For a full description of the strict syntax, refer to the Nextflow language specification. + +:::{note} +The "Nextflow language specification" is a strict specification of DSL2, not a new DSL version. The Nextflow language will be defined by this specification moving forward, rather than new DSL versions. +::: + +:::{note} +You can move unsupported code into the `lib` directory or a plugin, both of which support the full Groovy language. +::: + +### Excluded syntax + +**Import declarations** + +In Groovy, the `import` declaration can be used to import external classes: + +```groovy +import groovy.json.JsonSlurper + +def json = new JsonSlurper().parseText(json_file.text) +``` + +Instead, use the fully qualified name directly: + +```nextflow +def json = new groovy.json.JsonSlurper().parseText(json_file.text) +``` + +**Class declarations** + +Some users use custom classes in Nextflow to define helper functions or custom record types. Instead, helper functions can be defined as standalone functions in a script. Custom record classes must be moved to the `lib` directory. + +:::{note} +Record types will be addressed in a future version of the Nextflow language specification. +::: + +:::{note} +Enums, a special type of class, are supported, but cannot be included across modules at this time. +::: + +**Mixing script declarations and statements** + +A script may contain any of the following top-level declarations: + +- Feature flags +- Includes +- Parameter declarations +- Workflows +- Processes +- Functions +- Output block + +Alternatively, a script may contain only statements, also known as a "code snippet": + +```nextflow +println 'Hello world!' +``` + +Code snippets are treated as an implicit entry workflow: + +```nextflow +workflow { + println 'Hello world!' +} +``` + +Script declarations and statements cannot be mixed at the same level. All statements must reside within script declarations unless the script is a code snippet: + +```nextflow +process foo { + // ... +} + +// incorrect -- move into entry workflow +// println 'Hello world!' + +// correct +workflow { + println 'Hello world!' +} +``` + +:::{note} +Mixing statements and script declarations was necessary in DSL1 and allowed in DSL2. However, it is no longer supported in the Nextflow language specification in order to simplify the language and to ensure that top-level statements are only executed when the script is executed directly and not when it is included as a module. +::: + +**Assignment expressions** + +In Groovy, you can assign a variable as part of an expression: + +```groovy +foo(x = 1, y = 2) +``` + +The Nextflow language specification only allows assignment as statements: + +```nextflow +x = 1 +y = 2 +foo(x, y) +``` + +The increment (`++`) and decrement (`--`) operators are no longer supported. Use `+=` and `-=` instead: + +```nextflow +x += 1 // x++ +x -= 1 // x-- +``` + +**For and while loops** + +Groovy supports loop statements such as `for` and `while`: + +```groovy +for (rseqc_module in ['read_distribution', 'inner_distance', 'tin']) { + if (rseqc_modules.contains(rseqc_module)) + rseqc_modules.remove(rseqc_module) +} +``` + +The Nextflow language specification does not support loop statements. Use higher-order functions like the `each` method instead: + +```nextflow +['read_distribution', 'inner_distance', 'tin'].each { rseqc_module -> + if (rseqc_modules.contains(rseqc_module)) + rseqc_modules.remove(rseqc_module) +} +``` + +Lists, maps, and sets provide several functions (e.g., `collect`, `find`, `findAll`, `inject`) for iteration. See [Groovy standard library](https://docs.groovy-lang.org/latest/html/groovy-jdk/overview-summary.html) for more information. + +**Switch statements** + +Groovy supports switch statements for pattern matching on a value: + +```groovy +switch (aligner) { +case 'bowtie2': + // ... + break +case 'bwamem': + // ... + break +case 'dragmap': + // ... + break +case 'snap': + // ... + break +default: + // ... +} +``` + +The Nextflow language specification does not support switch statements. Use if-else statements instead: + +```nextflow +if (aligner == 'bowtie2') { + // ... +} else if (aligner == 'bwamem') { + // ... +} else if (aligner == 'dragmap') { + // ... +} else if (aligner == 'snap') { + // ... +} else { + // ... +} +``` + +**Slashy dollar strings** + +Groovy supports a wide variety of strings, including multi-line strings, dynamic strings, slashy strings, multi-line dynamic slashy strings, and more. + +The Nextflow language specification supports single- and double-quoted strings, multi-line strings, and slashy strings. Dynamic slashy strings are not supported: + +```groovy +def logo = /--cl-config 'custom_logo: "${multiqc_logo}"'/ +``` + +Use a double-quoted string instead: + +```nextflow +def logo = "--cl-config 'custom_logo: \"${multiqc_logo}\"'" +``` + +Slashy dollar strings are not supported: + +```groovy +$/ +echo "Hello world!" +/$ +``` + +Use a multi-line string instead: + +```nextflow +""" +echo "Hello world!" +""" +``` + +**Implicit environment variables** + +In Nextflow DSL1 and DSL2, you can reference environment variables directly in strings: + +```nextflow +println "PWD = ${PWD}" +``` + +The Nextflow language specification does not support implicit environment variables. Use `System.getenv()` instead: + +```nextflow +println "PWD = ${System.getenv('PWD')}" +``` + +### Restricted syntax + +The following patterns are still supported but have been restricted, i.e. some syntax variants have been removed. + +**Variable declarations** + +In Groovy, variables can be declared in many different ways: + +```groovy +def a = 1 +final b = 2 +def c = 3, d = 4 +def (e, f) = [5, 6] +String str = 'foo' +def Map meta = [:] +``` + +In Nextflow, variables should be declared with `def` and should not specify a type: + +```nextflow +def a = 1 +def b = 2 +def (c, d) = [3, 4] +def (e, f) = [5, 6] +def str = 'foo' +def meta = [:] +``` + +Similarly, functions should be declared with `def` and should not specify a return type or parameter types: + +```nextflow +/** + * You can use comments to denote types, for example: + * + * @param x: Map + * @param y: String + * @param z: Integer + * @return List + */ +def foo(x, y, z) { + // ... +} +``` + +To ease the migration of existing scripts, the language server only reports warnings for Groovy-style type annotations and implicit variable declarations. These warnings will become errors in the future. + +:::{note} +Type annotations and static type checking will be addressed in a future version of the Nextflow language specification. +::: + +**Process env inputs/outputs** + +In Nextflow DSL1 and DSL2, the name of a process `env` input/output can be specified with or without quotes: + +```nextflow +process PROC { + input: + env FOO + env 'BAR' +} +``` + +The Nextflow language specification requires the name to be specified with quotes: + +```nextflow +process PROC { + input: + env 'FOO' + env 'BAR' +} +``` + +**Implicit process script section** + +In Nextflow DSL1 and DSL2, the process `script:` section label can almost always be omitted: + +```nextflow +process greet { + input: + val greeting + + """ + echo '${greeting}!' + """ +} +``` + +The Nextflow language specification allows the `script:` label to be omitted only if there are no other sections: + +```nextflow +process sayHello { + """ + echo 'Hello world!' + """ +} + +process greet { + input: + val greeting + + script: + """ + echo '${greeting}!' + """ +} +``` + +### Deprecated syntax + +The following patterns are deprecated. The language server reports "future warnings" for these patterns. Future warnings are disabled by default. Enable them by deselecting **Nextflow > Suppress Future Warnings** in the [extension settings](#settings). These warnings may become errors in the future. + +**Implicit closure parameter** + +In Groovy, a closure with no parameters is assumed to have a single parameter named `it`. The Nextflow language specification does not support implicit closure parameters. Declare the parameter explicitly instead: + +```nextflow +ch | map { it * 2 } // deprecated +ch | map { v -> v * 2 } // correct +ch | map { it -> it * 2 } // also correct +``` + +**Using params outside the entry workflow** + +While params can be used anywhere in the pipeline code, they are only intended to be used in the entry workflow. + +Processes and workflows should receive params as explicit inputs: + +```nextflow +process foo { + input: + val foo_args + + // ... +} + +workflow bar { + take: + bar_args + + // ... +} + +workflow { + foo(params.foo_args) + bar(params.bar_args) +} +``` + +**Process each input** + +The `each` process input is deprecated. Use the `combine` or `cross` operator to explicitly repeat over inputs in the calling workflow. + +**Process when section** + +The process `when` section is deprecated. Use conditional logic, such as an `if` statement or the `filter` operator, to control the process invocation in the calling workflow. + +### Configuration syntax + +See {ref}`config-syntax` for a comprehensive description of the configuration language. + +Currently, Nextflow parses config files as Groovy scripts, allowing the use of scripting constructs like variables, helper functions, and conditional logic for dynamic configuration. For example: + +```groovy +def getHostname() { + // ... +} + +def hostname = getHostname() +if (hostname == 'small') { + params.max_memory = 32.GB + params.max_cpus = 8 +} +else if (hostname == 'large') { + params.max_memory = 128.GB + params.max_cpus = 32 +} +``` + +The strict config syntax does not support functions, and only allows statements (e.g., variables and if statements) within closures. You can achieve the same dynamic configuration by using a dynamic include: + +```groovy +includeConfig ({ + def hostname = // ... + if (hostname == 'small') + return 'small.config' + else if (hostname == 'large') + return 'large.config' + else + return '/dev/null' +})() +``` + +The include source is a closure that is immediately invoked. It includes a different config file based on the return value of the closure. Including `/dev/null` is equivalent to including nothing. + +Each conditional configuration is defined in a separate config file: + +```groovy +// small.config +params.max_memory = 32.GB +params.max_cpus = 8 + +// large.config +params.max_memory = 128.GB +params.max_cpus = 32 +``` + +## Troubleshooting + +In the event of an error, you can stop or restart the language server from the command palette. See [Commands](#commands) for the set of available commands. + +Report issues at [nextflow-io/vscode-language-nextflow](https://github.com/nextflow-io/vscode-language-nextflow) or [nextflow-io/language-server](https://github.com/nextflow-io/language-server). When reporting, include a minimal code snippet that reproduces the issue and any error logs from the server. To view logs, open the **Output** tab and select **Nextflow Language Server** from the dropdown. Enable **Nextflow > Debug** in the [extension settings](#settings) to show additional log messages while debugging. + +## Limitations + +- The language server does not detect certain filesystem changes, such as changing the current Git branch. Restart the language server from the command palette to sync it with your workspace. + +- The language server does not recognize configuration options from third-party plugins and will report "Unrecognized config option" warnings for them. + +- The language server provides limited support for Groovy scripts in the `lib` directory. Errors in Groovy scripts are not reported as diagnostics, and changing a Groovy script does not automatically re-compile the Nextflow scripts that reference it. Edit the Nextflow script or close and re-open it to refresh the diagnostics. + +## Commands + +The following commands are available from the command palette: + +- Restart language server +- Stop language server + +## Settings + +The following settings are available: + +`nextflow.debug` +: Enable debug logging and debug information in hover hints. + +`nextflow.files.exclude` +: Configure glob patterns for excluding folders from being searched for Nextflow scripts and configuration files. + +`nextflow.formatting.harshilAlignment` +: Use the [Harshil Alignment™️](https://nf-co.re/docs/contributing/code_editors_and_styling/harshil_alignment) when formatting Nextflow scripts and config files. + +`nextflow.java.home` +: Specifies the folder path to the JDK. Use this setting if the extension cannot find Java automatically. + +`nextflow.suppressFutureWarnings` +: Hide warnings for future changes, deprecations, and removals. + +## Language server + +Most of the functionality of the VS Code extension is provided by the [Nextflow language server](https://github.com/nextflow-io/language-server), which implements the [Language Server Protocol (LSP)](https://microsoft.github.io/language-server-protocol/) for Nextflow scripts and config files. + +The language server is distributed as a standalone Java application. It can be integrated with any editor that functions as an LSP client. Currently, only the VS Code integration is officially supported, but community contributions for other editors are welcome. diff --git a/docs/workflow.md b/docs/workflow.md index 9da30592f3..bfc99c1f8d 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -6,7 +6,7 @@ In Nextflow, a **workflow** is a function that is specialized for composing proc A script can define up to one *entry workflow*, which does not have a name and serves as the entrypoint of the script: -```groovy +```nextflow workflow { Channel.of('Bonjour', 'Ciao', 'Hello', 'Hola') | map { v -> "$v world!" } @@ -16,7 +16,7 @@ workflow { A *named workflow*, on the other hand, is a workflow that can be called from other workflows: -```groovy +```nextflow workflow my_workflow { foo() bar( foo.out.collect() ) @@ -39,7 +39,7 @@ Workflows were introduced in DSL2. If you are still using DSL1, see {ref}`dsl1-p Parameters can be defined in the script with a default value that can be overridden from the CLI, params file, or config file. Params should only be used by the entry workflow: -```groovy +```nextflow params.data = '/some/data/file' workflow { @@ -58,7 +58,7 @@ While params can also be used by named workflows, this practice is discouraged. The `take:` section is used to declare workflow inputs: -```groovy +```nextflow workflow my_workflow { take: data1 @@ -72,7 +72,7 @@ workflow my_workflow { Inputs can be specified like arguments when calling the workflow: -```groovy +```nextflow workflow { my_workflow( Channel.of('/some/data') ) } @@ -82,7 +82,7 @@ workflow { The `emit:` section is used to declare workflow outputs: -```groovy +```nextflow workflow my_workflow { main: foo(data) @@ -97,7 +97,7 @@ When calling the workflow, the output can be accessed using the `out` property, If an output is assigned to a name, the name can be used to reference the output from the calling workflow. For example: -```groovy +```nextflow workflow my_workflow { main: foo(data) @@ -120,7 +120,7 @@ Every output must be assigned to a name when multiple outputs are declared. Processes and workflows are called like functions, passing their inputs as arguments: -```groovy +```nextflow process foo { output: path 'foo.txt', emit: txt @@ -167,7 +167,7 @@ Processes and workflows have a few extra rules for how they can be called: The "return value" of a process or workflow call is the process outputs or workflow emits, respectively. The return value can be assigned to a variable or passed into another call: -```groovy +```nextflow workflow flow { take: data @@ -187,7 +187,7 @@ workflow { Named outputs can be accessed as properties of the return value: -```groovy +```nextflow workflow flow { take: data @@ -209,7 +209,7 @@ workflow { As a convenience, process and workflow outputs can also be accessed without first assigning to a variable, by using the `.out` property of the process or workflow name: -```groovy +```nextflow workflow flow { take: data @@ -239,7 +239,7 @@ Process and workflow outputs can also be accessed by index (e.g., `foo.out[0]`, Workflows can be composed in the same way: -```groovy +```nextflow workflow flow1 { take: data @@ -287,7 +287,7 @@ The following operators have a special meaning when used in a workflow with proc The `|` *pipe* operator can be used to chain processes, operators, and workflows: -```groovy +```nextflow process foo { input: val data @@ -311,7 +311,7 @@ The above snippet defines a process named `foo` and invokes it with the input ch The same code can also be written as: -```groovy +```nextflow workflow { ch1 = Channel.of('Hello','Hola','Ciao') ch2 = foo( ch1 ) @@ -323,7 +323,7 @@ workflow { The `&` *and* operator can be used to call multiple processes in parallel with the same channel(s): -```groovy +```nextflow process foo { input: val data @@ -359,7 +359,7 @@ In the above snippet, the initial channel is piped to the {ref}`operator-map` op The same code can also be written as: -```groovy +```nextflow workflow { ch = Channel.of('Hello').map { v -> v.reverse() } ch_foo = foo(ch) @@ -387,7 +387,7 @@ A workflow can publish outputs by sending channels to "publish targets" in the w Here is a basic example: -```groovy +```nextflow process foo { // ... @@ -439,7 +439,7 @@ Each publish target is saved into a subdirectory of the output directory. By def For example, given the following publish targets: -```groovy +```nextflow workflow { main: ch_foo = foo() @@ -467,7 +467,7 @@ Target names cannot begin or end with a slash (`/`). By default, all files emitted by a published channel will be published into the specified directory. If a channel emits list values, each file in the list (including nested lists) will be published. For example: -```groovy +```nextflow workflow { main: ch_samples = Channel.of( @@ -481,7 +481,7 @@ workflow { A workflow can also disable publishing for a specific channel by redirecting it to `null`: -```groovy +```nextflow workflow { main: ch_foo = foo() @@ -497,7 +497,7 @@ The output directory structure can be customized further in the "output block", For example: -```groovy +```nextflow workflow { // ... } @@ -530,7 +530,7 @@ The output block is only needed if you want to customize the behavior of specifi The `path` directive in a target block can also be a closure which defines a custom publish path for each channel value: -```groovy +```nextflow workflow { main: ch_fastq = Channel.of( [ [id: 'SAMP1'], file('1.fastq'), file('2.fastq') ] ) @@ -550,7 +550,7 @@ The above example will publish each channel value to a different subdirectory. I The closure can even define a different path for each individual file by returning an inner closure, similar to the `saveAs` option of the {ref}`publishDir ` directive: -```groovy +```nextflow output { 'fastq' { path { meta, fastq_1, fastq_2 -> @@ -572,7 +572,7 @@ A publish target can create an index file of the values that were published. An For example: -```groovy +```nextflow workflow { main: ch_fastq = Channel.of( @@ -605,7 +605,7 @@ The above example will write the following CSV file to `results/fastq/index.csv` You can customize the index file with additional directives, for example: -```groovy +```nextflow index { path 'index.csv' header ['id', 'fastq_1', 'fastq_1'] diff --git a/docs/working-with-files.md b/docs/working-with-files.md index 09254ede7a..aad769be96 100644 --- a/docs/working-with-files.md +++ b/docs/working-with-files.md @@ -6,7 +6,7 @@ To access and work with files, use the `file()` method, which returns a file system object given a file path string: -```groovy +```nextflow myFile = file('some/path/to/my_file.file') ``` @@ -14,7 +14,7 @@ The `file()` method can reference both files and directories, depending on what When using the wildcard characters `*`, `?`, `[]` and `{}`, the argument is interpreted as a [glob](http://docs.oracle.com/javase/tutorial/essential/io/fileOps.html#glob) path matcher and the `file()` method returns a list object holding the paths of files whose names match the specified pattern, or an empty list if no match is found: -```groovy +```nextflow listOfFiles = file('some/path/*.fa') ``` @@ -28,14 +28,14 @@ A double asterisk (`**`) in a glob pattern works like `*` but also searches thro By default, wildcard characters do not match directories or hidden files. For example, if you want to include hidden files in the result list, enable the `hidden` option: -```groovy +```nextflow listWithHidden = file('some/path/*.fa', hidden: true) ``` :::{note} To compose paths, instead of string interpolation, use the `resolve()` method or the `/` operator: -```groovy +```nextflow def dir = file('s3://bucket/some/data/path') def sample1 = dir.resolve('sample.bam') // correct def sample2 = dir / 'sample.bam' @@ -48,7 +48,7 @@ def sample4 = "$dir/sample.bam" // incorrect The `file()` method returns a [Path](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/file/Path.html), which has several methods for retrieving metadata about the file: -```groovy +```nextflow def path = file('/some/path/file.txt') assert path.baseName == 'file' @@ -69,25 +69,25 @@ See the {ref}`stdlib-types-path` reference for the list of available methods. Given a file variable, created with the `file()` method as shown previously, reading a file is as easy as getting the file's `text` property, which returns the file content as a string: -```groovy +```nextflow print myFile.text ``` Similarly, you can save a string to a file by assigning it to the file's `text` property: -```groovy +```nextflow myFile.text = 'Hello world!' ``` Binary data can managed in the same way, just using the file property `bytes` instead of `text`. Thus, the following example reads the file and returns its content as a byte array: -```groovy +```nextflow binaryContent = myFile.bytes ``` Or you can save a byte array to a file: -```groovy +```nextflow myFile.bytes = binaryContent ``` @@ -103,13 +103,13 @@ The above methods read and write the **entire** file contents at once, in a sing In order to append a string value to a file without erasing existing content, you can use the `append()` method: -```groovy +```nextflow myFile.append('Add this line\n') ``` Or use the left shift operator, a more idiomatic way to append text content to a file: -```groovy +```nextflow myFile << 'Add a line more\n' ``` @@ -117,7 +117,7 @@ myFile << 'Add a line more\n' In order to read a text file line by line you can use the method `readLines()` provided by the file object, which returns the file content as a list of strings: -```groovy +```nextflow myFile = file('some/my_file.txt') allLines = myFile.readLines() for( line : allLines ) { @@ -127,7 +127,7 @@ for( line : allLines ) { This can also be written in a more idiomatic syntax: -```groovy +```nextflow file('some/my_file.txt') .readLines() .each { println it } @@ -139,7 +139,7 @@ The method `readLines()` reads the **entire** file at once and returns a list co To process a big file, use the method `eachLine()`, which reads only a single line at a time into memory: -```groovy +```nextflow count = 0 myFile.eachLine { str -> println "line ${count++}: $str" @@ -152,7 +152,7 @@ The classes `Reader` and `InputStream` provide fine-grained control for reading The method `newReader()` creates a [Reader](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/Reader.html) object for the given file that allows you to read the content as single characters, lines or arrays of characters: -```groovy +```nextflow myReader = myFile.newReader() String line while( line = myReader.readLine() ) { @@ -163,7 +163,7 @@ myReader.close() The method `withReader()` works similarly, but automatically calls the `close()` method for you when you have finished processing the file. So, the previous example can be written more simply as: -```groovy +```nextflow myFile.withReader { String line while( line = it.readLine() ) { @@ -182,7 +182,7 @@ The `Writer` and `OutputStream` classes provide fine-grained control for writing For example, given two file objects `sourceFile` and `targetFile`, the following code copies the first file's content into the second file, replacing all `U` characters with `X`: -```groovy +```nextflow sourceFile.withReader { source -> targetFile.withWriter { target -> String line @@ -203,7 +203,7 @@ Methods for performing filesystem operations such as copying, deleting, and dire The simplest way to list a directory is to use `list()` or `listFiles()`, which return a collection of first-level elements (files and directories) of a directory: -```groovy +```nextflow for( def file : file('any/path').list() ) { println file } @@ -211,7 +211,7 @@ for( def file : file('any/path').list() ) { Additionally, the `eachFile()` method allows you to iterate through the first-level elements only (just like `listFiles()`). As with other `each*()` methods, `eachFile()` takes a closure as a parameter: -```groovy +```nextflow myDir.eachFile { item -> if( item.isFile() ) { println "${item.getName()} - size: ${item.size()}" @@ -237,13 +237,13 @@ Nextflow can work with many kinds of remote files and objects using the same int To reference a remote file, simple specify the URL when opening the file: -```groovy +```nextflow pdb = file('http://files.rcsb.org/header/5FID.pdb') ``` You can then access it as a local file as described previously: -```groovy +```nextflow println pdb.text ``` diff --git a/docs/your-first-script.md b/docs/your-first-script.md index 6b8cfcf10a..e0ded98a7c 100644 --- a/docs/your-first-script.md +++ b/docs/your-first-script.md @@ -7,7 +7,7 @@ This script defines two processes. The first splits a string into 6-character chunks, writing each one to a file with the prefix `chunk_`, and the second receives these files and transforms their contents to uppercase letters. The resulting strings are emitted on the `result` channel and the final output is printed by the `view` operator. Copy the following example into your favorite text editor and save it to a file named `tutorial.nf`: ```{literalinclude} snippets/your-first-script.nf -:language: groovy +:language: nextflow ``` Execute the script by entering the following command in your terminal: @@ -48,7 +48,7 @@ Nextflow keeps track of all the processes executed in your pipeline. If you modi For the sake of this tutorial, modify the `convertToUpper` process in the previous example, replacing the process script with the string `rev $x`, like so: -```groovy +```nextflow process convertToUpper { input: path x