Preview support for virtual threads

[pditommaso]

This PR implements a preliminary support for virtual threads

[pditommaso]

Doing some stress tests using this script that upload 300 files and download ~200 files.

Using platform (classic) thread it took ~ 9 mins

Completed at: 16-Apr-2023 10:50:15
Duration    : 8m 53s
CPU hours   : 0.3
Succeeded   : 581

Using virtual threads 7 min and 35 secs

Completed at: 16-Apr-2023 10:59:43
Duration    : 7m 35s
CPU hours   : (a few seconds)
Succeeded   : 581

Test script

process foo {
  publishDir "s3://nextflow-ci/outputs"
  input:
    val x
  output:
    path 'hello*'
  """
  head -c 50737410 </dev/urandom > hello${x}.txt
  """
}

process bar {
  input: 
    path x
  """
  head $x
  """
}

workflow {
  channel.of(1..300) | foo
  channel.fromPath('s3://nextflow-ci/uploads/*') | bar
}

nextflow-platform.log
nextflow-virtual.log

[abhi18av]

Hi Paolo 👋

Just curious about this improvement - going through the code it seems that it'll impact the file copy and task monitoring mechanism. However, for some reason when you mentioned about virtual threads last year, I thought it was in the context of replacing GPars library as it wasn't being frequently updated (ref https://github.com/GPars/GPars).

Did I misunderstand the overall intent then 🤔 ?

[pditommaso]

Also GPars will benefit from virtual thread thanks this

https://github.com/nextflow-io/nextflow/pull/3871/files#diff-3f9b4052b1a59668dd036b9281994bde863cab86884c8677f967d75a1e6f37b5R33-R65

[abhi18av]

Ah, I see now - so basically you're swapping the underlying primitives for the abstractions (DataFlow) in GPars library using the virtual threads!

[pditommaso]

Yeah!