Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Massive conda update #1663

Merged
merged 42 commits into from
Oct 3, 2024
Merged

Massive conda update #1663

merged 42 commits into from
Oct 3, 2024

Conversation

maxulysse
Copy link
Member

Replace #1662

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/sarek branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

CHANGELOG.md Outdated Show resolved Hide resolved
Copy link

github-actions bot commented Sep 23, 2024

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 08be956

+| ✅ 201 tests passed       |+
#| ❔  12 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!

❔ Tests ignored:

  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: conf/modules.config
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
  • actions_ci - actions_ci
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/sarek/sarek/.github/workflows/awstest.yml
  • template_strings - template_strings
  • modules_config - modules_config

✅ Tests passed:

Run details

  • nf-core/tools version 2.14.1
  • Run at 2024-10-03 15:03:57

@maxulysse maxulysse mentioned this pull request Sep 23, 2024
11 tasks
Copy link
Contributor

@edmundmiller edmundmiller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the non-autogenerated files look good to me!

@maxulysse maxulysse merged commit e9a0c1e into dev Oct 3, 2024
134 checks passed
@maxulysse
Copy link
Member Author

All the non-autogenerated files look good to me!

Good for me then

@@ -64,12 +64,12 @@ process {
// BCFTOOLS ANNOTATE
if (params.tools && params.tools.split(',').contains('bcfann')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

    // ALL ANNOTATION TOOLS
    if (params.tools && (params.tools.split(',').contains('snpeff') || params.tools.split(',').contains('vep') || params.tools.split(',').contains('merge') || params.tools.split(',').contains('bcfann'))) {
        withName: 'NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:.*:(TABIX_BGZIPTABIX|TABIX_TABIX)' {
            ext.prefix = { input.name - '.vcf' }
            publishDir = [
                    mode: params.publish_dir_mode,
                    path: { "${params.outdir}/annotation/${meta.variantcaller}/${meta.id}/" },
                    pattern: "*{gz.tbi}"
            ]
        }
    }

    if (params.tools && (params.tools.split(',').contains('snpeff') || params.tools.split(',').contains('merge'))) {
        withName: 'NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:VCF_ANNOTATE_SNPEFF:TABIX_BGZIPTABIX' {
            publishDir = [
                mode: params.publish_dir_mode,
                path: { "${params.outdir}/annotation/${meta.variantcaller}/${meta.id}/" },
                pattern: "*{gz,gz.tbi}",
                saveAs: { params.tools.split(',').contains('snpeff') ? it : null }
            ]
        }
    }
    ```
    
    This could be cleaned up I think?

@@ -50,7 +50,10 @@ workflow FASTQ_CREATE_UMI_CONSENSUS_FGBIO {
// Using newly created groups
// To call a consensus across reads in the same group
// And emit a consensus BAM file
CALLUMICONSENSUS(GROUPREADSBYUMI.out.bam)
// TODO: add params for call_min_reads and call_min_baseq
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be an issue

@@ -38,12 +45,13 @@ process DEEPVARIANT {
--output_gvcf=${prefix}.g.vcf.gz \\
${args} \\
${regions} \\
--intermediate_results_dir=. \\
${par_regions} \\
--intermediate_results_dir=tmp \\
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we sure about this one? If nextflow starts writing to /tmp here againit will break a lot of cluster. The . should ensure it writes to the current scratch directory

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update the module upstream

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link

@fellen31 fellen31 Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should not write to /tmp, it should write to a directory tmp in the work/scratch directory.

Comment on lines +12 to +16
tuple val(meta), path(input), path(input_index), path(intervals), path(recal_table)
tuple val(meta1), path(fasta)
tuple val(meta2), path(fai)
tuple val(meta3), path(dbsnp)
tuple val(meta4), path(dbsnp_tbi)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you test this? I am surprised this didn't require changes to the module call in bam_variant_calling_germline

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOT AT ALL, I thought we fixed sentieon server so we could do CI

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦

Anyhow, I can see that all sentieon modules are using oras, so we still need to update those one ore time

echo "Decoded and exported Sentieon test-license system environment variables"
fi
sentieon driver $args -t $task.cpus $input_list -r ${fasta} --algo LocusCollector $args2 --fun score_info ${prefix_basename}.score
sentieon driver $args3 -t $task.cpus $input_list -r ${fasta} --algo Dedup $args4 --score_info ${prefix_basename}.score --metrics ${metrics} ${prefix}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not using the suffix anymore. think the config here needs updating to have .cram in the prefix end:

ext.prefix = { "${meta.id}.dedup" }

@@ -64,7 +39,12 @@ process SENTIEON_BWAMEM {
-t $task.cpus \\
\$INDEX \\
$reads \\
| sentieon util sort -r $fasta -t $task.cpus -o ${prefix}.bam --sam2bam -
| sentieon util sort -r $fasta -t $task.cpus -o ${prefix} --sam2bam -
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar here. The prefix already should contain the final file ending. We could consider using cram here already. I assume people would always run sentieon for preprocessing if they have it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants