nextflow - nextflow 进程之一将无法一致地运行/完成

标签 nextflow

我有一个包含四个进程的 nextflow 管道，最后一个进程“INTERSECT”的行为很奇怪，有时它只是不执行任何操作，但也没有给出任何错误。 下面的例子:

nextflow run test_v3.nf 
N E X T F L O W  ~  version 22.10.6
Launching `test_v3.nf` [backstabbing_lumiere] DSL2 - revision: bb41e3dd9a
executor >  local (11)
[61/6edec1] process > GIMME_SCAN (3)    [100%] 5 of 5 ✔
[48/bfb62c] process > EXTRACT_GIMME (5) [100%] 5 of 5 ✔
[23/b911e5] process > EXTRACT_NR_MOTIFS [100%] 1 of 1 ✔
[-        ] process > INTERSECT

如果我 -resume，它会正常执行并输出预期结果。

nextflow run test_v3.nf  -resume
N E X T F L O W  ~  version 22.10.6
Launching `test_v3.nf` [festering_bardeen] DSL2 - revision: bb41e3dd9a
executor >  local (5)
[52/17a139] process > GIMME_SCAN (2)    [100%] 5 of 5, cached: 5 ✔
[46/61caed] process > EXTRACT_GIMME (1) [100%] 5 of 5, cached: 5 ✔
[23/b911e5] process > EXTRACT_NR_MOTIFS [100%] 1 of 1, cached: 1 ✔
[d8/42ae7a] process > INTERSECT (Ss4)   [100%] 5 of 5 ✔

我是 nextflow 的新手，不知道如何在执行中设置依赖项。有人可以帮我解决这个问题吗？

我的 nextflow 管道如下:

#!/usr/bin/env nextflow
nextflow.enable.dsl=2

params.input = "beds/*.bed"
params.pfm = file("nonredundant.motifs.pfm")
params.nr_count = file("motif.nr.count.txt")

params.gimme_scan_bed_files = './results/gimme_extract/*/*.bed'
params.coord_bed_files = './results/motifs_nr_coord/*.bed'
params.outdir = './results/intersect'

process GIMME_SCAN {
    publishDir 'results/gimme_scan', mode: 'copy', overwrite: false

    input:
    path BED
    path PFM
    
    output:
    file ("${BED}_gimme.scan.bed")

    script:
    """
    gimme scan $BED -g Ssal_v3.1 -p $PFM -f 0.05 -b > ${BED}_gimme.scan.bed
    """
}


process EXTRACT_GIMME {
    publishDir 'results/gimme_extract', mode: 'copy', overwrite: false

    input:
    path gimme_scan_file
 
    output:
    path '*'

    script:
    """
    Rscript '$baseDir/gimmeScan_extract_motifs.R' $gimme_scan_file gimme_extract
    """
}

process EXTRACT_NR_MOTIFS {
    publishDir 'results/', mode: 'copy', overwrite: false

    input:
    path nr_count

    output:
    path '*'

    script:
    """
    Rscript '$baseDir/extract_motifs_maelstrom.R' $nr_count
    """
}


process INTERSECT {

    tag { scan_id }

    publishDir "${params.outdir}", mode: 'copy'

    input:
    tuple val(scan_id), path('bed_dir/*')
    path 'coord_dir/*'

    output:
    tuple val(scan_id), path("${scan_id}/*")

    shell:
    '''
    mkdir "!{scan_id}"
    for bed in bed_dir/*; do

        bedtools intersect \\
            -a "${bed}" \\
            -b "coord_dir/$(basename "${bed}")" \\
            -wa |
        sort \\
            -u \\
            > "!{scan_id}/$(basename "${bed}" '.bed')_intersected.bed"
    done
    '''
}

workflow {
    input_ch = Channel.fromPath(params.input)
    pfm_file = params.pfm
    nr_count_file = params.nr_count

    GIMME_SCAN(input_ch, pfm_file)
    EXTRACT_GIMME(GIMME_SCAN.out.flatten())
    EXTRACT_NR_MOTIFS(nr_count_file)
    
    Channel
        .fromFilePairs( params.gimme_scan_bed_files, size: -1) {
            it.parent.name.substring(it.parent.name.lastIndexOf('_') + 1)
        }
        .set { gimme_scan_bed_files }

    Channel
        .fromPath( params.coord_bed_files )
        .collect()
        .set { coord_bed_files }

    INTERSECT(gimme_scan_bed_files, coord_bed_files)
}

感谢您的帮助。

ps:.nextflow.log内容如下:

$ cat .nextflow.log
Apr-03 16:32:55.054 [main] DEBUG nextflow.cli.Launcher - $> nextflow run extract_motif_loc.nf
Apr-03 16:32:55.124 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 22.10.6
Apr-03 16:32:55.144 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/pdewari/.nextflow/plugins; core-plugins: <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="046a62296569657e6b6a44352a35352a37" rel="noreferrer noopener nofollow">[email protected]</a>,<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="afc1c982ced5daddcaef9f819e9b819d" rel="noreferrer noopener nofollow">[email protected]</a>,<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="87e9e1aae4e8e3e2e4e8eaeaeef3c7b7a9b6a9b5" rel="noreferrer noopener nofollow">[email protected]</a>,<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="2d434b004e42435e4241486d1c031d0319" rel="noreferrer noopener nofollow">[email protected]</a>,<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="aec0c883c9cf9ac9c6ee9f809e809a" rel="noreferrer noopener nofollow">[email protected]</a>,<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="d4bab2f9b3bbbbb3b8b194e5fae0fae1" rel="noreferrer noopener nofollow">[email protected]</a>,<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="82ece4aff6edf5e7f0c2b3acb7acb4" rel="noreferrer noopener nofollow">[email protected]</a>,<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="3a545c174d5b4c5f7a0a140f1409" rel="noreferrer noopener nofollow">[email protected]</a>
Apr-03 16:32:55.155 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Apr-03 16:32:55.156 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Apr-03 16:32:55.160 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Apr-03 16:32:55.172 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Apr-03 16:32:55.239 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Apr-03 16:32:55.261 [main] INFO  nextflow.cli.CmdRun - Launching `extract_motif_loc.nf` [furious_wescoff] DSL2 - revision: 938a4809dd
Apr-03 16:32:55.261 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Apr-03 16:32:55.262 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Apr-03 16:32:55.272 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /home/pdewari/.nextflow/secrets/store.json
Apr-03 16:32:55.279 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@6f3f0fae] - activable => nextflow.secret.LocalSecretsProvider@6f3f0fae
Apr-03 16:32:55.364 [main] DEBUG nextflow.Session - Session UUID: df0e66b3-cf2f-4b38-864f-084e5853d983
Apr-03 16:32:55.364 [main] DEBUG nextflow.Session - Run name: furious_wescoff
Apr-03 16:32:55.364 [main] DEBUG nextflow.Session - Executor pool size: 8
Apr-03 16:32:55.378 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Apr-03 16:32:55.407 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 22.10.6 build 5843
  Created: 23-01-2023 23:20 UTC (23:20 BST)
  System: Linux 5.14.0-1054-oem
  Runtime: Groovy 3.0.13 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
  Process: 2840906@MVM-EB-L20071 [127.0.1.1]
  CPUs: 8 - Mem: 15 GB (8.2 GB) - Swap: 2 GB (960.1 MB)
Apr-03 16:32:55.441 [main] DEBUG nextflow.Session - Work-dir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work [ext2/ext3]
Apr-03 16:32:55.441 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/bin
Apr-03 16:32:55.451 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Apr-03 16:32:55.463 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Apr-03 16:32:55.489 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Apr-03 16:32:55.500 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
Apr-03 16:32:55.589 [main] DEBUG nextflow.Session - Session start
Apr-03 16:32:56.659 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Apr-03 16:32:56.795 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Apr-03 16:32:56.795 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Apr-03 16:32:56.804 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Apr-03 16:32:56.811 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=8; memory=15 GB; capacity=8; pollInterval=100ms; dumpInterval=5m
Apr-03 16:32:56.933 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Apr-03 16:32:56.933 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Apr-03 16:32:56.937 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Apr-03 16:32:56.938 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Apr-03 16:32:56.993 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Apr-03 16:32:56.993 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Apr-03 16:32:57.002 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: EXTRACT_GIMME, GIMME_SCAN, EXTRACT_NR_MOTIFS, INTERSECT
Apr-03 16:32:57.003 [main] DEBUG nextflow.Session - Igniting dataflow network (7)
Apr-03 16:32:57.009 [PathVisitor-1] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: beds/; pattern: *.bed; options: [:]
Apr-03 16:32:57.009 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > GIMME_SCAN
Apr-03 16:32:57.010 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > EXTRACT_GIMME
Apr-03 16:32:57.011 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > EXTRACT_NR_MOTIFS
Apr-03 16:32:57.013 [PathVisitor-2] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: ./results/gimme_extract/; pattern: */*.bed; options: [:]
Apr-03 16:32:57.013 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > INTERSECT
Apr-03 16:32:57.013 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Apr-03 16:32:57.013 [main] DEBUG nextflow.Session - Session await
Apr-03 16:32:57.014 [PathVisitor-3] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: ./results/motifs_nr_coord/; pattern: *.bed; options: [:]
Apr-03 16:32:57.019 [PathVisitor-3] DEBUG nextflow.file.PathVisitor - No such file: ./results/motifs_nr_coord/ -- Skipping visit
Apr-03 16:32:57.020 [PathVisitor-2] DEBUG nextflow.file.PathVisitor - No such file: ./results/gimme_extract/ -- Skipping visit
Apr-03 16:32:57.217 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:32:57.219 [Task submitter] INFO  nextflow.Session - [8c/e5a0ac] Submitted process > GIMME_SCAN (5)
Apr-03 16:32:57.228 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:32:57.229 [Task submitter] INFO  nextflow.Session - [19/c27748] Submitted process > GIMME_SCAN (1)
Apr-03 16:32:57.235 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:32:57.236 [Task submitter] INFO  nextflow.Session - [5d/28c5ab] Submitted process > EXTRACT_NR_MOTIFS
Apr-03 16:32:57.244 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:32:57.245 [Task submitter] INFO  nextflow.Session - [1e/3343c1] Submitted process > GIMME_SCAN (4)
Apr-03 16:32:57.252 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:32:57.253 [Task submitter] INFO  nextflow.Session - [d2/849345] Submitted process > GIMME_SCAN (3)
Apr-03 16:32:57.261 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:32:57.276 [Task submitter] INFO  nextflow.Session - [0d/5513d1] Submitted process > GIMME_SCAN (2)
Apr-03 16:35:18.500 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: GIMME_SCAN (4); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/1e/3343c128e22ab3b05d96efc9e221e8]
Apr-03 16:35:18.650 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:35:18.727 [Task submitter] INFO  nextflow.Session - [8e/a8ba92] Submitted process > EXTRACT_GIMME (1)
Apr-03 16:35:18.963 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Apr-03 16:35:47.499 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: GIMME_SCAN (2); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/0d/5513d184a8883c6a4b7c53955e9606]
Apr-03 16:35:47.514 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:35:47.522 [Task submitter] INFO  nextflow.Session - [15/f4712f] Submitted process > EXTRACT_GIMME (2)
Apr-03 16:36:01.440 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: EXTRACT_NR_MOTIFS; status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/5d/28c5abf48258d0e8b2063d3ddc258c]
Apr-03 16:36:06.195 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 6; name: GIMME_SCAN (5); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/8c/e5a0acd7d95e67f0df1fd82d3e6e5b]
Apr-03 16:36:06.210 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:36:06.211 [Task submitter] INFO  nextflow.Session - [70/35fd33] Submitted process > EXTRACT_GIMME (3)
Apr-03 16:36:09.921 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: GIMME_SCAN (1); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/19/c277486903547af29fb04c06bf6f54]
Apr-03 16:36:09.950 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:36:09.951 [Task submitter] INFO  nextflow.Session - [69/8092c2] Submitted process > EXTRACT_GIMME (4)
Apr-03 16:36:17.948 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 7; name: EXTRACT_GIMME (1); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/8e/a8ba92563bf47fca01e65e84af3a61]
Apr-03 16:36:19.946 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: GIMME_SCAN (3); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/d2/849345d841e15d00eb8cdca9e3ca71]
Apr-03 16:36:19.954 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Apr-03 16:36:19.954 [Task submitter] INFO  nextflow.Session - [f8/d5b8e7] Submitted process > EXTRACT_GIMME (5)
Apr-03 16:36:20.143 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 8; name: EXTRACT_GIMME (2); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/15/f4712f5cf2134f66d468414cef6cb0]
Apr-03 16:36:28.895 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 9; name: EXTRACT_GIMME (3); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/70/35fd33f651416f11d043e139520bb6]
Apr-03 16:36:32.813 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 10; name: EXTRACT_GIMME (4); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/69/8092c20050d98b4871f2fc5c4be5c2]
Apr-03 16:36:41.082 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 11; name: EXTRACT_GIMME (5); status: COMPLETED; exit: 0; error: -; workDir: /home/pdewari/Documents/2022/Final_analysis_Salmon/Salmon/chromHMM_2023/model_Ss_12_blacklist_filtered/act_enh_12states/Act_Enh_12states_JASPAR2022_vertebrates.out/test/gimme_scan/nextflow_pipe/work/f8/d5b8e75f4a2be2b4838e5aa43875a8]
Apr-03 16:36:41.086 [main] DEBUG nextflow.Session - Session await > all processes finished
Apr-03 16:36:41.089 [main] DEBUG nextflow.Session - Session await > all barriers passed
Apr-03 16:36:41.116 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Apr-03 16:36:41.122 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=11; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=20m 37s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=6; peakCpus=6; peakMemory=0; ]
Apr-03 16:36:41.164 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Apr-03 16:36:41.171 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Apr-03 16:36:41.172 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

最佳答案

问题是您正在尝试访问publishDir中的一些输出文件。 ，但在(第一次)运行完成之前它们并不存在。解决方案是确保仅使用一个或多个 channel 访问输出文件:

Files are copied into the specified directory in an asynchronous manner, so they may not be immediately available in the published directory at the end of the process execution. For this reason, downstream processes should not try to access output files through the publish directory, but through channels.

我不清楚你的 Rscript 的输出是什么(使用 output: path('*') 并没有告诉我们太多)，但如果我理解正确的话，以下 < em>可能适合你。它至少应该让您开始:

arams.input_beds = "./beds/*.bed"
params.pfm = "nonredundant.motifs.pfm"
params.nr_count = "motif.nr.count.txt"

params.outdir = './results'

process GIMME_SCAN {

    tag { sample }

    publishDir "${params.outdir}/gimme_scan", mode: 'copy'

    input:
    tuple val(sample), path(bed)
    path pfm
    
    output:
    tuple val(sample), path("${bed.baseName}_gimme.scan.bed")

    """
    gimme scan \\
        -g Ssal_v3.1 \\
        -p "${pfm}" \\
        -f 0.05 \\
        -b \\
        "${bed}" \\
        > "${bed.baseName}_gimme.scan.bed"
    """
}

process EXTRACT_GIMME {

    tag { sample }

    publishDir "${params.outdir}/gimme_extract", mode: 'copy'

    input:
    tuple val(sample), path(gimme_scan_file)
 
    output:
    tuple val(sample), path("${sample}/*.bed")

    """
    gimmeScan_extract_motifs.R \\
        "${gimme_scan_file}" \\
        "${sample}"
    """
}

process EXTRACT_NR_MOTIFS {

    publishDir "${params.outdir}/extract_nr_motifs", mode: 'copy'

    input:
    path nr_count

    output:
    path '*.bed'

    """
    extract_motifs_maelstrom.R "${nr_count}"
    """
}

process INTERSECT {

    tag { sample }

    publishDir "${params.outdir}/intersect", mode: 'copy'

    input:
    tuple val(sample), path('bed_dir/*')
    path 'coord_dir/*'

    output:
    tuple val(sample), path("${sample}/*")

    shell:
    '''
    mkdir "!{sample}"
    for bed in bed_dir/*; do

        bedtools intersect \\
            -a "${bed}" \\
            -b "coord_dir/$(basename "${bed}")" \\
            -wa |
        sort \\
            -u \\
            > "!{sample}/$(basename "${bed}" '.bed')_intersected.bed"
    done
    '''
}

workflow {

    input_beds = Channel.fromFilePairs( params.input_beds, size: 1 )
    pfm_file = file( params.pfm )
    nr_count_file = file( params.nr_count )

    EXTRACT_NR_MOTIFS( nr_count_file )

    GIMME_SCAN( input_beds, pfm_file )
    EXTRACT_GIMME( GIMME_SCAN.out )
    
    INTERSECT( EXTRACT_GIMME.out, EXTRACT_NR_MOTIFS.out )
}

请注意，您可以将 Rscript 移动到项目存储库根目录中名为“bin”的文件夹中。 Nextflow 会自动将此文件夹添加到执行环境中的PATH 1 。如果它们尚未可执行，您可以使用 chmod +x your_script.R 使其可执行。

关于nextflow - nextflow 进程之一将无法一致地运行/完成，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/75920706/

上一篇：R - 生成数据帧，使每个值出现相同的次数

下一篇：ios - 在 Swift 中通过 Promise 传递值

相关文章：

nextflow - 如何强制 Nextflow 流程重新计算并忽略恢复工作流程中的缓存

module - 如何在 nextflow dsl2 的同一模块中重复使用同一进程两次，但以不同的名称保存输出？

python - 在 Snakemake 工作流程中作为输入的值数组

pipeline - Nextflow 中的条件管道

python-3.x - 如何在 Nextflow 中运行 Python 脚本

bioinformatics - 如何在 Nextflow 中将一个进程的一个输出传递到另一进程？

channel - Nextflow:分割或子集包含元组的 channel

jobs - 设置 slurm 在 nextflow 中跨节点分配作业

nextflow - 使用命令行参数覆盖 Nextflow 参数

python-3.x - nextflow 没有找到我所有的 python 模块

©2024 IT工具网  联系我们