Skip to content

Commit

Permalink
Implement Weblog feature as an external plugin
Browse files Browse the repository at this point in the history 
This commit remove the weblog as a core feature and move
it into a plugin maintained separately. See the link below

https://github.com/nextflow-io/nf-weblog

Signed-off-by: Paolo Di Tommaso <[email protected]>
  • Loading branch information
@pditommaso
pditommaso committed Jul 18, 2023
1 parent 9a1c584 commit f9f2c33
Show file tree
Hide file tree
Showing 8 changed files with 5 additions and 623 deletions.
14 changes: 0 additions & 14 deletions docs/config.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1410,20 +1410,6 @@ trace {

Read the {ref}`trace-report` page to learn more about the execution report that can be generated by Nextflow.

(config-weblog)=

### Scope `weblog`

The `weblog` scope allows you to send detailed {ref}`trace <trace-fields>` information as HTTP POST requests to a webserver, shipped as a JSON object.

Detailed information about the JSON fields can be found in the {ref}`weblog description<weblog-service>`.

`weblog.enabled`
: If `true` it will send HTTP POST requests to a given url.

`weblog.url`
: The url where to send HTTP POST requests (default: `http:localhost`).

(config-miscellaneous)=

### Miscellaneous
Expand Down
202 changes: 0 additions & 202 deletions docs/tracing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -426,205 +426,3 @@ And the final image produced with the [Mermaid Live Editor](https://mermaid-js.g

```{image} images/dag-mermaid.png
```

(weblog-service)=

## Weblog via HTTP

Nextflow can send detailed workflow execution metadata and runtime statistics to a HTTP endpoint. To enable this feature, use the `-with-weblog` as shown below:

```bash
nextflow run <pipeline name> -with-weblog [url]
```

Workflow events are sent as HTTP POST requests to the given URL. The message consists of the following JSON structure:

```json
{
"runName": "<run name>",
"runId": "<uuid>",
"event": "<started|process_submitted|process_started|process_completed|error|completed>",
"utcTime": "<UTC timestamp>",
"trace": { },
"metadata": { }
}
```

The JSON object contains the following attributes:

`runName`
: The workflow execution run name.

`runId`
: The workflow execution unique ID.

`event`
: The workflow execution event. One of `started`, `process_submitted`, `process_started`, `process_completed`, `error`, `completed`.

`utcTime`
: The UTC timestamp in ISO 8601 format.

`trace`
: *Included only for the following events: `process_submitted`, `process_started`, `process_completed`, `error`*
: The task runtime information as described in the {ref}`trace fields<trace-fields>` section.
: The set of included fields is determined by the `trace.fields` setting in the Nextflow configuration file. See the {ref}`Trace configuration<config-trace>` and [Trace report](#trace-report) sections to learn more.

`metadata`
: *Included only for the following events: `started`, `completed`*
: The workflow metadata including the {ref}`config manifest<config-manifest>`. For a list of all fields, have a look at the bottom message examples.

### Example `started` event

When a workflow execution is started, a message like the following is posted to the specified end-point. Be aware that the properties in the parameter scope will look different for your workflow. Here is an example output from the `nf-core/hlatyping` pipeline with the weblog feature enabled:

```json
{
"runName": "friendly_pesquet",
"runId": "170aa09c-105f-49d0-99b4-8eb6a146e4a7",
"event": "started",
"utcTime": "2018-10-07T11:42:08Z",
"metadata": {
"params": {
"container": "nfcore/hlatyping:1.1.4",
"help": false,
"outdir": "results",
"bam": true,
"singleEnd": false,
"single-end": false,
"reads": "data/test*{1,2}.fq.gz",
"seqtype": "dna",
"solver": "glpk",
"igenomes_base": "./iGenomes",
"multiqc_config": "/Users/sven1103/.nextflow/assets/nf-core/hlatyping/conf/multiqc_config.yaml",
"clusterOptions": false,
"cluster-options": false,
"enumerations": 1,
"beta": 0.009,
"prefix": "hla_run",
"base_index": "/Users/sven1103/.nextflow/assets/nf-core/hlatyping/data/indices/yara/hla_reference_",
"index": "/Users/sven1103/.nextflow/assets/nf-core/hlatyping/data/indices/yara/hla_reference_dna",
"custom_config_version": "master",
"custom_config_base": "https://raw.githubusercontent.com/nf-core/configs/master"
},
"workflow": {
"start": "2019-03-25T12:09:52Z",
"projectDir": "/Users/sven1103/.nextflow/assets/nf-core/hlatyping",
"manifest": {
"nextflowVersion": ">=18.10.1",
"defaultBranch": "master",
"version": "1.1.4",
"homePage": "https://github.com/nf-core/hlatyping",
"gitmodules": null,
"description": "Precision HLA typing from next-generation sequencing data.",
"name": "nf-core/hlatyping",
"mainScript": "main.nf",
"author": null
},
"complete": null,
"profile": "docker,test",
"homeDir": "/Users/sven1103",
"workDir": "/Users/sven1103/git/nextflow/work",
"container": "nfcore/hlatyping:1.1.4",
"commitId": "4bcced898ee23600bd8c249ff085f8f88db90e7c",
"errorMessage": null,
"repository": "https://github.com/nf-core/hlatyping.git",
"containerEngine": "docker",
"scriptFile": "/Users/sven1103/.nextflow/assets/nf-core/hlatyping/main.nf",
"userName": "sven1103",
"launchDir": "/Users/sven1103/git/nextflow",
"runName": "shrivelled_cantor",
"configFiles": [
"/Users/sven1103/.nextflow/assets/nf-core/hlatyping/nextflow.config"
],
"sessionId": "7f344978-999c-480d-8439-741bc7520f6a",
"errorReport": null,
"scriptId": "2902f5aa7f297f2dccd6baebac7730a2",
"revision": "master",
"exitStatus": null,
"commandLine": "./launch.sh run nf-core/hlatyping -profile docker,test -with-weblog 'http://localhost:4567'",
"nextflow": {
"version": "19.03.0-edge",
"build": 5137,
"timestamp": "2019-03-28T14:46:55Z"
},
},
"stats": {
"computeTimeFmt": "(a few seconds)",
"cachedCount": 0,
"cachedDuration": 0,
"failedDuration": 0,
"succeedDuration": 0,
"failedCount": 0,
"cachedPct": 0.0,
"cachedCountFmt": "0",
"succeedCountFmt": "0",
"failedPct": 0.0,
"failedCountFmt": "0",
"ignoredCountFmt": "0",
"ignoredCount": 0,
"succeedPct": 0.0,
"succeedCount": 0,
"ignoredPct": 0.0
},
"resume": false,
"success": false,
"scriptName": "main.nf",
"duration": null
}
}
```

### Example `completed` event

When a task is completed, a message like the following is posted to the specified end-point:

```json
{
"runName": "friendly_pesquet",
"runId": "170aa09c-105f-49d0-99b4-8eb6a146e4a7",
"event": "process_completed",
"utcTime": "2018-10-07T11:45:30Z",
"trace": {
"task_id": 2,
"status": "COMPLETED",
"hash": "a1/0024fd",
"name": "make_ot_config",
"exit": 0,
"submit": 1538912529498,
"start": 1538912529629,
"process": "make_ot_config",
"tag": null,
"module": [

],
"container": "nfcore/hlatyping:1.1.1",
"attempt": 1,
"script": "\n configbuilder --max-cpus 2 --solver glpk > config.ini\n ",
"scratch": null,
"workdir": "/home/sven1103/git/hlatyping-workflow/work/a1/0024fd028375e2b601aaed44d112e3",
"queue": null,
"cpus": 1,
"memory": 7516192768,
"disk": null,
"time": 7200000,
"env": "PATH=/home/sven1103/git/hlatyping-workflow/bin:$PATH\n",
"error_action": null,
"complete": 1538912730599,
"duration": 201101,
"realtime": 69,
"%cpu": 0.0,
"%mem": 0.1,
"vmem": 54259712,
"rss": 10469376,
"peak_vmem": 20185088,
"peak_rss": 574972928,
"rchar": 7597,
"wchar": 162,
"syscr": 16,
"syscw": 4083712,
"read_bytes": 4096,
"write_bytes": 0,
"native_id": 27185
}
}
```
3 changes: 1 addition & 2 deletions modules/nextflow/src/main/groovy/nextflow/config/ConfigBuilder.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ import nextflow.trace.GraphObserver
import nextflow.trace.ReportObserver
import nextflow.trace.TimelineObserver
import nextflow.trace.TraceFileObserver
import nextflow.trace.WebLogObserver
import nextflow.util.HistoryFile
import nextflow.util.SecretHelper
/**
Expand Down Expand Up @@ -678,7 +677,7 @@ class ConfigBuilder {
if( cmdRun.withWebLog != '-' )
config.weblog.url = cmdRun.withWebLog
else if( !config.weblog.url )
config.weblog.url = WebLogObserver.DEF_URL
config.weblog.url = 'http://localhost'
}

// -- sets tower options
Expand Down
15 changes: 0 additions & 15 deletions modules/nextflow/src/main/groovy/nextflow/trace/DefaultObserverFactory.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ class DefaultObserverFactory implements TraceObserverFactory {
createReportObserver(result)
createTimelineObserver(result)
createDagObserver(result)
createWebLogObserver(result)
createAnsiLogObserver(result)
return result
}
Expand All @@ -36,20 +35,6 @@ class DefaultObserverFactory implements TraceObserverFactory {
}
}

/**
* Create workflow message observer
* @param result
*/
protected void createWebLogObserver(Collection<TraceObserver> result) {
Boolean isEnabled = config.navigate('weblog.enabled') as Boolean
String url = config.navigate('weblog.url') as String
if ( isEnabled ) {
if ( !url ) url = WebLogObserver.DEF_URL
def observer = new WebLogObserver(url)
result << observer
}
}

/**
* Create workflow report file observer
*/
Expand Down
Loading

0 comments on commit f9f2c33

Please sign in to comment.