Skip to content

Nextflow Tower data structure#

Your data#

Your data stays strictly within your infrastructure itself. When you launch a workflow through Tower, you need to connect your infrastructure (HPC/VMs/K8s) by creating the appropriate credentials and compute environment in a workspace.

Tower then uses this configuration to trigger a Nextflow workflow within your infrastructure similar to what is done via the Nextflow CLI, therefore Tower does not manipulate any data itself and no data is transferred to the infrastructure where Tower is running.

It may be possible to access some data within your storage from the Nextflow Tower interface - for example, viewing logs and reports generated in a pipeline run - however, this data is never stored within the Tower infrastructure.

Metadata stored by Nextflow Tower#

Workflow execution metadata is sent by the Nextflow runtime to Nextflow Tower when:

  • Launching workflow with Tower
  • Using the -with-tower option at the command line
  • When a Nextflow Tower is specified in the Nextflow config

The following sections describe the data structure and metadata fields collected by Tower.

Workflow metadata#

The following metadata fields are collected and stored by the Tower backend during a workflow execution:

Name Description
command_line The command line used to launch the workflow execution
commit_id The workflow project commit Id at the time of the execution
complete The workflow execution completion timestamp
config_files The nextflow config file paths(s) involved in the workflow execution
config_text The nextflow config content used for the workflow execution. Note: secrets, such as, AWS keys are stripped and not included in this field.
container The container image name(s) used for the pipeline execution
container_engine The container engine name used for the pipeline execution
duration The workflow execution overall duration (wall time)
error_message The error message reported in the case of nextflow execution failure
error_report The extended error message reported in case of workflow execution error.
exit_status The workflow execution (POSIX) exit code
home_dir The launching user home directory path
launch_dir The workflow launching directory path
manifest_author The workflow project author as defined in the nextflow config manifest file
manifest_default_branch The workflow project default Git branch as defined in the nextflow config manifest file
manifest_description The workflow project description as defined in the nextflow config manifest file
manifest_gitmodules The workflow project Git submodule flag in the nextflow config manifest file
manifest_home_page The workflow project Git home page as defined in the nextflow config manifest file
manifest_main_script The workflow project main script file name as defined in the nextflow config manifest file
manifest_name The workflow project name as defined in the nextflow config manifest file
manifest_nextflow_version The workflow project required Nextflow version defined in the nextflow config manifest file
manifest_version The workflow project version string as defined in the nextflow config manifest file
nextflow_build The build number of the Nextflow runtime used to launch the workflow execution
nextflow_timestamp The build timestamp of the Nextflow runtime used to launch the workflow execution
nextflow_version The version string of the Nextflow runtime used to launch the workflow execution
params The workflow params used to launch the pipeline execution
profile The workflow config profile string used for the pipeline execution
project_dir The directory path where the workflow scripts are stored
project_name The workflow project name
repository The workflow project repository
resume The flag set when a resume execution was submitted
revision The workflow project revision number
run_name The workflow run name as given by the Nextflow runtime
script_file The workflow script file path
script_id The workflow script checksum number
script_name The workflow script filename
session_id The workflow execution unique UUID as assigned by the Nextflow runtime
start The workflow execution start timestamp
stats_cached_count The number of cached tasks upon completion
stats_cached_duration The aggregate time of cached tasks upon completion
stats_cached_pct The percentage of cached tasks upon completion
stats_compute_time_fmt The overall compute time as a formatted string
stats_failed_count The number of failed tasks upon completion
stats_failed_count_fmt The number of failed tasks upon completion as a formatted string
stats_failed_duration The aggregate time of failed tasks upon completion
stats_failed_pct The percentage of failed tasks upon completion
stats_ignored_count The number of ignored tasks upon completion
stats_ignored_count_fmt The number of ignored tasks upon completion as a formatted string
stats_ignored_pct The percentage of ignored tasks upon completion
stats_succeed_count The number of succeeded tasks upon completion
stats_succeed_count_fmt The number of succeeded tasks upon completion as a formatted string
stats_succeed_duration The aggregate time of succeeded tasks upon completion
stats_succeed_pct The percentage of succeeded tasks upon completion
status The workflow execution status
submit The workflow execution submission timestamp
success The flag reporting whether the execution completed successfully
user_name The POSIX user name launching that launched the workflow execution
work_dir The workflow execution scratch directory path

Task Metadata#

Name Description
attempt Number of execution attempt of the task
cloud_zone Cloud zone where the task execution was allocated
complete Task execution completion timestamp
container Container image name used to execute the task
cost Estimated task compute cost
cpus Number of CPUs requested
disk Amount of disk storage requested
duration Amount of time for the task completion
env Task execution environment variables
error_action Action applied on task failure
executor Executor requested for the task execution
exit_status Task POSIX exit code on completion
hash Task unique hash code
inv_ctxt Number of involuntary context switches
machine_type Cloud virtual machine type
memory Amount of memory requested
module Environment Module requested
name Task unique name
native_id Task unique ID as assigned by the underlying execution platform
pcpu Percentage of CPU used to compute the task
peak_rss Peak of real memory during the task execution
peak_vmem Peak of virtual memory during the task execution
pmem Percentage of memory used to compute the task
price_model The cloud price model applied for the task
process The nextflow process name
queue The compute queue name requested
rchar Number of bytes the process read, using any read-like system call from files, pipes, tty, etc.
read_bytes Number of bytes the process directly read from disk
realtime The time required to compute the task
rss Real memory (resident set) size of the process
scratch Flag reporting the task was executed in a local scratch path
script The task command script
start Task execution start timestamp
status The task execution status
submit Task submission timestamp
syscr Number of read-like system call invocations that the process performed
syscw Number of write-like system call invocations that the process performed
tag Nextflow tag associated to the task execution
task_id Nextflow task ID
time Task execution timeout requested
vmem Virtual memory size used by the task execution
vol_ctxt Number of voluntary context switches
wchar Number of bytes the process wrote, using any write-like system call
workdir Task execution work directory
write_bytes Number of bytes the process written to disk
Back to top