Introduction
systemPipeR
provides flexible utilities for designing, building, and running automated nd-to-end analysis
workflows for a wide range of research applications, including next-generation
sequencing (NGS) experiments (H Backman and Girke 2016). Important features include a
uniform workflow interface across different data analysis applications, automated
report generation, and support for running both R and command-line software,
on local computers or compute clusters (see Figure 1).
The latter supports interactive job submissions and batch submissions to queuing
systems of clusters.
It has been designed to improve the reproducibility of large-scale data analysis
projects while substantially reducing the time it takes to analyze complex omics
data sets. Its unique features include a uniform workflow interface and management
system that allows the user to run selected steps, customize, and design entirely
new workflows. Also, the package features take advantage of central community S4
classes of the Bioconductor ecosystem and command-line-based software support.
The main motivation and advantages of using systemPipeR for complex data analysis tasks are:
- Facilitates the design of complex workflows involving multiple R/Bioconductor packages
- Common workflow interface for different applications
- Makes analysis with Bioconductor utilities more accessible to new users
- Simplifies usage of command-line software from within R
- Reduces the complexity of using compute clusters for R and command-line software
- Accelerates runtime of workflows via parallelization on computer systems with multiple CPU cores and/or multiple compute nodes
- Improves reproducibility by automating analyses and generation of analysis reports
## Warning in knitr::include_graphics(system.file("extdata/images", "utilities.png", : It is
## highly recommended to use relative paths for images. You had absolute paths: "/tmp/RtmpbUOpjz/
## Rinst1ccd542002e128/systemPipeR/extdata/images/utilities.png"
A central concept for designing workflows within the systemPipeR environment
is the use of workflow management containers.
Workflow management containers allow the automation of design, build, run and
scale different steps and tools in data analysis.
systemPipeR adopted the widely used community standard Common Workflow Language (CWL)
(Amstutz et al. 2016) for describing parameters analysis workflows in a generic and reproducible
manner. Using this community standard in systemPipeR
has many advantages. For instance, the integration of CWL allows running systemPipeR
workflows from a single specification instance either entirely from within R,
from various command-line wrappers (e.g., cwl-runner) or from other languages
(, e.g., Bash or Python).
systemPipeR includes support for both command-line and R/Bioconductor software
as well as resources for containerization, parallel evaluations on computer clusters
along with the automated generation of interactive analysis reports.
An important feature of systemPipeR's CWL interface is that it provides two
options to run command-line tools and workflows based on CWL. First, one can
run CWL in its native way via an R-based wrapper utility for cwl-runner or
cwl-tools (CWL-based approach). Second, one can run workflows using CWL’s
command-line and workflow instructions from within R (R-based approach). In the
latter case the same CWL workflow definition files (e.g. *.cwl and *.yml)
are used but rendered and executed entirely with R functions defined by
systemPipeR, and thus use CWL mainly as a command-line and workflow
definition format rather than software to run workflows. In this regard
systemPipeR also provides several convenience functions that are useful for
designing and debugging workflows, such as a command-line rendering function to
retrieve the exact command-line strings for each data set and processing step
prior to running a command-line.
This overview introduces the design of a workflow management container, an S4
class in systemPipeR, as well as the custom command-line interface,
combined with the overview of all the common analysis steps of NGS experiments.
New workflow management interface
systemPipeR allows creation (multi-step analyses) and execution of workflow
entirely for R, with control, flexibility, and scalability of all processes.
Furthermore, the workflow execution can be integrated with compute clusters
from R, accelerating results acquisition.
The flexibility of systemPipeR's new interface workflow management class is
the driving factor behind the use of as many steps necessary for the analysis
as well as the connection between command-line- or R-based software. The
connectivity among all workflow steps is achieved by the SYSargsList workflow
management class.
SYSargsList S4 class is a list-like container where each instance stores all the
input/output paths and parameter components required for a particular data
analysis step (see Figure 2).
The SYSargsList constructor function will generate the instances, using as data
input initial targets files, as well as two-parameter files (for details, see below).
When running preconfigured workflows, the only input the user needs to provide
is the initial targets file containing the paths to the input files (e.g., FASTQ)
along with unique sample labels. Subsequent targets instances are created
automatically, based on the connectivity establish between the steps. The
parameters required for running command-line software is provided by the
parameter (*.cwl and *.yml)) files described below.
The class store one or multiple steps, allowing central control for running,
checking status, and monitor complex workflows from start to finish. This design
enhances the systemPipeR workflow framework with a generalized, flexible, and
robust design.
## Warning in knitr::include_graphics(system.file("extdata/images", "SPR_WF.png", : It is highly
## recommended to use relative paths for images. You had absolute paths: "/tmp/RtmpbUOpjz/
## Rinst1ccd542002e128/systemPipeR/extdata/images/SPR_WF.png"
Quick Start
This section will demonstrate how to build a basic workflow. The main features
will be briefly illustrated here. The following section will discuss all the
alternatives to design and build the workflow and the support features available
in systemPipeR.
- Load sample data and directory structure
systemPipeRdata::genWorkenvir(workflow = "new", mydirname = "spr_project")
## [1] "Generated spr_project directory. Next run in new directory, the R code from *.Rmd template interactively. Alternatively, workflows can be exectued with a single command as instructed in the vignette."
setwd("spr_project")
sal <- SPRproject()
## Creating directory '/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project/.SPRproject'
## Creating file '/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project/.SPRproject/SYSargsList.yml'
- Build workflow from R Markdown file
sal <- importWF(sal,
file_path = system.file("extdata", "spr_simple_wf.Rmd", package = "systemPipeR"),
verbose = FALSE)
sal <- runWF(sal)
## Running Step: load_library
## Running Session: Management
##
|
| | 0%
|
|==========================================================================================| 100%
## Step Status: Success
## Running Step: export_iris
## Running Session: Management
##
|
| | 0%
|
|==========================================================================================| 100%
## Step Status: Success
## Running Step: gzip
## Running Session: Management
##
|
| | 0%
|
|============================== | 33%
|
|============================================================ | 67%
|
|==========================================================================================| 100%
## ---- Summary ----
## Targets Total_Files Existing_Files Missing_Files gzip
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## Step Status: Success
## Running Step: gunzip
## Running Session: Management
##
|
| | 0%
|
|============================== | 33%
|
|============================================================ | 67%
|
|==========================================================================================| 100%
## ---- Summary ----
## Targets Total_Files Existing_Files Missing_Files gunzip
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## Step Status: Success
## Running Step: stats
## Running Session: Management
##
|
| | 0%
|
|==========================================================================================| 100%
## Step Status: Success
plotWF(sal, width = "80%", rstudio = TRUE)
statusWF(sal)
## $load_library
## DataFrame with 1 row and 2 columns
## Step Status
## <character> <character>
## 1 load_library Success
##
## $export_iris
## DataFrame with 1 row and 2 columns
## Step Status
## <character> <character>
## 1 export_iris Success
##
## $gzip
## DataFrame with 3 rows and 5 columns
## Targets Total_Files Existing_Files Missing_Files gzip
## <character> <numeric> <numeric> <numeric> <matrix>
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## $gunzip
## DataFrame with 3 rows and 5 columns
## Targets Total_Files Existing_Files Missing_Files gunzip
## <character> <numeric> <numeric> <numeric> <matrix>
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## $stats
## DataFrame with 1 row and 2 columns
## Step Status
## <character> <character>
## 1 stats Success
sal <- renderLogs(sal)
Getting Started
Installation
systemPipeR
environment can be installed from the R console using the BiocManager::install
command. The associated data package systemPipeRdata
can be installed the same way. The latter is a helper package for generating systemPipeR
workflow environments with a single command containing all parameter files and
sample data required to quickly test and run workflows.
if (!requireNamespace("BiocManager", quietly=TRUE)) install.packages("BiocManager")
BiocManager::install("systemPipeR")
BiocManager::install("systemPipeRdata")
Please note that if you desire to use a third-party command-line tool, the particular
tool and dependencies need to be installed and exported in your PATH.
See details.
Loading package and documentation
library("systemPipeR") # Loads the package
library(help="systemPipeR") # Lists package info
vignette("systemPipeR") # Opens vignette
How to get help for systemPipeR
All questions about the package or any particular function should be posted to
the Bioconductor support site https://support.bioconductor.org.
Please add the “systemPipeR” tag to your question, and the package authors will
automatically receive an alert.
We appreciate receiving reports of bugs in the functions or documentation and
suggestions for improvement. For that, please consider opening an issue at
GitHub.
Project structure
systemPipeR expects a project directory structure that consists of a directory
where users may store all the raw data, the results directory that will be reserved
for all the outfiles files or new output folders, and the parameters directory.
This structure allows reproducibility and collaboration across the data science
team since internally relative paths are used. Users could transfer this project
to a different location and still be able to run the entire workflow. Also, it
increases efficiency and data management once the raw data is kept in a separate
folder and avoids duplication.
Directory Structure
systemPipeRdata,
helper package, provides pre-configured workflows, reporting
templates, and sample data loaded as demonstrated below. With a single command,
the package allows creating the workflow environment containing the structure
described here (see Figure 3).
Directory names are indicated in green.
Users can change this structure as needed, but need to adjust the code in their
workflows accordingly.
- workflow/ (e.g. myproject/)
- This is the root directory of the R session running the workflow.
- Run script ( *.Rmd) and sample annotation (targets.txt) files are located here.
- Note, this directory can have any name (e.g. myproject). Changing its name does not require any modifications in the run script(s).
- Important subdirectories:
- param/
- param/cwl/: This subdirectory stores all the parameter and configuration files. To organize workflows, each can have its own subdirectory, where all
*.cwl and *input.yml files need to be in the same subdirectory.
- data/
- Raw data (e.g. FASTQ files)
- FASTA file of reference (e.g. reference genome)
- Annotation files
- Metadata
- etc.
- results/
- Analysis results are usually written to this directory, including: alignment, variant and peak files (BAM, VCF, BED); tabular result files; and image/plot files
- Note, the user has the option to organize results files for a given sample and analysis step in a separate subdirectory.
## Warning in knitr::include_graphics(system.file("extdata/images", "spr_project.png", : It is
## highly recommended to use relative paths for images. You had absolute paths: "/tmp/RtmpbUOpjz/
## Rinst1ccd542002e128/systemPipeR/extdata/images/spr_project.png"
The following parameter files are included in each workflow template:
targets.txt: initial one provided by user; downstream targets_*.txt files are generated automatically*.param/cwl: defines parameter for input/output file operations, e.g.:
hisat2/hisat2-mapping-se.cwlhisat2/hisat2-mapping-se.yml
*_run.sh: optional bash scripts- Configuration files for computer cluster environments (skip on single machines):
.batchtools.conf.R: defines the type of scheduler for batchtools pointing to template file of cluster, and located in user’s home directorybatchtools.*.tmpl: specifies parameters of scheduler used by a system, e.g. Torque, SGE, Slurm, etc.
Structure of initial targets data
The targets data defines all input files (e.g. FASTQ, BAM, BCF) and sample
comparisons of an analysis workflow. It can, also, store any number of descriptive
information for each sample used in the workflow.
The following shows the format of a sample targets file included in the
package. It also can be viewed and downloaded
from systemPipeR’s GitHub repository here.
Please note that here it is represented a tabular file, however systemPipeR can
import the inputs information from a YAML and Google Sheets files, as well as
SummarizedExperiment object. For more details on how to create custom targets,
please find here.
Users should note here, the usage of targets files is optional when using
systemPipeR's new workflow management interface. They can be replaced by a standard YAML
input file used by CWL. Since for organizing experimental variables targets
files are extremely useful and user-friendly. Thus, we encourage users to keep using
them.
Structure of targets file for single-end (SE) samples
In a target file with a single type of input files, here FASTQ files of
single-end (SE) reads, the first column describe the path and the second column
represents a unique id name for each sample. The third column called Factor
represents the biological replicates. All subsequent columns are additional
information, and any number of extra columns can be added as needed.
targetspath <- system.file("extdata", "targets.txt", package = "systemPipeR")
showDF(read.delim(targetspath, comment.char = "#"))
## Loading required namespace: DT
To work with custom data, users need to generate a targets file containing
the paths to their own FASTQ files and then provide under targetspath the
path to the corresponding targets file.
Structure of targets file for paired-end (PE) samples
For paired-end (PE) samples, the structure of the targets file is similar, where
users need to provide two FASTQ path columns: FileName1 and FileName2
with the paths to the PE FASTQ files.
targetspath <- system.file("extdata", "targetsPE.txt", package = "systemPipeR")
showDF(read.delim(targetspath, comment.char = "#"))
Structure of targets file for “Hello World” example
In this example, targets file presents only two columns, which the first one
are the different phrases used by the echo command-line and the second column
it is the sample id. The id column is required, and each sample id should be unique.
targetspath <- system.file("extdata/cwl/example/targets_example.txt", package = "systemPipeR")
showDF(read.delim(targetspath, comment.char = "#"))
Sample comparisons
Sample comparisons are defined in the header lines of the targets file
starting with ‘# <CMP>’.
targetspath <- system.file("extdata", "targetsPE.txt", package = "systemPipeR")
readLines(targetspath)[1:4]
## [1] "# Project ID: Arabidopsis - Pseudomonas alternative splicing study (SRA: SRP010938; PMID: 24098335)"
## [2] "# The following line(s) allow to specify the contrasts needed for comparative analyses, such as DEG identification. All possible comparisons can be specified with 'CMPset: ALL'."
## [3] "# <CMP> CMPset1: M1-A1, M1-V1, A1-V1, M6-A6, M6-V6, A6-V6, M12-A12, M12-V12, A12-V12"
## [4] "# <CMP> CMPset2: ALL"
The function readComp imports the comparison information and stores it in a
list. Alternatively, readComp can obtain the comparison information from
the corresponding SYSargsList step (see below). Note, these header lines are
optional. They are mainly useful for controlling comparative analyses according
to certain biological expectations, such as identifying differentially expressed
genes in RNA-Seq experiments based on simple pair-wise comparisons.
readComp(file = targetspath, format = "vector", delim = "-")
## $CMPset1
## [1] "M1-A1" "M1-V1" "A1-V1" "M6-A6" "M6-V6" "A6-V6" "M12-A12" "M12-V12" "A12-V12"
##
## $CMPset2
## [1] "M1-A1" "M1-V1" "M1-M6" "M1-A6" "M1-V6" "M1-M12" "M1-A12" "M1-V12" "A1-V1"
## [10] "A1-M6" "A1-A6" "A1-V6" "A1-M12" "A1-A12" "A1-V12" "V1-M6" "V1-A6" "V1-V6"
## [19] "V1-M12" "V1-A12" "V1-V12" "M6-A6" "M6-V6" "M6-M12" "M6-A12" "M6-V12" "A6-V6"
## [28] "A6-M12" "A6-A12" "A6-V12" "V6-M12" "V6-A12" "V6-V12" "M12-A12" "M12-V12" "A12-V12"
Downstream targets files description
After the step which required the initial targets file information, the downstream
targets files are created automatically (see Figure 4).
Each step that uses the previous step outfiles as an input, the new targets input
will be managed internally by the workflow instances, establishing connectivity
among the steps in the workflow.
systemPipeR provides features to automatically and systematically build this
connection, providing security that all the samples will be managed efficiently
and reproducibly.
## Warning in knitr::include_graphics(system.file("extdata/images", "targets_con.png", : It is
## highly recommended to use relative paths for images. You had absolute paths: "/tmp/RtmpbUOpjz/
## Rinst1ccd542002e128/systemPipeR/extdata/images/targets_con.png"
Structure of the new parameters files
The parameters and configuration required for running command-line software are
provided by the widely used community standard Common Workflow Language (CWL)
(Amstutz et al. 2016), which describes parameters analysis workflows in a generic
and reproducible manner. For R-based workflow steps, param files are not required.
For a complete overview of the CWL syntax, please see the section below.
Also, we have a dedicated section explain how to systemPipeR establish the
connection between the CWL parameters files and the targets files. Please see here.
Project initialization
To create a Workflow within systemPipeR, we can start by defining an empty
container and checking the directory structure:
sal <- SPRproject(projPath = getwd())
## Creating directory '/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project/.SPRproject'
## Creating file '/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project/.SPRproject/SYSargsList.yml'
Internally, SPRproject function will create a hidden folder called .SPRproject,
by default, to store all the log files.
A YAML file, here called SYSargsList.yml, has been created, which initially
contains the basic location of the project structure; however, every time the
workflow object sal is updated in R, the new information will also be store in this
flat-file database for easy recovery.
If you desire different names for the logs folder and the YAML file, these can
be modified as follows:
sal <- SPRproject(logs.dir= ".SPRproject", sys.file=".SPRproject/SYSargsList.yml")
Also, this function will check and/or create the basic folder structure if missing,
which means data, param, and results folder, as described here.
If the user wants to use a different names for these directories, can be specified
as follows:
sal <- SPRproject(data = "data", param = "param", results = "results")
It is possible to separate all the R objects created within the workflow analysis
from the current environment. SPRproject function provides the option to create
a new environment, and in this way, it is not overwriting any object you may want
to have at your current section.
sal <- SPRproject(envir = new.env())
In this stage, the object sal is a empty container, except for the project information. The project information can be accessed by the projectInfo method:
sal
## Instance of 'SYSargsList':
## No workflow steps added
projectInfo(sal)
## $project
## [1] "/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project"
##
## $data
## [1] "data"
##
## $param
## [1] "param"
##
## $results
## [1] "results"
##
## $logsDir
## [1] ".SPRproject"
##
## $sysargslist
## [1] ".SPRproject/SYSargsList.yml"
Also, the length function will return how many steps this workflow contains and
in this case it is empty, as follow:
length(sal)
## [1] 0
Workflow Design
systemPipeR workflows can be designed and built from start to finish with a
single command, importing from an R Markdown file or stepwise in interactive
mode from the R console.
In the next section, we will demonstrate how to build the workflow in an
interactive mode, and in the following section, we will show how to build from a file.
New workflows are constructed, or existing ones modified, by connecting each
step via appendStep method. Each SYSargsList instance contains instructions
needed for processing a set of input files with a specific command-line or R
software, as well as the paths to the corresponding outfiles generated by a
particular tool/step.
To build R code based step, the constructor function Linewise is used.
For more details about this S4 class container, see here.
Build workflow interactive
This tutorial shows a very simple example for describing and explaining all main
features available within systemPipeR to design, build, manage, run, and
visualize the workflow. In summary, we are exporting a dataset to multiple
files, compressing and decompressing each one of the files, and importing to R,
and finally performing a statistical analysis.
In the previous section, we initialize the project by building the sal object.
Until this moment, the container has no steps:
sal
## Instance of 'SYSargsList':
## No workflow steps added
Next, we need to populate the object created with the first step in the
workflow.
Adding the first step
The first step is R code based, and we are splitting the iris dataset by Species
and for each Species will be saved on file. Please note that this code will
not be executed now; it is just store in the container for further execution.
This constructor function requires the step_name and the R-based code under
the code argument.
The R code should be enclosed by braces ({}) and separated by a new line.
appendStep(sal) <- LineWise(code = {
mapply(function(x, y) write.csv(x, y),
split(iris, factor(iris$Species)),
file.path("results", paste0(names(split(iris, factor(iris$Species))), ".csv"))
)
},
step_name = "export_iris")
For a brief overview of the workflow, we can check the object as follows:
sal
## Instance of 'SYSargsList':
## WF Steps:
## 1. export_iris --> Status: Pending
##
Also, for printing and double-check the R code in the step, we can use the
codeLine method:
codeLine(sal)
## export_iris
## mapply(function(x, y) write.csv(x, y), split(iris, factor(iris$Species)), file.path("results", paste0(names(split(iris, factor(iris$Species))), ".csv")))
Adding more steps
Next, an example of how to compress the exported files using
gzip command-line.
The constructor function creates an SYSargsList S4 class object using data from
three input files:
- CWL command-line specification file (`wf_file` argument);
- Input variables (`input_file` argument);
- Targets file (`targets` argument).
In CWL, files with the extension .cwl define the parameters of a chosen
command-line step or workflow, while files with the extension .yml define the
input variables of command-line steps.
The targets file is optional for workflow steps lacking input files. The connection
between input variables and the targets file is defined under the inputvars
argument. It is required a named vector, where each element name needs to match
with column names in the targets file, and the value must match the names of
the input variables defined in the *.yml files (see Figure 5).
A detailed description of the dynamic between input variables and targets
files can be found here.
In addition, the CWL syntax overview can be found here.
Besides all the data form targets, wf_file, input_file and dir_path arguments,
SYSargsList constructor function options include:
step_name: a unique name for the step. This is not mandatory; however,
it is highly recommended. If no name is provided, a default step_x, where
x reflects the step index, will be added.dir: this option allows creating an exclusive subdirectory for the step
in the workflow. All the outfiles and log files for this particular step will
be generated in the respective folders.dependency: after the first step, all the additional steps appended to
the workflow require the information of the dependency tree.
The appendStep<- method is used to append a new step in the workflow.
targetspath <- system.file("extdata/cwl/gunzip", "targets_gunzip.txt", package = "systemPipeR")
appendStep(sal) <- SYSargsList(step_name = "gzip",
targets = targetspath, dir = TRUE,
wf_file = "gunzip/workflow_gzip.cwl", input_file = "gunzip/gzip.yml",
dir_path = system.file("extdata/cwl", package = "systemPipeR"),
inputvars = c(FileName = "_FILE_PATH_", SampleName = "_SampleName_"),
dependency = "export_iris")
Note: This will not work if the gzip is not available on your system
(installed and exported to PATH) and may only work on Windows systems using PowerShell.
For a overview of the workflow, we can check the object as follows:
sal
## Instance of 'SYSargsList':
## WF Steps:
## 1. export_iris --> Status: Pending
## 2. gzip --> Status: Pending
## Total Files: 3 | Existing: 0 | Missing: 3
## 2.1. gzip
## cmdlist: 3 | Pending: 3
##
Note that we have two steps, and it is expected three files from the second step.
Also, the workflow status is Pending, which means the workflow object is
rendered in R; however, we did not execute the workflow yet.
In addition to this summary, it can be observed this step has three command lines.
For more details about the command-line rendered for each target file, it can be
checked as follows:
cmdlist(sal, step="gzip")
## $gzip
## $gzip$SE
## $gzip$SE$gzip
## [1] "gzip -c results/setosa.csv > results/SE.csv.gz"
##
##
## $gzip$VE
## $gzip$VE$gzip
## [1] "gzip -c results/versicolor.csv > results/VE.csv.gz"
##
##
## $gzip$VI
## $gzip$VI$gzip
## [1] "gzip -c results/virginica.csv > results/VI.csv.gz"
Using the outfiles for the next step
For building this step, all the previous procedures are being used to append the
next step. However, here, we can observe power features that build the
connectivity between steps in the workflow.
In this example, we would like to use the outfiles from gzip Step, as
input from the next step, which is the gunzip. In this case, let’s look at the
outfiles from the first step:
outfiles(sal)
## $export_iris
## DataFrame with 0 rows and 0 columns
##
## $gzip
## DataFrame with 3 rows and 1 column
## gzip_file
## <character>
## SE results/SE.csv.gz
## VE results/VE.csv.gz
## VI results/VI.csv.gz
The column we want to use is “gzip_file”. For the argument targets in the
SYSargsList function, it should provide the name of the correspondent step in
the Workflow and which outfiles you would like to be incorporated in the next
step.
The argument inputvars allows the connectivity between outfiles and the
new targets file. Here, the name of the previous outfiles should be provided
it. Please note that all outfiles column names must be unique.
It is possible to keep all the original columns from the targets files or remove
some columns for a clean targets file.
The argument rm_targets_col provides this flexibility, where it is possible to
specify the names of the columns that should be removed. If no names are passing
here, the new columns will be appended.
appendStep(sal) <- SYSargsList(step_name = "gunzip",
targets = "gzip", dir = TRUE,
wf_file = "gunzip/workflow_gunzip.cwl", input_file = "gunzip/gunzip.yml",
dir_path = system.file("extdata/cwl", package = "systemPipeR"),
inputvars = c(gzip_file = "_FILE_PATH_", SampleName = "_SampleName_"),
rm_targets_col = "FileName",
dependency = "gzip")
We can check the targets automatically create for this step,
based on the previous outfiles:
targetsWF(sal[3])
## $gunzip
## DataFrame with 3 rows and 2 columns
## gzip_file SampleName
## <character> <character>
## SE results/SE.csv.gz SE
## VE results/VE.csv.gz VE
## VI results/VI.csv.gz VI
We can also check all the expected outfiles for this particular step, as follows:
outfiles(sal[3])
## $gunzip
## DataFrame with 3 rows and 1 column
## gunzip_file
## <character>
## SE results/SE.csv
## VE results/VE.csv
## VI results/VI.csv
Now, we can observe that the third step has been added and contains one substep.
sal
## Instance of 'SYSargsList':
## WF Steps:
## 1. export_iris --> Status: Pending
## 2. gzip --> Status: Pending
## Total Files: 3 | Existing: 0 | Missing: 3
## 2.1. gzip
## cmdlist: 3 | Pending: 3
## 3. gunzip --> Status: Pending
## Total Files: 3 | Existing: 0 | Missing: 3
## 3.1. gunzip
## cmdlist: 3 | Pending: 3
##
In addition, we can access all the command lines for each one of the substeps.
cmdlist(sal["gzip"], targets = 1)
## $gzip
## $gzip$SE
## $gzip$SE$gzip
## [1] "gzip -c results/setosa.csv > results/SE.csv.gz"
Getting data from a workflow instance
The final step in this simple workflow is an R code step. For that, we are using
the LineWise constructor function as demonstrated above.
One interesting feature showed here is the getColumn method that allows
extracting the information for a workflow instance. Those files can be used in
an R code, as demonstrated below.
getColumn(sal, step = "gunzip", 'outfiles')
## SE VE VI
## "results/SE.csv" "results/VE.csv" "results/VI.csv"
appendStep(sal) <- LineWise(code = {
df <- lapply(getColumn(sal, step = "gunzip", 'outfiles'), function(x) read.delim(x, sep = ",")[-1])
df <- do.call(rbind, df)
stats <- data.frame(cbind(mean = apply(df[,1:4], 2, mean), sd = apply(df[,1:4], 2, sd)))
stats$species <- rownames(stats)
plot <- ggplot2::ggplot(stats, ggplot2::aes(x = species, y = mean, fill = species)) +
ggplot2::geom_bar(stat = "identity", color = "black", position = ggplot2::position_dodge()) +
ggplot2::geom_errorbar(ggplot2::aes(ymin = mean-sd, ymax = mean+sd), width = .2, position = ggplot2::position_dodge(.9))
},
step_name = "iris_stats",
dependency = "gzip")
Build workflow from a {R Markdown}
The precisely same workflow can be created by importing the steps from an
R Markdown file.
As demonstrated above, it is required to initialize the project with SPRproject function.
importWF function will scan and import all the R chunk from the R Markdown file
and build all the workflow instances. Then, each R chuck in the file will be
converted in a workflow step.
sal_rmd <- SPRproject(logs.dir = ".SPRproject_rmd")
## Creating directory '/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project/.SPRproject_rmd'
## Creating file '/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project/.SPRproject_rmd/SYSargsList.yml'
sal_rmd <- importWF(sal_rmd,
file_path = system.file("extdata", "spr_simple_wf.Rmd", package = "systemPipeR"))
## Reading Rmd file
##
## ---- Actions ----
## Checking chunk eval values
## Checking chunk SPR option
## Ignore non-SPR chunks: 17
## Parse chunk code
## Now importing step 'load_library'
## Now importing step 'export_iris'
## Now importing step 'gzip'
## Now importing step 'gunzip'
## Now importing step 'stats'
Let’s explore the workflow to check the steps:
stepsWF(sal_rmd)
## $load_library
## Instance of 'LineWise'
## Code Chunk length: 1
##
## $export_iris
## Instance of 'LineWise'
## Code Chunk length: 1
##
## $gzip
## Instance of 'SYSargs2':
## Slot names/accessors:
## targets: 3 (SE...VI), targetsheader: 1 (lines)
## modules: 0
## wf: 1, clt: 1, yamlinput: 4 (inputs)
## input: 3, output: 3
## cmdlist: 3
## Sub Steps:
## 1. gzip (rendered: TRUE)
##
##
##
## $gunzip
## Instance of 'SYSargs2':
## Slot names/accessors:
## targets: 3 (SE...VI), targetsheader: 1 (lines)
## modules: 0
## wf: 1, clt: 1, yamlinput: 4 (inputs)
## input: 3, output: 3
## cmdlist: 3
## Sub Steps:
## 1. gunzip (rendered: TRUE)
##
##
##
## $stats
## Instance of 'LineWise'
## Code Chunk length: 5
dependency(sal_rmd)
## $load_library
## [1] NA
##
## $export_iris
## [1] "load_library"
##
## $gzip
## [1] "export_iris"
##
## $gunzip
## [1] "gzip"
##
## $stats
## [1] "gunzip"
codeLine(sal_rmd)
## gzip AND gunzip step have been dropped because it is not a LineWise object.
## load_library
## library(systemPipeR)
## export_iris
## mapply(function(x, y) write.csv(x, y), split(iris, factor(iris$Species)), file.path("results", paste0(names(split(iris, factor(iris$Species))), ".csv")))
## stats
## df <- lapply(getColumn(sal, step = "gunzip", "outfiles"), function(x) read.delim(x, sep = ",")[-1])
## df <- do.call(rbind, df)
## stats <- data.frame(cbind(mean = apply(df[, 1:4], 2, mean), sd = apply(df[, 1:4], 2, sd)))
## stats$species <- rownames(stats)
## plot <- ggplot2::ggplot(stats, ggplot2::aes(x = species, y = mean, fill = species)) + ggplot2::geom_bar(stat = "identity", color = "black", position = ggplot2::position_dodge()) + ggplot2::geom_errorbar(ggplot2::aes(ymin = mean - sd, ymax = mean + sd), width = 0.2, position = ggplot2::position_dodge(0.9))
targetsWF(sal_rmd)
## $load_library
## DataFrame with 0 rows and 0 columns
##
## $export_iris
## DataFrame with 0 rows and 0 columns
##
## $gzip
## DataFrame with 3 rows and 2 columns
## FileName SampleName
## <character> <character>
## SE results/setosa.csv SE
## VE results/versicolor.csv VE
## VI results/virginica.csv VI
##
## $gunzip
## DataFrame with 3 rows and 2 columns
## gzip_file SampleName
## <character> <character>
## SE results/SE.csv.gz SE
## VE results/VE.csv.gz VE
## VI results/VI.csv.gz VI
##
## $stats
## DataFrame with 0 rows and 0 columns
Rules to create the R Markdown to import as workflow
To include a particular code chunk from the R Markdown file in the workflow
analysis, please use the following code chunk options:
- `spr=TRUE'`: for code chunks with step workflow.
For example:
```{r step_1, eval=TRUE, spr=TRUE}
```{r step_2, eval=FALSE, spr=TRUE}
ImportWF function can ignore eval option in code chunk, and in this case,
both of the examples steps above would be incorporated in the workflow.
For spr = TRUE, the last object assigned must to be the SYSargsList, for example:
targetspath <- system.file("extdata/cwl/example/targets_example.txt", package = "systemPipeR")
appendStep(sal) <- SYSargsList(step_name = "Example",
targets = targetspath,
wf_file = "example/example.cwl", input_file = "example/example.yml",
dir_path = system.file("extdata/cwl", package = "systemPipeR"),
inputvars = c(Message = "_STRING_", SampleName = "_SAMPLE_"))
OR
appendStep(sal) <- LineWise(code = {
library(systemPipeR)
},
step_name = "load_lib")
Also, note that all the required files or objects to generate one particular
command-line step must be defined in a R code chunk imported.
The motivation for this is that when R Markdown files are imported, the
spr = TRUE R chunk will be evaluated and append and stored in the workflow control
class as the SYSargsList object.
The workflow object name used in the R Markdown (e.g. appendStep(sal)) needs to
be the same used for the importWF function. It is important to keep consistency.
If different object names are used, when running the workflow, you can see a error,
like Error: object not found..
Running the workflow
For running the workflow, runWF function will execute all the command lines
store in the workflow container.
sal <- runWF(sal)
## Running Step: export_iris
## Running Session: Management
##
|
| | 0%
|
|==========================================================================================| 100%
## Step Status: Success
## Running Step: gzip
## Running Session: Management
##
|
| | 0%
|
|============================== | 33%
|
|============================================================ | 67%
|
|==========================================================================================| 100%
## ---- Summary ----
## Targets Total_Files Existing_Files Missing_Files gzip
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## Step Status: Success
## Running Step: gunzip
## Running Session: Management
##
|
| | 0%
|
|============================== | 33%
|
|============================================================ | 67%
|
|==========================================================================================| 100%
## ---- Summary ----
## Targets Total_Files Existing_Files Missing_Files gunzip
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## Step Status: Success
## Running Step: iris_stats
## Running Session: Management
##
|
| | 0%
|
|==========================================================================================| 100%
## Step Status: Success
This essential function allows the user to choose one or multiple steps to be
executed using the steps argument. However, it is necessary to follow the
workflow dependency graph. If a selected step depends on a previous step(s) that
was not executed, the execution will fail.
sal <- runWF(sal, steps = c(1,3))
Also, it allows forcing the execution of the steps, even if the status of the
step is 'Success' and all the expected outfiles exists.
Another feature of the runWF function is ignoring all the warnings
and errors and running the workflow by the arguments warning.stop and
error.stop, respectively.
sal <- runWF(sal, force = TRUE, warning.stop = FALSE, error.stop = TRUE)
When the project was initialized by SPRproject function, it was created an
environment for all objects created during the workflow execution. This
environment can be accessed as follows:
viewEnvir(sal)
The workflow execution allows to save this environment for future recovery:
sal <- runWF(sal, saveEnv = TRUE)
Workflow status
To check the summary of the workflow, we can use:
sal
## Instance of 'SYSargsList':
## WF Steps:
## 1. export_iris --> Status: Success
## 2. gzip --> Status: Success
## Total Files: 3 | Existing: 3 | Missing: 0
## 2.1. gzip
## cmdlist: 3 | Success: 3
## 3. gunzip --> Status: Success
## Total Files: 3 | Existing: 3 | Missing: 0
## 3.1. gunzip
## cmdlist: 3 | Success: 3
## 4. iris_stats --> Status: Success
##
To access more details about the workflow instances, we can use the statusWF method:
statusWF(sal)
## $export_iris
## DataFrame with 1 row and 2 columns
## Step Status
## <character> <character>
## 1 export_iris Success
##
## $gzip
## DataFrame with 3 rows and 5 columns
## Targets Total_Files Existing_Files Missing_Files gzip
## <character> <numeric> <numeric> <numeric> <matrix>
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## $gunzip
## DataFrame with 3 rows and 5 columns
## Targets Total_Files Existing_Files Missing_Files gunzip
## <character> <numeric> <numeric> <numeric> <matrix>
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## $iris_stats
## DataFrame with 1 row and 2 columns
## Step Status
## <character> <character>
## 1 iris_stats Success
Parallelization on clusters
This section of the tutorial provides an introduction to the usage of the
systemPipeR features on a cluster.
The computation can be greatly accelerated by processing many files
in parallel using several compute nodes of a cluster, where a scheduling/queuing
system is used for load balancing. For this the clusterRun function submits
the computing requests to the scheduler using the run specifications
defined by runWF.
A named list provides the computational resources. By default, it can be defined
the upper time limit in minutes for jobs before they get killed by the scheduler,
memory limit in Mb, number of CPUs, and number of tasks.
The number of independent parallel cluster processes is defined under the
Njobs argument. The following example will run one process in parallel using
for each 4 CPU cores. If the resources available on a cluster allow running all
the processes simultaneously, then the shown sample submission will utilize in
total four CPU cores (NJobs * ncpus). Note, clusterRun can be used
with most queueing systems as it is based on utilities from the batchtools
package which supports the use of template files (*.tmpl) for defining the
run parameters of different schedulers. To run the following code, one needs to
have both a conf file (see .batchtools.conf.R samples here)
and a template file (see *.tmpl samples here)
for the queueing available on a system. The following example uses the sample
conf and template files for the Slurm scheduler provided by this package.
library(batchtools)
resources <- list(walltime=120, ntasks=1, ncpus=4, memory=1024)
sal <- clusterRun(sal, FUN = runWF,
more.args = list(),
conffile=".batchtools.conf.R",
template="batchtools.slurm.tmpl",
Njobs=1, runid="01", resourceList=resources)
Note: The example is submitting the jog to short partition. If you desire to
use a different partition, please adjust accordingly (batchtools.slurm.tmpl).
Visualize workflow
systemPipeR workflows instances can be visualized with the plotWF function.
This function will make a plot of selected workflow instance and the following
information is displayed on the plot:
- Workflow structure (dependency graphs between different steps);
- Workflow step status, *e.g.* `Success`, `Error`, `Pending`, `Warnings`;
- Sample status and statistics;
- Workflow timing: running duration time.
If no argument is provided, the basic plot will automatically detect width,
height, layout, plot method, branches, etc.
plotWF(sal, show_legend = TRUE, width = "80%", rstudio = TRUE)
For more details about the plotWF function, please see here.
Technical report
systemPipeR compiles all the workflow execution logs in one central location,
making it easier to check any standard output (stdout) or standard error
(stderr) for any command-line tools used on the workflow or the R code stdout.
Also, the workflow plot is appended at the beginning of the report, making it
easier to click on the respective step.
sal <- renderLogs(sal)
Exported the workflow
systemPipeR workflow management system allows to translate and export the
workflow build interactively to R Markdown format or an executable bash script.
This feature advances the reusability of the workflow, as well as the flexibility
for workflow execution.
R Markdown file
sal2rmd function takes an SYSargsList workflow container and translates it to
SPR workflow template R markdown format. This file can be imported with the
importWF function, as demonstrated above.
sal2rmd(sal)
Bash script
sal2bash function takes an SYSargsList workflow container and translates
it to an executable bash script, so one can run the workflow without loading
SPR or using an R console.
sal2bash(sal)
It will be generated on the project root an executable bash script, called by
default the spr_wf.sh. Also, a directory ./spr_wf will be created and store
all the R scripts based on the workflow steps. Please note that this function will
“collapse” adjacent R steps into one file as much as possible.
Project Resume and Restart
If you desire to resume or restart a project that has been initialized in the past,
SPRproject function allows this operation.
With the resume option, it is possible to load the SYSargsList object in R and
resume the analysis. Please, make sure to provide the logs.dir location, and the
corresponded YAML file name, if the default names were not used when the project was created.
sal <- SPRproject(resume = TRUE, logs.dir = ".SPRproject",
sys.file = ".SPRproject/SYSargsList.yml")
If you choose to save the environment in the last analysis, you can recover all
the files created in that particular section. SPRproject function allows this
with load.envir argument. Please note that the environment was saved only with
you run the workflow in the last section (runWF()).
sal <- SPRproject(resume = TRUE, load.envir = TRUE)
After loading the workflow at your current section, you can check the objects
created in the old environment and decide if it is necessary to copy them to the
current environment.
viewEnvir(sal)
copyEnvir(sal, list="plot", new.env = globalenv())
The resume option will keep all previous logs in the folder; however, if you desire to
clean the execution (delete all the log files) history and restart the workflow,
the restart=TRUE option can be used.
sal <- SPRproject(restart = TRUE, load.envir = FALSE)
The last and more drastic option from SYSproject function is to overwrite the
logs and the SYSargsList object. This option will delete the hidden folder and the
information on the SYSargsList.yml file. This will not delete any parameter
file nor any results it was created in previous runs. Please use with caution.
sal <- SPRproject(overwrite = TRUE)
Exploring workflow instances
systemPipeR provide several accessor methods and useful functions to explore
SYSargsList workflow object.
Accessor Methods
Several accessor methods are available that are named after the slot names of
the SYSargsList workflow object.
names(sal)
## [1] "stepsWF" "statusWF" "targetsWF" "outfiles"
## [5] "SE" "dependency" "targets_connection" "projectInfo"
## [9] "runInfo"
- Check the length of the workflow:
length(sal)
## [1] 4
- Check the steps of the workflow:
stepsWF(sal)
## $export_iris
## Instance of 'LineWise'
## Code Chunk length: 1
##
## $gzip
## Instance of 'SYSargs2':
## Slot names/accessors:
## targets: 3 (SE...VI), targetsheader: 1 (lines)
## modules: 0
## wf: 1, clt: 1, yamlinput: 4 (inputs)
## input: 3, output: 3
## cmdlist: 3
## Sub Steps:
## 1. gzip (rendered: TRUE)
##
##
##
## $gunzip
## Instance of 'SYSargs2':
## Slot names/accessors:
## targets: 3 (SE...VI), targetsheader: 1 (lines)
## modules: 0
## wf: 1, clt: 1, yamlinput: 4 (inputs)
## input: 3, output: 3
## cmdlist: 3
## Sub Steps:
## 1. gunzip (rendered: TRUE)
##
##
##
## $iris_stats
## Instance of 'LineWise'
## Code Chunk length: 5
- Checking the command-line for each target sample:
cmdlist() method printing the system commands for running command-line
software as specified by a given *.cwl file combined with the paths to the
input samples (e.g. FASTQ files) provided by a targets file. The example below
shows the cmdlist() output for running gzip and gunzip on the first sample.
Evaluating the output of cmdlist() can be very helpful for designing
and debugging *.cwl files of new command-line software or changing the
parameter settings of existing ones.
cmdlist(sal, step = c(2,3), targets = 1)
## $gzip
## $gzip$SE
## $gzip$SE$gzip
## [1] "gzip -c results/setosa.csv > results/SE.csv.gz"
##
##
##
## $gunzip
## $gunzip$SE
## $gunzip$SE$gunzip
## [1] "gunzip -c ./results/gzip/SE.csv.gz > results/SE.csv"
- Check the workflow status:
statusWF(sal)
## $export_iris
## DataFrame with 1 row and 2 columns
## Step Status
## <character> <character>
## 1 export_iris Success
##
## $gzip
## DataFrame with 3 rows and 5 columns
## Targets Total_Files Existing_Files Missing_Files gzip
## <character> <numeric> <numeric> <numeric> <matrix>
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## $gunzip
## DataFrame with 3 rows and 5 columns
## Targets Total_Files Existing_Files Missing_Files gunzip
## <character> <numeric> <numeric> <numeric> <matrix>
## SE SE 1 1 0 Success
## VE VE 1 1 0 Success
## VI VI 1 1 0 Success
##
## $iris_stats
## DataFrame with 1 row and 2 columns
## Step Status
## <character> <character>
## 1 iris_stats Success
- Check the workflow targets files:
targetsWF(sal[2])
## $gzip
## DataFrame with 3 rows and 2 columns
## FileName SampleName
## <character> <character>
## SE results/setosa.csv SE
## VE results/versicolor.csv VE
## VI results/virginica.csv VI
- Checking the expected outfiles files:
The outfiles components of SYSargsList define the expected outfiles files
for each step in the workflow, some of which are the input for the next workflow step.
outfiles(sal[2])
## $gzip
## DataFrame with 3 rows and 1 column
## gzip_file
## <character>
## 1 ./results/gzip/SE.cs..
## 2 ./results/gzip/VE.cs..
## 3 ./results/gzip/VI.cs..
- Check the workflow dependencies:
dependency(sal)
## $export_iris
## [1] NA
##
## $gzip
## [1] "export_iris"
##
## $gunzip
## [1] "gzip"
##
## $iris_stats
## [1] "gzip"
- Check the sample comparisons:
Sample comparisons are defined in the header lines of the targets file
starting with ‘# <CMP>’. This information can be accessed as follows:
targetsheader(sal, step = "Quality")
- Get the workflow steps names:
stepName(sal)
## [1] "export_iris" "gzip" "gunzip" "iris_stats"
- Get the Sample Id for on particular step:
SampleName(sal, step = "gzip")
## [1] "SE" "VE" "VI"
SampleName(sal, step = "iris_stats")
## This step doesn't contain multiple samples.
- Get the
outfiles or targets column files:
getColumn(sal, "outfiles", step = "gzip", column = "gzip_file")
## SE VE VI
## "./results/gzip/SE.csv.gz" "./results/gzip/VE.csv.gz" "./results/gzip/VI.csv.gz"
getColumn(sal, "targetsWF", step = "gzip", column = "FileName")
## SE VE VI
## "results/setosa.csv" "results/versicolor.csv" "results/virginica.csv"
- Get the R code for a
LineWise step:
codeLine(sal, step = "export_iris")
## export_iris
## mapply(function(x, y) write.csv(x, y), split(iris, factor(iris$Species)), file.path("results", paste0(names(split(iris, factor(iris$Species))), ".csv")))
- View all the objects in the running environment:
viewEnvir(sal)
## <environment: 0x563667e898e8>
## [1] "df" "plot" "stats"
- Copy one or multiple objects from the running environment to a new environment:
copyEnvir(sal, list = c("plot"), new.env = globalenv(), silent = FALSE)
## <environment: 0x563667e898e8>
## Copying to 'new.env':
## plot
yamlinput(sal, step = "gzip")
## $file
## $file$class
## [1] "File"
##
## $file$path
## [1] "_FILE_PATH_"
##
##
## $SampleName
## [1] "_SampleName_"
##
## $ext
## [1] "csv.gz"
##
## $results_path
## $results_path$class
## [1] "Directory"
##
## $results_path$path
## [1] "./results"
Subsetting the workflow details
- The
SYSargsList class and its subsetting operator [:
sal[1]
## Instance of 'SYSargsList':
## WF Steps:
## 1. export_iris --> Status: Success
##
sal[1:3]
## Instance of 'SYSargsList':
## WF Steps:
## 1. export_iris --> Status: Success
## 2. gzip --> Status: Success
## Total Files: 3 | Existing: 3 | Missing: 0
## 2.1. gzip
## cmdlist: 3 | Success: 3
## 3. gunzip --> Status: Success
## Total Files: 3 | Existing: 3 | Missing: 0
## 3.1. gunzip
## cmdlist: 3 | Success: 3
##
sal[c(1,3)]
## Instance of 'SYSargsList':
## WF Steps:
## 1. export_iris --> Status: Success
## 2. gunzip --> Status: Success
## Total Files: 3 | Existing: 3 | Missing: 0
## 2.1. gunzip
## cmdlist: 3 | Success: 3
##
- The
SYSargsList class and its subsetting by steps and input samples:
sal_sub <- subset(sal, subset_steps = c( 2,3), input_targets = ("SE"), keep_steps = TRUE)
stepsWF(sal_sub)
## $export_iris
## Instance of 'LineWise'
## Code Chunk length: 1
##
## $gzip
## Instance of 'SYSargs2':
## Slot names/accessors:
## targets: 1 (SE...SE), targetsheader: 1 (lines)
## modules: 0
## wf: 1, clt: 1, yamlinput: 4 (inputs)
## input: 1, output: 1
## cmdlist: 1
## Sub Steps:
## 1. gzip (rendered: TRUE)
##
##
##
## $gunzip
## Instance of 'SYSargs2':
## Slot names/accessors:
## targets: 1 (SE...SE), targetsheader: 1 (lines)
## modules: 0
## wf: 1, clt: 1, yamlinput: 4 (inputs)
## input: 1, output: 1
## cmdlist: 1
## Sub Steps:
## 1. gunzip (rendered: TRUE)
##
##
##
## $iris_stats
## Instance of 'LineWise'
## Code Chunk length: 5
targetsWF(sal_sub)
## $export_iris
## DataFrame with 0 rows and 0 columns
##
## $gzip
## DataFrame with 1 row and 2 columns
## FileName SampleName
## <character> <character>
## SE results/setosa.csv SE
##
## $gunzip
## DataFrame with 1 row and 2 columns
## gzip_file SampleName
## <character> <character>
## SE ./results/gzip/SE.cs.. SE
##
## $iris_stats
## DataFrame with 0 rows and 0 columns
outfiles(sal_sub)
## $export_iris
## DataFrame with 0 rows and 0 columns
##
## $gzip
## DataFrame with 1 row and 1 column
## gzip_file
## <character>
## 1 ./results/gzip/SE.cs..
##
## $gunzip
## DataFrame with 1 row and 1 column
## gunzip_file
## <character>
## 1 ./results/gunzip/SE...
##
## $iris_stats
## DataFrame with 0 rows and 0 columns
- The
SYSargsList class and its operator +
sal[1] + sal[2] + sal[3]
Replacement Methods
- Update a
input parameter in the workflow
sal_c <- sal
## check values
yamlinput(sal_c, step = "gzip")
## $file
## $file$class
## [1] "File"
##
## $file$path
## [1] "_FILE_PATH_"
##
##
## $SampleName
## [1] "_SampleName_"
##
## $ext
## [1] "csv.gz"
##
## $results_path
## $results_path$class
## [1] "Directory"
##
## $results_path$path
## [1] "./results"
## check on command-line
cmdlist(sal_c, step = "gzip", targets = 1)
## $gzip
## $gzip$SE
## $gzip$SE$gzip
## [1] "gzip -c results/setosa.csv > results/SE.csv.gz"
## Replace
yamlinput(sal_c, step = "gzip", paramName = "ext") <- "txt.gz"
## check NEW values
yamlinput(sal_c, step = "gzip")
## $file
## $file$class
## [1] "File"
##
## $file$path
## [1] "_FILE_PATH_"
##
##
## $SampleName
## [1] "_SampleName_"
##
## $ext
## [1] "txt.gz"
##
## $results_path
## $results_path$class
## [1] "Directory"
##
## $results_path$path
## [1] "./results"
## Check on command-line
cmdlist(sal_c, step = "gzip", targets = 1)
## $gzip
## $gzip$SE
## $gzip$SE$gzip
## [1] "gzip -c results/setosa.csv > results/SE.txt.gz"
- Append and Replacement methods for R Code Steps
appendCodeLine(sal_c, step = "export_iris", after = 1) <- "log_cal_100 <- log(100)"
codeLine(sal_c, step = "export_iris")
## export_iris
## mapply(function(x, y) write.csv(x, y), split(iris, factor(iris$Species)), file.path("results", paste0(names(split(iris, factor(iris$Species))), ".csv")))
## log_cal_100 <- log(100)
replaceCodeLine(sal_c, step="export_iris", line = 2) <- LineWise(code={
log_cal_100 <- log(50)
})
codeLine(sal_c, step = 1)
## export_iris
## mapply(function(x, y) write.csv(x, y), split(iris, factor(iris$Species)), file.path("results", paste0(names(split(iris, factor(iris$Species))), ".csv")))
## log_cal_100 <- log(50)
For more details about the LineWise class, please see below.
renameStep(sal_c, step = 1) <- "newStep"
renameStep(sal_c, c(1, 2)) <- c("newStep2", "newIndex")
sal_c
## Instance of 'SYSargsList':
## WF Steps:
## 1. newStep2 --> Status: Success
## 2. newIndex --> Status: Success
## Total Files: 3 | Existing: 3 | Missing: 0
## 2.1. gzip
## cmdlist: 3 | Success: 3
## 3. gunzip --> Status: Success
## Total Files: 3 | Existing: 3 | Missing: 0
## 3.1. gunzip
## cmdlist: 3 | Success: 3
## 4. iris_stats --> Status: Success
##
names(outfiles(sal_c))
## [1] "newStep2" "newIndex" "gunzip" "iris_stats"
names(targetsWF(sal_c))
## [1] "newStep2" "newIndex" "gunzip" "iris_stats"
dependency(sal_c)
## $newStep2
## [1] NA
##
## $newIndex
## [1] "newStep2"
##
## $gunzip
## [1] "newIndex"
##
## $iris_stats
## [1] "newIndex"
sal_test <- sal[c(1,2)]
replaceStep(sal_test, step = 1, step_name = "gunzip" ) <- sal[3]
sal_test
Note: Please use this method with attention, because it can disrupt all
the dependency graphs.
sal_test <- sal[-2]
sal_test
## Instance of 'SYSargsList':
## WF Steps:
## 1. export_iris --> Status: Success
## 2. gunzip --> Status: Success
## Total Files: 3 | Existing: 3 | Missing: 0
## 2.1. gunzip
## cmdlist: 3 | Success: 3
## 3. iris_stats --> Status: Success
##
CWL syntax
This section will introduce how CWL describes command-line tools and the
specification and terminology of each file.
For complete documentation, please check the CommandLineTools documentation here
and here for Workflows and the user guide here.
CWL command-line specifications are written in YAML format.
In CWL, files with the extension .cwl define the parameters of a chosen
command-line step or workflow, while files with the extension .yml define
the input variables of command-line steps.
CWL Workflow
Workflow class in CWL is defined by multiple process steps, where can have
interdependencies between the steps, and the output for one step can be used as
input in the further steps.
cwl.wf <- yaml::read_yaml(file.path(dir_path, "example/workflow_example.cwl"))
- The
cwlVersion component shows the CWL specification version used by the document.
- The
class component shows this document describes a Workflow.
cwl.wf[1:2]
## $class
## [1] "Workflow"
##
## $cwlVersion
## [1] "v1.0"
- The
inputs section describes the inputs of the workflow.
cwl.wf[3]
## $inputs
## $inputs$message
## [1] "string"
##
## $inputs$SampleName
## [1] "string"
##
## $inputs$results_path
## [1] "Directory"
- The
outputs section describes the outputs of the workflow.
cwl.wf[4]
## $outputs
## $outputs$string
## $outputs$string$outputSource
## [1] "echo/string"
##
## $outputs$string$type
## [1] "stdout"
- The
steps section describes the steps of the workflow. In this simple example,
we demonstrate one step.
cwl.wf[5]
## $steps
## $steps$echo
## $steps$echo$`in`
## $steps$echo$`in`$message
## [1] "message"
##
## $steps$echo$`in`$SampleName
## [1] "SampleName"
##
## $steps$echo$`in`$results_path
## [1] "results_path"
##
##
## $steps$echo$out
## [1] "[string]"
##
## $steps$echo$run
## [1] "example/example.cwl"
How to connect CWL description files within systemPipeR
This section will demonstrate how to connect CWL parameters files to create
workflows. In addition, we will show how the workflow can be easily scalable
with systemPipeR.
SYSargsList container stores all the information and instructions needed for processing
a set of input files with a single or many command-line steps within a workflow
(i.e. several components of the software or several independent software tools).
The SYSargsList object is created and fully populated with the SYSargsList construct
function.
Full documentation of SYSargsList management instances can be found here
and here.
The following imports a .cwl file (here example.cwl) for running the echo Hello World!
example.
HW <- SYSargsList(wf_file = "example/workflow_example.cwl",
input_file = "example/example_single.yml",
dir_path = system.file("extdata/cwl", package = "systemPipeR"))
HW
## Instance of 'SYSargsList':
## WF Steps:
## 1. Step_x --> Status: Pending
## Total Files: 1 | Existing: 0 | Missing: 1
## 1.1. echo
## cmdlist: 1 | Pending: 1
##
cmdlist(HW)
## $Step_x
## $Step_x$defaultid
## $Step_x$defaultid$echo
## [1] "echo Hello World! > results/M1.txt"
However, we are limited to run just one command-line or one sample in this example.
To scale the command-line over many samples, a simple solution offered by systemPipeR
is to provide a variable for each of the parameters that we want to run with multiple samples.
Let’s explore the example:
yml <- yaml::read_yaml(file.path(dir_path, "example/example.yml"))
yml
## $message
## [1] "_STRING_"
##
## $SampleName
## [1] "_SAMPLE_"
##
## $results_path
## $results_path$class
## [1] "Directory"
##
## $results_path$path
## [1] "./results"
For the message and SampleName parameter, we are passing a variable connecting
with a third file called targets.
Now, let’s explore the targets file structure:
targetspath <- system.file("extdata/cwl/example/targets_example.txt", package = "systemPipeR")
read.delim(targetspath, comment.char = "#")
## Message SampleName
## 1 Hello World! M1
## 2 Hello USA! M2
## 3 Hello Bioconductor! M3
The targets file defines all input files or values and sample ids of an analysis workflow.
For this example, we have defined a string message for the echo command-line tool,
in the first column that will be evaluated, and the second column is the
SampleName id for each one of the messages.
Any number of additional columns can be added as needed.
Users should note here, the usage of targets files is optional when using
systemPipeR's new CWL interface. Since for organizing experimental variables targets
files are extremely useful and user-friendly. Thus, we encourage users to keep using them.
How to connect the parameter files and targets file information?
The constructor function creates an SYSargsList S4 class object connecting three input files:
- CWL command-line specification file (
wf_file argument);
- Input variables (
input_file argument);
- Targets file (
targets argument).
As demonstrated above, the latter is optional for workflow steps lacking input files.
The connection between input variables (here defined by input_file argument)
and the targets file are defined under the inputvars argument.
A named vector is required, where each element name needs to match with column
names in the targets file, and the value must match the names of the .yml
variables. This is used to replace the CWL variable and construct all the command-line
for that particular step.
The variable pattern _XXXX_ is used to distinguish CWL variables that target
columns will replace. This pattern is recommended for consistency and easy identification
but not enforced.
The following imports a .cwl file (same example demonstrated above) for running
the echo Hello World example. However, now we are connecting the variable defined
on the .yml file with the targets file inputs.
HW_mul <- SYSargsList(step_name = "echo",
targets=targetspath,
wf_file="example/workflow_example.cwl", input_file="example/example.yml",
dir_path = dir_path,
inputvars = c(Message = "_STRING_", SampleName = "_SAMPLE_"))
HW_mul
## Instance of 'SYSargsList':
## WF Steps:
## 1. echo --> Status: Pending
## Total Files: 3 | Existing: 0 | Missing: 3
## 1.1. echo
## cmdlist: 3 | Pending: 3
##
cmdlist(HW_mul)
## $echo
## $echo$M1
## $echo$M1$echo
## [1] "echo Hello World! > results/M1.txt"
##
##
## $echo$M2
## $echo$M2$echo
## [1] "echo Hello USA! > results/M2.txt"
##
##
## $echo$M3
## $echo$M3$echo
## [1] "echo Hello Bioconductor! > results/M3.txt"
## Warning in knitr::include_graphics(system.file("extdata/images", "SPR_CWL_hello.png", : It is
## highly recommended to use relative paths for images. You had absolute paths: "/tmp/RtmpbUOpjz/
## Rinst1ccd542002e128/systemPipeR/extdata/images/SPR_CWL_hello.png"
Creating the CWL param files from the command-line
Users need to define the command-line in a pseudo-bash script format:
command <- "
hisat2 \
-S <F, out: ./results/M1A.sam> \
-x <F: ./data/tair10.fasta> \
-k <int: 1> \
-min-intronlen <int: 30> \
-max-intronlen <int: 3000> \
-threads <int: 4> \
-U <F: ./data/SRR446027_1.fastq.gz>
"
Define prefix and defaults
First line is the base command. Each line is an argument with its default value.
For argument lines (starting from the second line), any word before the first
space with leading - or -- in each will be treated as a prefix, like -S or
--min. Any line without this first word will be treated as no prefix.
All defaults are placed inside <...>.
First argument is the input argument type. F for “File”, “int”, “string” are unchanged.
Optional: use the keyword out followed the type with a , comma separation to
indicate if this argument is also an CWL output.
Then, use : to separate keywords and default values, any non-space value after the :
will be treated as the default value.
If any argument has no default value, just a flag, like --verbose, there is no need to add any <...>
createParam Function
createParam function requires the string as defined above as an input.
First of all, the function will print the three components of the cwl file:
- BaseCommand: Specifies the program to execute.
- Inputs: Defines the input parameters of the process.
- Outputs: Defines the parameters representing the output of the process.
The four component is the original command-line.
If in interactive mode, the function will verify that everything is correct and
will ask you to proceed. Here, the user can answer “no” and provide more
information at the string level. Another question is to save the param created here.
If running the workflow in non-interactive mode, the createParam function will
consider “yes” and returning the container.
cmd <- createParam(command, writeParamFiles = FALSE)
## *****BaseCommand*****
## hisat2
## *****Inputs*****
## S:
## type: File
## preF: -S
## yml: ./results/M1A.sam
## x:
## type: File
## preF: -x
## yml: ./data/tair10.fasta
## k:
## type: int
## preF: -k
## yml: 1
## min-intronlen:
## type: int
## preF: -min-intronlen
## yml: 30
## max-intronlen:
## type: int
## preF: -max-intronlen
## yml: 3000
## threads:
## type: int
## preF: -threads
## yml: 4
## U:
## type: File
## preF: -U
## yml: ./data/SRR446027_1.fastq.gz
## *****Outputs*****
## output1:
## type: File
## value: ./results/M1A.sam
## *****Parsed raw command line*****
## hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz
If the user chooses not to save the param files on the above operation,
it can use the writeParamFiles function.
writeParamFiles(cmd, overwrite = TRUE)
How to access and edit param files
Print a component
printParam(cmd, position = "baseCommand") ## Print a baseCommand section
## *****BaseCommand*****
## hisat2
printParam(cmd, position = "outputs")
## *****Outputs*****
## output1:
## type: File
## value: ./results/M1A.sam
printParam(cmd, position = "inputs", index = 1:2) ## Print by index
## *****Inputs*****
## S:
## type: File
## preF: -S
## yml: ./results/M1A.sam
## x:
## type: File
## preF: -x
## yml: ./data/tair10.fasta
printParam(cmd, position = "inputs", index = -1:-2) ## Negative indexing printing to exclude certain indices in a position
## *****Inputs*****
## k:
## type: int
## preF: -k
## yml: 1
## min-intronlen:
## type: int
## preF: -min-intronlen
## yml: 30
## max-intronlen:
## type: int
## preF: -max-intronlen
## yml: 3000
## threads:
## type: int
## preF: -threads
## yml: 4
## U:
## type: File
## preF: -U
## yml: ./data/SRR446027_1.fastq.gz
Subsetting the command-line
cmd2 <- subsetParam(cmd, position = "inputs", index = 1:2, trim = TRUE)
## *****Inputs*****
## S:
## type: File
## preF: -S
## yml: ./results/M1A.sam
## x:
## type: File
## preF: -x
## yml: ./data/tair10.fasta
## *****Parsed raw command line*****
## hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta
cmdlist(cmd2)
## $defaultid
## $defaultid$hisat2
## [1] "hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta"
cmd2 <- subsetParam(cmd, position = "inputs", index = c("S", "x"), trim = TRUE)
## *****Inputs*****
## S:
## type: File
## preF: -S
## yml: ./results/M1A.sam
## x:
## type: File
## preF: -x
## yml: ./data/tair10.fasta
## *****Parsed raw command line*****
## hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta
cmdlist(cmd2)
## $defaultid
## $defaultid$hisat2
## [1] "hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta"
Replacing a existing argument in the command-line
cmd3 <- replaceParam(cmd, "base", index = 1, replace = list(baseCommand = "bwa"))
## Replacing baseCommand
## *****BaseCommand*****
## bwa
## *****Parsed raw command line*****
## bwa -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz
cmdlist(cmd3)
## $defaultid
## $defaultid$hisat2
## [1] "bwa -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz"
new_inputs <- new_inputs <- list(
"new_input1" = list(type = "File", preF="-b", yml ="myfile"),
"new_input2" = "-L <int: 4>"
)
cmd4 <- replaceParam(cmd, "inputs", index = 1:2, replace = new_inputs)
## Replacing inputs
## *****Inputs*****
## new_input1:
## type: File
## preF: -b
## yml: myfile
## new_input2:
## type: int
## preF: -L
## yml: 4
## k:
## type: int
## preF: -k
## yml: 1
## min-intronlen:
## type: int
## preF: -min-intronlen
## yml: 30
## max-intronlen:
## type: int
## preF: -max-intronlen
## yml: 3000
## threads:
## type: int
## preF: -threads
## yml: 4
## U:
## type: File
## preF: -U
## yml: ./data/SRR446027_1.fastq.gz
## *****Parsed raw command line*****
## hisat2 -b myfile -L 4 -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz
cmdlist(cmd4)
## $defaultid
## $defaultid$hisat2
## [1] "hisat2 -b myfile -L 4 -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz"
Adding new arguments
newIn <- new_inputs <- list(
"new_input1" = list(type = "File", preF="-b1", yml ="myfile1"),
"new_input2" = list(type = "File", preF="-b2", yml ="myfile2"),
"new_input3" = "-b3 <F: myfile3>"
)
cmd5 <- appendParam(cmd, "inputs", index = 1:2, append = new_inputs)
## Replacing inputs
## *****Inputs*****
## S:
## type: File
## preF: -S
## yml: ./results/M1A.sam
## x:
## type: File
## preF: -x
## yml: ./data/tair10.fasta
## k:
## type: int
## preF: -k
## yml: 1
## min-intronlen:
## type: int
## preF: -min-intronlen
## yml: 30
## max-intronlen:
## type: int
## preF: -max-intronlen
## yml: 3000
## threads:
## type: int
## preF: -threads
## yml: 4
## U:
## type: File
## preF: -U
## yml: ./data/SRR446027_1.fastq.gz
## new_input1:
## type: File
## preF: -b1
## yml: myfile1
## new_input2:
## type: File
## preF: -b2
## yml: myfile2
## new_input3:
## type: File
## preF: -b3
## yml: myfile3
## *****Parsed raw command line*****
## hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz -b1 myfile1 -b2 myfile2 -b3 myfile3
cmdlist(cmd5)
## $defaultid
## $defaultid$hisat2
## [1] "hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz -b1 myfile1 -b2 myfile2 -b3 myfile3"
cmd6 <- appendParam(cmd, "inputs", index = 1:2, after=0, append = new_inputs)
## Replacing inputs
## *****Inputs*****
## new_input1:
## type: File
## preF: -b1
## yml: myfile1
## new_input2:
## type: File
## preF: -b2
## yml: myfile2
## new_input3:
## type: File
## preF: -b3
## yml: myfile3
## S:
## type: File
## preF: -S
## yml: ./results/M1A.sam
## x:
## type: File
## preF: -x
## yml: ./data/tair10.fasta
## k:
## type: int
## preF: -k
## yml: 1
## min-intronlen:
## type: int
## preF: -min-intronlen
## yml: 30
## max-intronlen:
## type: int
## preF: -max-intronlen
## yml: 3000
## threads:
## type: int
## preF: -threads
## yml: 4
## U:
## type: File
## preF: -U
## yml: ./data/SRR446027_1.fastq.gz
## *****Parsed raw command line*****
## hisat2 -b1 myfile1 -b2 myfile2 -b3 myfile3 -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz
cmdlist(cmd6)
## $defaultid
## $defaultid$hisat2
## [1] "hisat2 -b1 myfile1 -b2 myfile2 -b3 myfile3 -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz"
Editing output param
new_outs <- list(
"sam_out" = "<F: $(inputs.results_path)/test.sam>"
)
cmd7 <- replaceParam(cmd, "outputs", index = 1, replace = new_outs)
## Replacing outputs
## *****Outputs*****
## sam_out:
## type: File
## value: $(inputs.results_path)/test.sam
## *****Parsed raw command line*****
## hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz
output(cmd7)
## $defaultid
## $defaultid$hisat2
## [1] "./results/test.sam"
Internal Check
cmd <- "
hisat2 \
-S <F, out: _SampleName_.sam> \
-x <F: ./data/tair10.fasta> \
-k <int: 1> \
-min-intronlen <int: 30> \
-max-intronlen <int: 3000> \
-threads <int: 4> \
-U <F: _FASTQ_PATH1_>
"
WF <- createParam(cmd, overwrite = TRUE, writeParamFiles = TRUE, confirm = TRUE)
## *****BaseCommand*****
## hisat2
## *****Inputs*****
## S:
## type: File
## preF: -S
## yml: _SampleName_.sam
## x:
## type: File
## preF: -x
## yml: ./data/tair10.fasta
## k:
## type: int
## preF: -k
## yml: 1
## min-intronlen:
## type: int
## preF: -min-intronlen
## yml: 30
## max-intronlen:
## type: int
## preF: -max-intronlen
## yml: 3000
## threads:
## type: int
## preF: -threads
## yml: 4
## U:
## type: File
## preF: -U
## yml: _FASTQ_PATH1_
## *****Outputs*****
## output1:
## type: File
## value: _SampleName_.sam
## *****Parsed raw command line*****
## hisat2 -S _SampleName_.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U _FASTQ_PATH1_
## Written content of 'commandLine' to file:
## param/cwl/hisat2/hisat2.cwl
## Written content of 'commandLine' to file:
## param/cwl/hisat2/hisat2.yml
targetspath <- system.file("extdata", "targets.txt", package = "systemPipeR")
WF_test <- loadWorkflow(targets = targetspath, wf_file="hisat2.cwl",
input_file="hisat2.yml", dir_path = "param/cwl/hisat2/")
WF_test <- renderWF(WF_test, inputvars = c(FileName = "_FASTQ_PATH1_"))
WF_test
## Instance of 'SYSargs2':
## Slot names/accessors:
## targets: 18 (M1A...V12B), targetsheader: 4 (lines)
## modules: 1
## wf: 0, clt: 1, yamlinput: 9 (inputs)
## input: 18, output: 18
## cmdlist: 18
## Sub Steps:
## 1. hisat2 (rendered: TRUE)
cmdlist(WF_test)[1:2]
## $M1A
## $M1A$hisat2
## [1] "hisat2 -S _SampleName_.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446027_1.fastq.gz"
##
##
## $M1B
## $M1B$hisat2
## [1] "hisat2 -S _SampleName_.sam -x ./data/tair10.fasta -k 1 -min-intronlen 30 -max-intronlen 3000 -threads 4 -U ./data/SRR446028_1.fastq.gz"
Visualize Workflow - Full details
Color and text
On the plot, different colors and numbers indicate different status. This information
can be found also in the plot legends.
Shapes:
- circular steps: pure R code steps
- rounded squares steps:
sysargs steps, steps that will invoke command-line calls
- blue colored steps and arrows: main branch (see main branch section below)
Step colors
- black: pending steps
- Green: successful steps
- Green: failed steps
Number and colors
There are 4 numbers in the second row of each step, separated by /
- First No.: number of passed samples
- Second No.: number of warning samples
- Third No.: number of erros samples
- Forth No.: number of total samples
Duration
This is shown after the sample information, as how long it took to run this step.
Units are a few seconds (s), some minutes (m), or some hours (h).
on hover
When the mouse is hovering on each step, detailed information will be displayed.
logging
The workflow steps will also become clickable if in_log = TRUE. This will create links
for each step that navigate to corresponding log section in the SPR
workflow log file. Normally this option
is handled by SPR log file generating function to create this plot on top of the log file,
so when a certain step is click, it will navigate to the detailed section down the page.
Here is only an example to demo how the plot can be clickable (will not navigate you to
anywhere). Visit this page to see a real example.
plotWF(sal, in_log = TRUE)
Plot Method
The default plotting method is svg. It means the plot is generated by svg embedding.
Sometimes certain browsers may not display svg correctly. In this case, the other option
is to use png to embed the plot. However, you will lose hovering, clicking and some
responsiveness (plot auto resizing ability) of the plot.
plotWF(sal, plot_method = "png")
Rstudio
By default, even if you are working inside Rstudio the plot is not displayed in Rstudio viewer.
This is because the workflow steps will be too small inside Rstudio viewer too
see the details. We recommend to view it in a larger space, so by default it will
open up your web browser to display it. You can enforce rstudio = TRUE to see it
in Rstudio Viewer.
plotWF(sal, rstudio = TRUE)
Responsiveness
This is a term often used in web development. It means will the plot resize itself
if the user resize the document window? By default, plotWF will be responsive,
meaning it will fit current window container size and adjust the size once the window
size has changed. To always display the full sized plot, use responsive = FALSE,
useful for embedding the plot in a full-screen mode.
plotWF(sal, responsive = FALSE)
For the plot above, you need to scroll to see the plot.
Layout
There a few different layout you can choose. There is no best layout. It all depends
on the workflow structure you have. The default is compact but we recommend you
to try different layouts to find the best fitting one.
compact: try to plot steps as close as possible.vertical: main branch will be placed vertically and side branches will be placed
on the same horizontal level and sub steps of side branches will be placed
vertically.horizontal: main branch is placed horizontally and side branches and sub
steps will be placed vertically.execution: a linear plot to show the workflow execution order of all steps.
vertical
plotWF(sal, layout = "vertical", height = "600px")
The plot is very long, use height to make it smaller.
horizontal
plotWF(sal, layout = "horizontal")
execution
plotWF(sal, layout = "execution", height = "600px", responsive = FALSE)
The plot is very long but if we use height to limit to a smaller size, details are
hard to see. Then it will be good to use height and responsive = FALSE together.
Main branch
From the plots above, you can that there are many steps which do not connect to any
other steps. These dead-ends are called ending steps. If we connect the first step,
steps in between and these ending step, this will become a branch. Imagine the workflow is
a upside-down tree structure and the root is the first step. Therefore, there are
many possible ways to connect the workflow. For the convenience of plotting, we
introduce a concept of “main branch”, meaning one of the possible connecting
strategies that will be placed at the center of the plot. Other steps that are not
in this major branch will surround this major space.
This main branch will not impact the compact layout so much but will have a huge
effect on horizontal and vertical layouts.
The plotting function has an algorithm that will automatically choose a best branch for
you by default. In simple words, it favors: a. branches that connect first and last step;
b. as long as possible.
You can also choose a branch you want by branch_method = "choose". It will first
list all possible branches, and then give you a prompt to ask for your favorite branch.
Here, for rendering the Rmarkdown, we cannot have a prompt, so we use a second argument
in combination, branch_no = x to directly choose a branch and skip the prompt. Also,
we use the verbose = TRUE to imitate the branch listing in console. In a real case,
you only need branch_method = "choose".
Watch closely how the plot change by choosing different branches. Here we use vertical
layout to demo. Remember, the main branch is marked in blue.
plotWF(sal, layout = "vertical", branch_method = "choose", branch_no = 1, verbose = FALSE)
Unmark main branch
The main branch concept may not represent the main workflow. It is introduced
for the convenience of plotting. Most times by auto detecting, it will find the
major steps in a workflows, sometimes it does not. It depends on how the users
design the workflow. If you think this is not a good representation, you can mute it
by mark_main_branch = FALSE. You will no longer see the blue-colored steps on
plot and on legends.
plotWF(sal, mark_main_branch = FALSE, height = "500px")
Legends
The legend can also be removed by show_legend = FALSE
plotWF(sal, show_legend = FALSE, height = "500px")
Inner Classes
SYSargsList steps are can be defined with two inner classes, SYSargs2 and
LineWise. Next, more details on both classes.
SYSargs2 Class
SYSargs2 workflow control class, an S4 class, is a list-like container where
each instance stores all the input/output paths and parameter components
required for a particular data analysis step. SYSargs2 instances are
generated by two constructor functions, loadWF and renderWF, using as data
input targets or yaml files as well as two cwl parameter files (for
details see below).
In CWL, files with the extension .cwl define the parameters of a chosen
command-line step or workflow, while files with the extension .yml define
the input variables of command-line steps. Note, input variables provided by a
targets file can be passed on to a SYSargs2 instance via the inputvars
argument of the renderWF function.
The following imports a .cwl file (here hisat2-mapping-se.cwl) for
running the short read aligner HISAT2 (Kim, Langmead, and Salzberg 2015). For more details about the
file structure and how to design or customize our own software tools, please
check systemPipeR and CWL pipeline.
The loadWF and renderWF functions render the proper command-line strings for
each sample and software tool.
library(systemPipeR)
targetspath <- system.file("extdata", "targets.txt", package = "systemPipeR")
dir_path <- system.file("extdata/cwl", package = "systemPipeR")
WF <- loadWF(targets = targetspath, wf_file = "hisat2/hisat2-mapping-se.cwl",
input_file = "hisat2/hisat2-mapping-se.yml",
dir_path = dir_path)
WF <- renderWF(WF, inputvars = c(FileName = "_FASTQ_PATH1_",
SampleName = "_SampleName_"))
Several accessor methods are available that are named after the slot names of
the SYSargs2 object.
names(WF)
## [1] "targets" "targetsheader" "modules" "wf"
## [5] "clt" "yamlinput" "cmdlist" "input"
## [9] "output" "files" "inputvars" "cmdToCwl"
## [13] "status" "internal_outfiles"
Of particular interest is the cmdlist() method. It constructs the system
commands for running command-line software as specified by a given .cwl file
combined with the paths to the input samples (e.g. FASTQ files) provided by a
targets file. The example below shows the cmdlist() output for running
HISAT2 on the first SE read sample. Evaluating the output of cmdlist() can
be very helpful for designing and debugging .cwl files of new command-line
software or changing the parameter settings of existing ones.
cmdlist(WF)[1]
## $M1A
## $M1A$`hisat2-mapping-se`
## [1] "hisat2 -S ./results/M1A.sam -x ./data/tair10.fasta -k 1 --min-intronlen 30 --max-intronlen 3000 -U ./data/SRR446027_1.fastq.gz --threads 4"
The output components of SYSargs2 define the expected output files for each
step in the workflow; some of which are the input for the next workflow step,
here next SYSargs2 instance.
output(WF)[1]
## $M1A
## $M1A$`hisat2-mapping-se`
## [1] "./results/M1A.sam"
The targets components of SYSargs2 object can be accessed by the targets
method. Here, for single-end (SE) samples, the structure of the targets file is
defined by:
FileName: specify the FASTQ files path;SampleName: Unique IDs for each sample;Factor: ID for each treatment or condition.
targets(WF)[1]
## $M1A
## $M1A$FileName
## [1] "./data/SRR446027_1.fastq.gz"
##
## $M1A$SampleName
## [1] "M1A"
##
## $M1A$Factor
## [1] "M1"
##
## $M1A$SampleLong
## [1] "Mock.1h.A"
##
## $M1A$Experiment
## [1] 1
##
## $M1A$Date
## [1] "23-Mar-2012"
as(WF, "DataFrame")
## DataFrame with 18 rows and 6 columns
## FileName SampleName Factor SampleLong Experiment Date
## <character> <character> <character> <character> <character> <character>
## 1 ./data/SRR446027_1.f.. M1A M1 Mock.1h.A 1 23-Mar-2012
## 2 ./data/SRR446028_1.f.. M1B M1 Mock.1h.B 1 23-Mar-2012
## 3 ./data/SRR446029_1.f.. A1A A1 Avr.1h.A 1 23-Mar-2012
## 4 ./data/SRR446030_1.f.. A1B A1 Avr.1h.B 1 23-Mar-2012
## 5 ./data/SRR446031_1.f.. V1A V1 Vir.1h.A 1 23-Mar-2012
## ... ... ... ... ... ... ...
## 14 ./data/SRR446040_1.f.. M12B M12 Mock.12h.B 1 23-Mar-2012
## 15 ./data/SRR446041_1.f.. A12A A12 Avr.12h.A 1 23-Mar-2012
## 16 ./data/SRR446042_1.f.. A12B A12 Avr.12h.B 1 23-Mar-2012
## 17 ./data/SRR446043_1.f.. V12A V12 Vir.12h.A 1 23-Mar-2012
## 18 ./data/SRR446044_1.f.. V12B V12 Vir.12h.B 1 23-Mar-2012
Please note, to work with custom data, users need to generate a targets file
containing the paths to their own FASTQ files and then provide under
targetspath the path to the corresponding targets file.
In addition, if the Environment Modules is
available, it is possible to define which module should be loaded, as shown
here:
modules(WF)
## module1
## "hisat2/2.1.0"
Additional information can be accessed, as the parameters files location and the
inputvars provided to generate the object.
files(WF)
inputvars(WF)
LineWise Class
LineWise was designed to store all the R code chunk when an RMarkdown file is
imported as a workflow.
rmd <- system.file("extdata", "spr_simple_lw.Rmd", package = "systemPipeR")
sal_lw <- SPRproject(overwrite = TRUE)
## Recreating directory '/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project/.SPRproject'
## Creating file '/tmp/RtmpbUOpjz/Rbuild1ccd5458e4ece6/systemPipeR/vignettes/spr_project/.SPRproject/SYSargsList.yml'
sal_lw <- importWF(sal_lw, rmd, verbose = FALSE)
codeLine(sal_lw)
## firstStep
## mapply(function(x, y) write.csv(x, y), split(iris, factor(iris$Species)), file.path("results", paste0(names(split(iris, factor(iris$Species))), ".csv")))
## secondStep
## setosa <- read.delim("results/setosa.csv", sep = ",")
## versicolor <- read.delim("results/versicolor.csv", sep = ",")
## virginica <- read.delim("results/virginica.csv", sep = ",")
- Coerce methods available:
lw <- stepsWF(sal_lw)[[2]]
## Coerce
ll <- as(lw, "list")
class(ll)
## [1] "list"
lw <- as(ll, "LineWise")
lw
## Instance of 'LineWise'
## Code Chunk length: 3
length(lw)
## [1] 3
names(lw)
## [1] "codeLine" "codeChunkStart" "stepName" "dependency" "status"
## [6] "files" "runInfo"
codeLine(lw)
## setosa <- read.delim("results/setosa.csv", sep = ",")
## versicolor <- read.delim("results/versicolor.csv", sep = ",")
## virginica <- read.delim("results/virginica.csv", sep = ",")
codeChunkStart(lw)
## integer(0)
rmdPath(lw)
## character(0)
l <- lw[2]
codeLine(l)
## versicolor <- read.delim("results/versicolor.csv", sep = ",")
l_sub <- lw[-2]
codeLine(l_sub)
## setosa <- read.delim("results/setosa.csv", sep = ",")
## virginica <- read.delim("results/virginica.csv", sep = ",")
replaceCodeLine(lw, line = 2) <- "5+5"
codeLine(lw)
## setosa <- read.delim("results/setosa.csv", sep = ",")
## 5 + 5
## virginica <- read.delim("results/virginica.csv", sep = ",")
appendCodeLine(lw, after = 0) <- "6+7"
codeLine(lw)
## 6 + 7
## setosa <- read.delim("results/setosa.csv", sep = ",")
## 5 + 5
## virginica <- read.delim("results/virginica.csv", sep = ",")
- Replacement methods for
SYSargsList
replaceCodeLine(sal_lw, step = 2, line = 2) <- LineWise(code={
"5+5"
})
codeLine(sal_lw, step = 2)
appendCodeLine(sal_lw, step = 2) <- "66+55"
codeLine(sal_lw, step = 2)
appendCodeLine(sal_lw, step = 1, after = 1) <- "66+55"
codeLine(sal_lw, step = 1)
Workflow design structure using SYSargs: Previous version
Instances of this S4 object class are constructed by the systemArgs function
from two simple tabular files: a targets file and a param file. The
latter is optional for workflow steps lacking command-line software. Typically,
a SYSargs instance stores all sample-level inputs as well as the paths to
the corresponding outputs generated by command-line- or R-based software
generating sample-level output files, such as read preprocessors
(trimmed/filtered FASTQ files), aligners (SAM/BAM files), variant callers
(VCF/BCF files) or peak callers (BED/WIG files). Each sample level input/output
operation uses its own SYSargs instance. The outpaths of SYSargs usually
define the sample inputs for the next SYSargs instance. This connectivity is
established by writing the outpaths with the writeTargetsout function to a
new targets file that serves as input to the next systemArgs call.
Typically, the user has to provide only the initial targets file. All
downstream targets files are generated automatically. By chaining several
SYSargs steps together one can construct complex workflows involving many
sample-level input/output file operations with any combination of command-line
or R-based software.
## Warning in knitr::include_graphics(system.file("extdata/images", "SystemPipeR_Workflow.png", : It
## is highly recommended to use relative paths for images. You had absolute paths: "/tmp/RtmpbUOpjz/
## Rinst1ccd542002e128/systemPipeR/extdata/images/SystemPipeR_Workflow.png"
Funding
This project is funded by NSF award ABI-1661152.
References
Amstutz, Peter, Michael R Crusoe, Nebojša Tijanić, Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, et al. 2016. “Common Workflow Language, V1.0,” July. https://doi.org/10.6084/m9.figshare.3115156.v2.
Kim, Daehwan, Ben Langmead, and Steven L Salzberg. 2015. “HISAT: A Fast Spliced Aligner with Low Memory Requirements.” Nat. Methods 12 (4): 357–60.
