Watchdog

The development of high-throughput experimental technologies, such as next-generation sequencing, have lead to new challenges for handling, analyzing and integrating the resulting large and diverse datasets. Bioinformatical analysis of these data commonly requires a number of mutually dependent steps applied to numerous samples for multiple conditions and replicates. To support these analyses, a number of workflow management systems (WMSs) have been developed to allow automated execution of corresponding analysis workflows. Major advantages of WMSs are the easy reproducibility of results as well as the reusability of workflows or their components.

Here, we present Watchdog, a WMS for the automated and distributed analysis of large-scale experimental data. Watchdog is implemented in Java and is thus platform-independent.

Main features include:

straightforward processing of replicate data
support for distributed computer systems
remote storage support
customizable error detection
manual intervention into workflow execution
a GUI for workflow construction using pre-defined modules
a helper script for creating new module definitions
no restriction to specific programming languages
provides a flexible plugin system for extending without modifying the original sources

New features in Watchdog 2.0:

[new] execution: execute only altered or unfinished tasks (resume mode)
[new] execution: detach from workflow execution and attach later on (detach/reattach mode)
[new] execution: use custom before and after command scripts
[new] execution: implementation of XML plugin supporting package managers and virtualizer
[new] execution: support of Conda package manager for module dependencies
[new] execution: support of Docker container
[new] graphical user interface for module creation
[new] documentation: standard documentation format for modules
[new] documentation: documentation template generator
[new] documentation: generation of module reference book
[new] reporting: module versioning
[new] reporting: retrieval of third-party software versions
[new] reporting: create report on workflow execution
[new] sharing: community platform for sharing of modules and workflows

Manual:
More information on all available features can be found in the manual.

Download:
Download Watchdog
Watchdog on github
Watchdog on anaconda
Watchdog on Docker hub

Sharing of Watchdog modules and workflows:
Watchdog community modules on github
Watchdog community workflows on github

Contact:
If you have any questions or suggestions, please feel free to contact Michael Kluge.

Publications:

Michael and Friedel Caroline C. Kluge, Watchdog - a workflow management system for the distributed analysis of large-scale experimental data, BMC Bioinformatics, vol. 19, no. 97, 2018.

Michael Kluge, Marie-Sophie Friedl, Amrei L Menzel, Caroline C. Friedel, Watchdog 2.0: New developments for reusability, reproducibility, and workflow execution, GigaScience, vol. 9, no. 6, 2020.