Collate and MantleBio share a mission for developing technologies that help bring life-saving innovations to patients faster. Together, they are building an AI-first platform that streamlines the life science product lifecycle from concept to market.

About MantleBio

Biotech companies generate vast amounts of complex data: a single DNA sequencing experiment can create 7 TB of files. However, the majority of biotech companies lacked the tools necessary to fully leverage this data.

Founded in 2023, MantleBio developed a modern data engineering platform purpose-built for life sciences. MantleBio's platform enabled research teams to organize and analyze complex biological data in one centralized location while integrating seamlessly with existing research tools like electronic lab notebooks, instruments, and databases. This infrastructure helped teams accelerate discovery by making critical research data more accessible and actionable across the entire organization.

Emily Damato and Madeline Schade founded MantleBio to make the best data engineering technology accessible to all life science research. They brought over a decade of experience in biotech and big tech, including software development at GRAIL, Google, ArsenalBio, MIT, and the Broad Institute.

Following a $5 million funding round led by Y Combinator and Initialized, MantleBio quickly scaled their team and product capabilities. Mantle's platform has helped scientists process complex multi-omics experiments, prepare data for machine learning, and accelerate critical analysis workflows. The platform enabled research teams to focus on discovery, transforming how life science teams perform data-driven research.

AI for Every Step of Life Sciences Innovation

At every step of research and development, life science companies generate tremendous amounts of data and documentation. This administrative burden creates bottlenecks that delay new technologies from reaching patients.

Recent advances in AI technology now make it possible to streamline these processes at scale. Collate leverages artificial intelligence to automate the extensive documentation burdens faced by life science companies. Organizations developing drugs and medical devices require vast amounts of paperwork across their entire operations—from initial research through clinical trials, manufacturing, and regulatory submissions. By applying generative AI to these workflows, companies can significantly reduce manual documentation work and accelerate the timeline for bringing critical treatments to market.

MantleBio's team brings expertise in developing software specifically for life sciences, with deep understanding of the unique challenges that R&D and clinical teams face daily. Their experience building data infrastructure for complex biological research, combined with years working directly with scientists and regulatory requirements, makes them ideal partners in advancing Collate's mission.

Together, we're building AI solutions that understand both the science and the compliance needs of modern life sciences development, accelerating the path from discovery to patients.

Thank You

Thank you to our partners, investors, team, and the scientific community who pursued this mission with us. Together, we will continue to develop technology that helps bring life science innovations to the patients who need them.

Please stay in touch,

Emily Damato

emily@mantlebio.com

Madeline Schade

madeline@mantlebio.com

Understanding the problem

The first step in the design process is to clearly understand the problem. Scientific research generates vast amounts of unstructured data. Computational biologists and bioinformaticians  create pipelines to process these types of data, and scientists run these pipelines to analyze their research. Bench scientists and computational biologists may be on the same team or on different teams.

User research

At Mantle, our user research involved shadowing scientists working in the lab, allowing us to understand their workflows and challenges firsthand. By asking scientists to walk us through their processes, we gained valuable insights. Our team’s strong science background played a crucial role in these visits by facilitating in-depth discussions.

A key finding was that scientists generate data of different types (using different instruments or assays) for different samples, and they prefer to group data by sample or sample property. For example, instead of categorizing by data type (e.g. all flow cytometry data), they organized datasets related to a specific sample, which could involve multiple data types (e.g. sample 123’s flow cytometry, sequencing, and ELISA data).This insight helped us design a data management system that aligns with their natural organizational habits, making their workflows more intuitive and efficient.

Define the problem and ideate

After brainstorming with the team, we identified critical "how might we" questions that guided our design process:

• How might we make searching and filtering for datasets fast and intuitive?

• How might we make pipeline versioning and releases easy?

• How might we better communicate the science and progress to biotech leaders?

These questions helped us define the core problems: scientists struggle with unstructured data and need an intuitive way to search and filter datasets. They also require an easy method for pipeline versioning and releases. Their leaders need clearer communication of scientific progress. Addressing these issues became our primary goal as we began ideating solutions to enhance data management, streamline workflows, and improve overall efficiency in the lab.

Design and prototype

Working at a fast-paced startup like Mantle, our approach is to learn rapidly by releasing products and gathering feedback directly from our users. We don't have the luxury of conducting in-depth user studies, creating detailed user journeys or spending a lot of time on medium fidelity prototypes. Thus, we prioritize solutions that meet the minimum viable requirements, are easy and quick to build, and can be launched as fast as possible.

Instead, here are some solutions we focused on that target each of our personas.

Once we settled on promising ideas, I created high-fidelity prototypes on Figma.

Data management

Goals:

• Easily search and filter by creator, usage, and modification date.

• Save frequently used filters for quick access.

• Identify and organize datasets by type (e.g., Flow, Sequencing, Image).

• View IDs, names, creators, and last modified dates.

• Show additional dataset properties for more detailed information.

• Quickly upload new datasets.

• Star important datasets or archive inactive ones.

Pipeline versioning

Goals:

• Easily switch between different pipeline versions from a dropdown menu.

• View and manage all pipeline runs, including details such as run ID, name, date, status, creator, and updater.

• Quickly see the status of each run (e.g., running, completed, queued, error).

• Access detailed information about the current pipeline version, including repository name, GitHub URL, config info, and description.

Import templates and create new runs with the click of a button.

Dashboard

Goals:

• Displays key pipelines with their last update, status indicators, and quick access to all pipelines.

• Lists data types with a tabbed interface for quick navigation, showing recent dataset entries and creation dates.

• Easy access to create a pipeline run, upload datasets, import data types, and import pipelines. 

• Lists recent analyses with creation dates and quick links to detailed views.

• Overview of ongoing and successful runs, providing a snapshot of current activity.

Implementation

As both a designer and a coder, I am aware of the challenges that arise during the implementation phase and keep them in mind when designing. At Mantle, we leverage Ant Design for our UI components. Although Ant Design has some design limitations, it allows us to implement features rapidly, which is crucial in our fast-paced environment.

To further enhance our workflow, I utilize ChatGPT for problem-solving and ideation. For those interested in improving their prompt engineering skills, consider exploring dedicated courses. 

My approach involves setting the context for ChatGPT as an expert in the relevant field and then posing specific questions related to that expertise. This method has been instrumental in quickly overcoming complex challenges and accelerating our development process.

Continuous improvement

The design journey never truly ends. We plan to conduct further usability tests, iterate on our designs, and uncover new user insights. Continuous improvement will help us stay aligned with the evolving needs of scientists and computational biologists, ultimately driving more impactful scientific discoveries. Our journey is just beginning, and we are excited to continue refining and expanding our platform to support the future of biotech.

If you’d like to try Mantle, sign up for a free account here.

But is the pipeline reliable? Will it be easy to update? Is it efficient? In this article, we will be covering further considerations for creating a high-quality production pipeline.

Introduction

As a bioinformatics engineer, a typical day often involves getting a new set of data from the lab, determining what insights you need to make, producing the code to generate the insights, and presenting metrics, graphs, and results to the rest of the team. Since you are constantly working with new data for new experiments, you must adapt to changes and produce results that reflect the data and meaningfully impact the next experiment.

At some point, some experimentation may solidify around an essential process. Whether you do Nanopore sequencing every week or are using a standardized flow cytometry analyte panel daily, you realize you are running some of the same processes every time. This is when (as described in Part 1) you are going from a hike to the road, so now you have to build the road!

In this article, we will discuss some of the best practices for developing production pipelines your teammates will trust and you can easily maintain.

When and when not to consider a Pipeline

As a recap from part 1 of this series, creating a production-ready pipeline is not always the right direction for the given task. Creating a reliable and scalable pipeline takes time. If you spend time working on this and run the pipeline only once, running the code in a Jupyter notebook would have been faster and easier.

Here are a couple of reasons you might consider productionizing a pipeline:

• The size of the data is massive

  If you work with single cell sequencing, you know that the preprocessing from BCL or FASTQ file to a count matrix is done through sets of standard pipelines, no matter how many times you run it. This is because even for the smallest datasets, the computing time can take hours and each dataset can produce GBs to TBs of data. 

• It is run by multiple different people

  If you need to share the code and environment with many different people, it's worth considering creating a standard pipeline. This will allow everyone to run the same version in the same environment, no matter who is running it, allowing for effective comparison of results.

What is a WDL?

A Workflow Definition Language (WDL) is a specialized scripting language that defines, manages, and automates complex computational workflows. These languages provide a standardized, human-readable format for specifying the sequence of tasks, dependencies, inputs, and outputs involved in data processing pipelines. 

The syntax of a WDL is typically intuitive and straightforward. This readability promotes collaboration and ensures that workflows are easy to understand, share, and modify. By explicitly defining each component of a workflow, WDLs help eliminate ambiguity, ensuring precise and consistent execution.

WDLs support parallel execution and efficient resource management, allowing workflows to scale effectively across various computing environments, such as local clusters, cloud platforms, or high-performance computing (HPC) systems. Additionally, WDLs can provide enhanced reproducibility. Putting your script in a git repository may allow you to track which version of the code you ran, but do you know what environment it ran in? What versions of the packages you had installed? A workflow definition language can track both the code and the environment.

Picking a WDL

There are many different workflow management systems out there that were developed for specific use cases—so many that it’s almost overwhelming. A long list of workflow languages can be found here. Its even not unheard of for a company to build its own in house; for example, GRAIL has reflow and Insitro has redun.

The most commonly used WDLs in bioinformatics are Airflow, Nextflow, Snakemake, and Cromwell.

There are a few things to consider when picking which WDL to use:

• Architecture support

  Different languages work best for running on cloud vs. on HPC clusters, with differing levels of support for cloud infrastructure and containerization.

• Scalability

  Understanding how the language will scale to more samples and larger samples.

• Community support

  All of these tools are open source. Picking one that is actively maintained and well-documented can ease a lot of pressure. This can also be a good indicator of the ability to find engineers with experience.

• Flexibility

  Bioinformatics tools require many languages and must be run in multiple locations. The flexibility to run different workflows can be a huge advantage.

Nextflow and Snakemake are growing in popularity and are heavily supported by the bioinformatics community. Because of this, they are generally the best choice if you are setting up a new tech stack.

Scaling your pipeline

Clarifying the code

When writing bioinformatics analyses, you often end up with a long script that does many things. The first step to productionizing and scaling your script is to clean the code to a structure ready for scale.

Below are some tricks we have used over the years to help with that process.

1. Removing “magic numbers” / creating arguments

In software development, a “magic number” means using a value (a number or a string) within your code without specifying its meaning. These are often an indication that something is a variable within your code. Collecting each of these and assigning a name to the variable at the top of your script can provide clarity to the reader.

For example, see this snippet:

def main():
  # ...
  imgs = ["input_images/img1.png","input_images/img1.png"]
  for img in imgs:
    # ...
    img = skimage.io.imread("input_images/img1.png")
    norm = np.zeros_like(img)
    cv2.normalize(img, norm, 0, 255, cv2.NORM_MINMAX)
    # ...

We can convert this to the following, adding the image as an input and the 0, 255 as constant values.

# Constants
NORM_MIN = 0
NORM_MAX = 255
DEFAULT_NORM_TYPE = cv2.NORM_MINMAX

def main():

  # Arguments.
  parser.add_argument('input_dir', nargs='+',
                      help='List of input image paths')

  for img_path in args.input_imgs:
    # ...
    img = skimage.io.imread(img_path)
    norm = np.zeros_like(img)
    cv2.normalize(img, norm, norm_min, norm_max, norm_type)
    # ...

This is particularly critical in life science, where a “magic number” might have been difficult to determine and may dramatically impact results. Is it a metric from a publication? Which one? Is it something you experimentally tuned over time? Is it the value from a control from a different experiment? Documenting this well will help you update this code if the number changes and will reduce the chance of a teammate misinterpreting the number and introducing a bug.

2. Single responsibility principle

Each function should perform a single action. For example, from the code above, we can split to have a function for processing inputs and a function for evaluating the model:

def process_input_dir(input_dir):
  input_image_list = glob.glob(os.path.join(args.input_image_dir, "*"))
  input_image_list.sort()
  return [io.imread(f) for f in input_image_list]


def eval_model(imgs, cyto_channel, nucl_channel):
  channels = [args.cyto_channel, args.nucl_channel]
  return model.eval(imgs, diameter=None, channels=channels)


def main():
  # argparsing stays in main
  parser.add_argument(
        'input_image_dir',
        type=str,
        help="Input image directory.")

  parser.add_argument(
          '-c',
          '--cytoplasm',
          dest='cyto_channel',
          type=int,
          help="Integer representing the cytoplasm channel. Grayscale=0, R=1, G=2, B=3.")

  parser.add_argument(
          '-n',
          '--nucleus',
          dest='nucl_channel',
          type=int,
          help="Integer representing the nucleus channel. None=0, Grayscale=0, R=1, G=2, B=3.")

  imgs = process_input_dir()

  masks, flows, styles, diams = eval_model(imgs, args.cyto_channel,
  args.nucl_channel)

3. Function within loops

If you perform the same action within a loop, extract the code into its own function:

def process_img(img_path):
  # ...
  img = skimage.io.imread(img_path)
  norm = np.zeros_like(img)
  cv2.normalize(img, norm, norm_min, norm_max, norm_type)
  # ...


def main():
  # Arguments.
  parser.add_argument('input_dir', nargs='+',
                      help='List of input image paths')

  for img_path in args.input_imgs:
    process_img(img_path)

4. Repetitive code

 Create a function for any piece of code that is repeated.

def apply_threshold(data, column, threshold):
  return data[data[column] < threshold]

df = apply_threshold(data, 'p_value', 0.05)
df = apply_threshold(data, 'score', 0.1)

Documentation and version control

Having well-documented code allows for the next person who edits the code to quickly learn the code and provide improvements. Each function should be a verb and describe in simple terms what it does, with a comment at the top describing the inputs and outputs.

Storing each pipeline in its own Git repository allows for a clear separation of concerns. Adding a README.md to the directory's base with instructions on running and expected inputs and outputs can also be helpful.

Parallelization

Parallelizing a bioinformatics pipeline enhances performance by distributing tasks across multiple processors or machines, which is essential for large datasets and time-sensitive projects.

Parallelization in software can take many different forms:

• Task level: Processing multiple tasks in parallel. This is common within data engineering workflows, where data flows into a system that must process the data for many different use cases.

• Data level: Processing the same task on different pieces of data in parallel. This is common when you need to process large data all at once.

Data level parallelism is the most common use case in bioinformatics, where you get in hundreds of samples at once and must pre-process them to an analyzable state simultaneously.

The following Nextflow workflow takes in a channel of FASTQ files, grouping them by read 1 and 2 and processing them through PROCESS_A and PROCESS_B simultaneously. The workflow will fan out these steps to the maximum allowed compute.

// Create a channel for input FASTQ files
Channel
  .fromFilePairs("${params.inputDir}/*_{1,2}.fastq.gz")
  .set {read_pairs_ch}

workflow {
  PROCESS_A(read_pairs_ch)

  PROCESS_B(read_pair_ch)

  PROCESS_C(PROCESS_A.out)
}

In this example, PROCESS_A will run in parallel for each pair of input FASTQ files, potentially processing multiple samples simultaneously.

Conclusion

This is the final article in our Intro to Bioinformatics series. We’ve covered from a high level of what it means to be a bioinformatics engineer to practical tips on containerization and building robust pipelines.

We hope you have enjoyed and learned something to take away into your next project!

Subscribe now

Madeline Schade is the CTO and Co-Founder of Mantle. Her favorite organism is Carcharodon carcharias.

You have great Jupyter notebooks you reuse constantly for data analysis…but which version did you use to make the graph in last month’s presentation? How will you reprocess the last 20 datasets with your newest version? And how can your teammates use this algorithm for their datasets? Using a pipeline can solve these problems – here’s how to get your notebook code into a Nextflow pipeline.

Introduction

If you’ve analyzed data using Python, you’ve probably used Jupyter notebooks. When you get the very first pieces of data from your initial experiments for a new project, a Jupyter notebook lets you explore and iterate quickly. As you refine your experiments and the resulting data gets more standardized, you might come up with a definitive Jupyter notebook that you use over and over with different input files. You’ve gone from bushwhacking to following a paved trail with signposts and a map. This is great! You don’t need to put in a ton of new effort every time you get new data.

As your notebook becomes a crucial part of your workflow and your colleagues’ workflows, you might start to notice a few issues creeping in. You make some improvements to your algorithm. But now what version of the notebook you used to make past graphs is not documented anywhere. Or, you want to run the new version of the analysis on the last 20 datasets, and it’d be way quicker if you could parallelize. And you want to make sure that other people who are using your algorithm start using the updated version.

It’s time to turn your Jupyter notebook into your first pipeline. In this post, we’ll show you how.

Pipelines are written in workflow definition languages, such as Nextflow

Workflow definition languages provide a structured framework for describing and orchestrating the series of computational tasks needed to handle and analyze data. For data engineering in general, these tasks typically encompass data extraction, transformation, loading, and analysis.

There are many workflow definition languages that data and bioinformatics engineers use to write computational pipelines. We’ll use Nextflow as an example in this article because it is especially popular with bioinformaticians and computational biologists. As free open-source software, Nextflow is supported by a large community of developers. There’s also nf-core, a large library of open-source bioinformatics pipelines written in Nextflow.

A Nextflow pipeline consists of one or more modules or processes. A Nextflow process allows you to execute a script, which can be written in any popular scripting language, including Bash, Python, and R.

Let’s get to it!

Here are the steps for turning code from your Jupyter notebook into a Nextflow pipeline:

1. Jupyter notebook
   You already have a trusty Jupyter notebook for data processing and analysis. In this example, we’ll use a notebook written in Python that performs a simple image thresholding task.

2. Python script
   Turn your notebook into one or more executable scripts. If this is your first Nextflow pipeline, you may want to write one script instead of splitting your workflow into multiple modules. In this example, we’ll make one script.

3. Nextflow pipeline
   Write a Nextflow pipeline that executes your script(s) for you.

Example code and images are available on our GitHub.

Jupyter notebook

Let’s take a look at an example of a Jupyter notebook that takes in images, segments them, and saves the resulting masks. Here’s the code for reference (we’ll break it down in the next section):

Notable features

Inputs

The notebook takes two inputs that the user must change every time they want to run the notebook: input_dir, the path to the directory where the input images are stored, and output_dir, the path to the directory where the processed images should be saved.

Imports

Depending on the user’s Python environment, tqdm, numpy, scikit-image (skimage), and opencv-python (cv2) may not be installed. Additionally, the notebook does not by default store any information about what versions of each package are installed.

Python script

The first step in building the Nextflow pipeline is to turn the Jupyter notebook into one or more executable scripts. As a first pass, we’ll turn our example notebook into a single script:

#!/usr/bin/env python3

import argparse
import os
import glob
import tqdm
import numpy as np
import skimage
import cv2

def preprocess(img_path: str) -> np.array:
    """
    Reads in, normalizes, and thresholds a single image.
    Returns np.array of preprocessed image.
    """

    # Read in
    img = skimage.io.imread(img_path)

    # Normalize
    norm = np.zeros_like(img)
    cv2.normalize(img, norm, 0, 255, cv2.NORM_MINMAX)

    # Threshold
    _, thresh = cv2.threshold(img, 0, 255, cv2.THRESH_OTSU)

    return thresh

def main():

    # Instatiate argument parser
    parser = argparse.ArgumentParser(
        prog='example_script',
        description="Loads images and preprocesses by normalizing and 
        thresholding."
    )
    # Add arguments to the argument parser
    parser.add_argument(
        'input_dir',
        type=str,
        help="Directory containing input images."
    )
    parser.add_argument(
        '-o',
        '--output_dir',
        dest='output_dir',
        type=str,
        help="Directory to which to save outputs."
    )
    # Run argument parser and extract data
    args = parser.parse_args()

    all_image_paths = glob.glob(os.path.join(args.input_dir, "*.tif"))

    # Make sure output directory exists
    if not os.path.exists(args.output_dir):
        os.mkdir(args.output_dir)

    # Preprocess all the images in the input directory
    # and write out to the output directory
    for path in all_image_paths:

        # Apply preprocessing
        processed_img = preprocess(path)

        # Save
        basename = os.path.basename(path)
        extension_idx = basename.rfind(".")
        fname = os.path.join(
            args.output_dir, 
            f"{basename[:extension_idx]}_preprocessed.tif"
        )
        skimage.io.imsave(fname, processed_img, check_contrast=False)

if __name__ == "__main__":
    main()

Notable features

Shebang

The shebang #!/usr/bin/env python3 indicates the interpreter that the program loader should use to run the script (Python3 in this case).

Command line arguments

We use the argparse library to create an argument parser so that the script can take command line arguments as inputs.

If we were to run this script on its own, the usage would be:

./example_script.py <input_dir> -o <output_dir>

Versioning

After you have your script, you can check it into GitHub for version control. You can iterate on it and push new versions, and if you’ve shared the repository with colleagues, they can pull in your changes. To maximize reproducibility, you can add a requirements.txt file with package versions, or create a Docker container that others can run your script in.

If you want to go further and put your script into a pipeline, read on:

Nextflow pipeline

Now, we want to write a pipeline that will run the script we wrote in the last step.

We create a directory with the following structure:

example_nextflow_pipeline
│─── main.nf [1]
│─── nextflow.config [2]
│
│─── bin
│    └─── example_script.py [3]
│
└─── modules
     └─── preprocessing
          └───main.nf [4]

To run the Nextflow pipeline, using the command line, change directory to example_nextflow_pipeline, then run:

nextflow run main.nf --input_dir <input_dir> --output_dir <output_dir>

Now, we’ll go through each component in detail.

[1] main.nf

This is the main pipeline script:

include { PREPROCESS_IMAGES } from './modules/preprocessing'

log.info """\
    EXAMPLE PIPELINE
    ---------------------
    input_dir: ${params.input_dir}
    output_dir: ${params.output_dir}
"""
.stripIndent(true)

workflow {
    PREPROCESS_IMAGES ( params.input_dir )
}

The first line imports the PREPROCESS_IMAGES process, which is contained in modules/preprocessing/main.nf.

The next line outputs information to the console.

Finally, we have the workflow block, which calls the PREPROCESS_IMAGES process with the input input_dir that was specified in the command line.

params contains the command line arguments, which are anything specified like this: --<argument> value.

[2] nextflow.config

This is the Nextflow configuration file:

process.container = "<docker_image:tag>"
docker.enabled = true
docker.runOptions = '-u $(id -u):$(id -g) -v /Users:/Users'

In this example, we specify that processes should run in the given container. Additionally, we specify that Docker should always be used when executing hte pipeline, and give some options for Docker.

We’re only scratching the surface here, though this suffices for our simple example pipeline — for more information on Nextflow configuration, refer to the Nextflow documentation.

[3] bin/example_script.py

This is the example script that we wrote above.

Note that you need to make the script executable in order for Nextflow to run it. In Unix-like systems, you can do this in the command line by changing to the bin directory and running:

chmod +x example_script.py

[4] modules/preprocessing/main.nf

This is the module that contains the process that the Nextflow pipeline will execute:

process PREPROCESS_IMAGES {

    publishDir params.output_dir, mode: 'copy'
    
    input:
    path process_input_dir

    output:
    path('*')

    script:
    """
    example_script.py \\
    ${process_input_dir} \\
    -o "./" \\
    """
}

All files produced by the process script are stored in a work directory. The publishDir directive indicates that the output files of this process (specified in the output block) should be published to output_dir, which we specified as a command line argument.

The input block defines the input channels of a process, similar to function arguments. Inputs are specified by a qualifier (the type of data) and a name. The name is similar to a variable name. In our example, the input is a path.

The output block defines the output channels of a process. These can be accessed by downstream processes, or published to the directory specified by the publishDir directive. Here, the outputs are all the paths to files produced by the process. You can be as specific as you want here—e.g. you could specify path('*.tif') to emit only TIF files if other types were produced as well, or path('image006.tif')to emit only that single file.

The script block defines the script that the process executes. In this case, we are running the example_script.py script, with the input_dir process input as the path to the directory with the input files, and with the current working directory ./ as the directory to which to write the output files (the output files are then published according to the publishDir directive).

For more information on Nextflow processes, refer to the Nextflow Documentation.

Notable features

Containerization is automatically supported

A container is an isolated virtual environment for your code. It allows you to run your code in a reproducible way by having the same packages and the same versions of your packages installed in the environment every time. One of the most common container platforms is Docker. To learn more about containers and their application to bioinformatics, make sure to catch up with the previous post in our series:

In the Nextflow configuration, we specified a global Docker container in which the whole pipeline should run. If we had multiple process modules, Nextflow also would allow us to specify a Docker container for each module, if they had different requirements.

You could use Docker with Jupyter notebooks on your own, but Nextflow makes it easy to always use the same environment each time the pipeline runs.

Versioning

We didn’t explicitly touch on this in our tour through the example Nextflow pipeline, but we can implement version control for the pipeline through Nextflow’s integration with GitHub (along with BitBucket and GitLab). That way, if you’re sharing the pipeline with colleagues, they can always be up to date (or explicitly run a past version, if that’s what suits their needs).

Wrapping up

Nextflow is a powerful tool for creating data processing and analysis pipelines. By simplifying containerization and versioning, it helps you to increase the reproducibility and portability of your code. This post should help you get started with writing pipelines through turning a Jupyter notebook you use over and over into a script and then a Nextflow pipeline.

We only included one process in our simple example pipeline here. In future posts, we’ll highlight how (and why) you can make your pipelines modular and scalable, and how to run them in the cloud.

Subscribe now

Lealia Xiong is a Senior Applications Engineer at Mantle. Her favorite organism is Hypsibius exemplaris.

A Brief History of Containers

This decade saw containers become a permanent part of the software engineer’s toolkit. The latest in a series of virtualization technologies that began with the virtual machine, containers proved a popular building block for software applications, providing functional isolation and clean interfaces for hardware and operating system resources like CPU, RAM, networking, and storage. Cloud providers and other software vendors quickly took advantage of this new specification, the Open Container Initiative, and developed an ecosystem of new technologies. One product in particular, Docker, became the de-facto standard for containers due to its simple configuration language and well-built tooling. 

Engineers found that containers enabled powerful new workflows. They could run the same code nearly anywhere, on multiple different architectures, because they could treat the operating system as an interface instead of a hard dependency. Software was reproducible because all software dependencies were baked into the container. In addition, containers could start up without rebooting the underlying operating system, speeding up launch times and enabling rapid scaling.

More than 15 years after Docker began, containers are finding their way into the bioinformatician’s toolkit. Analyzing large multi-omic datasets and applying new machine learning techniques demand sophisticated software, and today, containers are what make it up.

Applications for Bioinformatics

One of the core benefits of using Docker is reproducibility – in practical terms: your code, in a compiled container, will behave the same in five years as it does today. Imagine your favorite computer game from childhood being available to you on your new laptop, or a Python script from graduate school that still runs despite your colleague changing their library. That is what Docker enables.

Case Study: Sharing An Analysis With Colleagues

You’ve done all the hard work – you’ve completed a rigorous statistical analysis of your biological dataset and found a few surprising results. You’ve even done some extra stuff – you use Git to version control your code, and you store your datasets in the cloud so that it is backed up and accessible to your collaborators. 

You reach out to your colleague and have her pull your code and data; you give her a list of Python dependencies to install and have her run the code. To your dismay, she messages you back with an error message: 

read_csv() got an unexpected keyword argument 'error_bad_lines'

“But…” you cry out, “it worked on my machine!”

A cursory Google search reveals that this particular argument to `read_csv()` was deprecated in Pandas 1.4.0; you were running version 1.3.0 on your computer while your colleague had upgraded to 1.5.3. A sympathetic bug, but one that could have been avoided had you shared a Docker container instead of your code directly.

You decide to package your code into a container using Docker. You read that you need a requirements.txt file to explicitly list your Python dependencies and their versions. It looks something like:

pandas==2.2.2
numpy==1.26.2
scanpy==1.9.3

Within your Dockerfile, you copy over the requirements.txt file and your updated Python script and install the dependencies:

FROM python:3.11

# Copy and install pip requirements
COPY requirements.txt /src/
RUN pip install -r /src/requirements.txt

# Copy your python script
COPY main.py /src/

# Run this command when the container runs
CMD ["python3", "main.py"]

You pull up an old blog post that taught you how to build your container and store it in a repository for your colleague to access (see below). You message your colleague with instructions for how to pull and run the container, which she does successfully. She sees the results of your analysis and is confident that you will win the Nobel Prize.

Case Study 2: Running Computational Pipelines in the Cloud

Another core benefit of using Docker is the ability to run your code “anywhere.” As mentioned earlier, containers are the latest in a series of virtualization technologies that began with the virtual machine (or, one could argue, the operating system itself). Virtual machines broke the link between hardware and the host operating system. This link was replaced with a piece of software called a ‘hypervisor’, which provided virtual interfaces for hardware resources like CPU and RAM. This technology allows you to run multiple operating systems on top of the same hardware. 

Containers take this one step further. Operating systems are cumbersome to install and don’t fully abstract away networking or storage. Technologies like Docker provide something called a “container engine” that sits on top of the host operating system to share operating-system-level resources with multiple isolated applications.

As a computational biologist in an industry lab, you are tired of running the same script every few days to generate results for your scientists. You research pipeline-running software and learn about an open-source tool called Nextflow. 

You do the difficult work of understanding the Nextflow model and refactoring your scripts to take in standard inputs and produce consistent outputs. You understand where you can modularize your pipeline to take advantage of parallelization. You learn from the documentation that you can leverage containers to create consistent script environments. 

You realize quickly the new capabilities you possess – simply by learning a little bit about container registries and Dockerfiles, you can now run your scripts in production ready and highly parallel Nextflow pipelines on your on-prem hardware or in the cloud. All you need to do to support a large influx of data is click a few buttons to increase the compute capacity of your cloud pipeline, which you set up in a day by following the Nextflow documentation. 

Using containers allowed you to run your pipeline on different architectures with little overhead and allowed you to rapidly scale with just a few clicks. What a time to be alive.

A Software Engineer’s Real-world Advice For Using Docker Effectively

Use a Container Registry to Version Your Images

Just as repositories like GitHub and GitLab provide a place to store and access code, container registries allow developers to store images and binaries. Previously, you would have to manually ensure your production instance was configured exactly like your local workstation, checking dependencies and configuration files line by line. Using a container registry allows you to access the same environment across the cloud, your local machine, or your specialized hardware; all that is needed is an operating system with Docker installed and a network connection.

Many software vendors and cloud providers offer a container registry service, differentiating on access controls, price, and integrations with other services. Below, we present a common workflow using AWS Elastic Container Registry (ECR) and Docker to demonstrate how you might build, push, and access a container using a registry.

#!/bin/bash

# Build the Docker image for a specific platform and label it as 'my_server'
docker build --platform linux/amd64 -f path/to/Dockerfile -t my_server .

# Log in to Amazon ECR
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com

# Tag the 'my_server' image with a label that specifies where it will be pushed
# and what version it is (1.0.0)
docker tag my_server:latest $AWS_ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com/my_server_repository:1.0.0

# Push the 'my_server' image to the specified ECR repository
docker push $AWS_ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com/my_server_repository:1.0.0

There are a few subtleties in the above script that demonstrate the power of this workflow.

• In the first step, we built the image locally and labeled it `my_server`; by default, Docker assigns this image a tag called `latest`, so we reference the local image as `my_server:latest`. 

• In the second step, we use the AWS CLI to get our login credentials for ECR (this requires us to be locally authenticated against AWS). We then provide those credentials to our local Docker client to pull and push images to ECR. Think of this like having a registered ssh key on your local machine so you can pull and push to GitHub.

• In the third and fourth steps, we re-label our local image with a fully qualified domain name so that our Docker client knows where to push the image. This step will vary across container registry providers, but the structure is similar, usually `<registry_domain_name.url>/<repository_name>:<version>`

With just a few lines of code, we now have versioned images that we can pull anywhere. Note that the source code is not versioned, just the compiled image. Much like having development and production branches for code, you may want to have a development and production repository. The former would be accessible to developers, while the latter may only be accessible within a CI/CD pipeline. This ensures that production images are being built in a consistent environment and that it is only building images from validated code that has been merged into your production branch.

Optimize your Dockerfile

It may be enough to copy/paste examples from the internet or a chatbot to get started writing your first Dockerfile, but understanding a few of the underlying mechanics pays dividends. Below are a few strategies to help you optimize your Dockerfile.

Caching

Software engineers often talk about containers being composed of layers. Think of Photoshop: the final image is a composition of independent layers, one for the background, one for the foreground, and one for intricate styling. Docker containers are constructed in the same way. There is a base layer, which can be an operating system interface or a language runtime, and intermediate layers that define new behavior. When building images, Docker creates a new layer for each instruction in the Dockerfile. 

To optimize build times, Docker caches these layers, and only re-builds the layer if it detects a change. Suppose a container has 7 layers; if Docker detects a change to layer 3, it will rebuild all layers from 3-7 while reading the first two layers from its cache.

Consider the following Dockerfile snippet:

FROM python:3.11

# Copy and install pip requirements
COPY requirements.txt /app/
RUN pip install -r /app/requirements.txt

# Copy the application code
COPY src/ /app/src/
COPY main.py /app/

...

In this example, we first copy and install the Python requirements before we copy our application code. In a common local development workflow, developers would iterate on the application code and may repeatedly build the image locally to test it end-to-end. If we had to re-install the dependencies for each build, iterating would take significantly longer. Therefore, it matters where we install the dependencies in our Dockerfile. If we had copied and installed the Python requirements after we copied our application code, Docker would see that the application code had changed and would invalidate the cache for all subsequent instructions, thereby forcing a reinstall for each build. We can iterate significantly faster by simply installing Python requirements before we copy the application code. 

Use Environment Variables (Wisely)

In containerized applications, environment variables provide a powerful way to configure applications at runtime. Unlike monolithic servers where environment variables are shared across the entire host, containers isolate these variables, eliminating conflicts and race conditions. This isolation allows each container to serve a single functional purpose with its own specific configuration, enhancing security and manageability.

Consider the following Dockerfile snippet:

FROM golang:1.20

ENV APP_ENV=development
ENV DB_URL=""

...

A developer running this container would pass in arguments for their local database, enabling local development while ensuring access only to resources they have been explicitly granted.

docker run -e DB_URL="your_database_url" my_image

Environment variables can be used for various purposes. We use them to adapt our application to different environments (development, testing, production) and pass configuration information and, when appropriate, secrets. If you decide to store sensitive information in environment variables, it is important to understand a few principles to keep that information secure.

First, secrets should not be stored unencrypted; that means you should not hard-code them in your Dockerfile and instead store them in an encrypted key store that you access at runtime, such as AWS KMS, Azure Key Vault, or HashiCorp Vault.

Also, remember that you can retrieve secrets at runtime in different ways. You could programmatically access the key store within your application instead of using a variable, and if you run on a cloud provider, you may find opportunities to delegate some permissions to them (for example, AWS IAM handles AWS internal access for you). 

Third, remember that the host operating system that runs your containers generally has superuser permissions for running containers; be careful where you deploy production services.

If you decide to pass sensitive information as an environment variable, here is an example of how to mount the secret at runtime using AWS Secrets Manager:

aws secretsmanager get-secret-value --secret-id my_secret_id --query SecretString --output text | docker run -e DB_URL="your_database_url" \
           -e SECRET_KEY="$(jq -r '.SECRET_KEY' -)" \
           -e AWS_REGION="us-west-2" \
           -e APP_ENV=production \
           my_image

This command might be run as part of the launch script for your production service in an environment that has been explicitly granted the correct permissions, making it more secure. 

Prevent Bloat

A common criticism of Docker is the significant disk space required by images, often ranging from a few hundred megabytes to a gigabyte. Preventing image sizes from getting too large requires careful optimization. Here are a few best practices:

1. Use Minimal Base Images

   A language runtime (like python:3.11 or golang:1.20) is more lightweight than a full-fledged OS image (like ‘ubuntu’ or ‘debian’). Unless you need a package that requires access to OS internals, opt for a language runtime. Use official images from trusted sources whenever possible as they are generally better maintained and more secure than the alternatives.

2. Use a `.dockerignore` File

   Create a `.dockerignore` file to exclude unnecessary files and directories from the build context (the set of files sent to the Docker daemon during build).

   For example:

   .git
   node_modules
   tmp

3. Combine Commands

   It’s important to remember that Docker creates a separate layer for each command. Although combining commands may sacrifice readability, it can save significant space. It is also a good idea to delete vestigial build artifacts and unnecessary files at runtime.

   RUN apt-get update && apt-get install -y \
       package1 \
       package2 \
       package3 && \
       apt-get clean && rm -rf /var/lib/apt/lists/*

Conclusion

Learning any new technology takes time, but at Mantle, we believe the gains from using containers are disproportionate to the effort required to learn. We hope that we’ve provided some practical insight into how containers are used by professional engineers so that you can get up to speed as quickly as possible.

If you want to learn more about how to get the most from your multi-omic data, stay tuned for more bioinformatics tips in upcoming posts in our Intro to Bioinformatics Engineering series and beyond.

Subscribe now

Aakash Shah is a Senior Software Engineer at Mantle. His favorite organism is Ornithorhynchus anatinus.

Introduction: Hiking or Driving

Analyzing new data is a bit like hiking in unmapped woods. You’re hoping to find something, but you might not know what that something is or where it will be. Armed with a compass and a blank map (or whining laptop and Jupyter Notebook), you set out to explore. If you find something interesting you might make a note, and if you find something exciting you might start a new trail.

Running a production bioinformatics pipeline is more like driving on a road. You (hopefully) know where you are going, and it will be faster than walking. Most drivers on the road are not the engineers who built it. Your priorities differ from a hiker’s; it’s more important to travel quickly and reliably than to investigate something off the route.

For a traveler, the tradeoffs between trail and road are usually clear. But the considerations become far more complicated if, instead, you are a civil engineer.

In this post, I will discuss what makes an amazing bioinformatics engineer, what they might consider when “city planning,” and a few of the most common patterns we have seen for developing a successful bioinformatics system. 

Why and When to Build a Pipeline

“Analysis” and “pipeline” can mean many things and are sometimes used interchangeably. Without claiming that these are the only definitions, for the sake of this article, they mean the following:

Analysis (hiking): Code that is written and run in a single setting. Examples: analyzing data in a Jupyter Notebook, editing and running an R script locally.

Pipeline (driving): Code that is written ahead of time and used at runtime. Examples: running a Python script with Snakemake, using 10x’s Cell Ranger Count, running Illumina’s BCL2FASTQ.

End-to-end data processing often requires a combination of analyses and pipelines.

As an example, processing single cell transcriptomics data often begins with the raw output of a DNA sequencer (BCL).

1. The BCL2FASTQ pipeline converts the raw observations of the DNA sequencer into human-readable files containing DNA sequences and quality scores (FASTQ).

2. The Cell Ranger COUNT pipeline groups the reads in the FASTQ file by cell and by gene to create a table of gene counts per cell (Count Matrix).

3. To understand and visualize this data, someone does an analysis of the count matrix using the Scanpy package in a Jupyter Notebook. After gaining understanding, they create a visualization (UMAP) of the data to communicate their insights to others.

This example captures a few benefits of pipelines:

• Shareable. A well-built pipeline can be written by a domain expert and then run by anyone. Illumina and 10x developed BCL2FASTQ and Cell Ranger COUNT respectively, enabling many scientists to analyze data without developing them from scratch.

• Consistent. If two different people analyze the same dataset, they will likely make slightly different choices. This may be as superficial as different line colors in a graph or as significant as choosing different statistical tests. Using versioned pipelines makes processing consistent and results comparable, and may facilitate interoperability with other systems.

• Testable. Almost everyone who works with sequencing data has used the output of BCL2FASTQ and trusts that the pipeline worked correctly. Because the pipeline has been benchmarked and tested by many use cases, it is now highly reliable and trustworthy. 

• Scalable. Suppose someone asks you to process thousands of BCL files with the process above. Though the analysis step will likely stay time-consuming, the pipeline steps can be run in parallel and automated.

• Reproducible. If you capture the input, environment, and pipeline version you can reproduce a pipeline run. Analyses may also be reproducible, but it can require meticulous manual documentation.

But there are also advantages to analyses:

• Flexible. Flexibility is naturally a tradeoff with consistency. Everyone loves being able to change something quickly. No one likes when a coworker’s quick change introduces a bug and blocks operations. If you want to move fast (and break things), an analysis is the answer. In an analysis, it is much easier to pull in additional files and packages on the go.

• No “Upfront Cost”. A road must be planned and paved before others can drive on it. Similarly, building a pipeline requires more up-front work than using an analysis. An analysis only needs to support one exact use case; a pipeline should be designed to handle a range of values and inputs.

• No Maintenance Cost. Roads must be maintained to be usable. Pipelines are not built in a vacuum; requirements, inputs, and infrastructure will change, and the pipeline will need to change to keep up. If you live somewhere with cold winters, imagine driving on a road that has not been repaved after a few freeze-thaw cycles. It is similarly unpleasant to use a pipeline that has not been updated after a faster or better tool became available.

In the example above, when would you replace the Count Matrix analysis with a pipeline? If you build a pipeline you only run once, you’ve wasted effort and slowed progress. If you use analyses hundreds of times, the results may be too inconsistent to analyze in aggregate.

How to weigh the tradeoffs depends on your priorities and capabilities. The faster you can write a pipeline and the more easily you can maintain it, the sooner you should do so. If you plan to change requirements frequently, it may be better to use analyses for longer. By combining both pipelines and analyses in your workflows, you can have the best of both worlds.

Common Pipeline Patterns

If you’re working with a complex analysis, it may be unclear if and how you should convert it to a pipeline. Here are a few of the most common patterns we have seen for creating pipelines.

The Last Mile

Many people experience the last-mile problem while commuting: the train almost takes you to the office but doesn’t quite get you there, or you park in a structure a few blocks from your workplace.

Here are some symptoms that your workflow is suffering from the last mile problem:

• You feel like it’s “almost” a pipeline, but you want to be able to edit the code for the last step regularly. 

• There is an “intermediate file” that your pipeline produces.

  • You find yourself opening that intermediate file in a Notebook for an additional analysis.

  • Your script may take that intermediate file as an optional input, and when provided, a significant section of code is bypassed.

• You are frequently asked to reprocess data with a small change to a final step.

What is often happening in this case is that the analysis begins with data cleaning/preprocessing, but ends with visualization/postprocessing. While the first step is fairly stable, the second step undergoes frequent changes and development.

Solution:

• Create a pipeline for the “preprocessing” step.

  • If there is an intermediate file generated, use this as the checkpoint. If your pipeline has clear preprocessing logic but no intermediate file, think about a way you can create a standardized file to write out after the preprocessing. If you’re using Python, an easy option may be to pickle the intermediate object.

• The “postprocessing” step may either be a pipeline or an analysis.

  • This depends on how frequently you are updating this section.

  • A common pattern is to start with an analysis and then convert it to a pipeline when it is stable. If you need to do a bespoke analysis, you can always open your intermediate file in a Jupyter Notebook. More on that below.

Multiple Destinations

“Multiple Destinations” is very similar to “The Last Mile.” The key difference is whether the pipeline always produces outputs of the same general type. Here are a few other symptoms of the Multiple Destinations pattern:

• There are large sections of code that only run when certain conditions are met. For example, perhaps there is a graph that is only produced for data from a specific CRO, or there are different QC checks for different organisms.

• You are frequently changing the script for certain types of input, but for other types of input you can run the analysis as-is.

Solution:

• Go through the code, line by line if necessary, and determine which sections run for ALL inputs.

• Separate this logic into a separate function or pipeline. This may take some refactoring. This will also require determining or creating an intermediate file, similar to the Last Mile pattern.

• Examine the code that does not run for all inputs. What determines when each of these runs? Is it the data type of the input? Is it a property, like the organism? Is it whether the raw data passes or fails a QC check? Make a list of these categories.

• Look at the processing and output for each category. You may find that some categories have a very established analysis that can be converted to a pipeline while others are still exploratory.

Example:

Suppose you are processing videos of zebrafish to study behavior. You had been doing experiments to study feeding behavior, and now you are beginning to study social behavior. Your script always begins by converting the video into wireframe models of the fish. Then, depending on the type of experiment, either feeding or social behavior metrics are extracted from the wireframes.

Solution: 

Create a pipeline that takes any video and produces wireframes. Create a second pipeline that takes wireframes and calculates feeding metrics. Since you are still developing the social behavior experiments, continue to analyze the data in a Jupyter Notebook until the protocol is refined. Now, if your coworkers want to start analyzing a new type of behavior they can use your preprocessing pipeline!

The First Mile

The most common story for The First Mile pattern is the following:

You’d like to do an analysis involving many datasets. These datasets are all of the same general type, but they come from multiple collaborators, CROs, instrument types, public databases, or protocols. You are frequently editing your script because it does not quite work for a new dataset, or you are manually tweaking the dataset to make it a valid input to your pipeline.

The solution is basically the same as for the Multiple Destinations Pattern, but reversed. First, determine the part of your script that runs for all data, and separate it into a pipeline. The input to this pipeline may be obvious, or it may take a little creativity.

Once you have decided on the input to the pipeline, your next goal becomes clear. For each new raw dataset, convert the dataset into the input. Depending on your data, this might mean writing an analysis for every dataset! Or maybe you will find that some datasets are similar enough to create a preprocessing pipeline. Either way, you do not need to edit the processing pipeline for each new dataset.

Pipeline Building is One Part of Bioinformatics Engineering

When should a city build a new highway? Where should the road go? Is a highway worth the investment? Will the road be able to widen and scale when the city grows in the future?

Similar to civil engineering, great bioinformatics engineering means building reliable, scalable, and ready-for-change systems.

Bioinformatics engineers are not the only ones who do bioinformatics engineering. Bioinformatics engineering is often a critical skill for computational biologists, data engineers, software engineers, scientists, and research engineers.

Bioinformatics engineering overlaps with computational biology and software engineering. While it’s common for one person to wear multiple hats, it's helpful to consider these as separate skills.

Advice for Teams

Great bioinformatics engineers are often people who love working in a collaborative environment. Commonly, the creator of the data, developer of the algorithm, and stakeholder for the results are three different people. Here are a few thoughts on effective collaboration with these groups:

• If you ask someone where they would like a road, they will likely ask for it to be a straight line from their starting point to their destination. Unfortunately, if this was done for everyone, the countless overlapping roads would be nearly impossible to maintain or change. Requests from users are often focused on the fastest solution to the current problem; bioinformatics engineers need to balance this with efficiency and sustainability. It’s important that a team is structured to allow scientists and engineers to collaborate on project requirements.

• When developing a new pipeline, it’s important to have clearly communicated requirements from the teams that will use it. I know not every engineer loves documentation, but having a shared design doc can prevent timelines from slipping due to misunderstood requirements. Consider versioning the document, and having 1-2 stakeholders “sign off” on every new version.

• Be careful not to convert an analysis into a pipeline too early; you wouldn’t want to build a road that never gets used.  A good rule of thumb is to convert an analysis to a pipeline only after it has been run at least five times without significant changes. This has the additional benefit of providing testing data to validate the pipeline.

Bioinformatics is a BLAST

I had a great time writing this, and I hope you enjoyed reading it! We have more posts planned for our Intro to Bioinformatics Engineering Series including guides for leveraging essential tools, tips for being ready-to-scale, example developer workflows, and more. Please subscribe below if you would like to be notified, or visit us here for more bioinformatics tips.

Subscribe now

Emily Damato is the CEO and Co-Founder of Mantle. Her favorite organism is Drosophila melanogaster.

Understanding the needs of biotech companies

Biotech companies are at the forefront of scientific innovation, requiring substantial computational resources for tasks like genomic sequencing, drug discovery, and data analysis. These tasks demand high computing power and the ability to scale resources up or down based on the dynamic nature of research projects. Cloud computing is particularly beneficial for biotech companies as it offers unparalleled scalability and flexibility. With cloud computing, organizations can easily scale up their computing resources to handle large datasets and complex calculations during peak times, such as when running intensive genomic analyses or large-scale simulations. Conversely, they can scale down during periods of lower demand, optimizing costs and resource usage. Additionally, cloud services provide access to cutting-edge technologies and infrastructure without the need for substantial upfront investments, enabling biotech companies to stay at the forefront of innovation while maintaining financial efficiency.

With these benefits in mind, comparing the practical aspects of running workflows on the cloud versus on-premises is essential. For instance, running the nf-core RNA-seq pipeline on AWS offers dynamic scalability, enabling biotech companies to handle peak computational loads without the need for significant upfront hardware investments. In contrast, on-premises solutions require substantial initial capital expenditure for hardware and ongoing maintenance costs. While on-premises infrastructure might provide lower latency and more control over physical resources, it lacks the flexibility and cost optimization offered by cloud services. By evaluating the costs and benefits of each approach, biotech companies can make informed decisions about their data infrastructure. The next section will explore how to estimate the cost of running nf-core RNA-seq on AWS compared to on-premises solutions, providing insights into the factors that influence these expenses and strategies for optimizing them.

Comparing the costs of compute

Running nf-core RNA-seq on AWS

When running bioinformatics pipelines like nf-core RNA-seq on the cloud, understanding the associated costs is crucial for budgeting and resource allocation. Below is the cost estimation for running the nf-core RNA-seq pipeline on AWS using memory-optimized instances with a 100GB dataset.

Assumptions and setup

For our cost estimation, we assume the following setup:

• Instance type: Memory-optimized r5.large AWS Elastic Compute Cloud (EC2) instance

• Number of samples: 10 samples

• Data size per sample: Initially, 100 GB

• Runtime: Approximately 10 hours per sample

• Storage needs: Temporary storage to stage samples

• Data transfer: Depends on long-term storage setup; within AWS incurs no additional transfer costs

Cost breakdown

Compute costs:

Using the r5.large EC2 instance in the US East (N. Virginia) region, which costs approximately $0.126 per hour, we calculate the compute costs:

• 10 hours × $0.126/hour = $1.26 per sample

Storage costs:

A general-purpose SSD (gp2) on AWS Elastic Block Storage (EBS) costs $0.10 per GB per month. Let’s assume 200 GB per instance for 100 GB samples for temporary storage.

• Assuming 1 day of usage: 200 GB × $0.10/GB-month × (1/30 month) ≈ $0.67 per sample

Total cost

• Compute: $1.26 / sample

• EBS: $0.67 / sample

Total: $1.93 per sample

Building an on-prem machine to run nf-core RNA-seq

The summary table provides a detailed breakdown of the costs of building a high-performance PC for running the nf-core RNA-seq pipeline, including the estimated annual maintenance and upgrade costs.

Here is the summary of the costs:

Initial build cost: $2,200

Annual maintenance and upgrade costs:

• Annual maintenance: $200

• Annual upgrades: $300

Total annual cost: $500

Total cost over 3 years:

• Initial Cost: $2,200

• Annual Costs for 3 Years: $500 × 3 = $1,500

Total Cost Over 3 Years: $2,200 + $1,500 = $3,700

Break-even point analysis: AWS vs. PC build for RNA-seq

The plot above incorporates the estimated annual maintenance and hardware upgrade costs for a 3-year period, providing a comprehensive comparison between running the nf-core RNA-seq pipeline on AWS and building a dedicated PC.

Key points:

• AWS cost:

  • The cost of processing 100 GB samples on AWS is $1.93 per sample.

• PC build cost:

  • Initial investment: $2,200

  • Annual maintenance and upgrades: $500 per year

  • Total cost over 3 years: $3,700

The break-even point for a 3-year period occurs at approximately 1,917 samples. This means if you process more than 1,917 samples over 3 years, building a PC/on-prem machine becomes more cost-effective than using AWS. This is equivalent to running ~1.75 samples per day.

Comparing the costs of storage

Many biotechnology companies hesitate to migrate to cloud storage due to concerns over storage costs. However, a detailed comparison reveals that leveraging cloud services like AWS S3 Glacier for raw data and AWS S3 Standard for pipeline outputs can be more cost-effective than maintaining on-premise storage solutions. On-premise storage requires a significant initial investment in hardware, such as a 100TB server costing around $4,799.99 with additional hardware for backup storage, plus ongoing expenses for maintenance, power, and cooling.

In contrast, AWS S3 Glacier offers long-term data archiving at a low cost of $0.00099 per GB per month, making it possible to store 95TB for approximately $3,385.80 over three years. AWS S3 Standard provides immediate access to frequently used data at $0.023 per GB per month, costing around $4,140.00 for 5TB over the same period. Thus, the total cost of using AWS S3 Glacier and S3 Standard for 100TB over three years is roughly $7,525.80. Included in this price is data replication across multiple AWS Availability Zones, which protects against data loss.

Splitting storage between AWS S3 Glacier and S3 Standard

In bioinformatics, the data generated and used in analyses can be broadly categorized into raw data and processed output data. Here’s an explanation for the reasoning behind splitting cloud storage into 95TB for raw data and 5TB for pipeline output files:

1. Raw data (95TB)

• Volume: Most of the data (95%) used in bioinformatics is raw data, such as high-throughput sequencing data, which are typically large files. This can include BCL and FASTQ files from RNA-seq, DNA-seq, or other omics technologies.

• Usage: Raw data is essential for the initial stages of analysis and must be stored securely and reliably. However, once processed, it is not frequently accessed.

•Storage solution: Raw data can be stored in AWS S3 Glacier, which is a cost-effective storage solution for long-term archival. S3 Glacier is suitable for data that does not need to be accessed frequently but must be retained for compliance or future re-analysis.

2. Processed output data (5TB)

• Volume: Processed data, such as count matrices, variant call files (VCFs), and other summary files, constitutes a smaller portion of the total data (5%). These files are significantly smaller than the raw data files but are critical for downstream analysis.

• Usage: These files are used frequently for various downstream analyses, visualization, and reporting. They need to be readily accessible to researchers for further analysis and interpretation.

•Storage solution: Processed output files should be stored in a more accessible and faster cloud storage solution, such as AWS S3 Standard, to ensure quick and easy access for ongoing research and analysis.

Summary

The split between 95TB for raw data and 5TB for processed output data reflects the different storage needs based on the usage patterns:

• Raw data: Stored in AWS S3 Glacier for cost-effective, long-term storage. This data is not frequently accessed but needs to be retained.

• Processed output data: Stored in AWS S3 Standard for quick and frequent access required for ongoing bioinformatics analysis and research.

This approach optimizes storage costs while ensuring that the necessary data is available, without compromising accessibility or data integrity.

Limitations of on-prem computing

Traditionally, biotech firms have relied on on-premises infrastructure. However, this comes with significant drawbacks:

• High capital expenditure: Setting up and maintaining on-prem infrastructure requires a hefty initial investment, often hundreds of thousands of dollars for state-of-the-art servers and data storage solutions.

• Scalability issues: Scaling on-prem infrastructure can be slow and costly. It often involves purchasing additional hardware that might not be used at full capacity, leading to inefficiencies.

• Maintenance and upgrades: On-prem systems require ongoing maintenance by highly skilled IT staff, adding to operational costs. Keeping up with the latest technology also necessitates regular hardware upgrades.

Benefits of cloud computing for biotech

Cloud computing offers several advantages that align perfectly with the needs of the biotech sector:

• Flexibility and scalability: Cloud services provide the ability to quickly scale computing resources up or down. This is crucial for biotech firms that may need to ramp up resources for large-scale experiments or dial them back during less intensive periods.

• Cost-effectiveness: With cloud computing, companies pay only for the resources they use. This "pay-as-you-go" model can lead to significant cost savings compared to the fixed costs associated with maintaining on-prem infrastructure.

• Advanced technologies and collaboration: Cloud providers often offer advanced analytics and machine learning services that can be integrated seamlessly into biotech workflows. Additionally, the cloud facilitates easier data sharing and collaboration across distributed teams, which is essential for today's remote work environment.

• Built-in compliance: Cloud services with built-in compliance features, such as AWS HealthLake or Google Cloud’s healthcare solutions, help ensure adherence to industry regulations like HIPAA and GDPR, protecting sensitive patient and research data

Conclusion

Studies demonstrate that companies adopting cloud computing can reduce their IT expenses by 30-50% compared to maintaining on-premises infrastructure. Additionally, the agility of cloud computing significantly accelerates the time-to-market for new scientific findings or drugs, enhancing potential revenue generation. As these companies continue to push the boundaries of scientific research, the selection of computing infrastructure becomes paramount. Cloud computing offers a more flexible and cost-effective solution than traditional systems, boosting a company’s ability to innovate and collaborate globally. This not only aids in beating out the competition but also establishes cloud computing as a strategic imperative for biotech firms determined to lead in innovation and efficiency.

Patriss Moradi is a Senior Software Engineer at Mantle. His favorite organism is Vulpes vulpes.

Subscribe now

You’re on an R&D team that records lab notes in Google Docs. This was fine at first, but now that the group has grown it can feel overwhelming to try to find a detail from a few months ago. This solution is reaching a breaking point, but when should you change and what should you use?

Almost every biotech company I know has been in this position at some point. In the last ten years of being a software engineer in life science, I have been asked for ELN advice countless times. The problem is that “ELN” has come to mean different things to different people and there is no one answer that is perfect for everyone.

What is an ELN?

“I was hoping you would just tell me what ELN to use and I could go back to research”

– Almost everyone

An ELN is an electronic lab notebook. By taking lab notes in a Google Doc, it counts as an ELN and has many advantages over the paper notebook I kept in a locked drawer during my first research project:

• Version Controlled

  • Timestamped changes can help you track how an SOP changed over time as well as prove ownership in an IP dispute.

• Shareable

  • Documents can be easily shared with teammates, making research more transparent and collaborative.

• Searchable

  • The ability to search for information in past records can help you find information about an experiment you performed three years ago or see if a teammate previously explored a particular topic. 

When you are doing exploratory research where every experiment is unique, the blank-canvas Google Doc will let you move quickly. But if you are looking for a new ELN, you have likely felt the limitations of a blank sheet.

When you perform the same experiment 5 or 500 times, lack of structure can lead to inconsistency that will make comparing results difficult or impossible. While this was never something a paper notebook could provide, we have come to expect a modern ELN to have structured data.

What is the right amount of structure?

If you try to capture data from a 200 sample experiment in a Google Doc it creates chaos; if you try to capture a one-off experiment in a database it creates unnecessary work. So what is the right amount of structure? Here is how I think about it:

• 1 - 5 samples / assays

  • Do something completely unstructured, but not untracked.

  • This phase is like finding a new hiking trail: you might not know where you are going the first time, but you still need to draw a map to retrace your path in the future.

  • Examples: Google Doc, Benchling Entry, Notion page

• 5 - 50

  • This should be structured, but there should also be a way to capture unstructured notes. 

  • This phase is like going on a hike: you should be following a route, but there may be unexpected events that require you to go off trail.

  • Examples: Google Sheets, Benchling Entry + Registry Tables, Notion Table

• 50 - 500

  • This should be structured, and structured carefully. Someone with data engineering experience and someone with detailed domain knowledge should both contribute to the design.

  • This phase is like driving on a road. Most of the decisions will be made when you pave the road, and driving down it should be very consistent. This is often when lab automation equipment is purchased.

  • Examples: Benchling, Notion, AirTable

    • 500 is the upper limit of what many traditional ELNs support. If you do not know the limit of a particular ELN, you can ask the customer service team for benchmark data or run a benchmarking experiment yourself.

• 500 +

  • This is very well structured. At this point you want to have versioned SOPs (or even automation!) and versioned data schemas to track structure changes over time.

  • This phase is the highway. It should be similar to the road, but with fewer potholes and more lanes.

  • Examples: Airtable, Software databases (e.g.,  Postgres)

Most research groups have different projects that require different levels of structure, and that’s okay. It’s nice if you have one tool for everything, but it is much more important to have the right tool for each thing. Many companies have two data systems, one for low-throughput experiments and one for high-throughput experiments. The good news is you can focus on the 1-500 problem now and solve the 500+ problem later.

What does biology have to do with this?

There are two categories of ELN / data tools:

• Domain Agnostic

  • Examples: Notion, AirTable

• Biology Specific

  • Examples: Benchling, SciSpot

Generally speaking, domain agnostic tools will be cheaper but you will need to do more work to configure them for your needs. There are also biology specific features that the domain-agnostic tools do not provide out the box, including:

• Customer support services with biology experience

  • E.g. pre-configured schemas for in vivo experiments, advice on how to set up a reagent tracking database

• Analysis and visualization tools for biological data

  • E.g. a plasmid viewer, BLAST tool, small molecule database

• Life science regulatory compliance

  • E.g. HIPAA, GxP

Whether to use a domain agnostic or biology specific tool depends on your requirements and how much time you are willing to invest in setting up the tool. In general, I don’t recommend using a domain agnostic tool unless someone on your team has data engineering or software experience.

I chose an ELN. Am I done now?

Unfortunately, no.

Your research is unique – that is why you are doing it. Even a biology specific tool like Benchling will need to be customized for your research. As your research evolves over time, you will need to keep your ELN up to date. Scientists will need to learn to use it, adopt it into their workflow, and enter data consistently.

Maintaining your ELN is likely to cost more than the ELN

Biotech companies that ask for initial implementation help from applications engineers or data engineers tend to have a better long term ELN experience. Biotech companies over 50 people tend to hire an informatics specialist or data manager to be responsible for the ELN and other information systems. 

Don’t forget to factor usability and maintainability into the price. A poorly implemented ELN that slows down your research and takes multiple employees to maintain will cost your company far more than any SaaS.

(Self Promotion) What about computational biology?

Computational research needs an ELN too – and that’s why we built Mantle. 

Traditional ELN systems are designed for research done at the bench. This is great for recording sample preparation for a Western blot, but not for capturing the analysis of single cell transcriptomics FASTQ files.

Mantle connects “big biological data” (FASTQs, PDB, micrographs, etc), analysis notebooks (Jupyter, R Studio), and bioinformatics pipelines (Nextflow). With automated workflows, you can run a DNA sequencer, go grab coffee, and receive reproducible results in your inbox. We integrate with the wet lab ELNs above to keep the entire lab connected.

Learn More

How do you manage data?

What ELN do you use? We’d love to hear what has worked for you and what lessons you’ve learned along the way. Thank you!

Subscribe now