# scRNAvigator: Interactive exploration, processing, and analysis of your scRNA-seq data

A series of R notebooks for analysing single cell RNA sequencing data.

## Contents

- [Introduction](#introduction)
- [Workflow structure](#workflow-structure)
    - [Input data](#input-data)
    - [Quality control](#quality-control)
    - [Dataset integration](#dataset-integration)
    - [Analysis](#analysis)
- [How to use the Quarto notebooks](#how-to-use-the-quarto-notebooks)
    - [Platform](#platform)
    - [Rendering documents](#rendering-documents)
- [Installation](#installation)
    - [Local installation](#local-installation)
    - [Using the `if89` R libraries on NCI](#using-the-if89-r-libraries-on-nci)
    - [Installation on NCI](#installation-on-nci)
- [Running on ARE](#running-on-are)

## Introduction

This collection of R notebooks has been designed to guide you through processing and analysing your single cell RNA (scRNA) sequencing data. They are designed to be worked through in the following order:

1. Quality control
2. Doublet detection
3. Dataset integration
4. Cell annotation
5. Pseudobulking and differential gene expression analysis
6. Pathway enrichment analyses.

Each notebook explains what is happening in each step, complete with code and rationales for the choices we have made in our approach.

It is important to note there is no one single way to pre-process scRNA data - there are as many ways as there are different software packages and libraries for scRNA analysis, and the limitless ways to use each of their tools and functions.

The workflow presented in these notebooks is the synthesis of best practices, studies, and discussions of how to analyse scRNA data with a focus on using Seurat in R. Footnotes and external links accompany the text throughout the document - please view these for useful additional information and rationale on why steps are done in certain ways.

This content primarily uses the Seurat R package, but the way and order things are run differs vastly from their tutorials. We like to note that the Satija lab Seurat tutorials are instructions on how to use the package, but not how to conduct robust scRNA pre-processing and analysis. This content leverages the flexibility of the Seurat package, but is supplemented by the practices outlined in existing resources. These resources are the most influential:

-   [Current best practices in single-cell RNA-seq analysis: a tutorial (Luecken and Theis, 2019)](https://www.embopress.org/doi/full/10.15252/msb.20188746)
-   [scRNAseq analysis in R with Seurat (Williams and Perlaza, 2024)](https://swbioinf.github.io/scRNAseqInR_Doco/)
-   [Spatial Sampler (Williams, 2025)](https://swbioinf.github.io/spatialsnippets/)

## Workflow structure

The workflow is split into six sections: quality control, doublet detection, dataset integration, cell annotation, pseudobulking and differential gene expression, and functional enrichment analysis:

![scRNA-seq workflow overview](img/workflow_overview.png)

### Input data

This workflow has been designed to be run after initial pre-processing with the [`nf-core/scrnaseq`](https://github.com/nf-core/scrnaseq/) Nextflow pipeline. That pipeline takes the raw sequencing data and performs genome alignment and counting of reads/UMIs per gene and per cell. The `nf-core/scrnaseq` pipeline outputs each sample's count matrices and metadata in multiple formats, including an R data file (`.Rds`) containing a Seurat data object. Your data must be in this format to begin working through these notebooks. **Importantly**, this pipeline expects the **filtered** Seurat data, which has had non-cell barcodes removed. For version `4.0.0` of the `nf-core/scrnaseq` pipeline (the latest version at the time of publishing these notebooks), these RDS files can be found in the `results/` output directory at `results/cellranger/mtx_conversions/<SAMPLE_ID>/<SAMPLE_ID>_filtered_matrix.seurat.rds`.

### Quality control

The first notebook in this workflow takes the input `.Rds` files containing your pre-processed Seurat data - one file per sample - and performs some basic quality control analyses to detect and remove low quality cells. This is an interactive process where you will be required to select thresholds for filtering. The notebook includes some interactive plots and figures to help guide your decisions in this process. We also perform initial normalisation and transformation of your count data to account for library size differences between samples.

The output from this first stage is a new series of `.Rds` files - again, one per sample - containing your filtered data.

### Doublet detection

The second notebook takes the output files from the quality control notebook and works through identifying doublets in your data. The cell capture process for single cell sequencing is not perfect and can result in multiple cells being captured together and given the same cellular barcode. This typically only affects a small proportion of the cell barcodes, and methods are available to detect these **doublet** and **multiplet** barcodes. In this notebook, we use the R library `DoubletFinder` for this purpose. By default, we remove these doublets from your data, as they will confound your results, although we give you the option of leaving them in and simply having them annotated as such.

The output from this stage is another series of `.Rds` files containing the doublet-free (or doublet-annotated) data - one file per sample.

### Dataset integration

The third notebook in this workflow takes the output of the doublet detection stage and performs dataset integration. This is a vital step that merges your data into a single object and helps to account for batch effects between samples. Without this step, you may find that cells will form clusters based solely on the sample they are from rather than by true biological differences. These batch effects will confound your analyses and make interpretation of results difficult or impossible.

In addition, this notebook works through a final round of data transformation and normalisation, followed by dimensionality reduction and cell clustering. These normalised data and clusters will be used in all downstream analyses.

The output from this stage is a single `.Rds` file containing the merged and integrated Seurat dataset. All downstream analyses will be performed on this single dataset.

### Cell annotation

The fourth notebook in this workflow takes the merged, integrated, and normalised data from the previous notebook and performs cell annotation.

We automatically annotate cells by cell cycle and cell type using public databases. We also provide you with an opportunity to supply curated marker gene lists for cell types that you are interested in, which we use to score and annotate your cells with.

The output from this notebook is an `.Rds` file containing your annotated single cell data

### Pseudobulking and differential gene expression analysis

After annotation, we perform pseudobulking, which sums together the counts from all cells within a cluster and treats the cluster like a single sample in a bulk RNA sequencing analysis. This has some important advantages, primarily allowing us to use existing bulk RNA sequencing tools and simpler, higher-powered statistical tests for analysing your data.

The pseudobulked data is then used to perform differential gene expression analysis and pathway enrichment analysis.

The outputs from this notebook are:

- An `.Rds` file containing your pseudobulked data
- An `.Rds` file containing your differential expression results.

### Functional enrichment analysis

The final notebook runs functional enrichment analysis (FEA), which helps to identify common pathways or gene sets that are enriched for differentially expressed genes. We run two forms of FEA: over-representation analysis (ORA) and gene set enrichment analysis (GSEA), which differ in their methodologies and provide two complementary analyses of the gene sets that are enriched in your data.

The output from this notebook is a collection of web reports summarising the pathway enrichment analyses.

## How to use the Quarto notebooks

The notebooks are written in the Quarto format - a format very closely related to R markdown. This format allows code to be interspersed with human-friendly text that explains what we are doing at each step. It also allows you to generate a styled HTML document at the end to save a record of the analyses you have run.

We recommend using [RStudio](https://posit.co/download/rstudio-desktop/) to run each notebook within the [notebooks/](notebooks/) directory. We also recommend running the notebooks in `Visual` mode, as it helps to distinguish between the text and code blocks and reduces distractions from formatting code:

![Viewing Quarto notebooks in RStudio in source mode](img/rstudio.qmd.source.png)

![Viewing Quarto notebooks in RStudio in visual mode](img/rstudio.qmd.visual.png)

Each chunk must be run sequentially. This ensures reproducibility and that objects saved in your R environment do not get mixed up.

At the end of each notebook, we also recommend restarting your R session to clear large objects from the workspace.

Some chunks will require your input for setting parameters that will be unique to your data.

At other points, we will generate template files within the [inputs/](inputs/) directory that you will need to edit in order to proceed.

In both cases, the notebooks will use the following alerts to let you know when an action is required:

---

**❱❱❱ ACTION ❰❰❰**

1. This is an example of an action that you will need to complete before proceeding.

---

### Platform

Single cell sequencing data is typically quite large, and processing more than a handful of samples can quickly require more computing resources than your typical laptop or desktop computer will have.

While these notebooks will work on your local computer, we have designed them with high-performance computing environments in mind. We recommend using a cloud- or HPC-hosted RStudio server to run these notebooks. We have tested the notebooks successfully on NCI's Australian Research Environment (ARE) - a web-based interface to the Gadi HPC, with the ability to run an RStudio server with the resources necessary to process large numbers of samples together.

As a consequence, we also only recommend running on Unix-like systems (e.g. Linux and Mac). These notebooks should also run on Windows, but they haven't been fully tested on that platform and so it is possible you may run into unexpected issues. Most HPC- and cloud-based environments are based on Linux and as such these notebooks will run well on these platforms.

### Rendering documents

Once you have run through all of the notebooks, you can render ("Knit") everything into a human-friendly HTML document. You can do this by running the following command in a terminal, within the top-level project directory:

```{bash}
quarto render
```

When rendering, the notebooks will avoid running expensive operations and will instead use the saved data objects created when running the notebooks interactively. This ensures that they render quickly and efficiently.

## Installation

These notebooks are based on the R programming language and use a number of bioinformatics R packages, in particular Seurat for single cell sequencing analysis. We have provided an R script in this repository at `install/install.R` which will install all the required packages.

The notebooks have been tested with R version `4.4.2`. Other versions of R may work but are untested at this stage. We recommend using this version of R if possible.

### Local installation

If you are running the notebooks locally on a desktop or laptop computer, you can simply run the installation R script like so:

```bash
Rscript install/install.R
```

On Mac, this install script should run without any other prerequisites.

If you are installing the libraries on Windows, you will additionally need to install [RTools](https://cran.r-project.org/bin/windows/Rtools/) prior to running the installation script.

**Please note** that local computers may struggle to handle larger datasets due to the large memory requirements, especially during the doublet detection and integration steps. If running locally, we recommend having at least 32GB of memory available.

### Using the `if89` R libraries on NCI

The notebooks are intended to be run on NCI's Australian Research Environment (ARE) platform, as this provides a way of running a web-based interactive R session on a high-performance computing system using an RStudio server. However, installing the required R packages on this system can be a little tricky, so we have pre-installed the necessary libraries on the `if89` NCI project. This is the Australian BioCommons Tools and Workflows project, which hosts a range of common bioinformatics tools, reference datasets, containers, and workflows that are available to all NCI users. If you are already an NCI user, you may request access to this project via the [NCI web portal](https://my.nci.org.au/mancini/project/if89). You can find more information about this project on the [Australian BioCommons GitHub Pages site](https://australianbiocommons.github.io/ables/if89/).

The pre-installed R libraries are located in the `if89` gdata storage: `/g/data/if89/R/scrna-analysis`.

### Installation on NCI

If you would prefer to install the required packages on NCI yourself into your own directories, we have also provided a bash script for this purpose - `install/install_nci.sh`. To use this script, you will first need to log into NCI's gadi:

```bash
# Replace "user" with your NCI username
ssh user@gadi.nci.org.au
```

Next, clone this repository to a convenient location. This will also be where you will be running the notebooks, so it is a good idea to choose a location with a large amount of storage space. We recommend using the `scratch` filesystem as a temporary location for running these notebooks.

```bash
# Replace "project" with your NCI project code
# or choose another location
cd /scratch/project/

git clone https://github.com/Sydney-Informatics-Hub/scrna-analysis.git

cd scrna-analysis/install
```

The installation script can be run interactively on the login node with the following command:

```bash
./install_nci.sh
```

By default, it will perform a dry run of the installation by telling you where the R libraries will be installed. It will also print out the `R_LIBS_USER` environment variable definition you will need to use later on to run the notebooks (see [Running on ARE](#running-on-are) below). The command to set this will also be saved in a new file called `install/setenv.sh`.

```console
R libraries will be installed to the following path:

/g/data/project/R/scrna-analysis/4.4

When running the notebooks, you will need to set the R_LIBS_USER environment variable to this path:

R_LIBS_USER=/g/data/project/R/scrna-analysis/4.4

*** DRY RUN ONLY ***
To submit the installation job to the cluster, run this script again with the --submit flag, or run the following command:

qsub -P project -l storage=gdata/project+scratch/project -v PREFIX='/g/data/project' install_nci.submit.sh
```

```bash
cat setenv.sh
```

```console
R_LIBS_USER=/g/data/project/R/scrna-analysis/4.4
```

Note that by default, the installation path will be `/g/data/project/R/scrna-analysis/4.4`, where `project` is your default NCI project code. You can override the project by using the `--project` parameter:

```bash
./install_nci.sh --project ab01
```

```console
R libraries will be installed to the following path:

/g/data/ab01/R/scrna-analysis/4.4

When running the notebooks, you will need to set the R_LIBS_USER environment variable to this path:

R_LIBS_USER=/g/data/project/R/scrna-analysis/4.4

*** DRY RUN ONLY ***
To submit the installation job to the cluster, run this script again with the --submit flag, or run the following command:

qsub -P ab01 -l storage=gdata/ab01+scratch/ab01 -v PREFIX='/g/data/ab01' install_nci.submit.sh
```

You can also select a different installation prefix with the `--prefix` parameter. The installation path will always be `${PREFIX}/R/scrna-analysis/4.4`:

```bash
./install_nci.sh --project ab01 --prefix /scratch/ab01
```

```console
R libraries will be installed to the following path:

/scratch/ab01/R/scrna-analysis/4.4

When running the notebooks, you will need to set the R_LIBS_USER environment variable to this path:

R_LIBS_USER=/scratch/ab01/R/scrna-analysis/4.4

*** DRY RUN ONLY ***
To submit the installation job to the cluster, run this script again with the --submit flag, or run the following command:

qsub -P ab01 -l storage=gdata/ab01+scratch/ab01 -v PREFIX='/scratch/ab01' install_nci.submit.sh
```

Once you are ready to submit the installation job to the cluster, add the `--submit` flag:

```bash
./install_nci.sh --project ab01 --prefix /scratch/ab01 --submit
```

The installation process may take ~2h to complete. Once finished, inspect the output logs to ensure all packages were correctly installed. A successful run should complete with an exit status of `0`. You can check this on NCI by looking at the resource usage summary that NCI adds to the end of the standard output log file, which will be named like `install_nci.submit.sh.o<JOBID>`, where `JOBID` is the numeric ID given to the job when it ran. For example:

```bash
tail -n 12 install_nci.submit.sh.o123456789
```

```console
======================================================================================
                  Resource Usage on 2025-01-01 12:00:00:
   Job Id:             123456789.gadi-pbs
   Project:            ab01
   Exit Status:        0
   Service Units:      13.36
   NCPUs Requested:    1                      NCPUs Used: 1               
                                           CPU Time Used: 02:05:06        
   Memory Requested:   8.0GB                 Memory Used: 3.36GB          
   Walltime requested: 04:00:00            Walltime Used: 03:20:25        
   JobFS requested:    64.0GB                 JobFS used: 687.52MB        
======================================================================================
```

It is possible that you may have received a non-zero exit status due to warning messages that can be safely ignored. To double-check that the required packages were installed correctly, you can run the `test_installation.R` script. This will try to load all the required packages and print a success or failure message at the end:

```bash
Rscript --vanilla test_installation.R
```

On success:

```console
...

=== REQUIRED PACKAGES WERE SUCCESSFULLY INSTALLED ===

```

On failure:

```console
...

=== REQUIRED PACKAGES FAILED TO INSTALL CORRECTLY ===

```

If the packages did not successfully install, you can try running the script again. Packages that were already installed should be skipped and so the install time will be considerably shorter. Also double-check the following:

- You have enough storage allocation for your install location.
    - By default, the script installs to the gdata allocation of your default NCI project, or your scratch allocation if you don't have a gdata allocation.
- You have enough compute allocation for the installation.
    - The installation process is very light on resources - on the order of 10SU - so this shouldn't be a problem unless you have already exceeded your quarterly quota.
- The job didn't run over the walltime allocation.
    - The script has 4 hours of walltime allocated to it, which should be more than enough to complete the process. You can check how long the job took by inspecting the resource usage summary as above. If the `Walltime Used` section is more than 4 hours, you can try updating the walltime allocation by following the instructions below under [Updating Installation Resource Allocations](#updating-installation-resource-allocations).
- The job didn't require more memory than requested.
    - The script has 8GB of memory allocated to it, which should also be enough to complete the installation. Check the resource usage summary to double check this wasn't exceeded. If it was, you can update the memory request. See [Updating Installation Resource Allocations](#updating-installation-resource-allocations) below for more details.

#### Updating Installation Resource Allocations

If your installation job failed because it ran out of resources (e.g. walltime or memory), you can manually update the installation script's header section to request more resources. The actual installation script - `install_nci.submit.sh` - has several lines at the top that start with `#PBS`. These lines are read by NCI's PBS scheduler software to determine the resources to give to the job.

```bash
#!/bin/bash
#PBS -q copyq
#PBS -l mem=8GB
#PBS -l jobfs=64GB
#PBS -l walltime=04:00:00
#PBS -l wd
```

If you need more memory you can change the line `#PBS -l mem=8GB`, e.g.:

```bash
#!/bin/bash
#PBS -q copyq
#PBS -l mem=16GB
#PBS -l jobfs=64GB
#PBS -l walltime=04:00:00
#PBS -l wd
```

Similarly, you can increase the requested walltime by updating the line `#PBS -l walltime=04:00:00`:

```bash
#!/bin/bash
#PBS -q copyq
#PBS -l mem=8GB
#PBS -l jobfs=64GB
#PBS -l walltime=08:00:00
#PBS -l wd
```

## Running on ARE

Here we provide step-by-step instructions for specifically running these notebooks on NCI's ARE platform. This assumes you have already installed all the required R packages (or are using the provided libraries on the `if89` project) and cloned the repository to a convenient location on Gadi by following the instructions above in [Installation on NCI](#installation-on-nci).

First, in a web browser, navigate to [are.nci.org.au](https://are.nci.org.au). Follow the prompts to log in using your NCI credentials.

On the main ARE dashboard, under "All Apps", select "RStudio". Do not select "RStudio (Rocker image)", as this is an older version of the RStudio app and isn't supported by these notebooks.

![ARE dashboard](img/are_dashboard.png)

On the new page that appears, you will be presented with a number of parameters to configure for your RStudio session. There is also a checkbox labelled "Show advanced settings", **which you will need to select**.

Use the table below to fill in the required parameters. If you don't see the input box for the parameter, ensure you have selected "Show advanced settings" first.

| parameter | value | notes |
| --------- | ----- | ----- |
| Walltime (hours) | `8` | It is better to request more than you will need as you won't be charged for time that isn't used. |
| Queue | `normalbw` |  |
| Compute Size | Custom (`cpus=1 mem=256G`) | Some of the steps in these notebooks require a lot of resources, so we recommend using a custom compute size with 1 CPU and 256GB. These notebooks are single-threaded, so more than 1 CPU is unnecessary, and 256GB is the maximum amount of memory you can request for a node on the `normalbw` queue. |
| Project | Your NCI project code |  |
| Storage | `gdata/project+scratch/project+gdata/if89` | Replace `project` with your NCI project code. If you installed the R libraries yourself rather than using the `if89` pre-installed libraries, you can omit the final `+gdata/if89` specification. |
| Modules | `R/4.4.2 gcc/14.2.0` | These notebooks are based on R version `4.4.2` and so **should only be used with this version**. Additionally, the pre-installed packages in `if89` use this version of R. The notebooks also require the `gcc` version 14.2.0 module to be loaded. |
| Environment variables | `R_LIBS_USER="/path/to/scrna-analysis/libraries",XDG_DATA_HOME="/scratch/<PROJECT>/<USER>/.local/share"` | There are two environment variables to set here. `R_LIBS_USER` tells R where to find the necessary libraries for running the notebooks, while `XDG_DATA_HOME` is used by RStudio to place various files and data. They should be separated by a comma. These variables are explained in further detail below. |

The two environment variables that need to be set are `R_LIBS_USER` and `XDG_DATA_HOME`.

The value for `R_LIBS_USER` will vary depending on whether you are using the `if89` pre-installed R libraries or if you ran `install/install_nci.sh`. If using the `if89` libraries, this should be set to the path `/g/data/if89/R/scrna-analysis`. If you installed the libraries yourself, use the path that was saved inside `install/setenv.sh` when you ran the installation script (see [Installation on NCI](#installation-on-nci) above).

Additionally, you will want to set the `XDG_DATA_HOME` environment variable. By default, RStudio will place working files and data in your home directory under `~/.local/share`, but on NCI your home directory has limited storage space and will quickly fill up. Instead, we recommend setting this variable to somewhere in the NCI scratch space, e.g. `/scratch/ab01/usr012/.local/share`, assuming a project ID of `ab01` and a user ID of `usr012`.

Once set up, your settings should look something like this:

![Typical ARE settings for RStudio](img/are_settings.png)

We recommend saving your settings so that you can quickly start a new session in the future. At the bottom of the page, click the checkbox labelled "Save settings". In the box below that, type a name for your saved settings and click "Save settings and close". This will take you to a new page with a list of your saved settings. At the top right of this list is a play button arrow. Click this to launch a new session of RStudio with your saved settings.

![Launching an RStudio session on ARE](img/are_launch.png)

You will be brought to a new page that shows the status of your session. It will start out as "Queued", but within a few minutes it should show the status as "Starting" and then "Running". Once running, a button will appear labelled "Connect to RStudio Server". Click this to open RStudio in a new browser tab.

![A queued RStudio job](img/are_queued.png)

![A running RStudio job](img/are_running.png)

Within RStudio, you can use the file browser at the lower right side to navigate to where you cloned the repository and start working through the notebooks.

![A brand new RStudio session](img/rstudio_home.png)

![Navigating to your repository in RStudio](img/rstudio_chdir.png)

![Your repository in RStudio](img/rstudio_repo.png)

![Opening the QC notebook in RStudio](img/rstudio_notebook.png)

You can access your saved settings anytime by going to the [My Interactive Sessions](https://are.nci.org.au/pun/sys/dashboard/batch_connect/sessions) page in the ARE dashboard. Under "Saved Settings" you should see the name you gave your settings. Clicking this link brings you back to the page where you can launch your session.

![Accessing your interactive sessions](img/are_navbar.png)

![Accessing your saved sessions](img/are_saved.png)

![Launching your saved sessions](img/are_launch.png)
