Whole fly brain dataset inside a tracing environment

We have imaged a full adult female fly brain at 4x4x40 nm resolution (see Zheng, Lauritzen et al., 2018). The whole brain dataset is able to be downloaded here preconfigured with a CATMAID instance so you can get started tracing. Instructions for tracing are available at the main CATMAID site here.

The dataset version here, is an alignment that improves in most places upon that present in Zheng, Lauritzen et al. (2018). In the interests of open data however, we also provide the Zheng, Lauritzen et al. (2018) alignment, instructions for how the installation differs are present at the bottom.

You will need at least 12 TB of storage space for the data and recommend Ubuntu is used. Instructions on how to set up a known system that it works on are below the general installation instructions below.

We are a small team and can't provide much individual tech support. But below we have a hardware configuration known to work. Some help may be available by emailing .

A fly through of the paper alignment is shown below

Installation

Installation can be done very simply by cloning the git repo and then running the installation scripts, however, this assumes the docker-compose system is already installed (see below under hardware)

First steps

The simplest installation method after docker has been installed as per above is to clone the repo.

  • git clone https://www.github.com/bocklab/temca2-catmaid

This will create a folder called temca2-catmaid in whatever directory you were in.

Pyramid download

From within the temca2-catmaid/download directory, we can run the download script. On Ubuntu this is done through first installing the required libraries

  • pip3 install requests progressbar2
and then running the directory to the specific drive 
  • python3 download_sections.py --datapath /TEMCA2_RAID/v14_align_tps

This will then proceed to download the sections sequentially. This is a large download (~12 TB) and will take a long time. For more details see the README.md in the download directory.

CATMAID setup

To set up the CATMAID docker we should move back to the main github directory (or open a new terminal, since the previous download will take awhile) and download the top levels (for faster access) and the database file with some initial tracing

  • cd temca2-catmaid
./catmaid_in_a_box_install.sh

After this has run it should be initalised.

  • ./catmaid_in_a_box_run.sh

This will then make the CATMAID instance accessible from http://localhost:8000

A default user with tracing privileges is availabe under the username/password of temca2/temca2. New users can be created through http://localhost:8000/admin after logging in with the admin account (admin/admin).

Hardware recommendations

For simplicity, we recommend a few pieces such as an external RAID for hosting the image stack, feel free to change upon this, but note that some configuration would be needed.

OS and hardware recommendations

The base docker image assumes a standard 16.04 Ubuntu system. This doesn't have to be overly powerful. We have used a Dell Precision 5720 with 16 GB of RAM, i5-7600 processor and Ubuntu 16.04 pre-installed.

The following commands should then be run to ensure the software is up to date.

  • sudo apt-get update
  • sudo apt-get upgrade

The instructions for docker compose install should then be followed from here.

Image pyramid storage

The image pyramid is approximately 12 TB in total size. We recommend (and the instructions assume that) these are placed in a RAID-5 setup. We have used a Mobius™ 5-Bay FireWire 800 enclosure, and five Seagate ST4000NM0035 4 TB HDDs available, for example here, and found acceptable performance.

Image pyramid setup

Once the drives have been set up in a RAID-5 system (as per the manual) they should be formatted and setup to allow easy mounting in the future. As a first step, the disk location will be needed. This can be determined by running the following

  • lsblk --output NAME,LABEL,UUID,MOUNTPOINT,SIZE

This will give an output along the lines of:

  • NAME   LABEL     UUID                                 MOUNTPOINT                SIZE
sda                                                                           465.8G
├─sda1 ESP       CA75-A471                            /boot/efi                 600M
├─sda2 OS        D88D-F5C7                                                        3G
├─sda3 UBUNTU    8cec81af-9187-4e4c-9e71-1efbf75053b7 /                       430.5G
└─sda4           da65590e-d2ce-42ad-9422-04697f272ed5 [SWAP]                   31.7G
sdb                                                                            14.6T

You will want to look for a drive which has no LABEL, and a size of 14.6T. Assuming this is located at sdb these can be formatted as following (NB: double check this location and change as appropriate, if it is listed as sdc change all sdb to sdc)

  • sudo parted -a optimal /dev/sdb mklabel gpt
sudo parted -a optimal /dev/sdb -- mkpart primary ext4 1 -1
sudo mkfs.ext4 -b 4096 -i 65536 -L TEMCA2_RAID /dev/sdb1 -m 0

The directory these drives will be mounted to should then be set up and fstab modified.

  • sudo mkdir /TEMCA2_RAID
sudo mount -L TEMCA2_RAID /TEMCA2_RAID
sudo chown -R $USER:$USER /TEMCA2_RAID

In order to mount these drives upon reboot, the /etc/fstab file should be modified. This should be done using the unique device ID to this file

  • UUID=$(lsblk /dev/disk/by-label/TEMCA2_RAID --output UUID | sed -n 2p)
echo "UUID=$UUID /TEMCA2_RAID      ext4      rw,noatime,data=writeback   0    0" | sudo tee -a /etc/fstab

The base level directory should be created. Assuming you are downloading the recommended v14 alignment

  • mkdir /TEMCA2_RAID/v14_align_tps

Paper alignment

The image alignment used in the paper, FAFB v13, is no longer available for download. The original v13 tracing (with low resolution imagery) remain available for replication of numerical results in the paper, but should not be used for future tracing.