Prepare Your Microscopy Data

Loon Will Guide You

To use Loon, you must format your microscopy data according to Loon's requirements before uploading it.

Loon will handle creating the full expected data format below for you.

When uploading data, the Loon software will prompt you to:

1. Create New Experiment
2. Select Files
   • Add one or more imaging locations, each with unique ID
   • For each imaging location, you select:
     • A .zip file of your cell images
     • A .zip file of your cell segmentations
     • A .csv file of your cell metadata
3. Define Cell Variables
   • You map required variables (e.g., frame, time, id, parent, mass, x, y) to columns in your cell metadata .csv file.
4. Review
5. Upload

note

If using Local Loon instead, Loon will not guide you. You must prepare your data into the expected data format yourself.
This removes the time burden of uploading to your server, however!

---

Expected Data Format

Loon expects:

• Experiment Index aa_index.json: Define your cell imaging experiments (E.g. Exp. 1, Exp. 2)
• Experiments .json: Metadata about this experiment (experiment name, cell attribute names e.g. time, mass)
  • Imaging Location folders each containing:
    • Cell Images as a folder of .tiff files
    • Cell Segmentations (boundaries outlining each cell) as a folder of as GeoJSON files
    • Cell Metadata as a .csv file (cell's id, mass, x, y etc. over time)

Recommended example file structure

your-microscopy-data/            # Your data file, choose any name
├── aa_index.json                # Defines your cell imaging experiments (e.g. Exp. 1, Exp. 2)
├── experiment_1/                # Your experiment file, any # of experiments, any names
│   ├── experiment_1.json        # Required metadata about this experiment (name, etc.)
│   ├── location_A/              # Your imaging location file, any # of loc's, choose any name
│   │   ├── images_A/            # Folder of .tiff files (cell images)
│   │   ├── segmentations_A/     # Folder of GeoJSON files (cell boundaries)
│   │   └── Table_A.csv          # Cell metadata (id, mass, x, y, etc. over time)
│   ├── location_B/
│   │   ├── images_B/
│   │   ├── segmentations_B/
│   │   └── Table_B.csv
│   └── ...                      # More locations as needed
├── experiment_2/
│   └── ...
└── ...

Cell metadata and segmentations are not generated by Loon.
As an initial step in your workflow, we recommend using software such as LiveCyte and/or TrackMate to generate metadata.

Loon Data Specification

Optional Read: If you'd like to learn more, we specify each data component below.

If using Local Loon, you will need to read these specifications.
Make sure to format your data file structure correctly (see above)

aa_index.json: Define Your Experiments

This file contains a list of your experiment metadata files.
Required:

• "experiments": attribute.
• File must be in .json format.

Choose any names for your experiment files e.g. "experiment_1.json"

aa_index.json File Example:

{
  "experiments": [
    "experiment_1.json",
    "experiment_2.json",
    "experiment_3.json",
    "experiment_4.json",
    "experiment_5.json",
    "experiment_6.json",
    "experiment_7.json",
  ]
}

Experiments

Each of your imaging experiments require an experiment metadata file as a .json file.
This describes the experiment for Loon.

In short, experiment metadata .json files require:

Attribute	Definition
name	Name of the experiment as it should appear in the Loon dashboard.
headers	The list of column names in the CSV feature tables. The order should match the CSV files.
headerTransforms	Defines the name of certain special columns (time, frame, id, parent, mass, x, y). This is optional if the name already exactly matches in headers. See the table below for information about these special columns.
locationMetadataList	A list of imaging location metadata. Each imaging location will include an id, tabularDataFilename, imageDataFilename, and segmentationsFolder. See the table below for more information on each of these.
compositeTabularDataFilename	Specifies the path to the combined tabular data file. See the section below for more information.

Example: Experiment Metadata .json File

{
  "name":"ExperimentOne",
  "headers": [
    "Frame",
    "Tracking ID",
    "Lineage ID",
    "Position X (µm)",
    "Position Y (µm)",
    "Pixel Position X (pixels)",
    "Pixel Position Y (pixels)",
    "Volume (µm³)",
    "Radius (µm)",
    "Area (µm²)",
    "Sphericity ()",
    "Dry Mass (pg)",
    "Track Length (µm)",
    "Parent ID"
  ],
  "headerTransforms": {
    "time": "Frame",
    "frame": "Frame",
    "id": "Tracking ID",
    "parent": "Parent ID",
    "mass": "Dry Mass (pg)",
    "x": "Pixel Position X (pixels)",
    "y": "Pixel Position Y (pixels)"
  },
  "compositeTabularDataFilename":"experiment1/composite_tabular_data_file.parquet",
  "locationMetadataList": [
    {
      "id": "Condition A",
      "tabularDataFilename": "experiment1/location_A/Table_A.csv",
      "imageDataFilename": "experiment1/location_A/images_A.companion.ome",
      "segmentationsFolder": "experiment1/location_A/segmentations_A",
      "tags":{
        "drug":"drug_1",
        "concentration":"0.1"
      }
    },
    {
      "id": "Condition B",
      "tabularDataFilename": "experiment1/location_B/Table_B.csv",
      "imageDataFilename": "experiment1/location_B/images_B.companion.ome",
      "segmentationsFolder": "experiment1/location_B/segmentations_B"
      "tags":{}
    },
    {
      "id": "Condition C",
      "tabularDataFilename": "experiment1/location_C/Table_C.csv",
      "imageDataFilename": "experiment1/location_C/images_C.companion.ome",
      "segmentationsFolder": "experiment1/location_C/segmentations_C",
      "tags":{
        "drug":"drug_2",
      }
    }
  ]
}

Experiment Metadata File Specifications

name

The name of the experiment as it should appear in the Loon dashboard.

headers

The list of column names in your cell metadata csv file

headerTransforms

Attribute	Definition
frame	The frame number indicates which number image the data row comes from in a sequence of images.
time	The time when the image was recorded. Often this is relative to the start of the experiment. If this is not explicitly recorded, then the the frame number can be used as a proxy.
id	The unique ID for a particular tracked cell. This should be the same across frames for that cell's lifetime.
parent	The id of the parent cell. If this is not tracked at all for an experiment, then map this column to the same one as the id column.
mass	The mass of the cell.
x	The X coordinate for the cell's center position in pixel space. (It does not matter what definition of center is used.)
y	Same, but for the Y coordinate.

locationMetadataList

Attribute	Definition
id	A unique name for this location. Can be anything, but will be displayed in the interface, so a more descriptive name is better.
tabularDataFilename	The location of the CSV file feature table for this experiment.
imageDataFilename	The location of the OME TIFF image file. This should be a *.companion.ome file.
segmentationsFolder	This folder contains all of the segmentation files for a given location. See the section on segmentations for more details.
tags	A JSON object containing key-value pairs that capture metadata about the particular location. See the tags section for more information and some examples.

tags (optional)

Tags are used to define metadata (such as conditions) about an individual location. This is used in the Loon UI to specify specific conditions corresponding to the location. The tags object has no restrictions. For example, locations have have completely different sets of tags, locations may have no tags, and locations can overlap on one more tags.

For example, suppose we have three locations. Your tags may look like this:

[
  {
    "id": "location_1",
    ...
    "tags": {
      "drug": "drug_1",
      "concentration":"0.1"
    }
  },
  {
    "id": "location_2",
    ...
    "tags": {}
  },
  {
    "id": "location_3",
    ...
    "tags": {
      "drug": "drug_3"
    }
  }
]

compositeTabularDataFile (optional)

This key specifies the location (relative to the root of the current experiment directory) of a "combined metadata table" as a parquet file.

This table must be the union of each of the individual location metadata csv files with an additional location column and the union of all tags separated as columns as well.

For example, suppose we use the example from tags section. Then, a sample of 6 rows of our parquet file would be something like this:

location	{rest_of_headers}	drug	concentration
location_1	. . .	drug_1	0.1
location_1	. . .	drug_1	0.1
location_2	. . .
location_2	. . .
location_3	. . .	drug_3
location_3	. . .	drug_3

Here, the empty spaces denote empty strings.

Imaging Location Folders

An Imaging Location is simply a folder containing a distinct imaging dataset of your choosing.
Created at a distinct location, time, or for a specific purpose.

Imaging Location folders each contain:

• Cell Images as a folder of .tiff files
• Cell Segmentations (boundaries outlining each cell) as a folder of as GeoJSON files
• Cell Metadata as a .csv file (cell's id, mass, x, y etc. over time)

Cell Images

Cell Images is a folder of .tiff (image) files or .ome.tiff (microscopy image) files.

We adhere to the ome-tiff library for cell images.

Segmentations Folder

Each imaging location should have a corresponding folder that contains all of the segmentation files.
The names of the files must correspond to the imaging frame.
That is, 1.json will contain all of the cell segmentations for the first frame, 2.json will contain the second frame, and so on.
Each JSON file must follow the GeoJSON specification.
In addition to the standard geometry attribute, the bbox attribute must be defined.
To link the segmentations with the corresponding metadata, the cell id defined in the feature table must be included as an ID in the GeoJSON properties object.

info

Often, segmentations are instead generated as *.roi files.
When uploading using MinIO, Loon will automatically convert the .roi files to proper GeoJSON format.
If you are instead using Loon without MinIO (i.e. using Local Loon), you will have to convert the .roi files to GeoJSON yourself.
Here is a Python script which can convert .roi to GeoJSON from one of our accompanying repositories.

Cell Metadata .csv

Cell Metadata is a .csv file of your choice containing:

• Rows: A cell at a certain time point
• Columns: Metadata about that cell

Example contents: example_metadata_table.csv

CELL_ID	LOCATION	PARENT_CELL	TIME	FRAME	MASS	POSITION_X	POSITION_Y	...
101	location_A	-	0	1	12.5	150	320	...
102	location_A	101	5	2	13.1	155	325	...
103	location_B	-	0	1	11.8	140	310	...
...	...	...	...	...	...	...	...	...

Loon requires these columns: frame, time, id, parent, mass, x, y
Those required columns can be named however you'd like in your file, in any order (see header transforms).
You can add any other metadata columns you would like.