Installation¶

Mosaic needs Python 3.12 or newer. Install it into a conda environment:

conda create -n mosaic python=3.12 -y
conda activate mosaic
conda install -c conda-forge ffmpeg av py-opencv -y
pip install "mosaic-behavior[all]"

The PyPI distribution is mosaic-behavior and the import name is mosaic. An unrelated project named mosaic exists on PyPI, so never run pip install mosaic.

av and py-opencv are installed from conda-forge, before pip runs, so that the environment holds a single ffmpeg build. conda's py-opencv registers itself as opencv-python, which lets pip accept it, and its av satisfies the av>=18,<20 requirement of mosaic-media. ffmpeg and ffprobe must be on PATH. They are used for media indexing and for mosaic media transcode.

Confirm the install:

mosaic --help
python -c "from mosaic.core.dataset import Dataset; print('OK')"

One ffmpeg build per environment¶

Two ffmpeg builds loaded into one process crash it at a different point on each run, often with Trace/BPT trap: 5. On macOS every import also warns that AVFFrameReceiver is implemented twice, with or without a crash. Either symptom means the environment contains two builds.

A pip install without the conda step produces two builds in two ways. pip install mosaic-behavior resolves the full opencv-python wheel and, through mosaic-media[io], the av wheel. Each bundles a complete ffmpeg, and on macOS the two collide. On Linux mosaic's CI runs use both wheels without a crash, and on Windows OpenCV loads its ffmpeg as a separate library on demand. Separately, the lightning-action and movement extras require opencv-python-headless beside mosaic's opencv-python, and pip installs both wheels into one cv2 package on every platform. The conda-forge builds avoid both, because they link the environment's ffmpeg and one py-opencv registers both OpenCV names.

On macOS, install av and py-opencv from conda-forge, and do not install either one with pip afterward. To repair an environment that shows either symptom, remove the wheels and reinstall the conda builds:

pip uninstall -y av opencv-python opencv-python-headless
conda install -c conda-forge --force-reinstall ffmpeg av py-opencv -y

--force-reinstall restores conda files that a wheel overwrote or the uninstall removed.

Optional extras¶

pip install mosaic-behavior on its own is a complete analysis install: every track and label converter, all per-frame and social features, wavelets, scaling, t-SNE, k-means, Ward, ARHMM, the XGBoost classifier, overlays and crops. What [all] adds is the deep-learning surface (the heatmap localizer, the identity models and the sequence classifier), which means PyTorch, and on Linux about 4 GB of CUDA wheels.

Extra	Adds
`all`	`deep-learning` + `faiss`: the documented install
`deep-learning`	`torch` + `timm` — the heatmap localizer, all three identity models and the `sequence-classifier`
`faiss`	The `"faiss"` kNN backend for `global-tsne`; its default `"annoy"` backend needs nothing
`movement`	The movement smoothing and filtering features, and the xarray / netCDF4 / pynwb / sleap-io stack they sit on
`lightning-action`	Lightning-Action temporal action classifier
`feral`	FERAL V-JEPA behavior classifier, training and inference
`mcp`	The MCP SDK behind `mosaic agent serve`, mosaic's tools for an MCP client such as Claude Code. Not part of `all`

Two things to know before choosing:

faiss installs faiss-cpu. On Linux with CUDA, install faiss-gpu yourself instead.
lightning-action installs only on a machine with CUDA. It requires NVIDIA's nvidia-dali-cuda110. PyPI serves that as a source package whose build needs CUDA, and the extra therefore fails to install on macOS and on any machine without CUDA.

Note: for an install without CUDA, take PyTorch from its own index:

pip install "mosaic-behavior[all]" --extra-index-url https://download.pytorch.org/whl/cpu

Spectral features, the SLEAP analysis reader and the DeepLabCut HDF5 reader are not extras: PyWavelets, h5py and PyTables are base dependencies, because each one gates reading a file you already have rather than an integration you opted into.

Extra names that no longer exist¶

An install command saved from an older mosaic may name an extra that is gone. pip warns about an unknown extra and installs the base without it. pip install "mosaic-behavior[recommended]" therefore gives an environment with no PyTorch. Replace each retired name:

Retired extra	Install instead
`recommended`	`all`
`identity`, `localizer`	`deep-learning`
`gpu`	`faiss`
`pose`, `polo`	`deep-learning`, and for YOLO work the Ultralytics or POLO environment
`yolo-augment`	`--extra augment` on that environment's `uv sync`

Install from source¶

To work on mosaic itself, or to build one of the tool environments below that come from the repository, clone it and install it editable into the same kind of conda environment:

git clone https://github.com/EcodylicScience/mosaic.git
cd mosaic
conda create -n mosaic python=3.12 -y
conda activate mosaic
conda install -c conda-forge ffmpeg av py-opencv -y
pip install -e ".[all]"

The environment notes of CONTRIBUTING.md cover working against an unreleased mosaic-media. The notebooks under notebooks/ are in the repository only, and the pip package does not carry them.

Tools that run in their own environment¶

Seven of the tools mosaic drives are not installed by mosaic. Install each one yourself, then tell mosaic where it went, and mosaic launches it there. The Ultralytics, POLO, keypoint-MoSeq and VAME environments are built from files that live in the repository and that the pip package does not carry, so they need a clone.

Install it yourself	Used by	Tell mosaic where it is
TRex	`mosaic track trex`	`MOSAIC_TREX_CONDA_ENV`, or `MOSAIC_TREX_BIN` for the binary itself
SLEAP	`mosaic track sleap`, the `train-sleap` op	`MOSAIC_SLEAP_CONDA_ENV`, or `MOSAIC_SLEAP_BIN`
Lightning Pose	`mosaic track litpose`, the `train-litpose` op	`MOSAIC_LITPOSE_CONDA_ENV`, or `MOSAIC_LITPOSE_BIN`
Ultralytics	`mosaic track ultralytics`, the `infer-pose` and `train-pose` ops	`MOSAIC_ULTRALYTICS_CONDA_ENV`, or `MOSAIC_ULTRALYTICS_BIN` for the environment's `yolo` script
POLO	the `infer-points` and `train-points` ops	`MOSAIC_POLO_CONDA_ENV`, or `MOSAIC_POLO_BIN` for the environment's `yolo` script
keypoint-MoSeq	the `kpms` feature	`MOSAIC_KPMS_CONDA_ENV`, or `MOSAIC_KPMS_PYTHON` for the environment's interpreter
VAME	the `vame` feature	`MOSAIC_VAME_CONDA_ENV`, or `MOSAIC_VAME_PYTHON` for the environment's interpreter

A _CONDA_ENV variable names a conda environment, which mosaic activates with conda run. A _BIN or _PYTHON variable names a path directly. With neither set, mosaic looks on $PATH. For keypoint-MoSeq it looks in the environment built under src/mosaic/behavior/feature_library/external of the clone that mosaic is installed from instead, and for VAME under .../external/vame-env. A tool's location does not enter a run_id. Two machines that place a tool differently therefore agree on what a run is called.

Mosaic looks for a conda environment in each directory that CONDA_ENVS_DIRS lists, then in the envs directory of the conda installation, then in ~/.conda/envs. Only the name base names the installation itself. An environment kept elsewhere, such as in an envs_dirs directory of .condarc, needs its directory in CONDA_ENVS_DIRS, or the tool's _BIN or _PYTHON variable instead. Otherwise keypoint-MoSeq and VAME refuse the environment, and any other tool runs by its bare name, which conda run resolves on $PATH.

Video codecs each tool reads¶

Every tool reads H.264. Mosaic's analysis transcodes, media variants and imgstore exports are AV1 by default, and the software in a tool's environment decides whether the tool reads AV1:

Tool	Decodes video with	Reads AV1
TRex	the ffmpeg of its conda environment, which links libdav1d	Yes
Ultralytics, `infer-pose`, `infer-points`	PyAV, in the tool's environment	Yes
`infer-localizer`	PyAV, in mosaic's process	Yes
SLEAP	OpenCV, through sleap-io	With conda-forge's OpenCV, as below
Lightning Pose	NVIDIA DALI's `fn.readers.video`	No

The GPU does not change the AV1 column, although two error messages name hardware:

OpenCV prints "Your platform doesn't support hardware accelerated AV1 decoding" when its build lacks a software AV1 decoder. The Linux OpenCV wheel from PyPI lacks one, and fails on every GPU.
DALI raises "Unhandled codec 225" for an AV1 file on every GPU, including GPUs whose hardware decodes AV1. Its reader accepts H.264, HEVC, MPEG-4, VP8, VP9 and MJPEG.

SLEAP, Lightning Pose and Ultralytics were measured on a GeForce GTX 1080 Ti (compute capability 6.1) and an RTX 4000 Ada (8.9), and TRex on the GTX 1080 Ti. The versions were SLEAP 1.6, Lightning Pose 2.3.1 and 2.4.2, DALI 1.50.0 and 2.3.0, and TRex 2.0. The infer-* ops read through the same PyAV reader as Ultralytics.

Before SLEAP is handed an AV1 file, mosaic reads one frame of it with SLEAP's reader in SLEAP's environment, once per run. SLEAP is handed the file when the frame decodes. Otherwise the run is refused with the reader's error. Lightning Pose is refused an AV1 file at once, without a test, and reads an H.264 media variant instead. MOSAIC_ALLOW_TOOL_CODECS=av1 skips the test for SLEAP and the refusal for Lightning Pose.

A GPU older than Turing (compute capability below 7.5), such as a GTX 1080 Ti, limits every codec. It needs builds that still contain kernels for it:

DALI's cuda130 build fails on it with cudaErrorNoKernelImageForDevice. The nvidia-dali-cuda110 build that pip install lightning-pose installs works.
PyTorch from PyPI fails with "no kernel image is available". Install PyTorch from the cu126 index into the tool's environment, as the Ultralytics environments README describes.

TRex¶

TRex's conda package pins python=3.11 and numpy=1.26, so it needs an environment of its own:

conda create -n trex -c conda-forge -c trexing trex -y
export MOSAIC_TREX_CONDA_ENV=trex

SLEAP¶

SLEAP 1.6 brings PyTorch and Qt, so it installs on its own:

uv tool install "sleap[nn]"

This puts sleap-convert on $PATH, where mosaic finds it. mosaic runs inference with sleap-nn track from the same environment, so sleap-nn itself does not need to be on $PATH. Installed into a conda environment instead, name it with export MOSAIC_SLEAP_CONDA_ENV=sleap.

Give it an OpenCV that decodes AV1. SLEAP reads video through OpenCV, and the Linux OpenCV wheel from PyPI lacks an AV1 decoder. In a conda environment, replace it with conda-forge's headless build:

pip uninstall -y opencv-python opencv-python-headless
conda install -c conda-forge "py-opencv=*=headless*" "libopencv=*=headless*"

Remove the pip wheels first. conda does not remove them, and a wheel left installed provides a second cv2 beside conda's build. If the solve fails on packages that the environment's history pins, such as ffmpeg, add --update-all to the conda install.

That build links the conda ffmpeg beside it, which carries libdav1d, and satisfies SLEAP's unpinned opencv-python requirement. Take the headless build. conda-forge's default build loads conda's Qt, and SLEAP's PySide6 then fails to import with "No Qt bindings could be found" whenever the two Qt versions differ. sleap-io reads through OpenCV whenever OpenCV is importable, and a working PyAV in the same environment does not help.

Lightning Pose¶

Lightning Pose brings PyTorch, Lightning and NVIDIA DALI, and its video inference needs a Linux CUDA GPU:

conda create -n litpose python=3.10 -y
conda activate litpose
pip install lightning-pose
export MOSAIC_LITPOSE_CONDA_ENV=litpose

Hand it H.264 video. Lightning Pose reads video through DALI's fn.readers.video, which reads AV1 on no GPU. Give it an H.264 media variant, made with "codec": "h264", or media that was not transcoded for analysis. mosaic's test before each run allows AV1 under a DALI release whose reader adds it.

Ultralytics and POLO¶

These are two separate environments, and one machine can hold both. Each is built from a directory of the repository that the pip package does not carry, so build them from a clone. Build whichever you need:

cd src/mosaic/tracking/external/ultralytics-env
uv sync --python 3.12
export MOSAIC_ULTRALYTICS_BIN="$PWD/.venv/bin/yolo"

cd src/mosaic/tracking/external/polo-env
uv sync --python 3.12
export MOSAIC_POLO_BIN="$PWD/.venv/bin/yolo"

Pass --python 3.12 rather than letting uv choose. The committed lock was resolved for that interpreter and a newer one may not resolve at all.

Set MOSAIC_POLO_BIN and MOSAIC_ULTRALYTICS_BIN explicitly rather than relying on $PATH. Both environments install a yolo script, and a $PATH lookup cannot tell them apart.

Add --extra augment to either uv sync to install albumentations. Ultralytics then applies Blur, MedianBlur, ToGray and CLAHE at p=0.01 during training. Nothing records which way a run went, so opt in deliberately.

Both tools read an entry's files themselves: the clips of a split recording, and the chunk files of an imgstore whose chunks are video. Neither needs export-joined or export-store for those. A store whose chunks hold images, raw arrays, or Bayer or YUV data needs an export-store run first, as it does for TRex, SLEAP and Lightning Pose. mosaic prerequisites prints its command before the run, and the run's error message names it. infer-localizer reads a store directly, without an export.

keypoint-MoSeq¶

Built the same way, from its own directory of a clone of the repository, since the pip package does not carry it:

cd src/mosaic/behavior/feature_library/external
uv sync --python 3.13
export MOSAIC_KPMS_LICENSE_ACCEPTED=1

Built in the clone that mosaic is installed from, mosaic finds the interpreter by itself. Built anywhere else, name it: export MOSAIC_KPMS_PYTHON=/path/to/env/bin/python, or, for a conda environment, export MOSAIC_KPMS_CONDA_ENV=kpms.

Mosaic will not start keypoint-MoSeq until MOSAIC_KPMS_LICENSE_ACCEPTED is set to exactly 1. Harvard OTD licenses keypoint-MoSeq for non-commercial research and academic use only, and setting the variable asserts that your use is permitted; external/README.md has the terms in full.

VAME¶

Built the same way, in a separate directory of the clone:

cd src/mosaic/behavior/feature_library/external/vame-env
uv sync --python 3.12

Built in the clone that mosaic is installed from, mosaic finds the interpreter by itself. Built anywhere else, name it: export MOSAIC_VAME_PYTHON=/path/to/env/bin/python, or, for a conda environment, export MOSAIC_VAME_CONDA_ENV=vame. On a GPU older than Turing, such as a GTX 1080 Ti, reinstall torch from the cu126 index as the environment's README describes, or run on the CPU.

VAME is licensed GPL-3.0, which does not restrict running it. Whoever conveys a built environment conveys VAME's source under the GPL. The environment's README and NOTICE describe the obligation.

vame's device parameter chooses cpu, cuda or mps directly, or auto, which tries CUDA, then MPS, then the CPU. It does not enter the run identifier. One recipe runs under one name on a laptop and on a GPU server. The device changes the trained model. CPU and MPS fits differ materially from each other and from a GPU fit. Only training on the CPU is bit-reproducible. Two CPU fits of one recipe give identical models. The model bundle and the run log record the device that ran.

FERAL¶

FERAL runs inside mosaic's own process, but install it in an environment of its own: FERAL pins exact dependency versions, and installing [feral] beside a normal mosaic environment downgrades several packages in it. Build a second environment holding mosaic and [feral], and point it at the same datasets.

Third-party licenses, and what mosaic does about each, are recorded in NOTICE.

Platform support¶

mosaic runs natively on macOS and Linux. On Windows the core analysis pipeline runs natively, but several capabilities depend on components with no native-Windows build and need WSL2:

Capability	Native Windows	WSL2 / Linux	macOS
Core analysis (indexing, tracks, features, clustering, ARHMM, XGBoost, visualization)	Yes	Yes	Yes
keypoint-MoSeq (`kpms`)	Untested	Yes	Yes
VAME (`vame`)	Untested, CPU only from PyPI	Yes	Apple silicon, macOS 14+
FERAL (`feral`) -- `decord`	No	Yes	Yes
GPU kNN (`faiss`, `faiss-gpu`)	No (`faiss-cpu` works)	Yes	n/a
imgstore recordings (reading is native)	Partial	Yes	Yes
TRex tracking and pose-model training	Partial	Yes	Yes

For any No, Partial or Untested capability, install WSL2 (wsl --install in an admin PowerShell) and follow the Linux setup above inside Ubuntu. Under WSL, keep the repository and your datasets under home (~/) rather than /mnt/c.