Social behavior classification using a pre-trained model

LISBET enables automated annotation of social behaviors in animal datasets using pre-trained or custom-trained classification models. This workflow is ideal when you have a set of human-defined behaviors and annotated examples, and you want to automate the scoring of large datasets with high consistency and speed.

Requirements:

• A pre-trained LISBET classification model (or a model you have fine-tuned)

• A keypoints dataset in a supported format (e.g., DeepLabCut, SLEAP, or movement)

Step 0: Prepare the data and model

Ensure your keypoint data is organized according to LISBET’s requirements (see Data Preparation). You can use your own data or fetch a public dataset, for example CalMS21 (Sun et al. 2021), using:

$ betman fetch_dataset CalMS21_Task1

To use a pre-trained model, download one from the LISBET model zoo:

$ betman fetch_model lisbet32x4-calms21UftT1-classifier

Step 1: Annotate behaviors

Run the LISBET classifier on your dataset to generate behavior annotations. The main command is:

$ betman annotate_behavior \
   --data_format=DATAFORMAT \      # Format of your keypoints dataset
   --fps_scaling=FPS_SCALING \     # Your dataset FPS / model training FPS
   --data_scale=DATA_SCALE \       # Original video frames dimensions in pixels
   -v \                            # Enable verbose mode
   DATA_PATH \                     # Path to the keypoints dataset
   MODEL_PATH \                    # Path to model config (YAML)
   MODEL_WEIGHTS                   # Path to model weights (.pt)

where DATAFORMAT indicates the format used to store the keypoints, FPS_SCALING is the ratio between the frame rate (FPS) of your dataset and the one used by the pre-trained model (e.g., 0.833 if your dataset was acquired at 25 FPS and you are using a model pre-trained on the 30 FPS CalMS21 dataset by Sun et al. 2021), and DATA_SCALE is the original video frames dimensions in pixels (e.g., 1024x570 for CalMS21). If –data_scale is not provided, LISBET will scale the data in the required (0, 1) range by computing the min and max values of the keypoints in the dataset.

For example:

$ betman annotate_behavior \
   --data_format=movement \
   --data_scale="1024x570" \
   -v \
   datasets/CalMS21/task1_classic_classification \
   models/lisbet32x4-calms21UftT1-classifier/model_config.yml \
   models/lisbet32x4-calms21UftT1-classifier/weights/weights_last.pt

The output will be CSV files with predicted behavior labels for each frame, saved in the specified output directory.

Example: Adapting Keypoints from a New Dataset

LISBET models require the input keypoints to match exactly (in both order and naming) the configuration used during model training. If your dataset uses different keypoint names, contains extra keypoints, or uses a different order, you can adapt it using the –select_coords and –rename_coords options.

1. Inspect the Model’s Expected Keypoints

Before running inference, check which keypoints and order the model expects:

$ betman model_info models/lisbet32x4-calms21U-embedder/model_config.yml

This will print the list of required keypoints (see the input_features field).

2. Adapt Your Dataset

Suppose your dataset has the following keypoints: body, nose, earL, earR, neck, hipsL, hipsR, tail, and the individuals are named experimental and stimuli. This is the configuration used in the SampleData dataset provided with LISBET. The model expects only nose, left_ear, right_ear, neck, left_hip, right_hip, tail for each individual, named resident and intruder.

You can adapt your dataset as follows:

$ betman compute_embeddings \
    --data_format=DLC \
    --fps_scaling=0.833 \
    --select_coords="*;*;nose,earL,earR,neck,hipsL,hipsR,tail" \
    --rename_coords="experimental:resident,stimuli:intruder;*;earL:left_ear,earR:right_ear,hipsL:left_hip,hipsR:right_hip" \
    datasets/sample_keypoints \
    models/lisbet32x4-calms21U-embedder/model_config.yml \
    models/lisbet32x4-calms21U-embedder/weights/weights_last.pt

• –select_coords drops the extra keypoint body and ensures the order matches the model.

• –rename_coords maps your dataset’s individual and keypoint names to those expected by the model.

3. What If You Include Extra Keypoints?

If you try to include a keypoint that the model was not trained on (e.g., body), LISBET will raise an error:

$ betman compute_embeddings \
    --data_format=DLC \
    --fps_scaling=0.833 \
    --select_coords="*;*;body,nose,earL,earR,neck,hipsL,hipsR,tail" \
    --rename_coords="experimental:resident,stimuli:intruder;*;earL:left_ear,earR:right_ear,hipsL:left_hip,hipsR:right_hip" \
    datasets/sample_keypoints \
    models/lisbet32x4-calms21U-embedder/model_config.yml \
    models/lisbet32x4-calms21U-embedder/weights/weights_last.pt

This will fail with an error about incompatible input features, because the model does not expect the body keypoint.

4. Summary

• Always match the keypoints (names and order) to the model’s input_features.

• Use –select_coords to drop extra keypoints and reorder as needed.

• Use –rename_coords to map your dataset’s names to the model’s expected names.

• Use betman model_info to inspect the model’s required keypoints before running inference.

Advanced: Fine-tuning a classifier

If you want to adapt LISBET to your own annotation style or behaviors, you can fine-tune a model using your labeled data. See Fine tuning a classification model on a custom dataset for a step-by-step guide.

References

Sun, J. J., Karigo, T., Chakraborty, D., Mohanty, S. P., Wild, B., Sun, Q., Chen, C., Anderson, D. J., Perona, P., Yue, Y., & Kennedy, A. (2021). The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions (arXiv:2104.02710). arXiv. https://doi.org/10.48550/arXiv.2104.02710