Building in Public: Model Distillation & INT8 Quantization on Raspberry Pi

Running complex computer vision models on low-power devices in communication-denied environments is a core challenge for secure cyber-physical infrastructure. For my collaborative wildlife tracking project, I designed a pipeline to run real-time segmentations on a Raspberry Pi 5 without external GPU clusters.

1. The Teacher-Student Framework

Manual annotation of wildlife masks in dense forests is time-prohibitive. To solve this, I deployed heavy, open-vocabulary architectures (Grounding DINO + Segment Anything Model / SAM) on a high-fidelity workstation to function as our Teacher.

The teacher automatically ingested high-resolution footage, outputting exact bounding boxes and pixel segmentation polygon coordinates. This auto-labeled dataset (4,000 polygon matrices) was then used to train a lightweight Student model (YOLOv8-seg / YOLO11-seg).

2. Post-Training INT8 Quantization

To run inference on the Raspberry Pi 5 CPU, the model needed further compression. I exported the student weights to ONNX format and performed dynamic INT8 Quantization using the ONNX Runtime quantization library.

import onnx
from onnxruntime.quantization import quantize_dynamic, QuantType

# Compress model weights to QInt8
quantize_dynamic(
  model_input="yolov8_seg_student.onnx",
  model_output="yolov8_seg_student_int8.onnx",
  weight_type=QuantType.QInt8
)

This quantization step resulted in a 75% model size reduction (compressing the model down to ~12MB) while incurring negligible mAP accuracy loss.

3. CPU Inference Speed

By compiling the quantized model to the NCNN library (an optimized high-performance neural network inference framework), the Raspberry Pi 5 achieved steady CPU inference speeds of 5-8 FPS (equivalent to 25-30 FPS on specialized NPU accelerators).