Efficient and semantically guided nuclei segmentation in histopathology: state space models and vision–language integration

Real-world deployment of AI-assisted nuclei segmentation faces three critical barriers: cross-center performance degradation (8–15% accuracy drops), computational constraints(large-scale pretraining on 104 million histology patches; quadratic memory scaling limiting patch sizes), and insufficient semantic discrimination (48.5% panoptic quality plateau). This dissertation addresses these barriers through two complementary innovations designed for practical deployment without large-scale pretraining: CellViM, a pretraining-free state-space model with linear-time complexity, and CellVLM, a vision–language approach integrating frozen biomedical text guidance via multi-scale fusion. Both systems were evaluated with three-fold cross-validation on PanNuke (7,904 patches; 19 tissue types), statistical significance testing, and cross-dataset checks on MoNuSeg. CellViM achieved statistically non-inferior accuracy to strong transformer baselines (mPQ 0.483 vs 0.485, p=0.231) while reducing wholeslide inference time by 62% (120s to 45s per slide). CellVLM significantly improved semantic discrimination (mPQ 0.504 vs 0.485, p=0.012, Cohen’s d=1.89) while maintaining stable detection performance (F1 0.823 vs 0.820). These results establish that competitive nuclei segmentation accuracy is achievable without pretraining dependencies, and that domain-specific language guidance yields practically meaningful semantic gains (≈3.9% panoptic quality increase). Together, these advances bridge the gap between research-grade performance and deployment constraints, providing a practical pathway for AI-assisted pathology workflows supported by a reproducible evaluation framework and comprehensive failure analysis.