Developing Reliable AI Tools for Coronary Artery Calcification Segmentation: A Retrospective and Summary

A Note on This Research Journey

This article documents an exploratory research project that did not reach its original goal of building a reliable VLM-assisted system for autonomous coronary calcium segmentation and repair.

The project began with a promising hypothesis: a vision-language model might identify errors made by a conventional CAC segmentation model, distinguish coronary calcium from other high-attenuation structures, and guide a local refinement process. We implemented and evaluated several approaches, including open-source and closed-source VLMs, MedSAM-based refinement, DeepCAC, HU-based candidate generation, and anatomical classification.

However, the experiments showed that the proposed system was not yet reliable enough for automatic deployment. The main difficulties were tiny CAC lesions, domain differences between datasets, inaccurate VLM localization, strong dependence on bounding-box quality, and the lack of high-quality annotations for stents, pacemaker leads, prosthetic valves, and other challenging artifacts.

Rather than presenting the project as a successful method, this post serves as an archive of the research process and its negative findings. It summarizes what was attempted, what failed, why it failed, and what can still be learned from the results. In medical AI, establishing that a promising idea is not yet ready for reliable use can be just as valuable as demonstrating a successful model.

Research Goal

Our project investigates whether a vision-language model (VLM) can improve coronary artery calcium (CAC) segmentation produced by an existing AI-CAC model.

The original idea was to use a VLM to identify segmentation errors, including:

• missed CAC near lesion boundaries;
• false positives caused by aorta or valve calcification;
• coronary stents, pacemaker leads, and other metallic implants;
• mixed components containing both CAC and non-coronary high-density structures.

The intended system was:

AI-CAC segmentation
        ↓
VLM error diagnosis
        ↓
MedSAM or local segmentation repair
        ↓
HU and anatomical constraints
        ↓
Final mask and Agatston score

The central research question is whether a VLM can provide useful semantic and anatomical reasoning beyond a conventional segmentation model.

Experiments Completed

1. COCA and AI-CAC baseline evaluation

We audited the COCA dataset and confirmed that it contains two substantially different CT domains:

• gated cardiac CT, usually with a small cardiac field of view;
• non-gated chest CT, with a mixture of small, medium, and full-chest fields of view.

This difference is important because AI-CAC was designed primarily for non-gated chest CT. Performance on gated small-FOV images therefore reflects domain shift rather than a direct reproduction of the original AI-CAC setting.

Using multi-class NRRD annotations as a coronary reference, the AICAC baseline achieved approximately:

• Dice: 0.4185
• Recall: 0.3349
• Precision: 0.5577
• Agatston score MAE: 196.6
• Pearson correlation: 0.7418

The dominant problem was under-segmentation. Approximately 71.5% of error voxels were false negatives. However, non-coronary false positives were still clinically relevant, particularly around the mitral valve, thoracic aorta, and aortic valve.

2. DeepCAC comparison

We also evaluated DeepCAC on the same COCA cases using the NRRD coronary annotations as reference.

DeepCAC performed substantially worse under the current COCA preprocessing and domain configuration:

• Dice: approximately 0.05
• Recall: approximately 0.04
• Pearson correlation: approximately 0.14

This suggests that the current DeepCAC pipeline is not well adapted to the COCA data, possibly because of differences in resampling, field of view, intensity normalization, heart localization, or training domain. AICAC therefore remains the more useful baseline for the current project.

3. Open-source VLM experiments

We tested Qwen3-VL-8B using several input configurations:

• local crop only;
• full CT slice;
• full slice plus local crop;
• three adjacent slices;
• five adjacent slices;
• five slices plus high-resolution local crop.

The results were highly conservative. Qwen frequently classified all candidates as non-CAC:

• CAC sensitivity: 0
• specificity: 1.0
• balanced accuracy: 0.5

This was not a successful result. The high specificity was mainly caused by the model failing to produce positive CAC predictions.

ClinFusion-8B performed better than Qwen and produced non-zero CAC predictions. On the larger 445-candidate experiment, its best configuration achieved approximately:

• sensitivity: 0.18
• specificity: 0.86
• macro-F1: 0.50

However, its sensitivity for tiny calcifications remained very low. Adding a 192×192 local crop did not reliably solve the problem and sometimes reduced the model’s global anatomical understanding.

4. Closed-source VLM experiments

A closed-source Gemini model was also tested.

When given a clean full CT slice and asked to locate CAC with a bounding box, Gemini could often recognize that CAC was present. However, precise localization was much weaker:

• positive detection among valid responses: approximately 77%;
• center-point hit rate: approximately 68%;
• mean bounding-box IoU: only 0.054.

This shows that recognizing “there is probably calcium here” is much easier than accurately localizing a tiny coronary lesion.

When the mask was already provided and Gemini only had to classify its anatomical origin, performance was more encouraging. It could distinguish coronary, aortic, and valvular regions with meaningful accuracy. However, classification of specific coronary branches remained unstable.

5. MedSAM and MedSAM-Agent

MedSAM was evaluated as a local refinement model.

With a reliable ground-truth bounding box, MedSAM could produce reasonable masks. Expanding the bounding box was important: an approximately 16-pixel expansion combined with an HU constraint produced a CAC Dice around 0.65 in the upper-bound experiment.

However, this was an oracle experiment because the correct bounding box was already known.

MedSAM-Agent showed the same limitation. With seeded candidates, it could refine masks, but in the fully automatic setting its predicted bounding boxes were poorly aligned with tiny CAC. Therefore, MedSAM is useful as a conditional refinement tool, but not as an autonomous CAC detector.

Why the Approach Failed

The main failure was not simply that the VLM was “too small.” Several deeper problems contributed.

Tiny-object localization

Many CAC lesions occupy only a few pixels in a 512×512 CT slice. Even human observers may need multiple adjacent slices, appropriate windowing, and anatomical context. A general-purpose VLM may recognize the overall cardiac anatomy while missing the very small high-density lesion.

Insufficient anatomical context

A local crop makes the lesion larger, but can remove the information needed to distinguish a coronary artery from the aorta or a valve. Conversely, a full slice provides context but makes tiny lesions visually insignificant. This explains why neither full images nor fixed local crops worked consistently.

HU information was not naturally available to the VLM

A VLM sees a rendered image, not the original numerical HU volume. Windowing compresses and sometimes obscures the relevant intensity information. The model therefore cannot reliably determine whether a pixel exceeds 130 HU or whether a very high-density structure is calcium or metal.

MedSAM depends on the prompt

MedSAM does not search the entire CT autonomously. It segments the region indicated by a box or point. If the box is too tight, misplaced, or covers the wrong structure, the final mask is also compromised.

Lack of hard-negative examples

The most important limitation was the lack of high-quality annotated data.

COCA and the available NRRD annotations provide useful coronary, aortic, and valvular calcium labels, but they do not provide enough carefully annotated examples of:

• coronary stents;
• pacemaker or ICD leads;
• prosthetic valves and TAVR;
• mixed stent-CAC components;
• metal artifacts;
• difficult false-positive structures;
• subtle false-negative CAC lesions.

Without these labels, we cannot train a reliable CAC-specific VLM error classifier or a robust repair model. We can perform zero-shot tests and oracle upper-bound experiments, but we cannot honestly claim to have trained an end-to-end artifact-aware correction system.

Current Conclusions

The experiments lead to four main conclusions.

First, AICAC’s largest problem on the current COCA data is under-segmentation, although non-coronary false positives remain important in selected high-impact cases.

Second, open-source VLMs are currently not reliable standalone CAC detectors or automatic mask repairers. ClinFusion is better than Qwen, but its performance is still insufficient for direct mask modification.

Third, closed-source VLMs have useful anatomical reasoning ability, especially when a candidate mask is already provided. Their main value may be candidate interpretation, uncertainty estimation, and human-review assistance rather than direct pixel-level segmentation.

Fourth, MedSAM can refine a reliable candidate, but it cannot replace candidate generation. Its performance depends strongly on box quality, padding, HU constraints, and anatomical filtering.

Recommended Next Direction

The most defensible next system is:

AI-CAC mask
    + HU ≥130 candidates
    + cardiac/coronary anatomical ROI
    + morphology and cross-slice continuity
    ↓
structured classifier or calibrated VLM
    ↓
keep / remove / add / local re-segment
    ↓
MedSAM or local U-Net refinement
    ↓
HU and independent quality-control constraints

Before QLoRA or large-scale VLM training, we need to build a dedicated expert-annotated hard-case dataset. It should contain patient-level splits and candidate-level labels for coronary calcium, aorta, valves, stents, pacemaker leads, prosthetic valves, artifacts, and uncertain cases.

Until that dataset exists, the current work should be presented as a feasibility study and failure analysis. The strongest contribution is not that a VLM already solves CAC segmentation, but that the experiments identify the precise conditions required for a reliable CAC-specific closed-loop system.