Issue with drawing overlapping bounding boxes during the prediction phase after training a multiclass detection using RF-DETR

[enescicek]

I trained an RF-DETR model using my own dataset. When I run predictions using the model I obtained after training, I get multiple overlapping bounding boxes (sometimes overlapping for the same class, sometimes for different classes). I know that RF-DETR is an NMS-free model, but how can I resolve this issue?

@Borda

[JESUSROYETH]

Hi @enescicek :

This looks like duplicate model predictions rather than a problem in the drawing code. You are correct that RF-DETR does not apply NMS inside predict(): post-processing ranks the scores for all query/class pairs, keeps the top num_select, and then applies the confidence threshold. So one query can appear with two labels, while two different queries can still predict almost the same box. The one-to-one matching trains the model to avoid this, but it is learned behaviour rather than a hard inference constraint.

Two things are worth checking first. In #187, training from 10 to 20 epochs reduced the duplicates. In #434, the reporter went from 20 to 85 epochs and also changed the ONNX opset where possible; the duplicates became less frequent but still happened. This means loss convergence alone is not enough to confirm they are gone... it would be useful to measure on validation how often pairs above your confidence threshold have IoU > 0.9.

Also, the FAQ documents that increasing num_select without increasing num_queries returns duplicate query/class pairs instead of more distinct objects. However, the cross-class boxes in the screenshot do not prove that this setting was changed, because the same query can be selected for more than one class even when num_select == num_queries.

Could you share a minimal example with the RF-DETR version, model construction, train/predict settings, and the raw detections for one image? If the cross-class boxes have exactly the same coordinates, that would be consistent with one query being selected for two classes. It would also help to check that the class definitions are mutually exclusive and that the annotations use them consistently.

If the model is converged and the query settings are correct, NMS is a reasonable practical fallback:

detections = model.predict(image, threshold=0.5)  # confidence threshold
detections = detections.with_nms(
    threshold=0.5,  # IoU threshold; tune this on validation data
    class_agnostic=True,
)

Use class_agnostic=True only if one object cannot validly have both labels and different-class objects are not expected to overlap heavily. Otherwise keep the default class-aware NMS. This is also the workaround recommended in #187.