Suggestion: merge dataset in `val` and `train` folder to train for final epochs

[LeMinhNgan]

Search before asking

• I have searched the RF-DETR issues and found no similar feature requests.

Description

Hello,

Practically, after completion of a normal training pipeline until the last epoch or early-stop condition, we will:

1. Check train\loss and val metrics to confirm if the training already hit its limits and satisfy our requirement
2. Manually copy dataset in folder .\dataset_dir\val into folder .\dataset_dir\train; and create new _annotations.coco.json file to merge them all. There is no change to val dataset, but only train dataset. Purpose: utilize all labeled dataset for training before ending and exporting the final best checkpoint.
3. Extend the training argument epochs 10-20 extra epochs and resume on training from last.ckpt
4. Finish and use the best checkpoint for prediction or ONNX conversion

Suggestion:
Simplify step 2 by establishing a new training argument, for example called train_all_dataset:bool to let the program either automatically include the labeled dataset in .\dataset_dir\val into training (if set True) or train on .\dataset_dir\train dataset only (if set False)

Use case

No response

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[isaacrob]

I would not recommend doing this. Part of the function of the validation set is to be the basis of a decision between multiple different algorithms, such as architecture variations. The test set is meant to give you the estimated performance after a decision has been made. Training on val would make it so you cannot make such a decision. Additionally, continuing a training run by merging train and val risks overfitting to train data. I would suggest either taking your final model and finetuning on val by itself, or taking the known-working hyperparameters of the running formulation such as architecture or epochs and training from scratch on merged train and val.

[LeMinhNgan]

@isaacrob sorry, may be my unclear writing make you confused. You're right about monitoring overfitting.
However what i am telling is a different approach: after we confirm that the training hit the limit and need to stop the training (use early stop condition for example). We stop the training process, then what should we do with the validation data? Since labeled data is expensive, we don't want to waste them; we merge the valid dataset into train dataset and train for further 10 epochs to enrich the final checkpoint. There is not any monitoring or making decision in this final step, just close the project after the extra 10 epochs.

[isaacrob]

If you are training multiple kinds of models or playing with hyperparameters, it's better to never train on the validation data and only use it to choose which model or hyperparameters to use. Once you know which model and hyperparameters you're using, including training time, you could consider merging the two and training from scratch with those settings, then reporting your final accuracy using the test set.

But if you merge val into train and keep training, you risk overfitting to the training split in that merged set. A less risky approach might be to just do a single epoch on the val set itself, but then you wouldn't be able to compare approaches. If you're not looking to compare, then I'd recommend using val to see how long it takes to converge and then training from scratch for that number of epochs with the merged set.

Any model, especially a model that has already seen most of the data, has the potential to overfit within 10 epochs, so I would be cautious about using that heuristic.

[isaacrob]

If you'd like to be maximally rigorous, I'd suggest looking into k fold cross validation