this is my first question here and I hope you can help me.

At the moment I'm training a binary classifier for medical images and my dataset is imbalanced with a ratio of roughly 0.8 (negative) to 0.2 (positive). My code is written with pytorch and pytorch lightning and I am using torchmetrics for evaluation.

So my question is if the torchmetrics AUROC is a good choice as evaluation metric for my model and if anyone has used it themselves for a similar dataset. The parameter average="weighted" should account for imbalance.

My implementation:

auroc_val = AUROC(num_classes=2, average="weighted")

auroc_val(prediction, target)

Torchmetrics link:

https://torchmetrics.readthedocs.io/en/latest/references/modules.html#auroc

Source code:

https://github.com/PyTorchLightning/metrics/blob/master/torchmetrics/classification/auroc.py#L27-L193

My understanding is that this implementation should work fine and the results look legit but I am asking because I already ran into some pitfalls when it comes to model evaluation.

Thanks in advance, comments on this are much appreciated!