How to use copy-paste augmentation for YOLO5 object detection

[Auth0rM0rgan]

Hey @conradry,
I'm curios to know how can i use the copy-paste augmentation in YOLO. Im using albumentations to do some augmentation on my custom dataset and I have added the copy-paste aug as you said in the readme file but im not able to make it work! im getting an error that needs the mask params... which I dont have it! any idea how can I use it in yolov5 detection??

transform = A.Compose([
       A.NoOp(p=0.4),
       A.RandomScale(p=0.2)
       CopyPaste(blend=True, sigma=1, pct_objects_paste=0.5, p=1)
], bbox_params=A.BboxParams(format="pascal_voc"))

Thanks!

[conradry]

Copy-paste requires masks, it's designed for use with instance segmentation not object detection. Are you trying to take the entire contents inside of a bounding box and paste them into another image? I expect that would make object detection performance worse, but if you want to try it, it is possible.

You'll have to create masks from the bounding boxes that you have. Code like this would go into load_example in coco.py.

image = load_image(index) #some image (H, W, 3)
bboxes = load_bboxes(index) #some bounding boxes (N, 4)
bbox_classes = load_bbox_classes(index) #(N,) object classes

masks = []
copy_paste_bboxes = []
for ix, (bbox, bbox_class) in enumerate(zip(bboxes, bbox_classes)):
    mask = np.zeros(image.shape[:2], dtype=np.bool)
    ymin, xmin, ymax, xmax = bbox[:4] #this is pascal format, not coco
    mask[ymin:ymax, xmin:xmax] = True #a rectangle
    masks.append(mask)
    copy_paste_bboxes.append(bbox + (bbox_class, ix)) #assumes bbox is a tuple, not a list

#apply transforms, etc. (see coco.py)

[glenn-jocher]

@Auth0rM0rgan @conradry good news 😃! Your original issue may now be fixed ✅ in PR ultralytics/yolov5#3845. This YOLOv5 update implements copy-paste augmentation using the new copy_paste hyperparameter (set from 0-1 for 0-100% of labels copied and pasted). Note this is only available for segment labels, not for box labels.

To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload with model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

[MarkPopovich]

Hello,

I have an add-on to this question.

How do shadows affect the performance when using copy-paste augmentation?

Lets say I have a dataset consisting of 10 differently colored marbles in high definition. For training purposes, the marbles have been laid out on 10 differently colored backgrounds.

In the test set, all marbles can appear on all backgrounds. In the training set, each marble only appears on its own colored background. Unfortunately, the model appears to be learning the color-keyed backgrounds.

If I use copy-paste augmentation, but I do not bring the shadows produced by the marbles, will it work?

[Gokul14092001]

@Auth0rM0rgan @conradry good news 😃! Your original issue may now be fixed ✅ in PR ultralytics/yolov5#3845. This YOLOv5 update implements copy-paste augmentation using the new copy_paste hyperparameter (set from 0-1 for 0-100% of labels copied and pasted). Note this is only available for segment labels, not for box labels.

To receive this update:

* **[Git](https://github.com/ultralytics/yolov5)** – `git pull` from within your `yolov5/` directory or `git clone https://github.com/ultralytics/yolov5` again

* **[PyTorch Hub](https://pytorch.org/hub/ultralytics_yolov5/)** – Force-reload with `model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)`

* **[Notebooks](https://github.com/ultralytics/yolov5/blob/master/tutorial.ipynb)** – View updated notebooks  [![Open In Colab](https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb) [![Open In Kaggle](https://camo.githubusercontent.com/a08ca511178e691ace596a95d334f73cf4ce06e83a5c4a5169b8bb68cac27bef/68747470733a2f2f6b6167676c652e636f6d2f7374617469632f696d616765732f6f70656e2d696e2d6b6167676c652e737667)](https://www.kaggle.com/ultralytics/yolov5)

* **[Docker](https://hub.docker.com/r/ultralytics/yolov5)** – `sudo docker pull ultralytics/yolov5:latest` to update your image [![Docker Pulls](https://camo.githubusercontent.com/280faedaf431e4c0c24fdb30ec00a66d627404e5c4c498210d3f014dd58c2c7e/68747470733a2f2f696d672e736869656c64732e696f2f646f636b65722f70756c6c732f756c7472616c79746963732f796f6c6f76353f6c6f676f3d646f636b6572)](https://hub.docker.com/r/ultralytics/yolov5)

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

What changes should be implemented in augmentations.py if i am using coco2017 dataset for training?