I am currently working on a convolutional neural network to classify images using Python 3.9, tensorflow 2.7 as well as its keras API. These images, however, not only consist of pixel data but also have one 'pmt-count'-value that is to be learned.

Unfortunately (since my network takes 2 inputs) I was unable to use keras ImageDataGenerator to augment data 'on the fly'. My workaround is to generate these images before feeding them to my network. Here's my (admittedly not too elegant) solution:

    def augment_data2(im_lb, im_pmt, im, size):
        df = pd.read_csv('data/pmt_ranges.csv')

        # Reading PMT ranges of each class for generating augmented PMT values
        class0_pmt_range = eval(df.iloc[0, 0])
        class1_pmt_range = eval(df.iloc[0, 1])
        class2_pmt_range = eval(df.iloc[0, 2])
        class3a_pmt_range = eval(df.iloc[0, 3])
        class3b_pmt_range = eval(df.iloc[0, 4])

        # Creating wrapper arrays, because network input has shape (len(images), im_height, im_width, channels) for images and (len(images), pmt value) vor pmt-values
        aug_im = np.empty((im.shape[0] * size, 11, 11, 1))
        aug_pmt = np.empty((im_pmt.shape[0] * size, 1))
        aug_lb = np.copy(aug_pmt)
        # The first i images, their labels and pmt are just copied to the array that will also contain the augmented images
        for i in range(im.shape[0]):
            aug_im[i] = im[i]
            aug_lb[i] = im_lb[i]
            aug_pmt[i] = im_pmt[i]
        

        datagen = ImageDataGenerator(vertical_flip=True,
                                     horizontal_flip=True,
                                     width_shift_range=.15,
                                     height_shift_range=.15,
                                     rotation_range=270,
                                     brightness_range=[.5, 1.0],
                                     zoom_range=[.75, 1.0]
                                     )
        # The rest of the array will be filled with augmented images using the datagen object
        for k in range(1, size):
            for j in range((im.shape[0] * k + 1), (im.shape[0] * (k + 1))):
                img_tmp = im[j - im.shape[0] * k]
                pmt_tmp = im_pmt[j - im.shape[0] * k]
                label = im_lb[j - im.shape[0] * k]
    
                reshape = np.expand_dims(img_tmp, 0)
                it = datagen.flow(reshape, batch_size=1)
                batch = it.next()
                image = batch[0].astype(np.uint)
    
                # Create random pmt value within the range of the images class
                if label == 0:
                    pmt_tmp = random.uniform(class0_pmt_range[0], class0_pmt_range[1])
                elif label == 1:
                    pmt_tmp = random.uniform(class1_pmt_range[0], class1_pmt_range[1])
                elif label == 2:
                    pmt_tmp = random.uniform(class2_pmt_range[0], class2_pmt_range[1])
                elif label == 3:
                    if class3a_pmt_range[0] <= pmt_tmp <= class3a_pmt_range[1]:
                        pmt_tmp = random.uniform(class3a_pmt_range[0], class3a_pmt_range[1])
                    else:
                        pmt_tmp = random.uniform(class3b_pmt_range[0], class3b_pmt_range[1])
    
                aug_lb[j] = label
                aug_im[j] = image
                aug_pmt[j] = pmt_tmp
    
        return aug_lb, aug_pmt, aug_im

Having split data into training, validation and test sets earlier, these augmented arrays are then passed as training data to the model like so:

history = model.fit([tr_im, tr_pmt], tr_lb, epochs=epochs, batch_size=32,
                        validation_data=([vd_im, vd_pmt], vd_lb))

Now first of all, it's not easy to read and also not too performant. I was wondering how to do it more efficiently. By the way, im not augmenting my validation data because I read that this type of data would be used to tune hyperparameters while training. Any thoughts on this too?

Should you be interested in more information about this project, please do not hesitate to ask.

Thank you very much for your time in advance.