My asm function works fine if its only one thread, but If try to use multi threading it makes the image almost fine, but there are some errors. I made sure that I push and pop registers that I use and I almost run out of ideas. My last idea is that things that I use and change in .data are same for both threads and maybe thats why image doesnt get processed the right way? The c++ code calling asm function and creating threads:

void prepareThreads(Sepia& ProcAdd, const HINSTANCE& hinstLib, 
    BOOL& fRunTimeLinkSuccess, int32& bytesPerPixel, const int32& width,
    const int32& height, byte* pixels, int thread)
{
    ProcAdd = (Sepia)GetProcAddress(hinstLib, "MyProc1");
    if (NULL != ProcAdd)
    {
        fRunTimeLinkSuccess = TRUE;
        int threadsAmount = 4;
        int sizeOfSubArray = (bytesPerPixel * width * height) / threadsAmount;
        std::vector<std::thread> threads;
        for (int i = 0; i < threadsAmount; i++) {

            threads.push_back(std::thread(ProcAdd, pixels + i * sizeOfSubArray, sizeOfSubArray, bytesPerPixel));
        }
        for (auto& th : threads) {
            th.join();
        }

    }
}

Start of asm procedure, in rbx i put pointer to array of bytes, in rcx I put number of pixels.

.data
vec1 real4 0.393, 0.769, 0.189, 0.0
vec2 real4 0.349, 0.686, 0.168, 0.0
vec3 real4  0.272, 0.534, 0.131, 0.0
vecReal real4 0.0, 0.0, 0.0, 0.0
vecResult real4 0.0, 0.0, 0.0, 0.0
vecMaxBytes dword 255, 255, 255, 255 
masks db 0, 4, 8, 12, 12 dup (-1)

.code
MyProc1 proc var1:QWORD, var2:WORD, jmpAddress:WORD
push rdx;
push rax
push rcx
push rbp
push rdi
push rsi
push rbx

mov rbx,RCX
mov rcx,RDX
mov rsi, 0