When using a real time kernel, if a program running in realtime wants to execute an instruction, the operating system will drop everything and execute that instruction. The process to drop everything, prepare itself to execute the coming instruction is called "context switching", and this is a very expensive process... How quickly it can do this is the latency test we're doing in Cyclictest.
The overall system performance should decrease if you're trying to do too much for this.
A workaround is to increase any audio buffers, and see if that helps. At the end of the day, the idea is not to try and run everything in real time, it's all about striking a good balance to get the best audio out.