The shader code was nothing to do with it, the glitch was obvious, one face of the environment map was wrong randomly.
I eventually fixed it by moving my render code from a separate thread onto the main thread. Annoying I have to do this, but only solution I can see.
I am now working on moving my input device management off the main thread to help re-balance core usage.