The matrix and quaternion calculations produce precision errors over time, these are typically rounding down errors so it will wind your position down.
You need to get the distance at the time you order the rotation. Like when you issue the order be it programatically or keypress. You save that distance ( Only ) at that time and keep reusing it later.
// class level declaration
float desiredOrbitalDistance = 1f;
then in the method
desiredOrbitalDistance = Vecor3.Distance(target - cameraPosition);
then as you rotate and you find the new direction as you do.
cameraDirection = Vector3.TransformNormal(cameraDirection,
Matrix.CreateFromAxisAngle(actor.world.Up, (-MathHelper.PiOver4 / 60) *
(Mouse.GetState().X - prevMouseState.X)));
You then normalize the result, in order to multiply the normalized unit length direction vector by the distance.
cameraPosition = cameraDirection * desiredOrbitalDistance;
then use your create look at