If I'm not mistaken the bearings have a collar attached to them.
Here's a question. Is it just the set screws in the collars and the bearings that control lateral movement of the shaft ?
In other words if you loosen alll the set screws on the collars and bearings there's nothing to stop the axle from sliding inward or outward.