1) You may simply not have your renderstates set up to act as you think.
GraphicsDevice.SamplerState = SamplerState.PointClamp;
Be aware of what antialiasing does?
There are two main categories of texture filtering, magnification filtering and minification filtering.
2) You are mis-understanding some vital things.
When you make a point to say...
it's impossible because spritebatch is only casting integer values.
Your physical screen is made up of discret pixels (physical integer amounts).
The monitor itself has a physical resolution of pixels in integer values.
3) To say that you want to make
real smoother position scale.
What does this mean if you are rendering at the maximum resolution on a 1 to 1 scale ?
What does it mean to say you want to render at .33 of a pixel or 1.33 then when each rectangle integer position is equal to a physical screen position. Do you know what this will mean in terms of your current filter settings and resolution ?
4) SpriteBatch cannot simply be overriden.
You can download monogame alter it and build it yourself from source but i don't think you realize that any alterations you make to it will be futile.
If you are determined to build your own and need proof of the above. You can take a look at a couple of my old experimental test projects that basically do the things spritebatch does.
These are neither made as tutorials or for use in a real game or app they were simply tests and they have multiple tests and versions in the same project so you will have to follow the flow from program thru execution as each project also has multiple game1 classes basically.