Thats a screen space square masking function it wont mask a circle in a sprite you draw. That function is masking a square area on the screen.
If your intent is to mask a circular area in a sprite you are drawing in a arbitrary location, on a per sprite basis then it would be something like.
float4 tex = tex2D(BaseTexture, texCoord);
float2 dist = distance (texCoord.xy, float2(0.5f,0.5f); // .5 .5 is the center of the image
float inbounds= saturate(sign(dist - threshold));
// if you are pre-multiplying your alpha and not using non premultiplied you might need to call clip(inbounds) and remove the above saturate.
// blendstate on the graphics device affects if the (alpha) portion of rgba is considered directly.
return tex * inbounds;
were threshold is some distance 0 to 1 in relation to the sprites edges to its center by which to shrink or expand the circle.
if you need something based on a square position like that one seems to do then you can actually just set the scissors rectangle on the graphics device (see jjags reply below) or use a shader.
Note the below forces spritebatch to use the vertex shader in the .fx as well you could probably make it much more efficient it was a test example though.