You can’t compile shaders on OS X. They have to be compiled on a Windows machine. I personally setup a little Amazon EC2 instance with a little web service that compiles the shaders. Throw in a simple little command line script on OS X to interact with the web service and you can then compile on Macs in a roundabout manner.
Just add the .fx files to the .mgcb file as you normally would. On windows just build it to the target platform and then copy the resulting .xnb files to you mac.
If you are going to be doing a lot of shader work I highly recommend using a VM running Windows and make yourself a little command line prog that compiles and spits out the shader right into your project.