Does no one remember the fact that the Phantoms were added because the community voted them in over three other mobs?
I personally have no issue with them.
^ This. Any time you're building, just bring a bed, and when it gets night time, lay in it for a second and then hop right out. If you do that consistently, then they should never spawn.