Wither is harder. The ender dragon can be killed with freaking beds... Just shoot down his pillars and hes toast whatever you do. Then again, the wither battle can be simplified if you fight him underground in a mine. If you have good gear, either can be killed fairly easily.
I've always found the Ender Dragon to be harder myself. I can always kill the Wither Boss without being killed, but the Ender Dragon always gets me. However, judging by the other replies, I this is just me.