I have a dedicated computer running a Minecraft server that I use to play with my brother (because hosting one from an OptiPlex GX280 is terrible). An even still, performance is bad. I put Lubuntu 17.04 64-bit on it, gave it 3 GB of RAM access (out of 4) and highest priority (10), but performance is still sluggish. The CPU usage is just about 50% maximum. A few times, we timed out because of this performance issue. What else can I do to improve this? Obviously, I'm missing something.
Quite honestly, that processor in your server isn't all too great. It has a rather slow clock speed and is nearly 10 years old. 4GB of ram, DDR2 nonetheless, also isn't very much once the operating system loads up.
Terrain generation is going to be your biggest hurdle. I would recommend pre-generating your world. Doing so will minimize lag spikes while people are on the server.
I later found out that the vanilla server is not worth it. Switching to the Spigot server resolved our problem.
Also, those specifications are the fastest I've got compared to the other computers I have in my collection. I host this server from this computer not only for speed but also in case we lose access to the internet.