Im tired of trying to beat my brain with blocks trying to identify this myself. i cant find ANYTHING about this ANYWHERE. So here we go. please let there be someone out there capable of assisting me with this.
Storage: 3x 240gb SSD raid 0 for OS and current files host (700gb volume)
Storage 2: 5x 250gb HDD raid 5 for backups and other files (1.5TB volume)
Network: all hardware from server to WAN is gigabit. observed speeds in speedtest.net are 1,050 ish Mbps up AND down. ping is between 3 and 20. never more than that.
server is hardwired into the network.
OS: windows server 2019
VM: windows server 2016
ipv6 disabled
firewalls have been verified and turned off for specific tests
port forwarding has been verified and completely opened up for specific tests.
The Minecraft server:
assigned 20 gb ram
stored on OS SSD raid drive on host
running with official java 64bit on host with -server flag (not in VM)
Observations of the problem:
while running i see a LOT of lag notifications on the console with and without datapacks enabled.
1player is smoothish
2players gets laggy
3 players gets unplayable
EVEN WITHOUT DATAPACKS ENABLED THIS DOES NOT CHANGE
while running ive observed a continuous 0% CPU usage on ALL CORES. NOTHING SPIKES EVER.
even when a player connects there appears to be NO load on any of the CPU cores
server only pulls 7gb RAM max now. it used to pull 15gb regularly with players. thats why i gave it 20gb all to itself.
due to CPU idle speeds it is underclocking itself to 0.17ghz and is not even coming close to boosting to normal speeds.
(ive observered the the same problem happening while the server was using more CPU speed and overclocking itself to max speeds. i dont think the speed stepping isnt the problem as it persists while boosting even for other reasons)
eventually the server crashes. to which i get a watchdog message meaning a single server tick took more than 60 seconds.
IDK whats going on. but here is what has become apparent:
with no datapacks running. the server crashes due to 3 players playing while not even touching the cpu max speed or any speed for that matter and without running more close to the max ram. and while stored on SSD raid. with no other services running on the server.
root cause, the server cant tick for some reason.
now this is weird, are you ready? while i can visibly see the server struggling with mobs not able to move more than just twitching in place instead of walking, the chat has 0 delay. normally chat suffers with the rest of the server. but not here. while the server is getting ready to crash the chat is faster than ever. its posted as soon as i type it. they can see it in real time. and they can respond. even while we know its crashing.
i have no idea whats going on at this point
if you can help please do. im at my wits end. i clearly have the hardware for it and this behavior is not typical of a normal issue that ive been able to find.
i pasted both logs from the day of the crash, i put a separation line indicating where the file split is.
again. while all of that is going on, server resources are far from maxed and there is NOTHING else running on this thing.
CPU=0% utilization
RAM=3% utilization (7gb/128gb)
1.5gb world on triple SSD RAID0
official java server (personal use)
From everything i can find, this server is capable of running servers MUCH more complicated than what i have going on. i have other servers i have been running but they arent used enough to tell if they suffer from the same thing. during the logs and crash above though, this server was the only one running. i was specifically keeping everything else at bay just so i could test it and duplicate the problem with the main one and if i find the issue to be accross all servers i host, then ill deal with it on a per server basis. but im at the point where ive ruled out all hardware involved. but i still dont know what is causing it.
Your server is using 2 gigs at max, assigning more RAM won't help as more RAM does not mean more FPS, it's a myth. Does it happen on one world or are newly created worlds affected aswell?
That's the thing though. I assigned 20gb because it normally uses 16gb with players. I'll test a new world and get back with toy but that ram usage is much lower than normal.
ok. leg went away after spawning a new world. added datapacks and minor lag came back with only 1 player. i also found the debug start/stop command so i ran that while i observed messages saying things like 11000 + ms behind skipping 200+ ticks.
so im struggling justifying datapacks causing lag because the CPU is litterally sitting at IDLE. i have no idea how a datapack can create lag without touching the cpu. i would expect AT LEAST 1 core to be pinned at 100% before that happens. perhaps i just need to be educated.
ok. so i think part of the problem was the CPU wasnt being allowed to change power states. i changed that and the server started running fine. it took 9000 TNT to crash it at that point
heres the latest terminal output. looks like a lot of complaining about the uncraft recipies for dye...
ok. so it was that. the only way i can get the CPU to run at normal speeds is to open the intel extreme tweak utility. for some reason that must opened at least once. otherwise the CPU stays at 0.17ghz and cannot change states. Once that tool is opened however, the CPU begins performing normally even if the tool is then closed. if i restart the host however I must open it again to restore normal operation.
we did identify a couple data packs that were particularly heavy and are working on optimizing them now.
Hello All,
Im tired of trying to beat my brain with blocks trying to identify this myself. i cant find ANYTHING about this ANYWHERE. So here we go. please let there be someone out there capable of assisting me with this.
The Server:
CPU: dual socket xeon 6core hyperthreaded 3.7ghz (24 virtual cores total)
RAM: 128gb ddr3 1600mhz Quad Channel
Storage: 3x 240gb SSD raid 0 for OS and current files host (700gb volume)
Storage 2: 5x 250gb HDD raid 5 for backups and other files (1.5TB volume)
Network: all hardware from server to WAN is gigabit. observed speeds in speedtest.net are 1,050 ish Mbps up AND down. ping is between 3 and 20. never more than that.
server is hardwired into the network.
OS: windows server 2019
VM: windows server 2016
ipv6 disabled
firewalls have been verified and turned off for specific tests
port forwarding has been verified and completely opened up for specific tests.
The Minecraft server:
assigned 20 gb ram
stored on OS SSD raid drive on host
running with official java 64bit on host with -server flag (not in VM)
Observations of the problem:
while running i see a LOT of lag notifications on the console with and without datapacks enabled.
1player is smoothish
2players gets laggy
3 players gets unplayable
EVEN WITHOUT DATAPACKS ENABLED THIS DOES NOT CHANGE
while running ive observed a continuous 0% CPU usage on ALL CORES. NOTHING SPIKES EVER.
even when a player connects there appears to be NO load on any of the CPU cores
server only pulls 7gb RAM max now. it used to pull 15gb regularly with players. thats why i gave it 20gb all to itself.
due to CPU idle speeds it is underclocking itself to 0.17ghz and is not even coming close to boosting to normal speeds.
(ive observered the the same problem happening while the server was using more CPU speed and overclocking itself to max speeds. i dont think the speed stepping isnt the problem as it persists while boosting even for other reasons)
eventually the server crashes. to which i get a watchdog message meaning a single server tick took more than 60 seconds.
IDK whats going on. but here is what has become apparent:
with no datapacks running. the server crashes due to 3 players playing while not even touching the cpu max speed or any speed for that matter and without running more close to the max ram. and while stored on SSD raid. with no other services running on the server.
root cause, the server cant tick for some reason.
now this is weird, are you ready? while i can visibly see the server struggling with mobs not able to move more than just twitching in place instead of walking, the chat has 0 delay. normally chat suffers with the rest of the server. but not here. while the server is getting ready to crash the chat is faster than ever. its posted as soon as i type it. they can see it in real time. and they can respond. even while we know its crashing.
i have no idea whats going on at this point
if you can help please do. im at my wits end. i clearly have the hardware for it and this behavior is not typical of a normal issue that ive been able to find.
Post the entire server log using paste.ubuntu.com and link it here
so sorry for the delay in reply. work got crazy for a second lol. anyway.
here are the crash reports: https://paste.ubuntu.com/p/XfkkRtmDrS/
Here ate the server logs: https://paste.ubuntu.com/p/yV6bMT8DFS/
i pasted both logs from the day of the crash, i put a separation line indicating where the file split is.
again. while all of that is going on, server resources are far from maxed and there is NOTHING else running on this thing.
CPU=0% utilization
RAM=3% utilization (7gb/128gb)
1.5gb world on triple SSD RAID0
official java server (personal use)
From everything i can find, this server is capable of running servers MUCH more complicated than what i have going on. i have other servers i have been running but they arent used enough to tell if they suffer from the same thing. during the logs and crash above though, this server was the only one running. i was specifically keeping everything else at bay just so i could test it and duplicate the problem with the main one and if i find the issue to be accross all servers i host, then ill deal with it on a per server basis. but im at the point where ive ruled out all hardware involved. but i still dont know what is causing it.
Your server is using 2 gigs at max, assigning more RAM won't help as more RAM does not mean more FPS, it's a myth. Does it happen on one world or are newly created worlds affected aswell?
That's the thing though. I assigned 20gb because it normally uses 16gb with players. I'll test a new world and get back with toy but that ram usage is much lower than normal.
ok. leg went away after spawning a new world. added datapacks and minor lag came back with only 1 player. i also found the debug start/stop command so i ran that while i observed messages saying things like 11000 + ms behind skipping 200+ ticks.
here are the results of that debug file.
https://paste.ubuntu.com/p/qSNt5b8nXJ/
i plan on running another one while we have more players on once they are available
ok, with 4 players on, i finally got a debug run. here is the link.
https://paste.ubuntu.com/p/ztXDqNN9dF/
thoughts?
so im struggling justifying datapacks causing lag because the CPU is litterally sitting at IDLE. i have no idea how a datapack can create lag without touching the cpu. i would expect AT LEAST 1 core to be pinned at 100% before that happens. perhaps i just need to be educated.
I'm no expert, but if a datapack is causing issues you either remove it or report it to the datapack creator so he can fix it.
so most of them we wrote. but how could a server lag without something being run without excessive resources. being used?
ok. the following has been applied. apparrently my testing was flawed.
the following link is the result of:
-New World Folder
-New server.properties file
-running server by double clicking server.jar
https://paste.ubuntu.com/p/xM545PVxJ8/
so ive eliminated config file, launch options, and datapacks as reason for lag. any suggestions for what to try next?
ok. so i think part of the problem was the CPU wasnt being allowed to change power states. i changed that and the server started running fine. it took 9000 TNT to crash it at that point
heres the latest terminal output. looks like a lot of complaining about the uncraft recipies for dye...
https://paste.ubuntu.com/p/dKNQzj2XJf/
UPDATE:
ok. so it was that. the only way i can get the CPU to run at normal speeds is to open the intel extreme tweak utility. for some reason that must opened at least once. otherwise the CPU stays at 0.17ghz and cannot change states. Once that tool is opened however, the CPU begins performing normally even if the tool is then closed. if i restart the host however I must open it again to restore normal operation.
we did identify a couple data packs that were particularly heavy and are working on optimizing them now.