Server Help Server high cpu usage not fixed? (Enraged Koala hotfix 5)

The server still shows very odd behavior when looking at the CPU graph.

From time to time, one core overloads entirely for 5 seconds, then loses its load instantly, then another core overloads in the same way. I can confirm that it is starbound, because one of its processes will spike to 100% then (overloaded planet?).

This issue can happen for like 5 minutes and then suddenly go away, which tells me it is player/planet related somehow.

I can't find anything in the logs, besides the usual mod errors of people with additional assets ("cheat matter" seems awfully common, which mod is that?)..

Code:

Error: Could not instantiate item '[cheatmatter, 1, {}]'. ItemException: No such item 'cheatmatter'
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN4Star13StarExceptionC2ERKSs+0xfe) [0xdf99ce]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNK4Star12ItemDatabase8itemDataERKNS_6StringE+0x59) [0xa4d399]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNK4Star12ItemDatabase10createItemERKNS_14ItemDescriptorE+0x30) [0xa4d8e0]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNK4Star12ItemDatabase4itemERKNS_14ItemDescriptorE+0x92) [0xa4e7c2]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNK4Star12ItemDatabase8readItemINS_4ItemEEEbRKNS_9ByteArrayERSt10shared_ptrIT_E+0xac) [0xa5ee5c]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server() [0xb962ac]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN4Star15PlayerInventory12getNetStatesEv+0x42) [0xb96392]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN4Star6Player12getNetStatesEv+0x1a1) [0xb7c111]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN4Star6PlayerC1ESt10shared_ptrINS_12PlayerConfigEERKNS_9ByteArrayE+0x234) [0xb7c6f4]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN9__gnu_cxx13new_allocatorIN4Star6PlayerEE9constructIS2_IRKSt10shared_ptrINS1_12PlayerConfigEERKNS1_9ByteArrayEEEEvPT_DpOT0_+0x43) [0xb8dc23]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNSt23_Sp_counted_ptr_inplaceIN4Star6PlayerESaIS1_ELN9__gnu_cxx12_Lock_policyE2EEC1IJRKSt10shared_ptrINS0_12PlayerConfigEERKNS0_9ByteArrayEEEES2_DpOT_+0x4b) [0xb8dcab]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNSt14__shared_countILN9__gnu_cxx12_Lock_policyE2EEC2IN4Star6PlayerESaIS5_EJRKSt10shared_ptrINS4_12PlayerConfigEERKNS4_9ByteArrayEEEESt19_Sp_make_shared_tagPT_RKT0_DpOT1_+0x4c) [0xb8dd1c]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNSt12__shared_ptrIN4Star6PlayerELN9__gnu_cxx12_Lock_policyE2EEC1ISaIS1_EJRKSt10shared_ptrINS0_12PlayerConfigEERKNS0_9ByteArrayEEEESt19_Sp_make_shared_tagRKT_DpOT0_+0x2b) [0xb8dd8b]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNK4Star13PlayerFactory13netLoadPlayerERKNS_9ByteArrayE+0x1f) [0xb8adcf]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZNK4Star13EntityFactory13netLoadEntityENS_10EntityTypeERKNS_9ByteArrayE+0xeb) [0xa1618b]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN4Star11WorldServer20handleIncomingPacketEjRKSt10shared_ptrINS_6PacketEE+0xf27) [0x868a07]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN4Star17WorldServerThread6updateEv+0x173) [0x899493]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN4Star17WorldServerThread3runEv+0x208) [0x899d38]
/home/starbound/steam/SteamApps/common/Starbound/linux64/starbound_server(_ZN4Star10ThreadImpl9runThreadEPv+0x21) [0xe00131]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f6e) [0x7f3a6aa63f6e]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f3a69d6c9cd]

 

Do you use any mods, mate?

 

Absolutely none right now.

The mod error above is from people trying to use their mods on the server.

 

I believe cheat matter is a super FTL fuel that is part of the creative mod. Beyond file access caused by the log spam, I don't think the modded items like that cause any trouble. They get turned into perfectly generic items.

The modded mining tools (like the super 5x5 extra-fast matter manipulator in creative mod) will kill your server with pure block destruction lag. As those modded tools are client authoritative, there is not much you can do to stop them. Perhaps one of the wrappers has cracked some of the packet codes and can throttle tool use, but I doubt it. The server software is very inefficient when it comes to rapid tool use. From what I can tell, every time the tool fires, the client generates some sort of block damage packet for each block being hit. Tools generally have rapid speeds (I believe the creative one is 0.05...so 20 per second). As you might imagine, a client spamming you with 20x5x5 (500) black damage packets per second is a BAD THING. Imagine what happens when a client decides to using a mining laser to clear half a screen of mountain away? Yeah... you just got hit with a small denial of service attack.

A reasonable dev would have narrowed that down to just a few packets. One packet to start tool use at location X,Y. One packet each time the aim of the tool is shifted to a new X,Y. One packet to stop tool use. Let the server handle the block damage iternally without having to handle a boatload of network traffic. And if the server doesn't have the data for you modded mining tool? Too bad buddy! You don't get to use your cheat item. Go find yourself a diamond drill.

I have no clue where all the CPU use comes from. I expect the starbound devs themselves don't know... I don't see any evidence of profiling tools built in at least. The basic sort of physics we have in starbound is easy to code. You can run tens of thousands of vector math operations per game tick without dragging a game engine down and honestly, a server simulation doesn't need to run at 60 FPS. You just need to crank through a few game frames per second and let the clients smooth out the data points. The sever labor is also very divisible. As worlds do not interact, you can spin off world simulations between game worlds. It should even be trivial to allow groups of physical machines to work together. One server acts as master, controlling auth, chat, transfers, and as many worlds as it can handle. Each slave takes excess worlds as they are loaded to keep the master server from being bogged down.... This was done with the fan-made Ragnarok Online server (before official Aegis server code leaked) and it worked wonderfully. I got to peek that source code and learn a lot. Its standard practice for MMOs now and starbound would lend itself well to the model.

 

Very informative post, for others (devs and server owners) as well.

I was aware of the super-tools and client mods already, the main reason why we switched to passwording the server was to prevent "server-list griefers" from picking ours and let them merrily destroy everything with for example those super-drills (or any other people that have no sense of "hey others play here too, perhaps I should stop using my modded super-tools").

Since clients can so elaborately edit their files and have it appear on the server, I feel this has a ton of potential for people that intend to harm servers, and I think that potential had been used in the early days already to cause random server instability.

The only reason people can do it, is because they can change their stock assets and the server won't lift a finger. My suggestion here revolves all around that: http://community.playstarbound.com/...trieve-all-config-data-from-the-server.72608/

 

It should go without saying that the client should not be able to use mods that affect gameplay elements without the server's consent. Thats been standard practice in games forever...and for good reason. I'm sure the devs implemented it the way the did because it was the quickest and easiest way to get a server working at all. We're still in alpha though. Maybe when beta is released we'll get some real server support.

Pulling down configs for the client as needed should be workable (as you describe) but it needs some additional logic to. One thing to note is that equipment, in particular, only spawns based on configs. It doesn't adhere to them after the fact. As an example, you can modify the config for a piece of armor, spawn the armor, and then revert the config file. The armor keeps the stats you spawned it with. Someone could still join the server with the modded gear even if config files were sent to the client. The server will have to check to make sure gear fits within constraints. For non-randomly generated stuff like tools and armor, they should be matched to configs as soon as loaded. I have no clue why they aren't now. Randomly generated weapons should be spawned based on rules and checked against those rules when loaded to make sure the item is possible at least. This wouldn't stop a player from forging a perfectly generated random weapon, but it would limit the power of the weapon at least.

Of course, we also need the option to have servers save characters and ships server-side... a feature that is coming eventually, the devs claim. If/when it does, be sure to add dye bucket mod to your server or fashion will suffer considerably!

 

Yes I noticed this funny behavior as well. Objects tend to do the same thing, unless they get broken and picked up again.. it's really odd how servers work atm, one half of the files comes from the server and the other half comes from the client. Lua scripts belonging to objects for example will be (thank god) retrieved from the server, because the object is placed on the server and asking for the script there.

So yeah, it purely depends on the context of the "asset" where the file will be pulled from.

It even goes on within the configs themselves... For instance, if you have an NPC with custom equipment and dialog, and spawn it on the server, it will have the custom equipment, but it will lack the dialog. I tried making some sort of welcoming bot that I could spawn without modding the server but it failed because of that, even though it had the correct equipment.

 

Go ahead and mod the server for your welcoming bot then. The server can do some things with mods even when the client doesn't have the mod... the dialog is probably one of those things. I modded out meteor and explosion damage. After having a few instances of meteor gun and thrown bombs grief, I nixed that. The guns still spawn meteors, but the meteors then use the server's configuration, which dictates that they don't do block damage now. This also solves the general annoyance of players spending hours building... only to suddenly get hit with meteor weather (large or small). Sometimes you can tell by craters in the background graphic, but not always. In any case, it has cut down grief significantly, including unintentional grief from thrown bombs that people use before they know what they do.

If only I had a way to defeat the modded mining tools...or even detect the behavior without being there to witness it. Oh yeah, and I'd like for people NOT to be invisible when wearing modded armor with modded assets. Make them look like hideous trolls or something!

 

Haha yeah, we did that server-side, but we had to replace an existing NPC for it to work, or it'd crash players : P

You can also mod quests server-side btw, which is pretty epic.

I'll look into modding out explosion damage, but so far that has been the least pressing issue.

That last thing you mentioned is also very beneficial for griefers.... It's as if the devs encourage griefing.

 

yep again an other poor linux server trying to get by but oh no look at that cpu usage

 

I also reported this months ago: http://community.playstarbound.com/...server-memory-leak-why-such-linux-hate.74879/

One of the several problems of the Linux Server Software, even windows is unstable, but windows got fixes, linux version is yet to have fixes.

 

maybe if we bitch more we get some updates xD?

 

Well Omni replied about such
http://community.playstarbound.com/...why-such-linux-hate.74879/page-2#post-1983519