Server Help Very high CPU load on planet

When a player is on a planet, that thread is already at around 70% CPU (as Linux counts it, so 100% == one core). Start moving around -- even if chunks are already generated -- it tops out at 100% and there are ingame problems with enemies stuttering and chunks not loading. Gods help you if you're in a hoverbike... And this is a personal server, there's ony two of us and we're generally playing together on the same planet.

And this CPU is no slouch either. Phenom II X4 965, the server process has the entire processor at its disposal.

This is on Linux. Some kind of optimization issue?

 

Post a copy of your starbound.log.

 

The Phenom II X4 965 actually is a kind of slouch. It's not an Atom but its single thread performance is not too high.

For reference, my server runs a processor that has about double the per-thread performance of the X4 965, and worlds still max a core sometimes. Starbound's CPU requirement is very high.

 

Shut down the server, restarted it, explored two worlds with a friend; one was just a moon and seemed fine, the other we only got partway into and it lagged pretty badly. Nothing in the log looks useful; just threads loading and unloading.

Code:

Start logging at: 2015-12-20 22:57:31.293
[22:57:31.293] Info: Star::Root using bootstrap file '/home/roothorick/Steam/steamapps/common/Starbound/linux32/sbboot.config'
[22:57:31.293] Info: Star::Root using storage directory '/home/roothorick/Steam/steamapps/common/Starbound/giraffe_storage'
[22:57:31.293] Info: Preparing Star::Root...
[22:57:31.307] Info: Loading Configuration with config file: 'starbound.config'
[22:57:31.308] Info: Loading Star::Configuration from 'Just (./../giraffe_storage/starbound.config)'
[22:57:31.335] Info: Writing Star::Configuration to './../giraffe_storage/starbound.config'
[22:57:31.337] Info: Initializing Star::Root with 2 assets sources
[22:57:31.337] Info: Done preparing Star::Root.
[22:57:31.338] Info: Loading Assets
[22:57:31.338] Info: Loading assets from: '../assets/packed.pak'
[22:57:31.338] Info: Loading assets from: '../assets/user'
[22:57:31.826] Info: Done loading Assets in 0.486 seconds
[22:57:31.826] Info: Loading NameGenerator
[22:57:31.903] Info: Done loading NameGenerator in 0.0770001 seconds
[22:57:31.903] Info: Loading ObjectDatabase
[22:57:43.667] Info: Done loading ObjectDatabase in 11.764 seconds
[22:57:43.667] Info: Loading PlantDatabase
[22:57:44.938] Info: Done loading PlantDatabase in 1.271 seconds
[22:57:44.939] Info: Loading ProjectileDatabase
[22:57:47.891] Info: Done loading ProjectileDatabase in 2.952 seconds
[22:57:47.891] Info: Loading MonsterDatabase
[22:57:50.373] Info: Done loading MonsterDatabase in 2.482 seconds
[22:57:50.373] Info: Loading NpcDatabase
[22:57:50.773] Info: Done loading NpcDatabase in 0.4 seconds
[22:57:50.774] Info: Loading StagehandDatabase
[22:57:50.775] Info: Done loading StagehandDatabase in 1.00816e-07 seconds
[22:57:50.775] Info: Loading VehicleDatabase
[22:57:50.784] Info: Done loading VehicleDatabase in 0.00899993 seconds
[22:57:50.784] Info: Loading PlayerFactory
[22:57:50.795] Info: Done loading PlayerFactory in 0.0110001 seconds
[22:57:50.795] Info: Loading EntityFactory
[22:57:50.795] Info: Loading VersioningDatabase
[22:57:50.798] Info: Done loading VersioningDatabase in 0.00299995 seconds
[22:57:50.798] Info: Done loading EntityFactory in 0.00299995 seconds
[22:57:50.798] Info: Loading ItemDatabase
[22:57:54.761] Info: Loading ImageMetadataDatabase
[22:57:54.761] Info: Done loading ImageMetadataDatabase in 1.12224e-07 seconds
[22:58:03.912] Info: Loading FunctionDatabase
[22:58:03.936] Info: Done loading FunctionDatabase in 0.0240001 seconds
[22:58:03.938] Info: Loading ParticleDatabase
[22:58:04.129] Info: Done loading ParticleDatabase in 0.191 seconds
[22:58:07.183] Info: Done loading ItemDatabase in 16.385 seconds
[22:58:07.183] Info: Loading MaterialDatabase
[22:58:07.346] Info: Done loading MaterialDatabase in 0.163 seconds
[22:58:07.347] Info: Loading TerrainDatabase
[22:58:07.398] Info: Done loading TerrainDatabase in 0.0499999 seconds
[22:58:07.398] Info: Loading BiomeDatabase
[22:58:07.510] Info: Done loading BiomeDatabase in 0.112 seconds
[22:58:07.510] Info: Loading LiquidsDatabase
[22:58:07.511] Info: Done loading LiquidsDatabase in 0.00100004 seconds
[22:58:07.511] Info: Loading StatusEffectDatabase
[22:58:07.565] Info: Done loading StatusEffectDatabase in 0.0540001 seconds
[22:58:07.565] Info: Loading DamageDatabase
[22:58:07.756] Info: Done loading DamageDatabase in 0.191 seconds
[22:58:07.757] Info: Loading EffectSourceDatabase
[22:58:07.782] Info: Done loading EffectSourceDatabase in 0.0250001 seconds
[22:58:07.782] Info: Loading TreasureDatabase
[22:58:07.815] Info: Done loading TreasureDatabase in 0.033 seconds
[22:58:07.815] Info: Loading DungeonDefinitions
[22:58:08.228] Info: Done loading DungeonDefinitions in 0.413 seconds
[22:58:08.228] Info: Loading TilesetDatabase
[22:58:08.228] Info: Done loading TilesetDatabase in 1.04541e-07 seconds
[22:58:08.228] Info: Loading EmoteProcessor
[22:58:08.228] Info: Done loading EmoteProcessor in 1.04541e-07 seconds
[22:58:08.228] Info: Loading SpeciesDatabase
[22:58:08.241] Info: Done loading SpeciesDatabase in 0.0130001 seconds
[22:58:08.241] Info: Loading QuestTemplateDatabase
[22:58:08.268] Info: Done loading QuestTemplateDatabase in 0.0270001 seconds
[22:58:08.268] Info: Loading AiDatabase
[22:58:08.296] Info: Done loading AiDatabase in 0.0279999 seconds
[22:58:08.296] Info: Loading TechDatabase
[22:58:08.299] Info: Done loading TechDatabase in 0.00300003 seconds
[22:58:08.299] Info: Loading CodexDatabase
[22:58:08.474] Info: Done loading CodexDatabase in 0.175 seconds
[22:58:08.474] Info: Loading BehaviorDatabase
[22:58:08.584] Info: Done loading BehaviorDatabase in 0.11 seconds
[22:58:08.584] Info: Loading DanceDatabase
[22:58:08.588] Info: Done loading DanceDatabase in 0.00399992 seconds
[22:58:08.588] Info: Loading SpawnTypeDatabase
[22:58:08.594] Info: Done loading SpawnTypeDatabase in 0.00599997 seconds
[22:58:08.596] Info: Done fully loading Star::Root
[22:58:08.596] Info: Server Version 'Beta v. Glad Giraffe - Update 3' Revision: 30555b17a491c2a97ab0afbff3b848a1fd937dbd Protocol: 711
[22:58:08.597] Info: UniverseServer: Acquiring universe lock file
[22:58:08.603] Info: UniverseServer: Loading settings
[22:58:08.605] Info: UniverseServer: No default world set; using randomized starter worlds
[22:58:08.606] Info: UniverseServer: Starting UniverseServer with UUID: 750f26304053053fbba296c3e05c250b
[22:58:30.193] Info: UniverseServer: Connection received from: 0000:0000:0000:0000:0000:ffff:ac18:000a:57992
[22:58:30.210] Info: UniverseServer: Logged in account '<anonymous>' as player 'Bartholomew Chesnington' from address 0000:0000:0000:0000:0000:ffff:ac18:000a
[22:58:30.215] Info: UniverseServer: Reviving player at ClientShipWorld:b804513a49e92941a1c91a94def685c5
[22:58:30.215] Info: UniverseServer: Client 'Bartholomew Chesnington' <1> (0000:0000:0000:0000:0000:ffff:ac18:000a) connected
[22:58:30.223] Info: UniverseServer: Loading client ship world ClientShipWorld:b804513a49e92941a1c91a94def685c5
[22:58:31.629] Info: UniverseServer: Loading celestial world -934676590:-557317097:24871800:11
[22:58:36.299] Info: UniverseServer: Connection received from: 0000:0000:0000:0000:0000:ffff:44bf:aaea:64820
[22:58:36.510] Info: UniverseServer: Logged in account '<anonymous>' as player 'Punch Fish' from address 0000:0000:0000:0000:0000:ffff:44bf:aaea
[22:58:36.522] Info: UniverseServer: Reviving player at ClientShipWorld:334d6d6a0c0893fd26b1b04ec9f064fd
[22:58:36.541] Info: UniverseServer: Client 'Punch Fish' <2> (0000:0000:0000:0000:0000:ffff:44bf:aaea) connected
[22:58:36.580] Info: UniverseServer: Loading client ship world ClientShipWorld:334d6d6a0c0893fd26b1b04ec9f064fd
[22:58:38.635] Info: UniverseServer: Loading celestial world -860158376:-550112781:-271505822:7
[22:58:48.034] Info: UniverseServer: Stopping world ClientShipWorld:b804513a49e92941a1c91a94def685c5 due to inactivity
[22:58:48.135] Info: UniverseServer: World thread has stopped due to inactivity, removing world ClientShipWorld:b804513a49e92941a1c91a94def685c5
[22:58:49.442] Info: UniverseServer: Stopping world CelestialWorld:-860158376:-550112781:-271505822:7 due to inactivity
[22:58:49.455] Info: UniverseServer: World thread has stopped due to inactivity, removing world CelestialWorld:-860158376:-550112781:-271505822:7
[22:59:07.366] Info: UniverseServer: Stopping world ClientShipWorld:334d6d6a0c0893fd26b1b04ec9f064fd due to inactivity
[22:59:07.466] Info: UniverseServer: World thread has stopped due to inactivity, removing world ClientShipWorld:334d6d6a0c0893fd26b1b04ec9f064fd
[23:06:06.094] Info: UniverseServer: Loading client ship world ClientShipWorld:b804513a49e92941a1c91a94def685c5
[23:06:11.459] Info: UniverseServer: Loading client ship world ClientShipWorld:334d6d6a0c0893fd26b1b04ec9f064fd
[23:06:23.529] Info: UniverseServer: Stopping world CelestialWorld:-934676590:-557317097:24871800:11 due to inactivity
[23:06:23.639] Info: UniverseServer: World thread has stopped due to inactivity, removing world CelestialWorld:-934676590:-557317097:24871800:11
[23:06:26.147] Info: UniverseServer: Creating celestial world -934676590:-557317097:24871800:10:1
[23:06:26.846] Info: Placing dungeon aviantower
[23:06:26.893] Info: Generating dungeon aviantower (NOT FORCED)
[23:06:26.897] Info: Placing dungeon at (2389, 875)
[23:06:27.045] Warn: Failed to place dungeon object: birdjar1 direction: 0 position: (2474, 958)
[23:06:47.412] Info: UniverseServer: Stopping world ClientShipWorld:334d6d6a0c0893fd26b1b04ec9f064fd due to inactivity
[23:06:47.512] Info: UniverseServer: World thread has stopped due to inactivity, removing world ClientShipWorld:334d6d6a0c0893fd26b1b04ec9f064fd
[23:06:48.636] Info: UniverseServer: Stopping world CelestialWorld:-934676590:-557317097:24871800:10:1 due to inactivity
[23:06:48.738] Info: UniverseServer: World thread has stopped due to inactivity, removing world CelestialWorld:-934676590:-557317097:24871800:10:1
[23:24:07.992] Info: UniverseServer: Loading celestial world -934676590:-557317097:24871800:10:1
[23:24:19.550] Info: UniverseServer: Stopping world ClientShipWorld:b804513a49e92941a1c91a94def685c5 due to inactivity
[23:24:19.650] Info: UniverseServer: World thread has stopped due to inactivity, removing world ClientShipWorld:b804513a49e92941a1c91a94def685c5
[23:24:20.339] Warn: Failed to place dungeon object: oillantern1 direction: 1 position: (1476, 892)
[23:24:46.161] Warn: Failed to place dungeon object: stonefurnace direction: 1 position: (1210, 869)
[23:27:26.212] Warn: Failed to place dungeon object: stonefurnace direction: 1 position: (793, 877)
[23:28:21.400] Info: UniverseServer: Client 'Bartholomew Chesnington' <1> (0000:0000:0000:0000:0000:ffff:ac18:000a) disconnected
[23:28:25.859] Info: UniverseServer: Client 'Punch Fish' <2> (0000:0000:0000:0000:0000:ffff:44bf:aaea) disconnected
[23:28:34.971] Info: Interrupt caught!
[23:28:35.036] Info: UniverseServer: Stopping UniverseServer
[23:28:35.839] Info: Server shutdown gracefully
[23:28:35.840] Info: Shutting down Star::Root

This is a small private server for friends, usually only two players, almost invariably in the same world -- I can't imagine this being significantly more stressful than simple singleplayer. It's no 6700k, but you're implying a 965 can't competently handle even singleplayer, and THAT seems like a stretch.

 

It depends on a variety of factors but it's not a stretch to say a single world can max a core. That said, there are a number of other things it could be--I've seen my players suggest a number of optimization WRT vsync, shaders, etc. But if you've observed the high CPU usage on the server box yourself, well...

anyways, does it happen on all worlds, all the time? Or is it just one world? etc.

It would be nice if starbound didn't need so much CPU... the box I run starbound on could easily support hundreds of players in, say, an FPS deathmatch server collection, but only a handful in this game. It's what we've got, however.

To illustrate:


The top server instance only had 3 players. 2 worlds still wanted a whole core to themselves, on a much more powerful processor.

Even the bottom server instance only had 12 players.

I'm hoping the 'polish patch' the front page posts talk about will help with this. Or if there is some secret tweak that drastically cuts the CPU requirements... please tell me.

 

It's most worlds, but not all. Atmosphereless worlds are fine, outpost can be weird when loading but is okay. Player ships are fine, surprisingly light even. This seems to line up... Most planets have quite a lot more going on; monster/NPC AI, insects and small animals wandering about, weather and its effects...

What proc are you running specifically? I run AMD in servers for free ECC support and I'm waiting for Zen to upgrade.

 

For starbound I'm running an E3-1270v3 bare metal. I've had some others call the E3 "dinky" but really for gaming it's the way to go. Other processor lines, be they Xeons or Opterons or whatever else--you can get a lot of cores but none that I've seen match the E3-1270v3 (or the 1271) for performance on single threads.

Even with all those things you say, I'd really like to know exactly what makes the requirement so high. For world generation I can understand... but well, Terraria (as the most comparable example) runs a whole world many times as large as any Starbound planet on minuscule CPU and not nearly so much memory. I don't know any game that comes anywhere close to starbound, even if they feature a lot of little details in their world.

Anyways, long story short... I'm afraid what you describe sounds normal. I've even had some residents of my server tell me they've tried to host from their PC and it's still laggy depending on where they are. I really hope things get better around 1.0 because starbound is just really hard to host as is.

 

You haven't heard of Space Engineers... an SE public server NOT being in a perpetual state of low simspeed (meaning performance-induced slow motion) is rather rare... but I digress.

In any case, it's a given that Starbound will be a fair bit heavier than Terraria: Monster and NPC AI is a great deal more complex, there's a good two or three times as many active entities in any given area, and there are entire active systems that Terraria doesn't even conceptually have, like weather that inflicts statuses (necessitating "shelter" detection) and modifies blocks (e.g. pools of water forming in the rain). On top of that, Starbound opts for Minecraft-style on-the-fly chunk generation whereas Terraria worlds must be pre-generated in their entirety before they can be loaded. But you're right, the difference shouldn't be THIS extreme.

 

My cheaper 4690k says otherwise. Just saying.

https://www.cpubenchmark.net/singleThread.html
http://cpuboss.com/cpus/Intel-Xeon-E3-1270-v3-vs-Intel-Core-i5-4690K



But anyhoo, to the thread. Please ingest your grains of salt now.

I have a running theory that Starbound does not perform well on AMD CPUs most of the time. Most of the people I've seen complain about performance have had AMD based systems. I don't have much technical knowledge to back this, only anecdotal evidence. From what I gather, it seems to come down to a fundimental design difference in the CPU itself vs Intel, meaning there is little Chucklefish can do about it.

 

Well, like I said, world generation I understand. It's post world generation that seems odd. But once it's done it's just a file usually 1 or 2mb in size. Interesting thought on shelter detection though, I can kinda see liquids too.

Not sure I agree with the AI. At least for monsters, Terraria AI feels more varied and complex (vs Starbound's 'charge at you and maybe use 1 special' AI). I suppose colony AI is a bit better, where applicable.

Hah, good point. I usually just look at other server procs, which often favor having a ton of cores (and/or features like ECC) at the expense of performance at the individual level. But none of us really know what starbound is doing behind the scenes... all we can do is at its utilization.

 

They're not using ICC are they? That's a known problem with Intel's compiler and they've actually gotten sued over it... and never did actually fix it.

With GCC there's definitely things you can do about it. -mtune=generic is the default, which tries to strike a compromise between the current most popular processor architectures, which currently should be Ivy Bridge, Haswell, Jaguar, and Puma, the latter two of which are AMD (their APUs are dominating the low end). However, one, -march changes -mtune at the same time generally necessitating specifying -mtune separately (which is why it's common to do things like -march=pentium4 -mtune=generic) which Chucklefish may not have anticipated, and two, 10h is a far cry from Puma, so modern GCC is likely to throw PhIIs under the bus in favor of newer hardware on both sides of the aisle.

Terraria only deals with movement as a function of player position (or even simple constants) and a handful of events (jumping, ranged attacks, idle wandering, etc). There's a wider variety of enemy "AI", but each is far simpler than any given Starbound opponent. And with wall climbers and more variations of flyers, that gap is closing.

Starbound's monsters in themselves are slightly more complicated, as there is such a thing as melee attacks as opposed to simply trying to make contact with players, and they make rudimentary attempts to set up their attacks instead of firing based on simple proximity (most noticeable with Ignomes, which prioritize getting vertically level with the player as opposed to simply approaching directly). Hostile NPCs have another layer of complexity, making decisions on when to use their shield, melee weapon, or gun based on the player's actions.

Well, desktop SKUs are favored for game servers for a reason data corruption from failing memory isn't exactly a game-ender and that extra one-thread performance is worth it.

 

You seem to know more than I on this matter, so I defer to you.

 

It's a fair enough point, but I have no idea where I would even start getting an i7 server. I mean I know I could probably google it but I don't actually know any hosts that run i7s

 

Well, you could always colo...

 

Nah I'm just going to stick with what I've got. Starbound's not worth that much and I don't have *that* much to burn

 

I believe it was said by a dev some time ago (sorry, no source at hand) that Starbound uses one base thread plus one for each open world. So yeah, single thread performance is a factor when you don't have many people online, but once you get more than a handful it's a moot point.

From what I've seen the devs seem to be pretty on-the-ball when it comes to details like this, but you could always mention it to them as an FYI just in case.

 

I can confirm each world has its own thread simply from watching how the process spawns and runs them (as in the tree-view screenshot in posts above).

But I have to disagree. Multithread performance is definitely a factor in supporting many people/many worlds but if a processor's single thread performance sucks, individual worlds can still lag. Just that one world's lag will not impact other worlds and cause other clients lag.

In short--Single thread performance: How well the server program can run each individual world
Multithread performance/number of cores: How many worlds you can run simultaneously without impacting other worlds.

Edit: In my own experience, watching server utilization, the best thing that helps with higher player counts--they tend to team up. When you've got 2, 3, or even 4 players working together--they're on the same world, instead of running 2/3/4 separate worlds, which really helps limit CPU utilization.