Server Help Developed Worlds Randomly Wiping and (Sometimes) Returning

Wait... could a backup script reading a planet file and adding it to an archive while the planet is active affect it in any way?
My backup script begins to read and copy the universe folder every automated reset, and the backup process can continue well into the server starting up.
It's usually after restarts that the corruption happens.

 

Given that it is happening to folks who I doubt are running the same/similar script, that's not likely. Nothing's not worth investigating, though, given that we're still grasping at straws, so at least it is another road to try travel down.

 

Well, you should not be reading from any .world file while Starbound is running. The moment Starbound opens the file, it may change the contents a lot. I wrote a tool that renders a world and the .world file changes so much that it fails to load any content at all the moment Starbound opens it (even if I stand completely still and do nothing in-game).

Maybe you could kindly ask the developers to enable a "dump" command which will allow the server to output a backup file that isn't corrupted.

 

Interesting, I didn't know that. Just to play it safe, I'm going to change how I do backups so that they're done when the server isn't running. I'll report back if the corruption continues or not.

 

Yes *absolutely* that absolutely *will* corrupt the world files that are going into the backup. We need to get admin commands enabled in the public build and have an admin command for copying a file to a backup (Like Blixt said), or a command to prevent starbound from making live writes to databases while backups are happening. In the meantime, do a scheduled shutdown of servers and do a backup during the shutdown.

Edit: Haha, you basically said the same thing further down, sorry

 

In the latest unstable push there is a new feature that adds a lock file for the universe directory to prevent multiple instances of starbound running and writing to the same directory. It is cooperative, so it doesn't prevent in any way other scripts or programs from reading from the universe folder at the wrong time or writing to it and messing everything up, it only prevents another copy of starbound or the starbound server from doing it.

 

We had one world corruption happen last night after updating. It was on our most populated hub world. I had turned off auto-restoring the world on restart to test if such corruption would still happen. We had a chain crash, of the sort where the server dies when 40 people or so attempt to reconnect at once. The corruption played out much different than the world resets of old and I have nothing to compare it to. When in orbit around the planet, there was only a solid black background (no stars/planets) and attempting to teleport down was having no response. I restored the world and it loaded fine after that. As I haven't seen anything like it before, it could be an isolated incident.
It also seems like that particular world is not having any progess saved to it (with auto-restore off). When are modifications saved to a world file? I had thought it was incrementaly saved due to the small rollbacks we've seen in the past after server crashes. Now it seems like maybe no progress is saved until a world unloads? This world is loaded practically from the moment the server boots until the server crashes, so if we're waiting for an unload that never happens, that would explain things.

In any case, I haven't seen resets on other worlds yet. So far, it seems like the issue has been at least reduced in frequency. I'll keep watching things and reporting what happens of course.

 

That with the black background on ships happens for me when the server lags

 

Just so you know the universe locking thing is *only* in unstable, you should see a universe.lock file when starbound is running if it's working.

 

Had a fresh world just get wiped about an hour ago. Fresh in the sense that I deleted the planet file so the players could start anew. The planet at these coords has experienced failures (wipes) in the past.
First error was the usual leaf block signature, and reportedly crashed the server or booted everyone. Did not confirm.
Logs show no sign of a server failure.
Later, the world was regenerated.
Even later, this error was given for the same world.

Code:

Error: UniverseServer: error during worker pooled job: VersionedVariantException: VersionedVariant identifier '' did not match expected 'WorldMetadata'

EDIT: Kyren, I started a new conversation with you linking you to a pastebin of the server log during this period.

 

I've been watching things for the few days since the patch. We're still seeing world resets as well, but they seem to be significantly less frequent now, excepting for our most populated planet that still gets hit hard. I think the lower frequency of resets is probably due to this latest version being a bit more stable. When things get very crowded, our prime time crashes are more like 1-2 hours apart, rather than 30 minutes or so.

 

Encountering a crash that seems to have resulted in a corrupt world.

Code:

Error: Segfault Encountered!
Error: Stack Trace...
./starbound_server(_ZN4Star15innerPrintStackEv+0x2c) [0xe0b3bc]
./starbound_server(_ZN4Star10fatalErrorEPKcb+0x1d) [0xe0a63d]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f863f8224a0]
./starbound_server(_ZNK4Star10ServerTile5writeERNS_10DataStreamE+0xd) [0x8f611d]
./starbound_server(_ZN4Star12WorldStorage15writeTileSectorERKNS0_15TileSectorStoreE+0x70) [0x8ad150]
./starbound_server(_ZN4Star12WorldStorage12unloadSectorERKNS_6VectorImLm2EEEb+0x493) [0x8ae8d3]
./starbound_server(_ZN4Star12WorldStorage9unloadAllEv+0x110) [0x8aefa0]
./starbound_server(_ZN4Star12WorldStorageD1Ev+0xf) [0x8aefff]
./starbound_server(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x49) [0x7ebd69]
./starbound_server(_ZN4Star11WorldServerD1Ev+0x29a) [0x87dcaa]
./starbound_server(_ZN4Star17WorldServerThreadD1Ev+0xa2) [0x8ab082]
./starbound_server(_ZNSt8_Rb_treeIN4Star19CelestialCoordinateESt4pairIKS1_St10shared_ptrINS0_17WorldServerThreadEEESt10_Select1stIS7_ESt4lessIS1_ESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x90) [0x853ba0]
./starbound_server(_ZNSt8_Rb_treeIN4Star19CelestialCoordinateESt4pairIKS1_St10shared_ptrINS0_17WorldServerThreadEEESt10_Select1stIS7_ESt4lessIS1_ESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x59) [0x853b69]
./starbound_server(_ZN4Star14UniverseServer3runEv+0x4c1) [0x84dcb1]
./starbound_server(main+0x582) [0x7e4dd2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f863f80d76d]
./starbound_server() [0x7eab1d]

 

Yeah you dev's made the game more stable, thanks! still hope we can get this wiping issue off our hands thou.. damn.. mysterious.... .wiping... damn Eco rangers (reference to http://community.playstarboundrp.com/threads/the-cause-of-city-wipes.1919/#post-22229 )

 

@kyren is linux server software going to get the Windows fixes? Or the Memory leak/crashes fixes are windows-only? At this moment it's a big issue with us.

It's frustrating for Linux Server Owners to read patch notes with the good news "Hey guys it's fixed" and them play the game and the issues are still there.

 

Any news on this?

 

No news yet... worlds are still being corrupted/reset. I still haven't found any good workaround. Mainly we're just keeping backups and restoring as needed. Popular worlds are restored automatically on each restart (which prevents new construction). We let people send in their worlds from singleplayer to be placed on the server as well which allows players to escape lag/crashes/reset issues while working on major projects.

Hopefully the issue will fade as the server software sees some stability improvements.

 

So... the issue still exists? Or did a recent unstable version fix it? TBH, thats the only usefull topic i found about that issue so far (still happens to me with stable build)

 

I haven't heard any indication that this is fixed yet in the unstable versions. Any stability improvements they made would help though as this seems related to crashing. I'm aware that unstable has fixed a few crash issues... assuming they didn't introduce more than they fixed, unstable should help.

 

I notice a trend that it seems to happen on planets with high traffic, or with a lot of content. Perhaps there is some sort of unintended capacity limit?

 

Dunno, i played with a friend privat multiplayer (2 players, always Stable version) it just happend in regular intervals like an hour, max. 2 hours of playing, crash. It didnt matter if we where together or on different planets. Out of 4 crashes their was 2 times the described corruption, so i assume it doesnt matter how mutch ppl, only more then 1 counts. Singleplayer still runs top, for endless hours no corruption or crash.