1. When making a thread, please tag your thread accordingly using the menu to the left of the textfield where you name your thread where applicable. Server Advertisements and Mod Releases should be contained to their respective subforums.

Server Help Developed Worlds Randomly Wiping and (Sometimes) Returning

Discussion in 'Multiplayer' started by Eberict, Feb 21, 2014.

  1. Eberict

    Eberict Void-Bound Voyager

    Would like to check if any one else is experiencing a weird bug in Enraged Koala in which a built-upon "settlement" world flips between its proper developed state (player structures intact) and a wild unexplored state (clean slate, no player structures). Furthermore, even when the settlements return either through rollbacks or inexplicably on their own, the wildlife changes--creatures which used to be underground are now overground, flyers which used to be in orbit are now on ground level, nocturnal and diurnal cycles switch, etc. Even after rollbacks and tests with different back-ups, the problem persists and strikes seemingly at random.

    Any insight on this issue would be greatly appreciated. Thank you.
     
    White935 likes this.
  2. ArchGaden

    ArchGaden Big Damn Hero

    I'm seeing this problem also. When the world is reverted to natural state, the old world is saved as .fail file, which is seemingly corrupted. I'm guessing some object on the world has bad data (a container maybe?) that causes the world to save improperly. As these highly developed worlds have tons of containers, this is a troublesome issue to attempt to fix. I'm waiting for the dust to settle on this patch before I attempt anything drastic. For now, I wrote an auto-restart script such that it auto-restores the affected worlds each time the server crashes. So this is what I do as a temporary measure. It beats have the worlds destroyed, but players still can't build on them.

    If we're very lucky, the devs will fix the patch bugs, but I don't have much hope of that. They only seem to make things worse. I'll be digging into this issue next week and if I find an answer, I'll post it.
     
    White935 and Eberict like this.
  3. Eberict

    Eberict Void-Bound Voyager

    Does it only happen to highly developed worlds on your server, or also to under-develop, small-time constructions? Did these planets feature custom assets like the Creative Mode spawners? Were they built through the use of mods on server, or purely through vanilla means (including importing blocks from SP, which is off-server use of creative mode)?

    On our end, it only seems to be the highly developed worlds that are being hit. We have a few test subjects for underdeveloped worlds (pre and post Enraged) which seem utterly unaffected.
     
  4. Symphony

    Symphony Giant Laser Beams

    Yes ^ This must be solved. Because Elberict is probably on the same server as myself. ... Bump for the sake of fixin'!
     
    White935 likes this.
  5. ArchGaden

    ArchGaden Big Damn Hero

    No fix yet... it only happens on worlds that are loaded at the time of the crash. More developed worlds tend to have more player traffic, so they are more likely to be loaded at the time of a crash, and thus more likely to be hit. Generally you only notice it on a developed world, as likely nobody would complain about a crash resetting NPC structures to their pristine conditions. There are no modded assets in use, so this seems to be purely a starbound server bug.

    My going theory is that the server is attempting a final world save on crash, but writing out corrupted data. Prior to enraged patch, no such final save was attempted, so you would see a few mintues of rollback after a crash... no big deal. Now we're dealing with an entirely corrupted world file. When the server attempts to load the corrupted file, it fails and renames it as a .fail file. To get around this for now, I have a script that restores the most visited world any time the server restarts. This prevents new work of course, which is a bad temporary measure, but it beats having the world destroyed. Other worlds that get corrupted are restored on demand from backups made twice daily or so.

    I'm hoping the devs will fix this issue, but if we dont' get a fix soon, I think the best workaround will be a script that backs up specific world files at intervals. It should work if I combine that with a script that detects the creation of a .fail file after the server crashes, which can then restore the world and delete all the .fail files before starting the server again. I'm new to linux so I don't know how to do this exactly, but I suspect it can be done with scripts. If not, I'll have to actually write a program to do it... *sigh* so much work fighting starbound to get multiplayer workable.
     
  6. White935

    White935 Big Damn Hero

    Server I'm on suffering badly from this too, really hope chucklefish fixes this soon, as its tearing our community apart, and folks getting egdy. not to mention the poor admin who has to constantly repair the broken worlds with backups.
     
  7. Dreganius

    Dreganius Void-Bound Voyager

    Couldn't have worded it better myself, White. Things have been really dicey, and this is basically making me lose interest in Starbound... That said it's going into Risk of Rain instead, but still, what's the point in building a nice base, or town, or even a city, when things just go haywire?
     
  8. ArchGaden

    ArchGaden Big Damn Hero

    I'm refusing to touch Risk of Rain or other chucklefish products until they get their starbound server issues sorted out. Starbound has/had such amazing potential, but they're really smashing it into the ground with this fiasco. I keep seeing dev boasts about future plans for new content, but we're barely getting any feedback on the catastrophic server issues. You can add all the paint you want to a sinking ship, but you still need to fix it before its going anywhere but down.

    Beyond these world file corruption issues, we're also having Universe.chunks and player *.db corruption during crashes. Fortunately, those are less frequent. Does anyone even know what kind of data goes into Universe.chunks? Deleting it seems to have no effect, by woe to you if it is corrupted... it will chain crash your server. If it happens again, I'm tempted to just add a delete for that to the restart script. We had the chunks file up into the hundreds of megabytes...of apparently useless data. I suspect this is somehow related to the starmap, but that is procedurally generated anyway... so a cache of some sort?

    Since we're doing player files and ship files client-side, why not make the player database files client-side to? That way, if it gets corrupted, it doesn't bring the server down. Beyond terrible execution, the server software seems like it was just writen on the fly with no design or cohesive approach to client vs server authority. Why are projectiles client authoritative? Why does the client get to dictate stats on non-random equipment like armor? Why should the client get to be invisible due to a modded armor or tech? ...I can understand the lack of features. Thats what beta is about, fleshing out the features beyond the core game. The server is nothing more than a shoddy prototype... more of a pre-planning proof of concept that isn't even in an alpha state. Ah well... the war to keep starbound multiplayer going wages on.
     
    zugz and White935 like this.
  9. White935

    White935 Big Damn Hero

    Well, i certainly believe that Starbound is the flagship of chucklefish, and they certainly will have to succede here if not to take a hit, and multiplayer is one pillar stones of it, as singleplayer groups, but the multiplayer groups are -huge- too, and i would guess atleast half of us play it multiplayer, so the netcoding being stable, and being able to handle a large number of people, is vital, people are comparing it to minecraft atm, and knowing it struggled so their giving this a try but i would love to hear the devs mention their trying to solve these mp problems we have, for they are as numerous as they are painful.
     
  10. ArchGaden

    ArchGaden Big Damn Hero

    Minecraft has some distinct advantages. You can play with whatever version you like. Server operators were able to stick with stable revisions until a new revision could be thorougjly tested. When a dud version popped out, it was ignored until it was fixed. Beyond that, minecraft gave a ton of information to modders and server operators, including packet structures and hints on how things operated internally... some of which was unintentional. For instance, you can decompile minecraft, because java is easy like that. When I ran a minecraft server that used a non-bukkit modset, I had to decompile and edit minecraft code to modify fire behavior. Similarly it let me track down a crash issue related to one of the mods I ran on the server. I'm not a huge Notch fan, but he at least gave enough slack to the community to be able to form workable multiplayer communities. Those communities made minecraft the massive hit that it is today. Players don't stick around long for single player. The fans that will stick around and spend money are almost all playing multiplayer.

    Starbound is stuck on steam's leash. Any update they push will be forced on players, and thus server operators....regardless of how terrible the patch is. Turning off auto-update isn't a viable solution, as unless you disconnect your network entirely, steam will refuse to let you play until you update. As they are opting for steam (no doubt for quicker access to a bigger customer base) the chucklfish devs have a responsabilitiy to test their work before they push it on us. They are failing miserably at that. They need to host a pre-patch-release official test server. It can be done by releasing a free-to-play test version of the game on steam. They can offer the keys to existing paid players. If they actually did their own server tests with many real players, they would quickly realize how terrible the server software is...I imagine the server would be targetted by the worst griefers imaginable....exactly the perfect people to test server stability with!
     
    The Grand Mugwump and White935 like this.
  11. Tyrakasaurus

    Tyrakasaurus Aquatic Astronaut

    Been seeing this issue, myself, on our own home server. After me and some friends did some private testing, we seem to have concluded the following...
    • It is PROBABLY NOT affected or caused by server/player mods, even broken ones. (Surprisingly. But it's happened to worlds that were mod-free.)

    • It is NOT affected by the number of people on a world, if any.

    • It is NOT fixed by any of the recent hotfixes.

    • It does NOT happen to every world on every server reset, but MIGHT happen with some, on some.

    • It CAN HAPPEN to worlds that have already been wiped once by this bug, AND newly generated worlds. This confirms it is NOT an issue with old world files conflicting with new ones, though universe files may be causing an issue. (We are testing whether new universe files are susceptible to this right now.)

    • It is CONFIRMED to happen on Linux servers, and MIGHT BE restricted to them. (The latter has not been concluded decisively, but I've not heard of it happening to any other servers. It would explain why it's not been reported as much.)

    • It is DEFINITELY caused by a server failing to load the world file, saving it as a .fail file, and then generating a new file in its place.

    • We do NOT know why it's doing this.
    I really, really hope this gets fixed soon. If I find out any useful information, I'll let you folks know.
     
  12. White935

    White935 Big Damn Hero

    Well, i would think their working on a fix atm Tyra, well i hope they are, for its a pretty serious issue <,< not to mention our community servers losing starbound players, and they thus drop out of starbound.. .we really need to keep this community breathing.. alas we need thus a working mp <.<
     
  13. Eberict

    Eberict Void-Bound Voyager

    Just going to point out that, while Risk of Rain is published by Chucklefish, it is developed by an entirely different studio. It's also a fantastic and well-polished game; don't let Starbound keep you from it as they essentially have zero in common.
     
  14. ArchGaden

    ArchGaden Big Damn Hero

    I've seen plenty of reports of this happening on windows in single player, so I'd be willing to bet it happens on windows servers as well. I have seen plenty of reports along the lines of 'I was playing with my friends on my server and aaaaah we lost our base!!!' which are probably windows servers, but no platform disclosed. Most any large server runs linux, and the large servers are much more likely to crash... so it would be no surprise that linux servers are reporting in here more often. If you want to see a ton of reports on world resets, check this;

    https://www.google.com/search?q=sta...laystarbound.com&safe=strict&biw=1539&bih=851

    The only temporary measure I can think of right now is cron job world backups from a list of known 'developed' worlds...set at a reasonably high interval. Then have it autorestore from the backup when the server crashes (stick it in an auto restart script). In theory, the worst you should experience is a rollback of as long as the save interval....of course you'll be thrashing your hard drive a lot if that world list gets big. I keep hoping i can figure out a better answer... but I think I'll have to fly with this for now.
     
  15. Eberict

    Eberict Void-Bound Voyager

    In our test on a Windows server, the rollback happened. Sadly not restricted to Linux. I'm beginning to suspect it may have to do with that auto-backup feature that got rolled out with hotfix 2, but have no empirical evidence yet, so this is an unfounded suspicion.
     
  16. ArchGaden

    ArchGaden Big Damn Hero

    If true, that would raise the vague hoping of maybe somehow forcing the feature off, but I doubt that is the case actually. It would be worth checking the configs for a new value related to backups. If I recall correctly, the backup feature is intended to backup player/ship files, not world files. Of course, its hard to believe anything I read in patch/hotfix notes now...they mostly seem to be random noise. The .fail files seem to be intended as a recoverable backup of sorts, should the corruption error that created them be reversed, but since there is some error that keeps them from loading in the first place, they don't work as functional backups currently. I haven't seen a single report of a .fail file every working when renamed as .world, despite the many suggestions to do so.... but that would be expected since the error hasn't been fixed yet.

    At first, I thought this was related to container issues, but since it does affect worlds generated after the latest hotfixes, this is probably not the case.
     
  17. SweFox

    SweFox Guest

    Yesterday a world that one of my players worked on for 10+hours regened itself. Now I made sure to back shit up every hour.
     
  18. Frigglish

    Frigglish Space Penguin Leader

    Well we'd all like to think that, but let's face it.... Saint Patrick's day is less than a month away, the dev priorities are pretty much set in (Blarney) stone. :facepalm:
     
  19. White935

    White935 Big Damn Hero

    Would they really kill off their population thou? as currently i don't know how many would be happy if this isn't hotfixed, i mean .. its like having several wipes a day, and we don't like wipes <.<.
     
  20. Calabrese

    Calabrese Guest

    Excuse me if I'm wrong, but can't you bypass the obnoxious Steam update mechanics by running the game's executable directly?
     

Share This Page