Closed Starbound takin 10-20 minutes to load

Is there any way to make this "faster"? Does the mods that I have installed make starbound to load this slow? If someone knows how to fix it plz help me because I really like starbound, another thing is that when I'm playing in multiplayer sometimes it crashes and I have to wait 10-20 minutes again.

If someone wants to know, I have a MacBook Pro with Yosemite 10.10.3

 

It's normal that it takes a while to initialize the game, but as that word suggests, this should only happen when nothing of that game is loaded into RAM anymore.
Yes, mods influence the loading time, universe changing mods do so pretty heavily.
One way you can slightly speed the startup up is packing your mods into modpaks.
How many mods are you using?

Another way would be closing any other application (be it a browser or another game, doesn't matter) that's running in the background or assigning a higher priority to the process (not really recommended).

 

Hard drive speed is a MAJOR factor in loading. Laptops typically have a 5400RPM drive where as the standard is 7200RPM for desktops. SSDs load the game in seconds.

Id wager your macbook has a small 2.5" HDD that is only 5400RPM

 

Right now I have 4 mods, 1 of those is fully customizable ship, I think that one is making the game to load that slow. How I pack mods into modpaks? Some mods are .modinfo but I think that isn't the same as .modpak, I'll delete the one with the ship because when starbound loads, it instantly crash.

EDIT: I also tried rebooting my mac, closing steam, etc.

 

Unfortunately, modpacks won't solve the issue, because the internal structure of Starbound archives contains holes of useless padding (currently 35 Megabytes out of 1 Gb archive) between the records and the files inside of archives are stored with no order at all, since these are not really archives but databases with filenames smeared all across the file. Not sure about other OSes, but in Windows NT 5.x you'll have to launch the game and let it load at least once so its data would be kept in RAM cache and load much (25-50 times) faster until the next reboot.

In other words, if you have a classic HDD, it will need to physically read tiny 64-128 byte long pieces to enumerate all the files inside of .pak (currently over 34 000 files) and .modpak.

This is ineffective by design, because modern hard drives are reading in huge blocks and would have to re-read them over and over again (note the internal HDD cache is usually limited to 32 Megabytes, and about 10-15 reads will be enough to purge it entirely).

The real solution is to discard the current .pak format and make new one, which would have:

• consolidated file allocation table stored at the end of archive (this would allow to append new files and reattach the table without the need to rebuild the archive from scratch and also to compress the filetable with zlib for faster loading and saving the space)
• script files (.lua, .dungeon, .json, .npctype, etc.) stored at the beginning of archive (preferably also compressed with zlib)
  
• no padding holes between the files

For anyone concerned, here's the easiest possible working prototype:

Code:

<Header>
4 bytes - char[4] - magic "SBPF" (StarBound Pack File)
4 bytes - dword - version (0x00000005)
8 bytes - qword - number of files (4 bytes would limit us to ~4 million files, 2 to 65535)
8 bytes - qword - filetable offset (4 bytes would limit us to 4 Gb archive size)
</Header>
<Files>
....
</Files>
<Table>
<TableEntry>
<TableEntry>
<TableEntry>
....
</Table>

Each table entry would consist of:
<TableEntry>
8 bytes - qword - offset
8 bytes - qword - filesize
8 bytes - qword - compressed filesize (required for proper extraction. If 0, then the file is not compressed)
8 bytes - qword - timedate stamp
2 bytes - word - filename size
? bytes - null-terminated Unicode string - filename, including path
</TableEntry>

I should also note such structure eliminates necessity to use database runtime for archive packer/unpacker - the code can be done with native read-write I/O functions and the compiled binaries will take less than 200 Kb with embedded zlib (right now packer and unpacker use 20-80 Mb each).

Edit: Also, in order to make the game itself run faster (and decrease the amount of reads), several other factors must be taken in account:

• all .png files of Starbound must be cleaned of Photoshop's junk metadata (it takes up to 99% of their size. No, I am not kidding) - this will shave off around 100-150 Mb off the pak file.
• all .wav files must be compressed to OGG Vorbis (music and several sound effects are already stored in them) - this will shave off another 100-150 Mb without any noticeable sound quality loss (since OGG is far superior than MP3 and capable of reproduction of 18 kHz data without audible artifacts even at ~128-160 kbps bitrate). Funny to note, most of Starbound's wave files were 128kbps MP3s in the past (possibly they were decompressed because CF ran into problem with licensing MP3 compression from Fraunhofer before its patent eventually expired), judging from their spectrogram: http://i.imgur.com/hM7cEqX.png
• all .json dungeon files must be converted to classic .png + .dungeon combos. This is the real problem with bloat ever since the dev team adopted the Tiled fork, because the generated .json files are outright monstrous. For example, if you'll try to compress monolith.json into ZIP archive, its size will drop from 7 Megabytes to 8 Kilobytes (same would happen if archive format used zlib compression for them). This happens because instead of storing tile map as pixels they store it as textual representations of integer values, which take not 4 bytes (like ARGB ), but up to 10 symbols and not compressed any further. And yes, this will shave off another 100-150 Mb.
• starter planet lookup mechanism must be either rewritten from scratch or slightly altered, so instead of looking up for the "Garden" world the game would look for "Gentle Star" type instead (which always spawns Garden world) and ignore the world's size (right now it searches for "small" planet, but these are very rare, which causes the game to "freeze" for 5-15 minutes on slow machines at first-time universe generation run). Just by changing the lookup values to "Forest" world of any size (yet kept the "Garden" and "Desert" in conditions) I've managed to make the game run first-time generation for least than 2 minutes.

 

Even if what MrChow said is true, that has nothing to do with @makiloja's problem. To solve his/her problem, we'll need a list of his/her mods and/or their starbound.log file. Mods can cause lengthy load issues and multiplayer crashes when the client (user) attempts to do something that a server without the same mods does not understand.

Here's a tutorial for finding the starbound.log file: http://community.playstarbound.com/...ors-fixing-most-problems-yourself-v1-1.69198/

Keep in mind that, if you do not see file extensions, the file may also just be named "starbound" in the same file folder.

And, yes, modpack files would help the issue since they do reduce the loading time significantly, regardless of their supposed subpar organization.

 

I know windows copies of starbound are slow to load, I'll make a pass at it today or this weekend to try and speed it up. A lot of what MrChow said is accurate, but a lot of it is not, specifically about the db format and the way that chunks of the universe are generated and planets are found. Yes I know it's slow, but WHY it's slow is not as simple as that.

I'll try to go through a (quick!) rundown of what is accurate and inaccurate with MrChow's post, it is not the lengthy answer it deserves (EDIT: okay it ended up being pretty lengthy)

Yes, it would be nice if we ran a png crusher as part of the build step before building the pak files. I run the build servers and all the infrastructure to maintain our complicated build process, I haven't done this yet though it would be a good idea.

We have never used mp3s in starbound, starbound has never had the capacity to decode mp3s ever. We use wavs for short sound effects to reduce the CPU time for decoding on sounds that are played all the time, though we do decode small sounds in memory and then cache the decoded sound instead of the compressed sound, so we COULD store all the sounds in starbound in .ogg and it would probably be okay, I might ask stephen when he gets back to look into just converting it all to ogg.

TMX dungeon files are huge, we have a compressor but we have to run the compressor in an automated fashion or incorporate it into the editor. This is a problem mostly with the nightly, so.. it's the NIGHTLY haha. This is a problem that we will solve before the next release.

Unfortunately, every chunk in the universe is generated as a complete chunk, not individually. Changing how the systems are enumerated doesn't help, the current generation algorithm generates one entire chunk at a time. It is slow, but do you know WHY it is slow? It is entirely because of JSON serialization when writing to the chunk store. NOT the generation, NOT the actual db write of the store, just the constant serialization and deserialization to JSON. It does need addressing though, I'll try to make some headway on it this weekend.

This is not entirely accurate.. the filenames of the entire db are stored in a single index record which like all of the records use 2048 byte blocks. The index is read in almost no time at all (WAY before the chucklefish logo appears). Now, there IS a problem with fragmentation of the pak format, but the thing that prevents me from addressing it is that during more active stable development people HATED the large downloads that occurred all the time, so one of our approaches was that when updating the pak format, it would be updated IN PLACE. This means that over time, it gets quite fragmented, and prevents me from running a defragmentation pass because the way steams binary delta algorithm works it prevents me from being able to re-organize the pak file between updates.

Now, the situation has changed somewhat as of late. We don't update stable with the frequency we used to, and we now have nightly and people who are on the nightly tend to care a little bit less about large steam downloads so here's what I'm gonna do:

The format that the pak file uses is not out of nowhere, it's the same database format that's used for world chunks and celestial chunks and all kinds of things, so I think it's fine to keep using it if it can be made fast, after I look at the performance numbers on windows on a platter hard drive, if I notice that this indeed is part of the cause, what I will do is make a way to defrag the current format with each record in contiguous blocks in order of largest to smallest starting from the beginning, with all the index blocks at the end of the file. It's not hard at all, I just haven't.. done it yet, there's even some comments about it in the assets packer about doing it later.

There are some caveats though. The difference in read speed between me on os x (even from a platter hd over usb!) and some people's load speed on windows is SO VAST, that it makes me think there's actually something more sinister going on. Alex thinks it may have something to do with some filesystem api changes that happened before the last stable update and I might be mis-using some windows API in some way, I'm not sure. Like I said I'll try and get to the bottom of it this weekend.

Anyway thanks for all your patience and for reading this far.

EDIT: Changed quotes to be actual quotes.

 

And thank you very much for the reply. All my conclusions and suggestions regarding the issues that plague Starbound were coming out of pure data analysis with available instruments and game engine's behavior observations, so I'm happy they all were confirmed (albeit some of them indirectly) and will actually be addressed.