Modding Help Someone explain this to me like I'm 5

Discussion in 'Starbound Modding' started by matrixparadox, Sep 2, 2019.

  1. matrixparadox

    matrixparadox Space Hobo

    Attempting to use a .pak mod. I'm new to modded starbound, and don't know an incredible amount of computers. Was told by a member of the group I'm in that I will need to unpack it, so I looked at the tutorial and was hit with "This guide will assume you know what you're doing and are a competent computer user." Which I do not, and am not. Could someone please explain this for someone who has a very basic level of computer skills, and is a fairly casual player?
    Not sure if this is needed, but I'm running Windows 10.
     
  2. Errors4l

    Errors4l Spaceman Spiff

    So pak files are basically zip files, but for Starbound. They're just a way to store many files (assets) in one file. This can (and usually does) reduce loading times, because your hard drive only has to find a single file.

    Unlike zip files (which are used world wide for just about anything), pak files are specific to Starbound. There isn't any native support for it, so you need software capable of reading the file before you can view its contents.
    Luckily, Chucklefish have bundled some tools that allow us to unpack (and pack) these pak files. Well, it wouldn't make sense for them not to because it's kind of vital (but not required) when releasing mods.

    For Windows, you can find these tools in ...\Steam\steamapps\common\Starbound\win32. They're called asset_unpacker.exe and asset_packer.exe for unpacking and packing respectively. We'll only look at unpacking for now.
    To use the unpacker, you can't just run them or drag files on them. You will need to open a command line window first.
    Windows key => Command Prompt (cmd.exe).

    Navigate to the folder using the CD command.
    CD F:\Steam\steamapps\common\Starbound\win32
    Is your folder on a different driver such as F:\ in my example? Just enter that drive letter to switch to the folder.
    F:

    [​IMG]

    Next up, to actually run the unpacker, just enter the file name (you can auto complete it using tab) and hit enter.

    [​IMG]

    This is the same as double clicking on the application, but it will immediately exit without doing anything. In the command prompt however, we can still see the information which tells us how to use the unpacker.
    The usage is very straightforward, just keep in mind that paths containing spaces should be wrapped in quotes.
    asset_unpacker.exe "C:\path\to\your\mod.pak" "C:\path\to\your\_unpack"

    Note that I'm using an underscore at the start of the output folder. Folders and pak files that start with an underscore are ignored by the game, so if you unpack to your mods folder it won't load the unpacked files. If you omit the underscore, make sure you check whether the game is loading the assets from your folder or from the pak file (cog icon in the bottom right corner of the main menu).

    [​IMG]

    Don't see anything happen when you hit enter? Just wait. As long as it's not showing you a new empty line with the current directory, it's busy.
     
  3. bk3k

    bk3k Oxygen Tank

    They're mistaken. Please inform them of this as well - please send them this thread. You can use either loose files in folders(named for the mod) or .pak files. Put the .pak files in \starbound\mods\
    While I do somewhat hate .pak files (as I like to look and tinker and they add additional steps to do so), they do have some advantages over loose files.

    1. They (effectively) take up less space. This is due to a quirk in how file storage systems work. There would be a set cluster size - the default system in Windows is NTFS and NTFS has a default cluster size of 4KB. It could be set higher though(which speeds loading larger files). Only 1 file can be used by a single cluster, but files often use multiple clusters as they usually don't fit in only 1 cluster. A 1KB file therefore effectively takes up 4KB of space. A 5KB file effectively takes up 8KB of space. Even worse with 64KB clusters - a 1KB file effectively takes 64KB. That wasted space can add up. HBP+ alone has close to 6,000 files (And that's after I shaved down quite a few). FU has nearly 70,000 files! But a packed file - as far as your operating system is concerned - is only 1 file, so you cut down on wasted space even without compression.

    2. They load faster. With loose files, Starbound is reading all the files individually. That means reading lots of partially empty clusters, and thus wasting time. If using a traditional spinning magnetic storage drive, the head needs to move around too and that makes makes loose files even slower by comparison.

    Now the disadvantages really are just a problem for mod makers and tinkerers.
    1. Proprietary format with proprietary tools. You cannot use great tools like 7zip (or anything with a GUI) to handle packing and unpacking. You must instead handle it with command line tools.
    2. You can't really browse through it without unpacking it. And of course you can't edit it this way. Unpack, edit, repack. Loose files you just edit and use as-is.
    3. You cannot unpack files made with older versions of the packing tool - unless you locate older versions of the packing tool! Unless you have them, you are SOL...

    So yes you can and should use those .pak files as is if they're already distributed that way. What you cannot use are .zip, .rar, and .7z etc files. Those are compressed archive files and need extracted. 7zip is an excellent little program that handles all 3. Typically I sent my mods out as a .7z file because that makes the mod upload and download faster, plus it is less trouble than bothering to pack it. HPB+ is the exception because of the high file count really demands I go ahead and pack it.

    And personal opinion here - Starbound SHOULD have been programmed to go ahead and use .zip files at least. "No Compression" is a valid mode if you want to maintain speed, and the devs should have used this instead of their own pointless proprietary format (of which changed several times over the course of development) and proprietary command line utilities to pack/unpack them. Zip files are well supported by modern OS's even without installing additional programs. That would make everything about this totally painless. Instead they created additional work for themselves and users as well - for no reason at all.

    Just imagine - you download a zip file. You (like everyone) knows exactly what a zip file is. You place it in \mods\ and you're done. If you feel like poking around the mod, you just double click on it.
     
  4. SpiroExDeus

    SpiroExDeus Cosmic Narwhal

    Before you go around wildly sharing opinions and, by extension, accusations it's always a good idea to get your facts straight.

    pak is not Chucklefish's own proprietary format it is, from what I can find from a few quick googles proprietary but was invented by electronic arts. As it was invented specifically for videogames (or computer games for those of us who aren't from america or unduly influencred by them) I suspect that it was picked as a form of compression which wouldn't slow the game down.

    Windows has inbuilt tools in its command line for packing and unpacking files and there are also free third party programs for doing so, so I really don't see your problem with the format. There are multiple types of compression available all of which have their benefits and disadvantages. Chucklefish picked one which they clearly felt was appropriate for mods. Command line is not hard and should really not be too much of a challenge to the kind of person who wants to use a mod.

    If using windows you can open command line from the folder you want to run commands in by shift and right-clicking and picking the right option from the menu that appears. If you have copied the command you want from your guide/wiki/forum post you can paste it by simply right clicking on the command line window. In many ways its almost as simple as double clicking and has the advantage that it's harder to do by accident.

    In terms of tools, 7zip is a downloadable tool and another quick google found a variety of tools for both Windows and Mac (and I'd be very surprised if there weren't any for linux - although a lot more Linux users are familiar with command line anyhow) that handle pak files so it boils down to whether or not you're famailiar with, and there are people who haven't a clue what 7zip is.
     
    Last edited: Sep 3, 2019
  5. Errors4l

    Errors4l Spaceman Spiff

    You realize a file extension says nothing about the contents of the file, right?

    The packer and unpacker released with the game are written by ChuckleFish and will only work on pak files made for Starbound. You seem misinformed on the topic even though you're criticizing someone else for the same thing.
    Other games that use pak files in a similar manner won't be able to read them because the data is stored differently in terms of raw bytes. Any integration in existing software such as 7zip would have to be programmed in.

    There are some alternative tools/libraries which have been written specifically for Starbound .pak files. They will not work for other pak files, just like the tools Chucklefish bundled with the game.

    https://community.playstarbound.com/threads/pakfs-pak-files-in-windows-file-explorer.154762/
    https://community.playstarbound.com/threads/starpakbrowser.154702/
    https://github.com/blixt/py-starbound
    https://community.playstarbound.com/threads/pakit-the-simple-graphical-tool-for-pak-files.142681/
     
  6. bk3k

    bk3k Oxygen Tank

    Indeed - but I have my facts straight. Do you?

    EA may have made their own proprietary format using the same file extension. However the format here is their own in-house one. If they had used a prebuilt solution, you wouldn't have problems like the current unpacker being unable to process older packed files made for Starbound earlier in development. Go ahead and try it. Find an old .modpack file (the extension they used to use). Try to unpack that with the current tools... You'll only get an error of course. Also I'll tell you their format has NO COMPRESSION. You can read plain text straight out of it when over a section containing JSON or lua files. I've used this fact to do string searches when problems come up. And that's why I mentioned that the zip file format offers a compression-less mode. Compression after all does increase CPU load somewhat.

    If you doubt me on this, find any small .pak file and open it in Notepad++. Then scroll down. Much of it is plain text you can read as-is. And Notepad++'s "find in files" feature works on a folder full of .pak files for this very reason. This can be useful for example finding duplicate references in your workshop folder.

    Indeed it isn't a challenge (in my opinion). It is however an annoyance. It is not as easy/fast as right click then left click. To make it a little more handy, I keep a file of commands to copy/paste into a command prompt. Or I should say "terminal" since I switched to Linux earlier this week. But that's not the point. There isn't any benefit to the way they've done this. They just made extra work for everyone - especially themselves. They reinvented the wheel here without any clear benefit. What's the upside?

    Anyone can easily Google 7zip to learn what it is, but that's irrelevant. That's just 1 example of a great tool that can be used if they had merely used something mature and well adopted like the zip format. You don't even need 7zip (though it is nice) to handle zip files. Any number of tools work on zip files. You can pick one, but even that is optional. In Windows where you haven't downloaded ANYTHING - you can double click on a zip file and use it as though it where a folder. And I just checked on my new Linux Mint install - I can also do the same. I can use zip files even though I have not installed any special tools to do so. Ditto with 7z files and probably rar files too. Like I said, zip is natively supported by modern operating systems. From the moment you install the OS, zip works.

    It is also common for different programs to use the zip format while renaming the extension to make it look proprietary. Firefox extensions for example - those .xpi files are actually zip files. IIRC Winamp doing the same with downloadable skins. Those 2 come to mind without searching. Or another example - directly relevant - is Minecraft. You can put .zip files or .jar files (which are essentially just zip files with a different extension intended as a Java archive) in the \mods\ folder and that's all you need to do. Drop either in, and you're done. Supposing you want to poke around a bit, the same utilities which open .zip files can also handle the .jar files (because they're just the zip format). Thus nothing is proprietary, and that's better on everyone.

    But yes, Minecraft players can just drop those .zip or .jar files directly in \mods\ and it could have been the same story here if they used an easy out-of-the-box solution like zip.
     

Share This Page