June 25 - Changing JSON Merging

The future is looking bright (Or dark if that's your thing.) From what I've read in this thread the new system is as easy to adapt to as changing a few lines on a file that's going to be rewritten to fit the new system anyway. And on top of that it makes rewriting easier while allowing even more functionality, (Potentially dangerous, but in science and backups of our backups backup we trust)

I say go for it, even if it all goes catastrophic, there's always that one functional backup to look back to with the knowledge you gained from testing.

 

Except, in the case of the most common operation, namely adding an item to a list so that the item can be found among vanilla items, the patch form is a lot LESS verbose than the current form, which is to completely replicate the affected structure with ones entries added in.

There is a reason why, with the __merge form of mods, people generally add the items to crafting tables and do not have the items spawn in the world. If they were to use the current system, adding the items to spawning tables means completely replacing those affected tables. If one has two mods which both change the same file in this way, one risks either the items not actually appearing or crashing the game entirely. So the patch form is also more stable, in addition to being, overall, less verbose for the most common use case.

You may respond saying that people simply don't add items to spawning tables. And you're right. And the reason isn't because they don't want to, but because it doesn't work correctly.

I've worked on compatibility patches for Frackin Flora + Vitsotu + Starscape. The problem here is that all of them modify liquids, biomes, and star types. With the current system, a set of files allowing the compatibility merge must be made as follows:
Vitsotu
Starscape
Frackin Flora
Vitsotu + Starscape
Frackin Flora + Starscape
Frackin Flora + Vitsotu
Frackin Flora + Vitsotu + Starscape

Let's say that you wanted to add another mod to that... Let's choose something simple like Wasteland. This is where it gets scary:
Vitsotu
Starscape
Frackin Flora
Wasteland
Starscape + Wasteland
Vitsotu + Starscape
Vitsotu + Wasteland
Vitsotu + Starscape + Wasteland
Frackin Flora + Wasteland
Frackin Flora + Starscape
Frackin Flora + Starscape + Wasteland
Frackin Flora + Vitsotu
Frackin Flora + Vitsotu + Wasteland
Frackin Flora + Vitsotu + Starscape
Frackin Flora + Vitsotu + Starscape + Wasteland

The amount of merges which must be made rises combinatorically, doubling in size for every mod added to this list. That's at 15. Add another mod, and you get 31. Another and you're at 63. And so on.

With a patch system, this kind of behavior is transparent, natual, and the need for compatibility patches between them in every various combination one might want VANISHES. One mod each, all would work with minimal modifications.

That's why RFC Patches are superior to __merge.

 

http://chbrown.github.io/rfc6902/ this generates patches, although it seems to omit [ , , , , , ] in the displayed output. (a text box to copy would be handy)

paste the same json file in both sides, edit the right side however you like, watch what the diff spits out.

if you have entire json files in your mods that replace a default asset with the same file name without using __merge, you could paste the default asset json in the left side and the modded json in the right side.

a big problem I have noticed is that neither of the tools I linked in this thread allow javascript comments in json

:%s/\/\/\(.\+\)$/\/* \1 *\//
I used this vim command to quickly transform player.config to test /* comments */
:%s/\/\/.\+$//
I had to remove the comments instead

 

The concept's sound, but I don't think it's safe to just throw it at every file:

Code:

//Generated. Will end in tears.
{
  "op" : "replace", "path" : "/surfaceBiomeParameters/forest/dungeonSelection/0",
  "value" : [ 0, [
    "scifidungeon", "apexresearchlab", "apextestchamber", "apextowerblock",
    "avianairship", "aviantemple", "aviantomb", "aviantower", "avianvillage",
    "floranhell", "floranhuntinggrounds", "floranvillagetower", "glitchcastle",
    "glitchsewer", "glitchvillage", "humanbunker", "humanprison", "naturalcave",
    "felinvillage"
  ] ]
}

//Hand-written. Will work.
{
  "op" : "add", "path": "/surfaceBiomeParameters/forest/dungeonSelection/0/1/-",
  "value" : "felinvillage"
}

 

good point. I encountered a slightly less nasty version where it did an "add" and "remove" instead of my handwritten "move"

Generated:

Code:

[
{"op":"add","path":"/defaultBlueprints/tier1/-","value":{"item":"silverarmorhead"}},
{"op":"remove","path":"/defaultBlueprints/tier2/0"}
]

Handwritten:

Code:

[
{"op" : "move", "from": "/defaultBlueprints/tier2/0", "path": "/defaultBlueprints/tier1/-"}
]

edit: attaching input file for reference

 

For those worrying about adapting their mods, the numbers don't exactly lie. At least with regards to a species mod. So depending on the kind of mod you got...

There are roughly 885 files in the Felins species mod, including:

• 3 Lua scripts
• 92 framesets
• 468 images
• 12 sounds (of which 8 the Laser Lyre)
• 4 entirely new NPCs
• 10 JSON files that alter existing ones, including 3 less original NPCs

NPCs were without a doubt the trickiest to figure out. But I did it so you don't have to. :critic:

The remaining ~300, all non-merging JSON files, include:

• 77 objects
• 17 upper body garments
• 17 lower body garments
• 12 headpieces
• 67 recipes
• 64 weapons

The Object+ Extension, consists of 782 individual JSON files, all merging. However, it is automatically generated and basically a single-line edit will do.

The Intro Extension does not alter anything. It only replaces.

 

From Omni's stream

 

I'd like to reference
http://community.playstarbound.com/index.php?threads/extensions-to-the-__merge-command.69962/
A number of list modification features are available in __merge but apparently not on the radar of modders.
These features explicitly avoid using fixed indexes to allow multiple mods to work on the same file.

 

can json patch be augmented to do things __merge can do?

 

You're welcome to ask Omni on stream if you want. THough he went to get lunch atm. Will post the stream here when he's back

 

Only thing I can imagine Patch can't do is removing certain kinds of things:

Code:

{
  "defaultBlueprints" : {
    "tier1" : [
      { "item" : "mininglantern" },
      { "item" : "copperarmorhead" },
      { "item" : "copperarmorchest" },
      { "item" : "copperarmorpants" },
      { "item" : "darkwoodmaterial" },
      { "item" : "yarnspinner" }
    ]
  }
}

Code:

[
  { "op" : "remove", "path" : "/defaultBlueprints/tier1/4" }
  //Removes darkwoodmaterial, assuming the default list doesn't change.
]

Whereas with __merge, you can refer to { "item" : "darkwoodmaterial" } to remove it.

 

He's back. Feel free to ask

 

I KNEW Omni was one of us (Linux weilding Vim users.)!!!!

 

Bartwe brought that up in my stream, and I asked him to cross post it here. For the record.

 

@Omni you even play the game like I do.

*code code code*, *change terminal*, ./starbound, *much cursing*, *code code code*

 

@Kawa
@Jeoshua

Yeah, my answer might be a bit late.

First of all, I'm not telling that current __merge model is good. I wrote two times against it, even being so rude to call it a bad design.
Second, I'm not against JSON patching completely.
Third, IMO this question is a matter of such tricky thing as API design.

Here, we have several existing problems, as I see them:
1. All mods co-exist in the same logical space. Though I won't be raising question on mods moved to their own namespaces here.
2. JSON patching makes both addition new records and manipulation on existing records to be operations of the same order and accessibility. To me this is like your text processor won't ask if you really wanna overwrite existing file.
3. __merge AFAIK performs silent overwrite if omitted accidentally.

Due to 3, __merge should die and be forgotten.
Though due to 1 and 2 we shouldn't accept JSON patching as-is.
I think that the default mod's config layout, which only adds mod's own entities, should be as plain as vanilla config. 'Add without overwrite' should be the default operation.
In case we need to influence existing entries, I'd propose to use separate config, named after influenced one, but with '.patch' suffix. And there we can have JSON patch.
Though, I'd propose to design patching more in terms of entities config describes, and not generic JSON.
Like:

Code:

{
  "remove": {
    "tier1": [
      { "item": "stone-pickaxe" }
    ]
  }
}

 

Considering the fact that this change will break all the mods currently using merge, would it be possible to provide early access and documentation to the modding community and give them a month or so to get used to the new system and have their updates ready to go before the changes go live for the whole community??

the ideas seem good but the major issue boils down to the delay between the update breaking the mods and the developers patching them, so giving the modding community a head start before the actual rollout would mostly take care of it and result in a much smoother changeover for players and server operators.

 

Well these are likely to come to the NIghtly builds first since this is part of the mega update they are working on so I think that'd happen anyway. The documentation would be nice tho

 

What i would like to eventually see is a plugin type system where you can write logic statements to override core game behavior at every level. I wanted to get into starbound modding but most of my ideas required a lot of scripting that the game simply cant support so i kinda gave up because there are simply not enough ways to add in logic statements to a mod, make scripted items that aren't considered tech, or significantly control things such as player re spawning behavior or warping.

but to be honest it might really help if starbound released a modding IDE similar to things like never winter nights toolset, or Fallout 3 / NV' GECK
They had a complete set of design tools that were fairly easy to use and extremely powerful in their ability to really script and manage game resources in a visual environment

 

@mikeloeven: there are IDE fanprojects, I think.