Modding Discussion __merge issue with nested arrays/lists

(I don't know if all this is correct, but thats how it looks like to me : )

Using __merge appends elements at THE END of arrays/lists by default. But some arrays need a very specific number of elements, e.g the "genders" in the .species files are exactly two.
It seems that the only way to manipulate data in those arrays is to overwrite the whole array. (Or has anyone a better idea?). The same goes for arrays nested in arrays, I guess.

Now the real problem is that one can not use any of the old data after overwriting the array. So if you have a complex stack of nested objects/maps in an array/list you have to copypasta the whole thing, wich - of course - torpedizes the whole merge-for-compatibility feature.

Code:

old foo is
"foo" : [
    {
        "bar" : [ "olditem1", "olditem2"]
    }
]

I need:
"foo" : [
    {
        "bar" : [ "olditem1", "olditem2","newitem1","newitem2"]
    }
]

Code:

{
"__merge":[],
"foo" : [
    {
        "bar" : [ "newitem1", "newitem2"]
    }
]

// This ends up in (I guess)
"foo" :[
{"bar":["olditem1","olditem2"]},
{"bar":["newlitem1","newitem2"]}
]

Code:

{
"__merge":[
  ["overwrite","foo"]
],
"foo" : [
    {
        "__merge":[],  <---------- does nothing, putting it one line higher gives a "bad array" log error
        "bar" : [ "newitem1", "newitem2"]
    }
]
}

/// bar would only be newitem1 and newitem2

Code:

{
"__merge":[
  ["overwrite","foo"]
],
"foo" : [
    {
        "bar" : [ "olditem1","olditem2","newitem1", "newitem2"]
    }
]
}

/// This obviously works, but really, makes the whole merge system pointless.

So, please correct me or post your ideas on how to ADD values to an array in an object in an array without destroying the old data or the array elements number... I'm lost.

A more practical example is adding new Hairs, Shirts etc to the gender Array in the species files.
Another discussion dealing with this in the default.treasurepools.
Another example: village guard dialogue.
And antother example with Biomes and Plants and a suggestion for an unorthodox solution

(code is only for visualization, there might be missing " , or stupid brackets. Also, the code is very simplified. Check out the genders in .species, there is a LOT more data that needs to be merged properly.)

 

And I still have no solution. For my mod I had to use a "halfway" implementation of the new merge stuff: Merging most of the file, but overwriting the whole section that I want to add a few items too. It's a bit more compatible now, but still... dirty.

 

Ahhh I was having a similar problem. Hopefully Bartwe will see this. Until then I'll have to get messy I guess.

 

Mhh... I think an elegant solution would be to make "overwritten" data still accessible with merge. Something like this example

Code:

{
"__merge":[
  ["overwrite","foo"]
],
"foo" : [
    {
        "__merge":[],  <---------- does nothing now,  putting it one line higher gives a "bad array" log error
        "bar" : [ "newitem1", "newitem2"]
    }
]
}

/// bar is now only newitem1 and newitem2 - should be "bar" : [ "olditem1", "olditem2","newitem1", "newitem2"]

This would be quite flexible, I guess. But this has to be adressed by the Devs in the code directly. I guess there is nothing we can do right now...

 

I hope this gets addressed soon.

 

Same, this seems to be what everyone's running into when trying to add new default hairstyles and clothing mods to character creation. Oh well! It'll be fixed soon enough I'm sure.

 

Thanks for your answer, but I recommend that you reread my first post. I'm totally aware how merging works. Exactly, it adds items to lists/arrays, but only at the bottom. And this is a major problem, when you need a list/array with a very specific number of items, when a lot more data is nested in this "root-list-array".

In other words - You can not really copy a nested array/list when doing an overwrite of the "root-list-array". And this negates the purpose of __merge and compatibility.

By the way, nice tutorial. I really hope all modders will read it and use the merging feature well.

 

Yes, that is exactly the problem we discovered. One has to ovewrite the whole array. But in cases like the genders in the .species files one has to copy over all the nested data: shirts, hairstyles, beards - even if you just want to add one single hairstyle... and this makes different mods incompatible again.

This is something that has to be adressed by the Devs, I'm afraid.

 

Here is another example. It also affects the village guard dialogue.

 

Sorry for any inconvenience, I changed the discussion title to a more clear description. Old links should still work.

(All code in here is a suggestion and does not actually work)
I was thinking a bit. There is also no way right now to delete/manipulate one specific entry in an array. (I'm also not so sure if this would really be a solution to the nesting issues.) The problem here is that one can only adress the array items with the index, but if one mod tries to delete index [10] all the following indices will be -1 (wont they?) and if another mod then tries to delete index 12 it actually deletes 13, wich is wrong... Maybe keeping track of deleted indices would be a way to avoid this... Or maybe it's possible to replace a "deleted" index with a "dead" entry that will be ignored in the end.
Even then, at the moment one could only delete/overwrite a specific value, to manipulate the nested data one has to "keep" it...

Code:

  "__merge":[
  ["overwrite",["fooarray",10]],
  ["delete",["fooarray", 12]]
  ]

Mh, I still think overwrite and delete have to be a different command, the only difference is the name right now. While deleting should delete the value right away, overwrite should somehow replace/keep the value step by step. So if I say:

Code:

"foo":["one","two","three","four"]
============
"__merge":[
  ["overwrite","foo"]
],
"foo" : ["A","B"]

It should simply take the old foo and go through the indices step by step, replacing them with the new values and only afer it hits the end of the new foo it drops the rest of the old. the result should still be "foo":["A","B"]. Or would it even be better if it keeps the rest? "foo":["A","B","three","four"]? This would be an alternative way to replace specific indices (but the problem mentioned above still persists).

Code:

"foo":["one","two","three","four"]
============
"__merge":[
  ["overwrite","foo"]
],
"foo" : ["A",{"__merge":[]},"C"]

///and the result would be
"foo":["A","two","C","four"]

I don't know if this makes sense, but it looks like a locical way to merge the nested stuff. Except when you want to replace one entry and ADD another one at the end at a very long array or even if you just want to replace index[137]...
Anyway, for the hair this could look like this:

Code:

/// mod that adds one new female haircut to a .species file

{"__merge":[
  ["overwrite","genders"]
],
"genders" : [
  {
  "__merge":[]  // keep all male stuff
  },
  {
    "__merge":[],  // keep all female stuff, but add
    "hair":["mycoolnewhair"]
  }
]

Those nested arrays that cause the merging problems right now are very specific anyway, so the chance of two mods interfering with each other would be minimal. There is no need to delete entries and to confuse the indices in this case... And adding new stuff is still always a new item at the end of the array, so multiple mods should be compatible...

arr, but what if one mod disrupts the nested layers of arrays and objects, but a second mod tries to delve into them and add something to some element that isn't there anymore... mmmmess.

A completely different approach would be a way to access nested elements directly through punctation (or something similar): "genders".[1].{0}."hair" : ["myadditionalhair"]. Like other languages do it. I have no idea if something like this would be possible in JSON.

Maybe someone has some better ideas here.

 

But there is still the problem to define wich array-entry / index to manipulate. There can be arrays with all the same entries, so one can only rely on the indices and those could have been changed be other mods already....

Code:

"foo":["one","two","three"]
========
"__merge":[
    ["arrayindex",["foo",1]] /// keeps whole array, but sets the pointer to index 1 for whatever is coming next. problem: what if more than 1 entry should be replaced
],
"foo": ["newentry1"]
=========
"foo":["one","newindex1","three"]

I'm not sure if the indices would really be that problematic: There are eiter "undefined" lists/arrays, like recipe lists, where people just need to add stuff, unsorted - no need to access specific indices here. Or "defined" arrays like the genders, where the entries need to be a very specific number anyway, where it is important to acces a specific index for nested data, but no need to delete/add indices. I guess...

 

I sure hope that there's a function in Starbound's game engine that's similar to JavaScript's .concat function, which takes the value of an array, and adds more indexes to the end. For example:

Code:

array1=[a,b,c];
array2=[d,e,f];
array3=[g,h];
array4=array1.concat(array2,array3);

//The values of array4 should be the following variables: a, b, c, d, e, f, g, h.

Such a feature would make community compilation mods alot more redundant.

 

Yes, if I'm not mistaken, this is the way __merge works right now with arrays/lists. Only that the arrays are in different files, with the same name... arrayOriginal=[a,b,c];arrayMod1=[d,e,f];arrayResult=arrayOriginal.concat(arrayMod1)....
The only problem here is that it doesn't help with hair-mods right now because of genders=[{male},{female}] - you don't want to concat this, you need to "go into" it....
I was so thrilled for the update, and now I have to wait again... sad sad.

 

You guys know that lua doesn't actually have arrays, right? they're called tables

..hahaha I'm just pulling your leg. please continue.

 

I don't know anything about lua. So I'm glad, that our issue is with JSON and the custom __merge feature.
To be honest, I also don't know where the "list"/"maps" name comes from, but I've seen it a lot. The W3C still call them arrays and objects.

 

I already said it before elsewhere, but I'll say it again: I think this kind of thing should be handled through lightweight Lua hooks which expose the assets to mod scripts handling the asset modification. Even if we managed to extend the merge syntax to allow deleting and merging entries from within an array, we'd either still not be able to perform more than one operation to an array (such as changing entries as well as adding new ones, in a single mod), or we'd end up with a horribly bloated and unmaintainable syntax to reach that level of generality.

I think this should be handled in the following manner. Suppose you have a file "/some/path/to/an.asset" relatively to the assets directory, consisting of

Code:

{
    "foo" : [ "one", "two", "three" ]
}

Suppose you want to replace the entry "two" (which could be another array, object or whatever, I'm using strings for simplicity). Using a Lua hook, you could easily access that entry semantically, maintaining compatibility with mods changing that same array, in the following way:

Code:

AddAssetHook("/some/path/to/an.asset", function(asset)
    for i, v in ipairs(asset.foo) do
        if v == "two" then
            asset.foo[i] = "something other than two"
            break
        end
    end
end)

I don't see how else (i.e., without resorting to a proper programming language) we could achieve the level of generality we need.

 

Yeah, I was half asleep when I wrote that. I donno what I was thinking. I'll crawl back into the hole from which I came now. Good luck with your JSON merging

 

I posted it in another thread already but...

If all this syntax-logic-progamming-code is too complicated, maybe it would be a simpler solution to change the structure of the critical configs, avoiding nested arrays alltogether. I know, this will break a few old mods, but...

Code:

"genders" : {
  "male" : {
      "image" : "/interface/title/plantmaleicon.png",
      "characterImage" : "/interface/title/plantmale.png",
      "hair" : [ "1", "2", "3" ],
      "shirt" : [ "floranfurnivourchest", "floranhunterchest"],
      ...
  },
  "female" :  {
      "image" : "/interface/title/plantfemaleicon.png",
      "characterImage" : "/interface/title/plantfemale.png",
      "hair" : [ "1", "2", "3" ],
      "shirt" : [ "floranfurnivourchest", "floranhunterchest" ],
      ...
  }
}

Instead of

Code:

"genders" : [
  {
    "name" : "male",
    ...
  },
  {
    "name" : "female",
    ...
  }
]

(probably not so clean in other cases...)
Edit: Here are examples from the biomes and plants

It would be easier to edit nested stuff like this, wouldn't it?