Hook into gltf loader and parser for custom extensions

[lunarraid]

My game needed to be able to play different sounds at certain frames of each animation, so I wrote a Blender plugin that exported all pose markers inside of each animation into the GLTF file as a GLTF Extension. Since pixi3d obviously doesn't support my custom extension, I needed to fork the library and add it here:

lunarraid@f4e1f97

This is not a great solution for the engine at large, though, and got me thinking. Would there be a way to hook into the default GLFT loader to process different extensions after the JSON data has been parsed? Possibly export overridden classes? I don't yet have any concrete ideas on the best structure for this, but was unable to come up with a clean way to simply override the classes and use my own with the current loader architecture.

[jnsmalm]

Would it be possible for you to share a gltf which have these custom extensions?

[lunarraid]

Here is the exported JSON of my original test export when I was writing the plugin (not including the binary, but that data is irrelevant. I can post a zip if needed). The important bits are extensionsUsed at the root, which tells the parser/renderer which extensions it needs to account for, and, in the case of my specific extension, each entry in animations has an extensions property, which lists per-animation extension data.

At the moment, I'm just trying to figure out the best way to hook into the gltf parser in a generalized way, not specifically a way to support my specific custom extension in the library.

{
    "asset" : {
        "generator" : "Khronos glTF Blender I/O v1.8.2",
        "version" : "2.0"
    },
    "extensionsUsed" : [
        "IWG_pose_markers"
    ],
    "scene" : 0,
    "scenes" : [
        {
            "name" : "Scene",
            "nodes" : [
                0
            ]
        }
    ],
    "nodes" : [
        {
            "mesh" : 0,
            "name" : "Cube"
        }
    ],
    "animations" : [
        {
            "channels" : [
                {
                    "sampler" : 0,
                    "target" : {
                        "node" : 0,
                        "path" : "translation"
                    }
                }
            ],
            "extensions" : {
                "IWG_pose_markers" : {
                    "markers" : [
                        {
                            "name" : "TestMarker",
                            "time" : 0.20833333333333334
                        }
                    ]
                }
            },
            "name" : "TestAction",
            "samplers" : [
                {
                    "input" : 4,
                    "interpolation" : "LINEAR",
                    "output" : 5
                }
            ]
        }
    ],
    "materials" : [
        {
            "doubleSided" : true,
            "name" : "Material",
            "pbrMetallicRoughness" : {
                "baseColorFactor" : [
                    0.800000011920929,
                    0.800000011920929,
                    0.800000011920929,
                    1
                ],
                "metallicFactor" : 0,
                "roughnessFactor" : 0.4000000059604645
            }
        }
    ],
    "meshes" : [
        {
            "name" : "Cube",
            "primitives" : [
                {
                    "attributes" : {
                        "POSITION" : 0,
                        "NORMAL" : 1,
                        "TEXCOORD_0" : 2
                    },
                    "indices" : 3,
                    "material" : 0
                }
            ]
        }
    ],
    "accessors" : [
        {
            "bufferView" : 0,
            "componentType" : 5126,
            "count" : 24,
            "max" : [
                1,
                1,
                1
            ],
            "min" : [
                -1,
                -1,
                -1
            ],
            "type" : "VEC3"
        },
        {
            "bufferView" : 1,
            "componentType" : 5126,
            "count" : 24,
            "type" : "VEC3"
        },
        {
            "bufferView" : 2,
            "componentType" : 5126,
            "count" : 24,
            "type" : "VEC2"
        },
        {
            "bufferView" : 3,
            "componentType" : 5123,
            "count" : 36,
            "type" : "SCALAR"
        },
        {
            "bufferView" : 4,
            "componentType" : 5126,
            "count" : 10,
            "max" : [
                0.4166666666666667
            ],
            "min" : [
                0.041666666666666664
            ],
            "type" : "SCALAR"
        },
        {
            "bufferView" : 5,
            "componentType" : 5126,
            "count" : 10,
            "type" : "VEC3"
        }
    ],
    "bufferViews" : [
        {
            "buffer" : 0,
            "byteLength" : 288,
            "byteOffset" : 0
        },
        {
            "buffer" : 0,
            "byteLength" : 288,
            "byteOffset" : 288
        },
        {
            "buffer" : 0,
            "byteLength" : 192,
            "byteOffset" : 576
        },
        {
            "buffer" : 0,
            "byteLength" : 72,
            "byteOffset" : 768
        },
        {
            "buffer" : 0,
            "byteLength" : 40,
            "byteOffset" : 840
        },
        {
            "buffer" : 0,
            "byteLength" : 120,
            "byteOffset" : 880
        }
    ],
    "buffers" : [
        {
            "byteLength" : 1000,
            "uri" : "MarkerTest.bin"
        }
    ]
}

[lunarraid]

MarkerTest.zip

[jnsmalm]

The most obvious/easy solution at first glance would be to expose the entire gltfParser in the API. That way you or anyone else could just override the needed parse-functions. I'm not too keen on doing this as this would probably mean that the entire gltf-folder needs to be exposed as well - and I have to support it as an external API.

One "hook" already exists in the gltf-parser, and that is the materialFactory. One solution could be to do something similar with the other parts (animation, mesh, textures).

Generally it's easier to design the API when you have a set of specific use-cases, otherwise the API may end up supporting stuff that no one really wants/needs. That is why I asked for an example file, it's a specific use case :)

As a side note: in your specific case I think you can do it without any hook at all. You actually already have all the data you need in that json. Below is just some test code - it probably doesn't work as is (just trying to show what I mean).

let gltf = resources["markertest.gltf"].gltf
let extensions = gltf.descriptor.animations[0].extensions
let markers = extensions.IWG_pose_markers?.markers

class MyCustomAnimation extends PIXI3D.Animation {
  constructor(gltfAnimation, markers) {
    super()
    this._gltfAnimation = gltfAnimation
  }

  update(delta) {
    super.update(delta)
    // custom markers logic goes here
    this._gltfAnimation.update(delta)
  }
}

let model = PIXI3D.Model.from(gltf)

let animation = new MyCustomAnimation(model.animations[0], markers)
animation.play()

[lunarraid]

This is interesting, since I was looking at it from the angle of the model creation not knowing about the extra extension properties, but looking at the raw gltf descriptor to decorate the model is an option. I'm still not huge on having to create a custom animation decorator class that has an undecorated animation as an internal property, but this is ironically close to a solution I was already looking into. I do like the idea of exposing a factory method in the model creation that allows me to read the extra animation data and export my own class instance that implements the Animation interface rather than having to construct a list of my own custom animations outside of the model.