Allow >4 bones to skin a vertex

[cstegel]

Description

Currently, ThreeJS limits how many bones can skin a vertex to at most 4. When a loader sees more than 4 it arbitrarily removes some of them and emits a warning. This results in broken rigs.

Some file formats and mesh/rig authoring tools allow vertices to be skinned by more than 4 bones. I have some meshes with this need so I would like to add support to ThreeJS for more than 4 bone weights per vertex.

I have started modifying ThreeJS to add this support and was wondering if it would likely be accepted?

How bone skinning currently works

Vertex shaders receive per-vertex skinning weights + indexes:

attribute vec4 skinIndex;
attribute vec4 skinWeight;

skinIndex is used to retrieve the bone matrices and then skinWeight weights them to get the final vertex.

The skinIndex and skinWeight vertex buffers are directly created by loaders at load time and attached to BufferGeometry.

How I propose supporting >4 weights per vertex

Instead of sending bone indexes and weights to the vertex shader as vertex buffer data, it would be uploaded as a data texture and a single int per-vertex for the starting point in the bone-index-weight-pair texture.

attribute int bonePairTexStartIndex;
uniform sampler2D boneIndexWeightPairs;

The shading code would read from the bone pair texture until it saw an index of -1 or the maximum number of vertices, MAX_BONES_PER_VERT, is reached:

        #define MAX_BONES_PER_VERT 32;

        ...
        vec4 skinVertex = bindMatrix * vec4( transformed, 1.0 );
	vec4 skinned = vec4( 0.0 );
	vec4 skinnedNormal = vec4( 0.0 );
        for (int i = 0; i < MAX_BONES_PER_VERT; i = i + 1) {
		int bonePairTexIndex = bonePairTexStartIndex + i;
		vec2 boneIndexWeight = 
			texelFetch(boneIndexWeightPairs, 
			           ivec2(bonePairTexIndex % boneTexWidth,
			           bonePairTexIndex / boneTexWidth),
			           0).xy;
		int boneIndex = int(boneIndexWeight.x);
		if (boneIndex < 0) break;
		mat4 boneMatrix = getBoneMatrix(boneIndex);
		skinned += (boneMatrix * skinVertex).xyz * boneIndexWeight.y;
		skinnedNormal += 
			normalize(mat3(bindMatrixInverse)
			        * mat3(boneMatrix)
			        * mat3(bindMatrix) * objectNormal)
			* boneIndexWeight.y;
	}

Instead of loading skinning index/weight data directly to a vertex buffer on BufferGeometry, loaders should create a data texture if a flag tells it to. It could also detect if any vertex has more than 4 bone weights and make this decision dynamically but it's easier to keep it as a configuration flag. A new texture on SkinnedMesh seems like the right place for this to live.

This change in the shaders would be controlled by a new #define so that the existing behavior is maintained when the extra bones aren't needed.

Alternatives

More shader attributes can be added to support additional bone weights but there's a limit on the amount of vertex data. MAX_VERTEX_ATTRIBS for WebGL is 16. The more attributes are used, the less likely a browser is to support the shader. Putting this data in a texture doesn't run into a similar limit although it might make performance worse.

Additional context

Unity supports unlimited blend weights as a parameter: https://docs.unity3d.com/2019.2/Documentation/Manual/class-QualitySettings.html#BlendWeights

Someone previously had an example of FBX deleting weights when there were more than 4 per vertex: https://discourse.threejs.org/t/is-the-limit-of-4-skinning-weights-per-vertex-a-hard-limit-of-webgl/801

Their related Github issue: #12127

[RemusMar]

Currently, ThreeJS limits how many bones can skin a vertex to at most 4. When a loader sees more than 4 it arbitrarily removes some of them and emits a warning. This results in broken rigs.

Well, that's not quite true.
Yes, most (if not all) of the WebGL based engines have this limit (ThreeJS, BabylonJS, PlayCanvas, Unity3D, etc).
The main reason is that if you have high detailed skinned meshes (over 50,000 polys) and a bunch of bones you'll need a HUGE processing power to animate them if you raise that limit to 6 or 8 for example.
On the other hand, the exporters/loaders do NOT arbitrarily remove anything.
They find the first 4 bones with the highest weights and then normalize those weights.
That gives the best result in >90% of the cases.
But yes, sometimes the 5th and the 6th bone has the same weight as the 4th one and you'll end up with an altered animation.
Anyway, I'm not saying that raising that limit to 6 or 8 is a bad idea, but I'm not sure it's a GOOD idea.
At least for now ...

[mrdoob]

I have started modifying ThreeJS to add this support and was wondering if it would likely be accepted?

Would be great if you can submit a PR with the changes and include an example with a model that uses more than 4 bones. Then we'll be able to measure performance and consider the maintenance costs.

[cstegel]

@RemusMar Thanks for elaborating on how the loaders pick which bones to remove. I'm not that familiar with the loader code yet so I wasn't fully aware of how it decided to remove them. I agree that 4 bones per vertex is usually enough and more is a special case. I'm also aware that this will be less performant, but to @mrdoob's comment, we don't know how much of a difference this is yet. Unity indicates support for it but I haven't tried it.

I want to make this a toggle in the shader so that the current shader setup is the default. If someone desires more than 4 bones per vertex then it would switch the shader implementation. This way, the performance difference is only seen when the feature is enabled.

I'll comment here once I having a working PR for this.

[RemusMar]

I agree that 4 bones per vertex is usually enough and more is a special case.

Hi Cory,
I don't know about Blender, but in 3DS Max (and Maya) there is a very useful setting for the Skin Modifier:
Bone Affect Limit.
By default, that value is 20 (not suitable for WebGL and browser based apps).
Before exporting (to GLTF) all you have to do is to set that value to 4 and 3DS Max will recompute and normalize everything before exporting.
So the exporters and loaders don't have to do anything here.

[cstegel]

Hi Remus,

That's good to know that authoring tools should have controls to limit the bone influence but unfortunately there are still some cases where the company I work at wants to use models with >4 bones per vertex. I've advocated for keeping things under 4 and this will be the majority of models, but there are still some cases where they want higher.

For some context, the company was previously using a custom WebGL-based renderer with the same shader approach to >4 bone influences as I've described. That renderer is being abandoned in favor of using Unity (not running on web). In some environments, Unity is undesirable so a 2nd rendering system is needed. Three.js fits well but the one feature missing is the ability to handle a larger number of bone weights.

[cstegel]

I found out that Babylon.js has support for >4 bone influences when I tried previewing a .glb with >4 bone influences in various renderers using VS Code's glTF Tools extension.

In the animations below, notice the jagged artifacts above the upper lip in Three.js that are not present in Babylon.js. This model was exported with max 16 bone influences although I don't know what the real max number was. When I use a copy of the mesh that was exported with max 4 bone influences, the animation has the same artifacts regardless of the renderer.

Babylon.js (5.6.1)	Three.js (r140)

Babylon's docs don't make this clear, but it seems to do this by using an extra vertex buffer for weights and indexes that holds 4 additional influences

Babylon's docs for their mesh class don't mention a max number for mesh bone influences other than it defaults to 4. There's also a seemingly-outdated doc about not supporting more than 4.

[RemusMar]

Cory,

babylon.js supports up to 4 bones influences per vertex.
It's using this formula:
Matrices weights: 4 floats to weight bones matrices
Matrices indices: 4 floats to index bones matrices
finalMatrix = worldMatrix * (bonesMatrices[index0] * weight0 + bonesMatrices[index1] * weight1 + bonesMatrices[index2] * weight2 + bonesMatrices[index3] * weight3);

In your 1st GIF, the first 4 weights are normalized and in the 2nd one they are not.
That's why the result in the 2nd one is wrong.
I get the feeling the GLTF loader used in Three.js is buggy.

[cstegel]

Hi Remus,

Thanks for your response! I dug deeper into the Three.js's GLTFLoader class and the Babylon.js codebase to see what's going on.

GLTFLoader is normalizing the first weights attribute buffer (the first 4 weights). Currently, only the glTF WEIGHTS_0 attribute is used as the Three.js skinWeights vertex buffer.

I inspected the ordering of weights in the glTF WEIGHTS_n buffers across each vertex of the skinned mesh I've exported from Blender. It orders the weights from highest to lowest influence.

For example, here are the 16 weights for an individual vertex in my mesh across the 4 glTF buffers:

x: 0.2424, y: 0.1997, z: 0.1924, w: 0.05653   // WEIGHTS_0 @ v = 2345
x: 0.0563, y: 0.0545, z: 0.0309, w: 0.02831   // WEIGHTS_1 @ v = 2345
x: 0.0281, y: 0.0224, z: 0.0214, w: 0.02091   // WEIGHTS_2 @ v = 2345
x: 0.0202, y: 0.0155, z: 0.0098, w: 0         // WEIGHTS_3 @ v = 2345

Notice that it is ordered by descending weight and if only the first 4 are used (and renormalized to 100%) then 30% of the original influence is gone. This is what creates the artifacts.

Since this mesh's weights are ordered from highest to lowest, GLTFLoader's current behavior of only normalizing the first 4 weights is CORRECT, yet it produces the artifacts shown in my previous comment. This is NOT due to a lack of normalization.

The code that I've seen in Babylon.js (1, 2, 3) indicates they are optionally using 1 extra vertex buffer to support up to 8 weights instead of 4. This explains why the animation artifacts are not easily visible when it loads my model exported for 16 weights. For the example vertex above, it would only lose 14% of the original influence instead of 30%. Most of the vertices don't have this much of a difference so the final artifacts in Babylon.js for this mesh/animation are minor.

When I have Babylon.js load a model that I've exported from Blender which limits the weights to 4 (a single buffer), it has the same artifacts as Three.js. Again, this shows that the artifacts are due to missing bone weights and not a lack of normalization.

Finally, I've implemented the skinning-weight-texture approach that I started this issue with and the artifacts completely disappear. See the new animations below:

Three.js single-buffer vertex skinning (current behavior)	Three.js skinning texture (WIP changes, all 16 bone weights)

I am currently working on an example which shows the difference in behavior and cleaning up the code for review.

Regards,
Cory

[RemusMar]

Notice that it is ordered by descending weight and if only the first 4 are used (and renormalized to 100%) then 30% of the original influence is gone. This is what creates the artifacts.

Cory,
0.7 for 4 bones and 0.3 lost is way too much!
Are you sure the skinning was done properly for that 3D model?

[cstegel]

Hi Remus,

I don't have much experience with creating rigs to know if the model could be skinned differently with less than four bones while still achieving the same quality of animation. This model was given to me by an artist as a sharable example to show the kind of artifacts that come up when bones are limited. The impression I got from them was that the 4 bone limit is always challenging and limits the expressiveness of the models they create. They frequently encounter this limit when creating highly expressive models.

This can be "fixed" for models by spending more artist time on rearranging and re-weighting bones, but some models will still have a trade off between expressiveness or < 4 bone influences. The choice depends on the use-case. For example, video games might want more expressiveness during cutscenes but otherwise limit to 4 bones during gameplay that has hundreds of models on screen. A V-tuber tracked-avatar system that renders a single model would want higher expressiveness.

Regards,
Cory

[cstegel]

I've created a draft PR for this now: #26222

[cstegel]

PR is no longer a draft. I am now waiting on reviews.