WebGPU: NodeMaterial BSDFs, revision and updates #21322

sunag · 2021-02-21T03:50:25Z

WebGPU - Selective Lights
https://raw.githack.com/sunag/three.js/nodematerial-light/examples/webgpu_lights_selective.html

Suggested syntax for selective lights:

const lightNodeA = new LightNode( lightA );
const lightNodeB = new LightNode( lightB );

// from nodes
material.lightNode = new LightsNode( [ lightNodeA, lightNodeB ] );

// from lights
material.lightNode = LightsNode.fromLights( [ lightA, lightB ] );

examples/jsm/renderers/nodes/accessors/Object3DNode.js

examples/jsm/renderers/nodes/functions/BSDFunctions.js

Mugen87 · 2021-02-28T16:56:47Z

examples/jsm/renderers/nodes/lights/LightContextNode.js

+
+		let RE_Direct = null;
+
+		if ( material.isMeshPhongMaterial === true ) {


Any reasons why you have decided for MeshPhongMaterial instead of MeshStandardMaterial? When I remember correctly, it was considered once to not support phong in WebGPU anymore.

@mrdoob Do you have a preference here? I personally would start with a PBR material as a first lit material. Although phong would be easier to implement/port. And we would need to support it for WebGL anyway.

Any reasons why you have decided for MeshPhongMaterial instead of MeshStandardMaterial?

It is a popular material and more easier to implement... but I can start with MeshStandardMaterial I see no problem about it.

Let's wait for @mrdoob's feedback. TBH, I'm not sure anymore if there was a conclusion about this topic and how it looked like.

Mugen87 · 2021-03-02T14:36:24Z

One question about LightNode. The code from your first post shows how selective lighting can be implemented.

But it is still the plan that materials will still respond to lights added to the scene graph, right? I mean this is obviously not yet implemented since the renderer needs to be enhanced. But I'd like to clarify this roadmap a bit.

Mugen87 · 2021-03-02T14:39:44Z

examples/jsm/renderers/nodes/lights/LightContextNode.js

+
+		if ( material.isMeshPhongMaterial === true ) {
+
+			RE_Direct = RE_Direct_BlinnPhong;


I have not yet debugged this part in detail but I assume the define RE_Direct in the phong shader will be set to the generated code of RE_Direct_BlinnPhong?

Yes, but RE_Direct is a FunctionCallNode. I am preserving the names to facilitate. NodeMaterial not need set a define for this. With LightContextNode and no defines we can adopt even two light model, like Phong and Physical runing simultaneously on the same material. - This is just adding a few nodes.

sunag · 2021-03-03T01:32:26Z

But it is still the plan that materials will still respond to lights added to the scene graph, right?

This should be very easy to implement, e.g:

// WebGPUNodeBuilder.js

// cache light context and share with all materials of the scene
const sceneLightNodeSlot = new NodeSlot( new LightContextNode( LightsNode.fromLights( sceneLightsArray ) ), 'LIGHT', 'vec3' );

_parseMaterial() {

	...

	if ( material.lightNode !== undefined ) {
		
		// selective light
		
		const materialLightContextNode = new LightContextNode( material.lightNode );
		
		this.addSlot( 'fragment', new NodeSlot( materialLightContextNode, 'LIGHT', 'vec3' ) );
		
	} else {
	
		// scene lights
	
		this.addSlot( 'fragment', sceneLightNodeSlot );
	
	}
	
}

sunag · 2021-03-06T04:20:36Z

I would like your option about keywords, what standard should we use:
@mrdoob @Mugen87

NormalLocal, NormalView, MaterialDiffuseColor, ReflectedLightDirectDiffuse
NORMAL_LOCAL, NORMAL_VIEW, MATERIAL_DIFFUSE_COLOR, REFLECTED_LIGHT_DIRECT_DIFFUSE
Suggestions?

1 is the current.

--

void RE_Direct_BlinnPhong( vec3 lightDirection, vec3 lightColor ) {

	float dotNL = saturate( dot( NormalView, lightDirection ) );
	vec3 irradiance = dotNL * lightColor;

#ifndef PHYSICALLY_CORRECT_LIGHTS

		irradiance *= PI; // punctual light

#endif

	ReflectedLightDirectDiffuse += irradiance * BRDF_Diffuse_Lambert( MaterialDiffuseColor.rgb );

}

void RE_Direct_BlinnPhong( vec3 lightDirection, vec3 lightColor ) {

	float dotNL = saturate( dot( NORMAL_VIEW, lightDirection ) );
	vec3 irradiance = dotNL * lightColor;

#ifndef PHYSICALLY_CORRECT_LIGHTS

		irradiance *= PI; // punctual light

#endif

	REFLECTED_LIGHT_DIRECT_DIFFUSE += irradiance * BRDF_Diffuse_Lambert( MATERIAL_DIFFUSE_COLOR.rgb );

}

mrdoob · 2021-03-06T07:05:11Z

I think I prefer NormalLocal otherwise it gets confused with defines like PHYSICALLY_CORRECT_LIGHTS.

sunag · 2021-03-06T10:32:40Z

Another thing... It is possible to maintain compatibility with a non nodes properties values like material.color in case of user not use a NodeMaterial property like material.colorNode. We are looking for this type of compatibility?

sunag · 2021-03-06T10:35:53Z

For example, in case of material.colorNode is undefined a material.colorNode = new ColorNode( material.color ) is added instead in WebGPUNodeBuilder process.

sunag · 2021-03-06T10:45:09Z

I create MaterialNode for this ( 07fdf79 ) but I don't know if I should keep it or use only fields intended for nodes.

Example of MaterialNode

 // added by user or WebGPUNodeBuilder if material.colorNode is empty
material.colorNode = new MaterialNode( MaterialNode.COLOR );

// user can change the values of the material in legacy mode (THREE.Color) 
// and MaterialNode will auto update the relative nodes
material.color = new THREE.Color( 0xFF00FF );

sunag · 2021-03-06T10:49:25Z

If that for interesting I would put other fields in MaterialNode like:
MaterialNode.MAP, MaterialNode.NORMAL_MAP, MaterialNode.EMISSIVE, MaterialNode.ROUGHNESS...

sunag · 2021-05-22T03:26:26Z

SkinningPositionNode missing normals

sunag · 2021-05-22T03:33:14Z

Seems that Uint16BufferAttribute still not work in WebGPU?

FBXLoader.js test:

if ( skeleton ) {

	geo.setAttribute( 'skinIndex', new Float32BufferAttribute( buffers.weightsIndices, 4 ) ); // work
	//geo.setAttribute( 'skinIndex', new Uint16BufferAttribute( buffers.weightsIndices, 4 ) ); // not work

	geo.setAttribute( 'skinWeight', new Float32BufferAttribute( buffers.vertexWeights, 4 ) );

	// used later to bind the skeleton to the model
	geo.FBX_Deformer = skeleton;

}

Mugen87 · 2021-05-22T11:45:16Z

Seems that Uint16BufferAttribute still not work in WebGPU?

Should be solved via Mugen87@006b2fc. WebGPURenderPipeline did not process all attribute types yet.

Mugen87 · 2021-05-22T11:56:31Z

@sunag Awesome work!

SkinningPositionNode missing normals

Um, do we need a SkinningNormalNode? The advantage of having a single skinning node class is the easier update process of the skeleton.

BTW: By doing this mvpNode.position = new SkinningPositionNode( object );, skinning won't work if the user defines a positionNode (maybe for some custom vertex displacement). I guess the user has to create an instance of SkinningPositionNode then and apply its own custom position node to SkinningPositionNode.position, right?

sunag · 2021-05-22T18:53:00Z

This is going to need a review. I think that the way forward would be to generalize to an super class for position/normal manipulation, like LightsNode is for lights.

At the moment I think about it for the next step:

const boneMatrixStructNode = new BoneMatrixStructNode( index, boneTexture, boneTextureSize );

position = new SkinningPositionNode( position, boneMatrixStructNode, weight, bindMatrix, bindMatrixInverse );
normal = new SkinningNormalNode( normal, boneMatrixStructNode, weight, bindMatrix, bindMatrixInverse );

Mugen87 · 2021-05-23T09:24:03Z

Where do we pass the instance of SkinnedMesh in this approach? It seems at least a single node class should hold it so it's possible to update the skeleton in the update() method.

I would also suggest to pass in the entire skinned mesh/node to SkinningPositionNode and SkinningNormalNode to simplfy the ctor signature. Except for the position and normal node, all other parameters are properties of the skinned mesh. How about this?

const skinningNode = new SkinningNode( skinnedMesh );

const positionNode = new SkinningPositionNode( position, skinningNode );
const normalNode = new SkinningNormalNode( normal, skinningNode );

Mugen87 · 2021-05-23T12:49:12Z

BTW: I think this is a good opportunity to find out how to use the node system to implement renderer features like skinning less hard-wired^^.

sunag · 2021-05-23T18:57:35Z

We just need one more intermediate vertex class for position/normal statics, I did not get to add that yet, so that gap remained.

Mugen87 · 2021-05-23T19:02:23Z

Okay, then let's continue with the skinning topic when this new class is ready.

sunag · 2021-08-24T19:07:53Z

@Mugen87 Is possible you merger your skinning PR so I can make some updates?

Mugen87 · 2021-08-24T20:25:42Z

The PR is quite outdated and it's probably better to make a new branch based on latest dev. I'm trying to prepare a PR tomorrow.

For stability reasons, it's probably better to merge it after r132 has be published anyway (which is planned for tomorrow^^).

sunag · 2021-08-24T21:11:27Z

For stability reasons, it's probably better to merge it after r132 has be published anyway (which is planned for tomorrow^^).

No problem we can wait after r132 has be published. There is a lot of thing to do here too.

Mugen87 · 2021-08-25T09:04:49Z

I've realized today that the skinning branch is broken and does not work with latest dev. Here is the rebased branch:

Mugen87@a00b186

It's probably better if you just copy these changes to your local branch and then apply the respective updates.

sunag · 2021-08-25T13:15:13Z

Thank you very much! Moreover, bring Skinning Mesh to WebGPU was a great initiative.

sunag · 2021-09-07T16:40:41Z

@Mugen87 I am having several problems using pointer with textures in GLSL to SPIRV.
I dont know when stopped working on WebGPU, but I havent been discovering the origin for a fews days.

layout(set = 0, binding = 0) uniform sampler nodeUniform2_sampler; 
layout(set = 0, binding = 1) uniform texture2D nodeUniform2;

vec3 nodeCode0 (  texture2D A, sampler B  )  {

	return vec3( 0.0 );

}

void main() {

	// Tint SPIRV reader failure:
	// Parser: error: function parameter of pointer type cannot be in 'none' storage class
	nodeCode0( nodeUniform2, nodeUniform2_sampler  )

}

Full code

#version 450

// <node_builder>

#define NODE_MATERIAL

// defines
#define NODE_CODE nodeVary0 = uv; 
#define NODE_CODE_MVP nodeVar0 = ( nodeUniforms.nodeUniform0 * nodeUniforms.nodeUniform1 ); PositionLocal = position; nodeVar1 = nodeCode0( nodeUniform2, nodeUniform2_sampler ); 
#define NODE_MVP ( nodeVar0 * vec4( nodeVar1, 1.0 ) )

// uniforms
layout(set = 0, binding = 0) uniform sampler nodeUniform2_sampler; layout(set = 0, binding = 1) uniform texture2D nodeUniform2; layout(set = 0, binding = 2) uniform NodeUniforms { uniform mat4 nodeUniform0; uniform mat4 nodeUniform1;  } nodeUniforms; 

// attributes
layout(location = 0) in vec3 position; layout(location = 1) in vec2 uv; 

// varys
layout(location = 0) out vec2 nodeVary0; 

// vars
mat4 nodeVar0; vec3 nodeVar1; vec3 PositionLocal; 

// codes
vec3 nodeCode0 (  texture2D A, sampler B  )  {

	return PositionLocal;

}

// </node_builder>

void main(){

	NODE_CODE

	NODE_CODE_MVP

	gl_Position = NODE_MVP;

}

sunag · 2021-09-07T16:52:56Z

Another example:

layout(set = 0, binding = 0) uniform sampler nodeUniform2_sampler; 
layout(set = 0, binding = 1) uniform texture2D nodeUniform2;

vec3 nodeCode0 (  texture2D A, sampler B  )  {

	// Tint SPIRV reader failure:
	// Parser: error: no matching call to textureSampleLevel(ptr<void, read_write>, ptr<void, read_write>, vec2<f32>, f32)
	return texture( sampler2D( A, B ), vec2( 0.0 ) ).xyz;

}

void main() {

	nodeCode0( nodeUniform2, nodeUniform2_sampler  )

Mugen87 · 2021-09-07T16:59:36Z

That is strange. I'll try to have a look at this tomorrow.

In the meanwhile, you might want to ask for advice directly at the Dawn or WebGPU chatrooms.

https://matrix.to/#/#webgpu-dawn:matrix.org
https://matrix.to/#/#WebGPU:matrix.org

For this issue, the Dawn chatroom seems more right. The second links is more about WebGPU standard related questions.

sunag · 2021-09-07T17:07:40Z

Thanks! I published a TextureNode PR needed for test this issue using NodeMaterial: #22501

sunag · 2021-09-08T04:59:35Z

It worked ( texture pointer ) at that time: #21322 (comment)

Mugen87 · 2021-09-08T11:14:03Z

Now I see the warning, too:

1 error(s) generated while compiling the shader:
error: no matching call to textureSampleLevel(ptr<void, read_write>, ptr<void, read_write>, vec2, f32)
15 candidate functions:
textureSampleLevel(texture: texture_2d, sampler: sampler, coords: vec2, level: f32) -> vec4
textureSampleLevel(texture: texture_2d, sampler: sampler, coords: vec2, level: f32, offset: vec2) -> vec4
textureSampleLevel(texture: texture_3d, sampler: sampler, coords: vec3, level: f32) -> vec4
textureSampleLevel(texture: texture_cube, sampler: sampler, coords: vec3, level: f32) -> vec4
textureSampleLevel(texture: texture_depth_2d, sampler: sampler, coords: vec2, level: i32) -> f32
textureSampleLevel(texture: texture_2d_array, sampler: sampler, coords: vec2, array_index: i32, level: f32) -> vec4
textureSampleLevel(texture: texture_3d, sampler: sampler, coords: vec3, level: f32, offset: vec3) -> vec4
textureSampleLevel(texture: texture_depth_2d, sampler: sampler, coords: vec2, level: i32, offset: vec2) -> f32
textureSampleLevel(texture: texture_depth_2d_array, sampler: sampler, coords: vec2, array_index: i32, level: i32) -> f32
textureSampleLevel(texture: texture_external, sampler: sampler, coords: vec2) -> vec4
textureSampleLevel(texture: texture_2d_array, sampler: sampler, coords: vec2, array_index: i32, level: f32, offset: vec2) -> vec4
textureSampleLevel(texture: texture_depth_2d_array, sampler: sampler, coords: vec2, array_index: i32, level: i32, offset: vec2) -> f32
textureSampleLevel(texture: texture_depth_cube, sampler: sampler, coords: vec3, level: i32) -> f32
textureSampleLevel(texture: texture_cube_array, sampler: sampler, coords: vec3, array_index: i32, level: f32) -> vec4
textureSampleLevel(texture: texture_depth_cube_array, sampler: sampler, coords: vec3, array_index: i32, level: i32) -> f32

The strange thing is that the texture sampling does not look different compared to the existing samplings in other examples (e.g. in webgpu_sandbox). We always use this pattern:

 texture( sampler2D( texture, sampler ), coords ) )

The only difference is that the skinning code samples in the vertex shader.

sunag · 2021-09-08T16:12:21Z

Strange that this pattern work if used without function argument in both shader stage.

sunag · 2021-09-28T13:09:38Z

I have just concluded some final steps of SkinnedMesh using VBO in WebGPU. I had to do many approaches to create a good integration and I think that use VBO instead of texture is better for performance too. Still missing, I finish the BufferNode API to publish the PR.

sunag · 2021-09-28T15:56:13Z

I still have many issues with WebGPU, this one I just noticed::

// 60fps
vec3 getSkinningPosition( ) {
	mat4 boneMatX = bones[ int( index.x ) ];
	...
}

// 5~ fps
vec3 getSkinningPosition( in mat4[ 52 ] bones ) {
	mat4 boneMatX = bones[ int( index.x ) ];
	...
}

mrdoob · 2021-09-28T16:20:49Z

/cc @Kangz

Kangz · 2021-09-28T16:35:35Z

Uh, this must be because the underlying code is doing a copy of the 52 mat4s when calling the function. You'd be able to more precisly control the costs of what's happening by using WGSL directly. If you want to see what code is produced you can run chrome with the additional --enable-dawn-features=dump_shaders and maybe --enable-dawn-features=dump_shaders,force_wgsl_step (if you're giving SPIR-V directly to Chromium).

Note that the SPIR-V input to createShaderModule is only in unsafe mode and not standard. WebGPU is only specified to ingest WGSL so it would be best for the NodeMaterial to output that if possible.

sunag · 2021-09-28T16:46:30Z

@Kangz Thanks! This makes a lot of sense and I was already suspicous that other issues I had like this ( #21322 (comment) ) one is related too with GLSL to SPIR-V compiler

sunag added 2 commits February 21, 2021 00:34

NodeMaterial BSDFs, revision and updates (WIP)

390d1b3

fix title

57de17c

mrdoob added this to the r127 milestone Feb 22, 2021

sunag added 3 commits February 28, 2021 01:29

add Math.LENGTH

464259a

ContextNode, VarNode, NodeKeywords, LightNode (WIP) and updates

9b25a44

a bit cleanup

a4e2a0e

Mugen87 reviewed Feb 28, 2021

View reviewed changes

examples/jsm/renderers/nodes/accessors/Object3DNode.js Outdated Show resolved Hide resolved

Mugen87 reviewed Feb 28, 2021

View reviewed changes

examples/jsm/renderers/nodes/accessors/Object3DNode.js Outdated Show resolved Hide resolved

Mugen87 reviewed Feb 28, 2021

View reviewed changes

examples/jsm/renderers/nodes/functions/BSDFunctions.js Outdated Show resolved Hide resolved

Mugen87 reviewed Feb 28, 2021

View reviewed changes

sunag added 6 commits March 1, 2021 04:35

modify order

80ef161

PI AND RECIPROCAL_PI keywords

8513a58

Rename BSDFunctions.js -> BSDFs.js

eec296c

optimization: not recompute the world matrix

8a3af04

use context material if need

632af07

PropertyNode, texture test (WIP)

fdd2dbf

Mugen87 reviewed Mar 2, 2021

View reviewed changes

sunag added 4 commits March 6, 2021 02:04

Flow code support, LightsNode WIP

02688ba

multiples lights test, decay and distance

c72628c

fix light.intensity

d80718e

test light.intensity

85e7275

MaterialNode

07fdf79

Mugen87 mentioned this pull request May 23, 2021

WebGPURenderPipeline: Support more vertex formats. #21879

Merged

sigmaxipi mentioned this pull request Jul 15, 2021

Some examples are broken on Firefox due to module importing #22137

Closed

sunag deleted the nodematerial-light branch September 28, 2021 09:17

sunag mentioned this pull request Sep 29, 2021

NodeMaterial: ShaderNode #22603

Merged


		let RE_Direct = null;

		if ( material.isMeshPhongMaterial === true ) {


		if ( material.isMeshPhongMaterial === true ) {

		RE_Direct = RE_Direct_BlinnPhong;

WebGPU: NodeMaterial BSDFs, revision and updates #21322

WebGPU: NodeMaterial BSDFs, revision and updates #21322

Conversation

sunag commented Feb 21, 2021 • edited Loading

Mugen87 Feb 28, 2021 • edited Loading

Choose a reason for hiding this comment

sunag Feb 28, 2021 • edited Loading

Choose a reason for hiding this comment

Mugen87 Feb 28, 2021

Choose a reason for hiding this comment

Mugen87 commented Mar 2, 2021 • edited Loading

Mugen87 Mar 2, 2021 • edited Loading

Choose a reason for hiding this comment

sunag Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

sunag commented Mar 3, 2021 • edited Loading

sunag commented Mar 6, 2021

mrdoob commented Mar 6, 2021

sunag commented Mar 6, 2021

sunag commented Mar 6, 2021

sunag commented Mar 6, 2021

sunag commented Mar 6, 2021

sunag commented May 22, 2021 • edited Loading

sunag commented May 22, 2021

Mugen87 commented May 22, 2021 • edited Loading

Mugen87 commented May 22, 2021 • edited Loading

sunag commented May 22, 2021 • edited Loading

Mugen87 commented May 23, 2021 • edited Loading

Mugen87 commented May 23, 2021 • edited Loading

sunag commented May 23, 2021

Mugen87 commented May 23, 2021

sunag commented Aug 24, 2021

Mugen87 commented Aug 24, 2021

sunag commented Aug 24, 2021

Mugen87 commented Aug 25, 2021 • edited Loading

sunag commented Aug 25, 2021

sunag commented Sep 7, 2021

Full code

sunag commented Sep 7, 2021

Mugen87 commented Sep 7, 2021 • edited Loading

sunag commented Sep 7, 2021

sunag commented Sep 8, 2021

Mugen87 commented Sep 8, 2021 • edited Loading

sunag commented Sep 8, 2021

sunag commented Sep 28, 2021

sunag commented Sep 28, 2021 • edited Loading

mrdoob commented Sep 28, 2021

Kangz commented Sep 28, 2021

sunag commented Sep 28, 2021 • edited Loading

sunag commented Feb 21, 2021 •

edited

Loading

Mugen87 Feb 28, 2021 •

edited

Loading

sunag Feb 28, 2021 •

edited

Loading

Mugen87 commented Mar 2, 2021 •

edited

Loading

Mugen87 Mar 2, 2021 •

edited

Loading

sunag Mar 3, 2021 •

edited

Loading

sunag commented Mar 3, 2021 •

edited

Loading

sunag commented May 22, 2021 •

edited

Loading

Mugen87 commented May 22, 2021 •

edited

Loading

Mugen87 commented May 22, 2021 •

edited

Loading

sunag commented May 22, 2021 •

edited

Loading

Mugen87 commented May 23, 2021 •

edited

Loading

Mugen87 commented May 23, 2021 •

edited

Loading

Mugen87 commented Aug 25, 2021 •

edited

Loading

Mugen87 commented Sep 7, 2021 •

edited

Loading

Mugen87 commented Sep 8, 2021 •

edited

Loading

sunag commented Sep 28, 2021 •

edited

Loading

sunag commented Sep 28, 2021 •

edited

Loading