LineSegments2: Fix raytracing when the geometry has instanceCount set.

[MixMasterMitch]

Description

The LineSegments2 mesh has support for raytracing. However, all segments are evaluated when checking for intersections, even if the geometry's instanceCount has been set to a lower value than the total segment count. Therefore, intersections are still detected with the non-visible line segments. This PR fixes this by considering the instanceCount when setting the iteration count during ray tracing.

The bounding box/sphere methods still do not consider the instanceCount, but this is a more challenging problem to solve correctly and does not affect the correctness of the ray tracing results.

The behavior before and after can be reproduced by applying a transformation in the Line2 ray tracing example (/examples/#webgl_lines_fat_raycasting). Here is an example patch that can be used:

diff --git a/examples/webgl_lines_fat_raycasting.html b/examples/webgl_lines_fat_raycasting.html
index 232623c744..a830a533a8 100644
--- a/examples/webgl_lines_fat_raycasting.html
+++ b/examples/webgl_lines_fat_raycasting.html
@@ -162,6 +162,7 @@
 				segments.scale.set( 1, 1, 1 );
 				scene.add( segments );
 				segments.visible = false;
+				segmentsGeometry.instanceCount = 50;

 				thresholdSegments = new LineSegments2( segmentsGeometry, matThresholdLine );
 				thresholdSegments.computeLineDistances();

[Mugen87]

It seems there is now a conflict after merging #24405. Do you mind updating the PR?

[MixMasterMitch]

It seems there is now a conflict after merging #24405. Do you mind updating the PR?

Done @Mugen87

[MixMasterMitch]

@Mugen87 Can this change be merged now?

[Mugen87]

https://raw.githack.com/MixMasterMitch/three.js/line-segments-2-instance-count-raycast-fix/examples/webgl_lines_fat_raycasting.html

[Mugen87]

The change looks good to me!

[Mugen87]

The bounding box/sphere methods still do not consider the instanceCount, but this is a more challenging problem to solve correctly and does not affect the correctness of the ray tracing results.

It has been discussed in #8690 whether bounding volumes should honor certain geometry properties or not. It seems BufferGeometry.groups should not be supported but there is no decision so far about drawRange (or related properties which determine what part of the geometry is going to be rendered).

If we add no support for drawRange, I doubt instanceCount is going to be supported, too.