topn with granularity regression fixes #17565
Conversation
changes: * fix issue where topN with query granularity other than ALL would use the heap algorithm when it was actual able to use the pooled algorithm, and incorrectly used the pool algorithm in cases where it must use the heap algorithm, a regression from apache#16533 * fix issue where topN with query granularity other than ALL could incorrectly process values in the wrong time bucket, another regression from apache#16533
| @@ -275,7 +275,7 @@ private static boolean canUsePooledAlgorithm( | |||
| final int numBytesToWorkWith = resultsBuf.capacity(); | |||
| final int numValuesPerPass = numBytesPerRecord > 0 ? numBytesToWorkWith / numBytesPerRecord : cardinality; | |||
|
|
|||
| return numValuesPerPass <= cardinality; | |||
| return numValuesPerPass >= cardinality; | |||
There was a problem hiding this comment.
this comparision change caught my eye...the new cmp seem to be more in line with the intention - but it seems to me that
depending on some conditions here ; this cardinality's value might be -1 in some cases
is that ok to answer true if cardinality is -1 ?
note: if cardinality == -1 ; then both the old and the new logic returns true - is that ok?
There was a problem hiding this comment.
yea this is fair, it is not really a problem in practice because currently the column capabilities should not report as dictionary encoded if the cardinality is -1, so we return false earlier in the method before we get here, but this seems worth explicitly checking for just in case.
| @@ -245,6 +245,11 @@ private static boolean canUsePooledAlgorithm( | |||
| final int numBytesPerRecord | |||
| ) | |||
| { | |||
| if (cardinality < 0) { | |||
| // unknown cardinality doesn't work with the pooled algorith which requires an exact count of dictionary ids | |||
| @@ -391,7 +391,7 @@ public boolean hasNext() | |||
| if (delegate != null && delegate.hasNext()) { | |||
| return true; | |||
| } else { | |||
| if (!cursor.isDone() && granularizer.currentOffsetWithinBucket()) { | |||
| if (granularizer.currentOffsetWithinBucket()) { | |||
There was a problem hiding this comment.
was the cursor.isDone() here removed simply because it's redundant?
There was a problem hiding this comment.
yea, i noticed granularizer.currentOffsetWithinBucket also checks isDone so it seemed nicer this way, this didn't cause any issues, just noticed while i was looking over stuff
| aggregator.init(resultsBuffer, position); | ||
| aggregator.aggregate(resultsBuffer, position); | ||
| positionToAllocate += aggregatorSize; | ||
| if (granularizer.currentOffsetWithinBucket()) { |
There was a problem hiding this comment.
was the granularizer.currentOffsetWithinBucket() check added here (& the other prototypes) to enable skipping of empty granular buckets?
There was a problem hiding this comment.
yea, this was a bug, i did it as an if statement wrapping the loop so we wouldn't need to check currentOffsetWithinBucket() twice per loop since the granularizer advance methods also call currentOffsetWithinBucket
Description
changes:
This PR has: