Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(crc): add crc hash fn #1136

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Conversation

ivor11
Copy link

@ivor11 ivor11 commented Nov 22, 2024

Summary

Add crc hash fn with an algorithm parameter
default algorithm used - CRC_32_ISO_HDLC

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

How did you test this PR?

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on
    our guidelines.
  • No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

  • Our CONTRIBUTING.md is a good starting place.
  • If this PR introduces changes to LICENSE-3rdparty.csv, please
    run dd-rust-license-tool write and commit the changes. More details here.
  • For new VRL functions, please also create a sibling PR in Vector to document the new function.

References

#997

Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @ivor11. This looks good. We will need a PR on Vector to document this new function.

fn parameters(&self) -> &'static [Parameter] {
&[Parameter {
keyword: "value",
kind: kind::ANY,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm I noticed this in other hash function. I don't think ANY is ideal but we can leave it here for consistency.

@ivor11
Copy link
Author

ivor11 commented Nov 30, 2024

Awesome @pront. I have created a PR on vector to document this function
vectordotdev/vector#21924

Copy link
Member

@bruceg bruceg Dec 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many kinds of CRC functions, even many kinds of 32-bit CRC functions. Instead of adding a separate function for each, would it make sense to instead have a single crc function with some "algorithm" parameter like decrypt, encrypt, and hmac use?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. @ivor11 should be trivial to modify this PR to do the above. Let me know if this makes sense.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes total sense @bruceg @pront , will modify the PR to add an algorithm parameter

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @bruceg @pront ,
I've added the algorithm parameter - keeping CRC_32_ISO_HDLC as the default which I suppose is the most popular one

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks. I will take a look at the updated code.

@ivor11 ivor11 changed the title feat(crc32): add crc32 hash fn feat(crc): add crc hash fn Dec 10, 2024
@ivor11 ivor11 requested review from bruceg and pront December 11, 2024 08:11
use crate::value;
use crc::Crc as CrcInstance;

fn crc(value: Value, algorithm: Value) -> Resolved {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 This is one of those (rare) cases where a #[proc_macro] can significantly reduce the boilerplate. We could iterate over valid_algorithms and keep pushing:

 $algo => CrcInstance::<u8>::new(&crc::$algo)
            .checksum(&value)
            .to_string(),

But this is just my observation. Feel free to ignore this comment.

tdef: TypeDef::bytes().infallible(),
}
];
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a test case with an invalid algorithm?

}

fn type_def(&self, state: &state::TypeState) -> TypeDef {
let valid_algorithms = [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move this at the top as a const VALID_ALGORITHMS: &[&str] = &[

"CRC_82_DARC",
];

let mut valid_static_algo = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can simplify this a little bit:

   fn type_def(&self, state: &state::TypeState) -> TypeDef {
        if let Some(algorithm) = self.algorithm.as_ref() {
            if let Some(algorithm) = algorithm.resolve_constant(state) {
                if let Ok(algorithm) = algorithm.try_bytes_utf8_lossy() {
                    let algorithm = algorithm.to_uppercase();
                    if VALID_ALGORITHMS.contains(&algorithm.as_str()) {
                        return TypeDef::bytes().infallible();
                    }
                }
            }
        } else {
            return TypeDef::bytes().infallible();
        }

        TypeDef::bytes().fallible()
    }

(as long as the tests are passing)

@pront
Copy link
Member

pront commented Dec 13, 2024

This also need a changelog to communicate this new feature to the users.

@ivor11
Copy link
Author

ivor11 commented Dec 14, 2024

@pront my bad, I've added the changelog file. Thanks for the thorough review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants