Add unique keyword option to all schemas and schema components

[cosmicBboy]

Is your feature request related to a problem? Please describe.

Currently, the allow_duplicates option enforces uniqueness if False, but it's only available in the Column schema component. Because value uniqueness is a fundamental data quality check and also has implications for the data synthesis strategies for a schema, it would make sense to (i) deprecate allow_duplicates in Columns and replace it with unique and (ii) add the unique option to all schemas and schema components.

Describe the solution you'd like

For Columns and Indexes, this can simply be True or False. However, at the dataframe-level, it would make sense to also accept List[str] representing the columns whose combined values should be unique.

Describe alternatives you've considered
The alternative, as described in #386 (comment), would be to have a built-in Check for uniqueness. This would be fine, except for the fact that it would complicate the data synthesis strategy for this validation check.

[abyz0123]

Hi is this feature still something you would like and be open to a PR for @cosmicBboy ? I've done this manually quite a bit and itd be nice to have built-in :)

At the dataframe level would you allow for a list of lists? Could one assume that the object will always be a pandas object and thus have access to dataframe/series-level attributes?

Any other considerations?

[cosmicBboy]

hi @fkrull8! yes a PR would be very much appreciated 🙂

At the dataframe level would you allow for a list of lists

The list of lists would be in the case of specifying multiple sets of columns that would be considered unique? I think that's a good idea! Feel free to add support for that to your PR.

We'll probably want to keep the allow_duplicates option in the Column, SeriesSchema, and Index classes for backwards compatability and forward the value to the unique property, raising a DeprecationWarning to let users know.

Re: dataframe-level unique option, the best place to start looking is probably around here after type coercian and right before the schema component checks. I think we'll also want to support lazy validation in this case too, see the parts of the validation routine that use error_handler to achieve this.

If you haven't already check out the contributing page, and let me know if you have any questions!

[schlich]

Would love to see this feature!!! Definitely needed for a Hypothesis strategy. In the meantime i'll see if i can't figure out a composite strategy/custom check
edit: just saw your reference to #386 (comment)
i'll give that a shot

[abyz0123]

Hi @cosmicBboy apologies I'm just getting around to this.

I have the dataframe level kwarg added and tested. However, adding the unique kwarg to the series object has proven to be more challenging. I'm not sure I understand entirely how the assignment is functioning for the SeriesBaseModel , as my implementation is failing two tests that check that copies of the object are being made.

What would be the best way to proceed? Should I submit the PR with the broken test? If so and you would be able to point me in the right direction, I could then resubmit with the corrections.

[cosmicBboy]

hey @fkrull8 thanks for your efforts on this!

Should I submit the PR with the broken test?

yes please create a PR against the release/0.7.0 branch and I can help you push this through the finish line.

[cosmicBboy]

fixed by #580