Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider having corner cases by default #231

Open
vch9 opened this issue Mar 11, 2022 · 1 comment
Open

Consider having corner cases by default #231

vch9 opened this issue Mar 11, 2022 · 1 comment

Comments

@vch9
Copy link
Contributor

vch9 commented Mar 11, 2022

For instance QCheck2.Genui64 that would be useful to have by default:

graft_corners ui64 [0L; 1L; 2L; Int64.max_int] ()

As we do not have a proper fuzzer, these cases are important and we can not trust the generation to cover theses. However, one answer can be: "they need to unit test the corner cases, then write pbt"

@jmid
Copy link
Collaborator

jmid commented Mar 18, 2022

I can certainly follow you.
In a sense, the folklore of "the tester needs to be mindful of testing corner cases" translates to
"the PBT tester needs to be mindful of generating corner cases".
A personal pet-peeve here is https://github.com/jmid/qc-ptrees where changing the int generator to the following made a difference between finding a bug and not finding it: https://github.com/jmid/qc-ptrees/blob/f3c01664e76bc3944f48e155cb410db8317baf06/qctest.ml#L84-L89
I admit this situation is not particularly satisfactory though.

That said, I'm very hesitant about making such a change as it makes generators stateful by default.
This means that simple generator transformations such as:

utop # let g = QCheck2.Gen.small_int;;
val g : int QCheck2.Gen.t = <abstr>
utop # QCheck2.Gen.(generate1 ~rand:(Random.State.make [|123|]) (pair g g));;
- : int * int = (6, 3)
utop # QCheck2.Gen.(generate1 ~rand:(Random.State.make [|123|]) (pair g g));;
- : int * int = (6, 3)

will no longer hold in the default case:

utop # let g = QCheck2.Gen.small_int_corners ();;
val g : int QCheck2.Gen.t = <abstr>
utop # QCheck2.Gen.(generate1 ~rand:(Random.State.make [|123|]) (pair g g));;
- : int * int = (1, 0)
utop # QCheck2.Gen.(generate1 ~rand:(Random.State.make [|123|]) (pair g g));;
- : int * int = (4611686018427387903, 2)

This will hurt QCheck/QCheck2's ability to reproduce pseudo-random runs, as behaviour now depends on both the random seed and the internal state of all involved generators.

A less invasive option is compose a non-stateful int-generator like above.
Here's an example of less trivial one - which I learned from @jlouis some years ago.
This has a greater chance of producing corner cases (and can be further composed with Gen.oneof):

(* sign * pow(2, k) + [-3;3] *)
let int_gen =
    (Gen.map3
       (fun sign expo addition ->
         let n = 1 lsl expo in
         let n = if sign then -n else n in
         n+addition)
       Gen.bool (Gen.int_range 0 63) (Gen.int_range (-3) 3));;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants