-
Notifications
You must be signed in to change notification settings - Fork 219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tweak sample counts and other details in tests #2433
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2433 +/- ##
=======================================
Coverage 86.41% 86.41%
=======================================
Files 22 22
Lines 1575 1575
=======================================
Hits 1361 1361
Misses 214 214 ☔ View full report in Codecov by Sentry. |
Pull Request Test Coverage Report for Build 12321043702Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
I'm hoping CI will now pass, except for Mooncake stack overflows. See updated OP for what this PR does. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, if you could bump Mooncake compat to 0.4.61, that would hopefully take care of CI 👍
I can approve and merge once CI passes :) |
Some Ubuntu tests are looking like they will time out, but I'm going to merge on the basis that the tests pass on Windows and macOS and the root cause for the timeout is known (TuringLang/DynamicPPL.jl#750) and will presumably be fixed soon (TuringLang/DynamicPPL.jl#751). |
This PR makes a number of small changes to the tests
Random.seed!
. Every@testset
resets the seed anyway.verbose = true
to many of the top-level@testset
s, to get some timing numbers even when tests pass.This started with me thinking that since we do a lot of samples in many tests, and our test suite is quite slow, clearly we should reduce the sample counts if we can. What I learned in the process is that most of the tests that do sampling are negligble compared to the tests that use particle methods. Anything involving PG/CSMC tends to take 10-100 times longer than comparable tests with other samplers.
In the end I've tried to adjust iteration counts only when either the count truly doesn't matter (we are not checking stats of the chain), or when doing so actually has a noticable, though not necessarily huge, impact on run time. I may have slipped a few times, and reduced something that didn' really make a difference. Also, when reducing iteration counts I tried to check that they pass if I pick a few different seeds, to make sure I'm not making them very brittle to future changes.