Add seed_aug parameter for ImageRecordItr to fix random seed for default augmentation #11247

wenyangchu · 2018-06-12T19:14:38Z

Description

Add random seed control to augmentations in io.ImageRecordIter.
Motivation: I am doing a medical imaging project where reproducibility of the complete training procedure is important for a legal purpose.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
(Add seed_aug parameter for ImageRecordItr to fix random seed for default augmentation)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

…entation

…t module (apache#11140) * gluon import * gluon tests * shape issues. * remove the dim_change list * onnx backend tests * changes to match onnx op set version 7 * fix * lint fix * add new folder * fix * fix * rename test file * tile topk clip * more ops * fix * fix

marcoabreu · 2018-06-12T23:34:26Z

Hi @wenyangchu ,

thanks a lot for your contribution! Would you mind adding a test to ensure that the behaviour is actually deterministic now and that everything is behaving as expected?

* Important ndarray feature * merge generic function Invoke * Pass the Scala Style test * add Experimental tags * Change with NDArgs addition * change dir for Experimental tag * reTrigger CI * add Symbol Macros change * Add some workaround on NDArray * Simplify the base part * add changes on ND and Symbols... * avoid vars * add Symbol Macros * Trigger the CI * Trigger CI

wenyangchu · 2018-06-13T09:17:21Z

Hi @marcoabreu ,
sure, I will add a test for it. Do you know where the existing tests for this c++ iterators right now? Or for this default augmenter? I couldn't find it.

wenyangchu · 2018-06-13T11:09:59Z

Hi @marcoabreu or any others, another question is that, I have a test failure in python3 cpu win and python3 MKL-DNN cpu. I have checked them and I do not think they have anything to do with my commit. Do you have a clue on this? Thanks!

Don't use tempdir on osx since this is not mountable by default in Docker in osx

marcoabreu · 2018-06-13T14:52:19Z

Hi, our tests are in https://github.com/apache/incubator-mxnet/tree/master/tests/cpp

Yes, sorry, we are encountering quite a decent number of flaky tests. Ammending your commit should be sufficient to retrigger a new job. Sorry for the inconvenience.

* Adding examples to list * Update links add demo * remove onnx-mxnet as it is now part of contrib * Trigger build * Adding Awesome-MXNet * Trigger build

* use nearest power of 2 for gpu memory pool sizes * add linear * add test

* disable scalatest on Spark * Kill the entire build * Disable test in test folder and add dummy label

* mkldnn support for quantization * fix output number in graph * update licsence * modify Jenkinsfile * modify Jenkinsfile * mkldnn has no int8 fc api, excluded_sym_names includes fc for cpu * add mkldnn uint8 pass for quantization graph * update ut * retrig ic * remove no mkldnn quantization test temp * seperate mkldnn quantization ut from gpu quantization ut * rm dev_id check for cpu * add mkl tests dictionary * resolve review comments * simplify DequantizeStorageType() logic * simplify quantize/quantized_conv storage type logic * Add mkldnn_OIhw4i16o4i type case (needed by int8) * INT8 conv/pooling: share with FP32 convolution/pooling class/function * minor indent changes * Remove unnecessary mkldnn_quantized_pooling-inl.h * Fix minor issue * Fix lint * delete duplicated data type * fix bugs and convert requantize data to NDArray * fix lint * fix requantize storgetype * fix requantize storge type * Fix coding style comments * Fix compile issue * Change to use quantized_dtype option to support uint8/int8 scenarios * fix gpu test quantization failure * Fix indent * fix quantized pooling param parser * Fix imagenet_gen_qsym.py option style * retrigger jenkins * retrigger again * trigger jenkins * Resolve further comments * share test code * remove unnecessary test code * add test_quantize_model for cpu * add comments in quantize_graph_pass.cc * jenkins * jenkins * improve coding style * improve coding style * Add naive CPU quantization test back and share quantization code between naive-CPU/MKLDNN/GPU * rename test_quantization_cpu.py to test_quantization_mkldnn.py * code style * trigger * Adjust variable naming for test quantization * add qdtype for quantized op test case to test/bypass all cases explicitly * change expressions to be consistent * revert unnecessary change

* add import_ for SymbolBlock * fix * Update block.py * add save_parameters * fix * fix lint * fix * fix * fix * fix * fix * Update save_load_params.md

* Fixes Scala memory leak (apache#10436)

* Adding the first set of Facebook OG tags to layout.html * Adding the direct link to the Facebook OG image * Re-arranging the og description tag * Changes to fix merge conflicts * Adding secure_url to the og:image tag * Adding the redundant og:image tag

* add test for mkldnnsum * add extra comment * fix verify * only run if mkldnn supported * divide size by size of type * filter out regular arrays * fix cond * add verify mes * filter views * different outputs arrays * move print message * add in place test * update copy fn * refactor copyfrom * use arr.copy instead of tmpmemmg * use InitMKLDNNArray helper * fix params * pass correct type to copyfrom * add print message for inplace sum * remove copyfrom refactor * remove redundant header * remove extra line * fix lint * retrigger

* Improve hybridblock doc * fix

* disable testcase test_gru_bidirectional temporarily * retrigger the build and see if permission denied /efs-ccache can pass

…rs() (apache#11210) * flipping official and alternative * Update block.py * Add new line * fix trailing whitespaces * clarify docs * Update block.py * Update block.py * fix typo * trailing whitespace * Trigger build

wenyangchu · 2018-06-14T21:31:53Z

Hi @marcoabreu or others, the c++ tests for iterators in mxnet are not there and iterator's code is entirely in cpp file which is very difficult to test in c++ without refactoring. I found other tests for c++ iterators were done in python. I follow the same way to perform tests on this augmentation seed.

Is it correct that the following document should be updated automatically after merging this PR?
https://mxnet.incubator.apache.org/api/python/io/io.html#mxnet.io.ImageRecordIter

* Enable shared ccache * Increase cache size * Don't fail if EFS ccache is not available

* improve transforms.Resize * fix * Trigger CI * Trigger CI * improve * Trigger CI * Trigger CI * fix unittest * keep_ratio is false by default, to keep consistency

marcoabreu · 2018-06-14T22:42:01Z

Yes, this will automatically update after merging.

A python test is totally fine as well

* add count_include_pad argument * add cound_include_pad in cudnn pooling and corresponding tests * add gluon support for the new option * switch to built-in functions for artificial NaN * add doc for gluon * change isAvg/getAvg to is_avg/get_avg

…e#11316) * MXNET_FORCE_ADDTAKEGRAD to disable AddTakeGradLargeBatchCaller If MXNET_FORCE_ADDTAKEGRAD is set, EmbeddingOpBackward will always use AddTakeGrad independently of gradient input and output shape * Read MXNET_FORCE_ADDTAKEGRAD to a static variable

…am.save(). Fix trainer init_kvstore (apache#11266) * clip sparse grad. fix _reduce for rowsparse param * fix kvstore init for local kv * trigger

zhreshold · 2018-06-18T22:27:20Z

src/io/image_aug_default.cc

@@ -136,6 +139,8 @@ struct DefaultImageAugmentParam : public dmlc::Parameter<DefaultImageAugmentPara
    DMLC_DECLARE_FIELD(pad).set_default(0)
        .describe("Change size from ``[width, height]`` into "
                  "``[pad + width + pad, pad + height + pad]`` by padding pixes");
+    DMLC_DECLARE_FIELD(seed_aug).set_default(-1)


better use dmlc::optional() and check has_value() rather than compare with -1.
Literally we can use any number as seed.

* Add first draft of profiler tutorial * Minor changes - Add images from web-data - Add  * Language corrections * Minor changes - Fix image URLs - Fix formatting of output * Minor changes - Add download button. - Hide profile_stats.png in notebook. * Add tutorial to index. * Add tutorial to tests. * Add a note about nd.waitall() * Remove the example build command. Link to installation page is sufficient. * Fix typo * Include info about env variables related to profiling * Add a further reading section

…11302) * Add ccache reporting to CI * Restructure dockcross dockerfiles to fix caching

* add resnet augmentation * add test * fix scope * fix warning * fix lint * fix lint * add color jitter and pca noise * fix center crop * merge * fix lint * Trigger CI * fix * fix augmentation implementation * add checks for parameters * modify training script * fix compile error * Trigger CI * Trigger CI * modify error message * Trigger CI * Trigger CI * Trigger CI * improve script in example * fix script * clear code * Trigger CI * set min_aspect_ratio to optional, move rotation and pad before random resized crop * fix * Trigger CI * Trigger CI * Trigger CI * fix default values * Trigger CI

wenyangchu · 2018-06-19T22:46:36Z

I think I have flaky test again

* add softmax imporvement * reuse CheckAxis code * update comment * add tests with negative axis

zhreshold · 2018-06-20T05:45:39Z

see if rebase fix the flaky test

…entation

zhreshold · 2018-06-20T23:13:34Z

@wenyangchu thanks for the contribution, this is merged.
A quick tip: please use rebase instead of merge to master because otherwise it's hard for people to review only your changes in this PR.

…ult augmentation (apache#11247) * add seed_aug parameter for ImageRecordItr to fix random seed for augmentation * remove white space * add test * fix test * improve according to review: using dmlc::optional and has_value() * missing header * change data type and way to get value

Wen-yang Chu and others added 3 commits June 12, 2018 20:39

add seed_aug parameter for ImageRecordItr to fix random seed for augm…

f610a41

…entation

remove white space

3a462b5

Fix build.py when CCACHE_DIR is set. (apache#11226)

fc102c3

Don't use tempdir on osx since this is not mountable by default in Docker in osx

ThomasDelteil and others added 7 commits June 13, 2018 10:39

[MXNET-534] Add examples to example list (apache#11225)

c54f9f7

* Adding examples to list * Update links add demo * remove onnx-mxnet as it is now part of contrib * Trigger build * Adding Awesome-MXNet * Trigger build

gpu mem pool strategy (apache#11041)

bf26886

* use nearest power of 2 for gpu memory pool sizes * add linear * add test

[MXNET-543] disable scalatest on Spark (apache#11264)

eb95d7b

* disable scalatest on Spark * Kill the entire build * Disable test in test folder and add dummy label

add import_ for SymbolBlock (apache#11127)

66ab27e

* add import_ for SymbolBlock * fix * Update block.py * add save_parameters * fix * fix lint * fix * fix * fix * fix * fix * Update save_load_params.md

Scala inference memory leak fix (apache#11204)

bbc7a22

* Fixes Scala memory leak (apache#10436)

wenyangchu changed the title ~~add seed_aug parameter for ImageRecordItr to fix random seed for augm…~~ [WIP] add seed_aug parameter for ImageRecordItr to fix random seed for augm… Jun 14, 2018

wenyangchu closed this Jun 14, 2018

wenyangchu reopened this Jun 14, 2018

piiswrong and others added 4 commits June 14, 2018 14:02

Improve hybridblock doc (apache#11236)

ee1a10f

* Improve hybridblock doc * fix

disable testcase test_gru_bidirectional temporarily (apache#11273)

4a81433

* disable testcase test_gru_bidirectional temporarily * retrigger the build and see if permission denied /efs-ccache can pass

add test

f3220fb

wenyangchu changed the title ~~[WIP] add seed_aug parameter for ImageRecordItr to fix random seed for augm…~~ Add seed_aug parameter for ImageRecordItr to fix random seed for default augmentation Jun 14, 2018

marcoabreu and others added 2 commits June 14, 2018 14:46

Enable shared ccache and fix test (apache#11269)

2eadfa2

* Enable shared ccache * Increase cache size * Don't fail if EFS ccache is not available

Improve data transform for gluon data loader (apache#11183)

b819fd9

* improve transforms.Resize * fix * Trigger CI * Trigger CI * improve * Trigger CI * Trigger CI * fix unittest * keep_ratio is false by default, to keep consistency

fix test

4908600

haojin2 and others added 3 commits June 18, 2018 11:10

[MXNET-514] Add clip_global_norm(row_sparse_grad). Fix row_sparse_par…

9226d91

…am.save(). Fix trainer init_kvstore (apache#11266) * clip sparse grad. fix _reduce for rowsparse param * fix kvstore init for local kv * trigger

zhreshold approved these changes Jun 18, 2018

View reviewed changes

zhreshold suggested changes Jun 18, 2018

View reviewed changes

indhub and others added 6 commits June 18, 2018 19:12

[MXNET-553] Restructure dockcross dockerfiles to fix caching (apache#…

47e2b89

…11302) * Add ccache reporting to CI * Restructure dockcross dockerfiles to fix caching

improve according to review: using dmlc::optional and has_value()

f0a4ad8

missing header

681bd1b

change data type and way to get value

ac9c1b7

zhreshold approved these changes Jun 19, 2018

View reviewed changes

Fix axis Bug in MKLDNN Softmax (apache#11335)

6307c00

* add softmax imporvement * reuse CheckAxis code * update comment * add tests with negative axis

Wen-yang Chu and others added 8 commits June 20, 2018 11:31

add seed_aug parameter for ImageRecordItr to fix random seed for augm…

5e544d2

…entation

remove white space

d85ee7e

add test

cca85ef

fix test

57811ce

improve according to review: using dmlc::optional and has_value()

e19226b

missing header

74b779a

change data type and way to get value

5103eca

Merge branch 'master' of https://github.com/wenyangchu/incubator-mxnet

585b21d

wenyangchu requested review from marcoabreu, szha and yzhliu as code owners June 20, 2018 09:35

zhreshold merged commit 9ffc03c into apache:master Jun 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add seed_aug parameter for ImageRecordItr to fix random seed for default augmentation #11247

Add seed_aug parameter for ImageRecordItr to fix random seed for default augmentation #11247

wenyangchu commented Jun 12, 2018 •

edited

Loading

marcoabreu commented Jun 12, 2018

wenyangchu commented Jun 13, 2018

wenyangchu commented Jun 13, 2018 •

edited

Loading

marcoabreu commented Jun 13, 2018

wenyangchu commented Jun 14, 2018 •

edited

Loading

marcoabreu commented Jun 14, 2018

zhreshold Jun 18, 2018

wenyangchu commented Jun 19, 2018

zhreshold commented Jun 20, 2018

zhreshold commented Jun 20, 2018

Add seed_aug parameter for ImageRecordItr to fix random seed for default augmentation #11247

Add seed_aug parameter for ImageRecordItr to fix random seed for default augmentation #11247

Conversation

wenyangchu commented Jun 12, 2018 • edited Loading

Description

Checklist

Essentials

Changes

Comments

marcoabreu commented Jun 12, 2018

wenyangchu commented Jun 13, 2018

wenyangchu commented Jun 13, 2018 • edited Loading

marcoabreu commented Jun 13, 2018

wenyangchu commented Jun 14, 2018 • edited Loading

marcoabreu commented Jun 14, 2018

zhreshold Jun 18, 2018

Choose a reason for hiding this comment

wenyangchu commented Jun 19, 2018

zhreshold commented Jun 20, 2018

zhreshold commented Jun 20, 2018

wenyangchu commented Jun 12, 2018 •

edited

Loading

wenyangchu commented Jun 13, 2018 •

edited

Loading

wenyangchu commented Jun 14, 2018 •

edited

Loading