Summary: For some reason I had been disabling the exhaustive search heuristic for cudnn for xray/resnet trainers. On BigBasin, this gives 10% perf boost. On BigSur maybe 5%.
Reviewed By: prigoyal
Differential Revision: D4338654
fbshipit-source-id: 3974dd612f5d4f4dc8b2febccb59664d3f276c3e
Summary:
It gives a significant perf boost to do the parameter update inside MomentumSGD, instead of with a separate WeightedSum op.
To ensure backwards compatibility, I made it a separate op.
Also added an unit test.
Reviewed By: prigoyal
Differential Revision: D4262446
fbshipit-source-id: 38e7ee6d7677b398658ac7fe9b7a59b569e033f4
Summary:
This examples writes a LMDB database of image data and labels (random). Then it reads them using Caffe2's TensorProtosDBINput and validates the checksums match. This example shows how to coerce image data into TensorProtos and be happy.
Before there was no clear example how to create databases for Caffe2.
Differential Revision: D4263614
fbshipit-source-id: 21e08066899095b4efcc2d23dbc3ede81e75914a
Summary:
When refactoring data parallel model, the division of LR by number of devices was dropped, and thus we ended up effectively multiplying gradients by the number of devices. Thus, we need to scale the LR by 1/numgpus.
Created a test to confirm that data_parallel_model produces exactly same results on different number of gpus, given the total batch size.
Reviewed By: prigoyal
Differential Revision: D4248907
fbshipit-source-id: af21ede113e6ac25f12c556de298cb18974548be
Summary: Just noticed that I had duplicate code in the example imagenet trainer. Removed the function.
Differential Revision: D4223070
fbshipit-source-id: 443a9401bf7e425f7a3a13a44c9d0f7e21e72303
