Summary:
We want it to be able to register children of layers who
are not direct children of ModelLayer.
This requires us to find subclasses of ModelLayer recursively.
Reviewed By: kittipatv, kennyhorror
Differential Revision: D5397120
fbshipit-source-id: cb1e03d72e3bedb960b1b865877a76e413218a71
Summary: This diff makes functional layer return scalar if only one output. This diff also corrects all other corresponding implementations.
Reviewed By: kittipatv
Differential Revision: D5386853
fbshipit-source-id: 1f00582f6ec23384b2a6db94e19952836755ef42
Summary:
Added device scope checks to data_parallel_model and data_parallel_rendevous
Added test to check that checks are working correctly to data_parallel_model_test
Fixed device_scope error in test_synchronization_barrier
Reviewed By: akyrola
Differential Revision: D5403936
fbshipit-source-id: 849c1cd7452692efbc5ef74d2d60ede090c9c017
Summary: the init method should also make _parameters_info shared between self and param_model, since params is shared. Otherwise it can cause a inconsistence between _parameters_info and params. Examples of using param_model can be find in rnn_cell.py.
Reviewed By: kennyhorror
Differential Revision: D5405327
fbshipit-source-id: ca8079058e898f529906452163cda234cb30a7df
Summary: this diff adds optimizer into param_info, and the associated implementations for modelhelper and brew to set optimizer for each individual parameter.
Reviewed By: kennyhorror
Differential Revision: D5385432
fbshipit-source-id: 5d682f9d1ab077e04a5d76a24d71470f4e64fc92
Summary:
akirillov again presented me with a memonger-bug: his model that has kind of a 'back-and-forth structure' where blobs are passed left and right in a ladder-like structure, revealed a bug in memonger: I should pass the set of free blobs as a reference, not a copy so that the recyclings are properly accounted for. Hard to explain.
Since we have the graph verifier, we can be more confident with these changes.
I also added some helpful debug to the graph verifier.
Differential Revision: D5396925
fbshipit-source-id: 0bffb3a0bf8532afcd6b5bc9331c779768a8c5c5
Summary: Implemented python logic and tests to create an RNNCell for GRU. Uses the preexisting GRU Unit Op code.
Reviewed By: salexspb
Differential Revision: D5364893
fbshipit-source-id: 2451d7ec8c2eacb8d8c9b7c893bfd21b65fb9d18
Summary:
Just an implementation of the forward pass of the GRU Unit Op, not the full RNNCell.
Functions were created to mimic LSTM implementation as closely as possible.
Backwards pass implementations are defined in GRU_unit_op.{h, cc}
assertGradientChecks call added to gru_cell_test.py
Reviewed By: salexspb
Differential Revision: D5364856
fbshipit-source-id: 09cff4478091827763b40cc331e4e0abf0ec258f
Summary:
Just an implementation of the forward pass of the GRU Unit Op, not the full RNNCell.
Functions were created to mimic LSTM implementation as closely as possible.
Implementation defined in GRU_unit_op.{h, cc}
tests put in gru_cell_test.py, which import rnn_cell_test_util.py for sigmoid, tanh, and _prepare_rnn functions.
Reviewed By: jamesr66a
Differential Revision: D5363697
fbshipit-source-id: f9ba9fe0be01ffc868dd22027be8be4975b84998
Summary:
Moved sigmoid, tanh, and _prepare_lstm (renamed) to a util file.
Also renamed _prepare_lstm to _preapare_rnn since it is being used for both setting up and LSTM and GRU model.
The reason for this commit is to allow the creation of GRU Op and testing code without copying and pasting code for sigmoid, tanh, and setting up an rnn unit op mode.
Reviewed By: jamesr66a
Differential Revision: D5363675
fbshipit-source-id: 352bd70378031f1d81606c9267e625c6728b18fd
Summary: Our existing serialization routines take a significant amount of time for large numpy arrays in order to verify the type of each element in the array as well as converting each element to a canonical type. For large floating-point tensors, such as model parameters, this checking and converting takes a significant amount of time. Adding a fast track path for just float32 arrays as this is the most common use case to worry about.
Reviewed By: akyrola
Differential Revision: D5389953
fbshipit-source-id: 26f44cb2426ea3efb849e7707b27d5485f69956c
Summary:
numpy.random.rand generates samples from [0, 1) and therefore, the leaky relu test cases weren't testing negative inputs. Tests still pass after change.
Leaky relu can be used in-place, but gradient took X rather than Y. Technically, the result is no different as it's just used for a sign test in the gradient, but updated it to take Y to reduce confusion.
Differential Revision: D5390126
fbshipit-source-id: d0c428abbb2797eb33902a7d2a2f59d5e85daaa6
Summary: GetComputedParams tests namescopes with equality while GetParams tests with a prefix. Switching GetComputedParams to also use a prefix so that both functions have similar usages.
Reviewed By: akyrola
Differential Revision: D5389816
fbshipit-source-id: 0e43e4b491fccbad3b855b6b735dc2b91d7626c9
Summary: Added a CUDA implementation of the PiecewiseLinearTransformOp.
Differential Revision: D5378537
fbshipit-source-id: 38857f59f5cc52e16e1ecc97983a0b0b82a46c74
Summary:
# Added the gradients of the operation for both CPU and CUDA kernels.
# Unified variable names across all ops.
# Added reference implementation in numpy.
# The gradient check needs a larger stepsize to succeed, is that normal?
Reviewed By: akyrola
Differential Revision: D5313682
fbshipit-source-id: aceb92649e01c5caeba8774e678f9095502d396c
Summary: replace params with sp, otherwise it will report an empty list
Reviewed By: akyrola
Differential Revision: D5382716
fbshipit-source-id: 34d8e6ee00cbe1718702e3d1f23ea12f8d65063e
Summary:
- Integrated RFF into the preprocessing workflow for dense features
- Developed Flow interface to input RFF parameters
- Created unit test for using RFF with sparseNN
Reviewed By: chocjy
Differential Revision: D5367534
fbshipit-source-id: 07307259c501a614d9ee68a731f0cc8ecd17db68
Summary:
To be used with predictor "online": C++ version of memonger for simple nets. Very simple greedy algorithm. Works well at least on Resnet-50 inference graph: only 3 shared blobs are used.
Next I will integrate this with predictor and run canary (separate diff).
Reviewed By: asaadaldien
Differential Revision: D5375392
fbshipit-source-id: d36e419e39a32e568e105657c27fb00c85a2535d
Summary:
As the title says.
Closes https://github.com/caffe2/caffe2/pull/879
Differential Revision: D5372787
Pulled By: akyrola
fbshipit-source-id: 0ff469c0d227f1b2252c1a0c4f6f8bebaac5580f
Summary: Add synchronization barrier API with configurable timeout. Users can call Synchronize() to join variable length execution before resuming multi-machine communication steps, i.e., resuming distributed training iterations after validation on a single machine.
Reviewed By: akyrola
Differential Revision: D5348387
fbshipit-source-id: 5826da10e6a60c50394c36c7cf47624f10191d11
Summary: Memonger had a bug that it crashes if an input blob was input to multiple ops. This fixes that and adds a test.
Reviewed By: asaadaldien
Differential Revision: D5374860
fbshipit-source-id: 1d5044001eacdbe6db43f69727da9297558f5c5c
Summary: Huge improvement in my tests, and it does not really hurt either.
Reviewed By: wesolwsk
Differential Revision: D5374925
fbshipit-source-id: c96a4ed2ca653120a82233c0037cbfded8a2d2a1
Summary:
b33894e95d removed this line:
```py
unittest.skipIf(workspace.NumCudaDevices() < 2, "Need at least 2 GPUs.")
```
but forgot to add it back later.
```
_________________________________ DataParallelModelTest.test_equiv __________________________________
...
if p2p_access_pattern is not None and not p2p_access_pattern[
> devices[0], peer
]:
E IndexError: index 1 is out of bounds for axis 1 with size 1
...
WARNING:data_parallel_model:** Only 1 GPUs available, GPUs [0, 1] requested
```
/cc akyrola
Closes https://github.com/caffe2/caffe2/pull/888
Reviewed By: akyrola
Differential Revision: D5341310
Pulled By: harouwu
fbshipit-source-id: 8d7f06913c7b5a42009a4033dbb6a48a8e812822
Summary:
- Created the random fourier features layer
- Generated a unit test to test the random fourier features layer is built correctly
- Inspired by the paper [[ https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf | Random Features for Large-Scale Kernel Machines]]
Reviewed By: chocjy
Differential Revision: D5318105
fbshipit-source-id: c3885cb5ad1358853d4fc13c780fec3141609176
Summary:
Otherwise was always added to main net instead of param_init_net when
desired (i.e. initial param sync)
Closes https://github.com/caffe2/caffe2/pull/894
Differential Revision: D5367451
Pulled By: akyrola
fbshipit-source-id: 3d82be6da687c736bd15f4852dbd272266eb4811
Summary: Allows to override the input/output record as long as the field blobs are the same.
Reviewed By: yangyangyyy
Differential Revision: D5362132
fbshipit-source-id: 3ac2ac22802902b7eed5c226b00a7e1971ad264c
Summary:
It is quite common question when users get some variant of "blob has version 2 but gradient expects version 1" in their backward pass. The error message is completely unhelpful.
To remedy this, I added proper debug information which tells user how the version number of a blob was incremented over time. i.e which ops caused the version to go op. This should help
understand the issue.
Reviewed By: dzhulgakov
Differential Revision: D5358227
fbshipit-source-id: bc09d048ac33200c35d56460e44e86c2f2888f3f
Summary: Added two operators that can be used to tranfer data into the input format of RNN and back.
Reviewed By: kittipatv
Differential Revision: D5329886
fbshipit-source-id: 07eac29416427b08c49989d4eeed50a6f18493a1
Summary: This was broken in a previous diff, fixing it to use model device type.
Reviewed By: asaadaldien
Differential Revision: D5356005
fbshipit-source-id: a4fcc932bae772076b57625a5fcc0d38eb702cc9
Summary:
this works as a standalone python script because args are
global. When used from Flow for monitoring purposes it doesn't
work. This diff fixes it
Reviewed By: zem7
Differential Revision: D5349996
fbshipit-source-id: f73842901d975b783e09e9db0565eb81880bbea1
Summary:
A couple of fixes to fix broken rerporting of lstm_benchmark:
- last_time must be recorded after warm up
- entry count was incorectly removed
Reviewed By: salexspb
Differential Revision: D5349890
fbshipit-source-id: 5dd5bdf46594c520b61bc3b57b153f90a6a17903
Summary:
Eliminates failures from overloaded machines from only
running a few examples before being timed out.
Reviewed By: tomdz
Differential Revision: D5349555
fbshipit-source-id: 89d1db063f58c72656b37157225a586c9e3f24bc
