Decomposing a problem into smaller and more manageable pieces to be solved independently and then merging the solutions has been a common motif inspiring many fundamental advancements (Search, ordinary differential equations, Kepler etc.).
In this survey, I will present several approaches to divide the parameter estimation problem for neural networks by back-propagation into simpler sub-problems.
Neural networks have recently had a strong impact in both our day to day life and in other fields, partially also due to this reason, the number of parameters of the modern models keeps increasing with no yet clear limits on performance, yet the hardware to train such networks won’t be able to keep up the pace of the software, the last 10 years have been supported by the move from CPUs to more specialized GPU hardware, recently dedicated hardware such as TPU or similar have managed to keep up with the demand but alas, there are limits to how many processing units we can fit in a small space, distributed algorithm have notoriously been the solution to such problem but we have yet to develop efficient optimization for dense problems like neural networks.
...