So I have been engineering my own neural network technology for some time, and I’ve been running tests and experiments sometimes. What I have discovered about backrpopagation, is that, when there are errors in backpropagation algorithm, they may not break backpropagation completely. It can work in partial completeness or with some amount of errors, to some extent.
The first attempt I made to write backpropagation some years ago, I used a math library called Armadillo in hopes that it would be fast. This code ended up sort of working, but clearly, was not actually working, and indeed there were so many errors since I wrote it during a 24 hour challenges.
Then, I branched and started from scratch without armadillo and just c++. This code worked too, but, it turns out there were bugs which in hindsight, introduced limits.
In the next branch, I revisted Armadillo using my understanding of my c++ code and more Armadillo research, to produce yet another version of backpropagation which seemed to work at least as good as my c++ version.
Finally, I created a branch to compare these two working methods to eachother and see what errors I can find. I found a lot of similarities between the two branches, and also, clear errors in both branches. I corrected them as far as I can tell, and found out my c++ code is some 10x faster than the Armadillo code, and I wonder if I am using Armadillo properly, IDK.
In my current project, in a pool of 2000 networks, if the backpropagation math is correct, then I generate maybe 30 xor networks with fewer than 100 trainings of specific xor function.
If I train circle function beforehand, then I end up training more like 200 xor networks with fewer than 100 trainings of specific xor function.
Some of the technology I developed related to genetic algorithms all functioned properly even though backpropagation was broke. I produced mnist network which could recognize 0,1,2 digits – with broken backpropagation process on CPU, probably mainly functioning through genetic features. It seemed to approach limits which I hoped were errors that I could fix, and perhaps I have. Errors or no errors, there is still the project.
Spot C++ 