Synapse: Modeling in Excel

One of the difficulties I have when modeling my different theories of neuronal network architectures is being able to visualize the relationships of the system. I was thinking of creating a Unity (video game engine) plugin that would be able to visualize the network with colors and shapes so that when the network processes input and learns, the relationships could be visualized in a way that allows us to interprets and identify the relationships between components much easier.

There is a tensor library attempting to do that here that I thought was interesting:

Creating that 3D visualization project has a long lead time before getting results, as I would have to change some accessibility of my network variables in order to do it properly, so I kept with my “lean” thinking and opted to try and model my network in an Excel file.

My Excel file draft is linked here so you can follow along with how I plan on implementing some of the concepts I described in my previous post – Synapse: The New Hypothesis

I created an “as simple” as I can network to try and prove out some of the concepts that looks like this:

  • Input (2 Features):
    • Green Light / Red Light
  • Action (1 Feature)
    • Go / Stop
  • Goal (1 Feature)
    • Feedback

The input of this system is a two dimension vector with one of the features being the “Green light” feature, and the other one being the “Red light” feature. The model has a single hidden layer with 4 neurons that then results in an output (action) that is a 1 dimension vector (is it a vector if it just a single number?) that has action features of either “Go” or “Stop”.

There is a “motor system” that interprets the output (action) of the network and decides to either “Go” or “Stop”. With the output guaranteed to be between 0-1 (sigmoid activation), my motor system will choose to go if the result is greater than half, and stop if it is less than half.

Motor System Interpretation of Output

  • Go > 0.5
  • Stop < 0.5

The goal sensor in this system is a “Feedback” sensor. This is a single dimension vector, that represents whether the system made a right choice (Go when Green), or a bad choice (Go when Red). In future implementations this can be used as a way of having real world interaction with the system and reward behaviors that you would want to duplicate.

From an architectural perspective, this system is largely just a feed forward network, with a couple changes. 1 – The “ideal” output is not initially known like a labeled data implementation, and 2 – you have an additional variable of a “goal sensor”. The goal sensor is needed so that an “ideal” output can be created with an appropriate weight based on how “close” the system is from achieving the goal.

In the linked (above) Excel file you should be able to piece apart how that is happening. There is some things that need explanation on how I’ve set the variables up, but the concept should be able to be interpreted from this file and how I plan to explicitly arrive at an “ideal action”.

I’ve defined some of the core components of the system to help clarify and guide.

Senses are: numeric interpretations of information that are read by sensors.

Inputs are: sensors of the world.

Actions are: sensors of the motor system.

Goals are: sensors of an internal state and require an ideal vector.
*This sensor is required to be coded in a way that a Euclidean distance between the current goal sensor reading and the ideal can be made.

Motor system is: a component that interprets actions in order to create behaviors in the system.

Hopefully the accessibility of the Excel file and the definitions give enough information to understand what is attempting to be built. Let me know if you have any questions on this, and when I get a version coded and implemented I’ll be posting here, so stay tuned!

Lesson Learned: Don’t use the Same number of Input as Output in a Feed Forward Network

My first official “failure post” – although I think I have had a couple already – is potentially an amateur lesson, but I learned a lot about the relationship between the inputs and outputs of a system.

I created a simple network in order to test my code. It was the “Red Light, Green Light” network where the input was a two feature vector that determined whether the green light was on, or the red light. The output was another vector with two features, whichever one would be a larger value would determine if it would “go” or “stop”.

The system would not work.

It kept getting a 50 / 50 success (completely random). I spent a ton of time looking into my logic in the code, and then resorted to putting a version of my logic in an Excel file to see if my understanding of the back propagation was correct, and it was still not working no matter what variations I made.

I hit the drawing board on almost everything and questioned everything, until I realized that the network I had setup was a linear network (2 inputs -> 2 Outputs) and not a multi-variable function network (2 inputs -> 1 output).

I changed my network to a single output with a midpoint (0.5) of the single output determining whether the system should go or stop. The system started working and within a couple of iterations was able to learn the rule.

The lesson learned is that there needs to be more inputs, than outputs in order for a feed forward network to work. By trying to simplify things, I overcomplicated things.

I’ve attached my Excel file with all my different attempts if you care to review all I did and what was happening in all the different iterations of the network.

I’ve attached my Excel file with all my different attempts if you care to review all I did and what was happening in all the different iterations of the network.

The Value of Failure

I am a big fan of the saying:

“Success stories are worthless, its the stories of failure that are valuable”.

I am not sure who said that, and a cursory Google search didn’t bring me an author, but stories of success feel more like humble brags then helpful guidance. With a success story, the lesson appears to be “do what I did, and maybe you can succeed with that too”.

If you want to copy and follow, then this is the type of advice you would seek out, but if you want to innovate and lead it is the stories of failure that should be sought out. With failure, there follows some lesson on why the failure occurred that can be generalized to other scenarios in hopes of avoiding similar failures going forward.

In a way, this is exactly what science is all about. It sounds counterintuitive, but we can never “prove” theories we can only disprove them, and only when evidence becomes significant enough is something accepted as a scientific “fact”.

An example of this to help illustrate the concept put together by Karl Popper, an Austrian-British philosopher, is a theory that all swans are white.

‘All swans are white cannot be proved true by any number of observations of white swan – we might have failed to spot a black swan somewhere – but it can be shown false by a single authentic sighting of a black swan. Scientific theories of this universal form, therefore, can never be conclusively verified, though it may be possible to falsify them.’ .
– Karl Popper

It is with this concept that I will try and post “Failure Posts” in hopes that others will be able to use my failures to avoid making their own, and shape their own theories on ‘thought’.

I was struggling with a simple issue while working on Synapse and I think it was telling, so I will be writing up something on that soon. Stay tuned.