Long Term Short Memory LTSM or FICO for short.

The anacronym FICO stands for Forget Include Copy Out these are the gates for the LTSM module based on the infamous LSTM module.

These are neurons with gates controlled by weights with sigmoid activation functions. An Input(Xt) is combined with the previous output of the last cell (Hin) or Hout at t-1. The gates determine whether the previous cells Activation (Ctin) is Forgotten or not (F - Forget Gate). They also determine whether the Input combine with Hin is included in computing the Activation(I - Include gate) of the unit and whether the Activation is passed to Hout (O - Output gate).

1. Include Previous Output?
ht-1 + Xt -> sigmoid -> forget 1/0 -> Adds PreOutput to Output(Ct)

ft = sigmoid(Wf.[ht-1,xt] + bf

2. Include new Input?
ht-1 + xt -> sigmoid -> include 1/0 -> Adds tanh(Input+PreHidden) to Output(Ct)

It = sigmoid(Wi.[ht-1,xt] + bi
~Ct = tanh(Wc.[ht-1,xt] +bc)
Ct = ft*Ct-1 + it*~Ct

3. Include Output as Hidden?
ht-1 + xt -> sigmoid -> include 1/0 -> Adds tanh(Output) to Hidden(ht)

Ot = sigmoid(Wo.[ht-1,xt] + bo
ht = Ot * tanh(Ct)

Four Weights: Wf  Wi  Wc  Wo  or F.I.C.O for short!

I connected everything up in this way and got it to learn an Identity function for one module and I am poised to use it on a time series which I have read is a good application for the LSTM.

I have found however that for Identity - a learnt mapping from input to output sigmoids insteads of the tanh functions work better.

Also for the Identity mapping test I fed back the previous Outputs to their Inputs Cout -> Cin  Hout -> Hin for Xt = X.

We shall see if it works out of the box for time series will it learn how to count and will it predict a sequence of numbers.

I have written the whole unit using vectors making it suitable for taking inputs and producing outputs for a convolutional process. Making it perhaps suitable for image processing.

The std::vector object is simply a faster and easier way of processing 2d arrays.

Also I have made the Weight arrays flexible to being 1d or 2d.  So its all go for Long Term Short Memeory!!

CNN trained on practical surveilance targets.

A stationary surveilance camera is position opposite the exit to a busy station its goal is to locate and track individuals matching a target group and behaving suspicously.

Each individual is given the task of leaving the station carrying a concealed item without being seen by a a set of cameras placed throughout the station.

Using an advanced form of deep learning a Convolutional Neural Network dynamically picks out those from the group who are trying to evade detection.

The group is instructed not to act in anyway to betray their location to surveilance.

The CNN correclty identify's isolates and tracks each individual using advance image processing and deep learning.

The algorithm first works by reading the body language of any individual moving differently to the other members of the public. It then uses a rule of thumb based on its previous training to determine whether they are a valid target.

Using a purpose built database it then matches the individual using rudimentary face recognition.

Source : Reuters

Will the Convolutor be a GAN - Generative Adversarial Network

My CNN Condor has been coming on leaps and bounds trained on MINST data performs admireably only after 205 iterations of training.

Fig 1 These are the Minst images after successive discriminating filters with preloaded weights.

However I cannot get it to perform as a GAN - Generative Adversarial Network.

I have created purpose built functions that assemble the GAN's elements -

Generator - A series of RGB Convolution layers in the literature they are of varying size and donot have a Pool layer or an MLP layer.

Discriminator - Well this is just the CNN pretrained with its preloaded weights which takes as its input the output of the generator - and feeds back the error.

I have tested it with an MLP layer and without but I have yet to vary the size of each layer.

What I get with training is simply an image that looks like an untuned TV with the contrast on high. Its dark mush but it may yet deliver some results after I vary the

size of the layers and perhaps play around with their transfer functions.

Interestingly the MLP included produces the same result even after training.

Random Code -> MLP1 -> Layer 1...3 -> Output Image 100x100

->Input to CNN -> MLP2 Generate Error -> Feedback Error to MLP1 and Layer 1..3

Nice try.

Making some Noise

A new and exciting project is now underway. Negative noise.

I had some time back been experimenting with combining oscilators to produce a chaotic signal and recalled being able to produce a signal that had a negative voltage.

I am not sure how I did it and frustrated having dismantled and forgotten how the original circuit worked. All I recall was that there were two oscillators and the output was passed through my sound card to the oscilloscope.

The soundcard (not being designed for this purpose) although revealling some interesting waveforms eventually caused the soudcard to develop a glitch as the current was too high even though it had passed through a filter.

I then went and brought an oscilloscope which uses the ps2302 usb to serial this works lovely.

I then spent some time trying to build a circuit that could replicate these previous attempts and after many failures I suddenly hit upon the right combination of components. Now I can use a variety of different transistor types to create a noise signal that has both a positive and a negative component. After passing it through another filter I can reveal more of the signal and change its underlying form as a high pass filter its really interesting that you can do so much with just a few components and this circuit design is flexible to use different  transistor types in order to create the noise and then amplify the signal using an op-amp.

I would now like to take it to the next stage and make the signal
audio with more amplification and create an A2D convertor so that the noisy signal can be read by a really LITE version of my Mlp-on-a-chip.

I'd like the Mlp-on-a-chip to read the noise and perform a time series prediction. It may be a lot to ask and it would be enough to get it to distinguish between the different noise types produced by each transistor. An ambitious project that I will hopefully begin with making the noise audible through a pair of headphones and an amplifier circuit addon that I must build.

I shall upload the circuit for my Negative Noise here for anyone interested.

For ages I could only build a working Multivibrator and still cannot get a good Sine Wave. But now I have Negative Noise!


Notes to self use Noisey Hyperparameters as-per Noisey transfer function.

Links within Links - Pointers the journey continues. (Wiring an RBM)

Having a renewed interest in using pointers for other than passing arrays and functions between functions. I am now using them to create a Link matrix that preserves symetry and equality between weights from i to j and from j to i.

I first decied to have each of my weights for each neuron pointing to a weight matrix. Having setup my array of pointers for each neuron I then had to award it the right amount of memory as * **Array needs *** much memory! and not this much * **!

I then had the problem of pointing each Weight to the same Weight as its symetric opposite and I found the simplest way was to turn the Weight matrix into another array of pointers that point to their own symetric weight values.

Now I am all sorted and everything points to something else and I feel complete I feel whole!

Its all connected and I feel great!

Neuron -> Weight Pointer[n] -> Weight Matrix [n][m] -> Weight Matrix [m][n]

Easy!

Now I need only write to and read from Each Neuron for updates and learning etc. In fact I can just isolate one neuron and run everything on that one neuron.

Another triumpth for the Neuron Centered Algo. [WARNING Weight Vectors STUNT GROWTH!]

The Wonder of Pointers and their use in Building Neuron Centered Algo's.

Well I have been looking at the wiring on one of my nets and I found it just wasnt up to the requirement of forward and backward updates. I need it to produce values at the input and the output nodes - this is for CD - contrastive divergence. A clever learning algorithm that uses samples taken at various phases :

Phase 1 clamp inputs generate hidden values (Sample h given v)
Phase 2 unclamp inputs and generate inputs  (Sample v' given h)
Phase 3 clamp to v' and generate h'

This process is repeated and the values are used to compute weight changes as with backpropogation.

My neuron centered design needed to update itself using weights that had formally been assigned to the neurons in successive layers.

The answer to this was the use of pointers as weights. I had tried using a function to equate the symetrical links between neurons but ofcourse all I needed ws these magic pointers. It was as though they were designed for this purpose.

Each neuron has an array of pointers to a weight matrix that holds all the weights of the network. For a restricted boltzman the pointers will point to the same weights allowing for both the forward and backward propogation and for the weight updates to affect both the forward and backward weights that belong to neurons in adjacent layers.

 If I create two pointers to the same weight value and I change either one of them then this single value is changed. Fantasticaly simple and to think I had only been using pinters to pass arrays between functions. They are so much more powerful for this purpose and fully compatable with the ANSI c Open-CL.

Here is my little test program to prove the point:

#include <stdio.h>
#include <stdlib.h>
#include <iostream>

using namespace std;

class neuron{


public:

double  *Wgt;
double* *LWgt;

void init(){
Wgt   = new double[26];
LWgt = new double*[26];
}

};

int main(){

neuron *node;
node = new neuron[10];

for(int i=0;i<10;i++){
node[i].init();
}

for(int i=0;i<5;i++){
node[i].LWgt[23] = &node[i+5].Wgt[12];
             }

      
node[6].Wgt[12]=50.6987;   //These are the initial values in the Weight matrix

node[8].Wgt[12]=0.999923;

cout<<*node[1].LWgt[23]<<*node[3].LWgt[23]<<"--\n";

*node[1].LWgt[23]=33.234; //Here is the pointer to this matrix see it change
                                              //the value on the Weight matrix like magic

cout<<node[6].Wgt[12]<<"---\n";

    }

Wiring a Neuron Centered Network just became a whole lot easier!

Secrets of Parallelisation using OpenCL - Intels SDK

Well this is exciting with my newly installed Scientific Fedora running on the Acer Aspire One Cloudbook (Low cost parallel computing) I have discovered how to use the N3050 Parallel features of this Intel chipset.

If you'd read my previous post you would know that although having a chipset capable of Paralell processing there are limitations on number precision solveable by using BOOST multiprecision. There are librarys developed to use OpenCL - Intels SDK for writing Parallel Programs. But I prefer not to use Boost - I like to keep things inhouse!

The main problem I had initially was that all my Algo's are written in C++ and use Objects with functions, I needed a way to get the Arrays contained in each Object or across many Objects processed by OpenCL to make each function parallel.

A good example is the Neuron Centered Algo that has each Neuron as a instantiation of the Neuron Class Object. So I did some testing and discovered that OpenCL uses C and will accept Structs which are the C equilivant of C++ Objects or Classes. A simple conversion between an Array of Objects to an Array of Structs allows you to do this easily and pass Object Variables  and Arrays of Object to be processed in Parallel using OpenCL.

A copy of the functions used by each Object in the Hierachy of Objects in your program can then be rewritten in C and listed within a seperate file with the *.cl prefix. This is then loaded into the Kernel together with a definition of each struct for each Object and its Variables.

After condensing some of the routines required to do this I then had a program that could update each Objects variables in parallel using a function listed in this file.

With this technique under my belt I can now examine my Algo and find all the functions and processes within functions that I can list and run in parallel. OpenCL simply needs to know the Struct type declaration at the start.

In the adjoining C++ coded Algorithm a Struct is declared for each object like so

class myclass{

public:

int valuex;
int valuey;
int valuez;

void function_x_y(){         //This is the function that is rewritten externally in functions.cl

valuez = valuex * valuey;

}

};

typedef struct{              //Here is the Struct that defines the Object
int valuex;
int valuey;
int valuez;

}myclass_;

int main(){

myclass* thisclass;                     //Create and Array of Objects
thisclass = new myclass[100]; 

myclass_* p_struct;                   //Create a matching Array of Structs
p_struct = new myclass_[100];

p_struct = (myclass_*)thisclass; //Type Convert Object to Struct


so that when you update in parallel the Struct you also update the Object in Parallel. Fantastico!

The definition of the Struct is made in the Open_CL.cc file and the adjoining C standard functions.cl file

 //Declar Structs for Classes
typedef struct{
int valuex;
int valuey;
int valuez;

}myclass_;

//Example function

__kernel void function_x_y_(__global myclass_* input,long n)
{

int gid = get_global_id(0);
if(gid<n)
input[gid].valuez = input[gid].valuey * input[gid].valuex * 8736;

}

Its early days and I have yet to bring all my Algo's over begining with the ANt and the fast CNN Condor.