LSTM with no learning - recall and count ahead.

Without any training my LSTM remembers for a specified delay and counts ahead it also has a go at predicting the future for some rudimentary time series.

But its not been trained because the wgt updates dont affect this output it does all this off the cuff!

Whats going on. I hope I can fix this.

Oh the memory thing is affected by the training to specify when to recall. I am uploading this doozy for anyone to play with and maybe get it to do the right thing!

This upload remembers stuff you can specify when to recall by seting ahead=1,2,3 steps ahead.

This upload counts ahead.

All with no learning - whats going on!

Here is roughly what Ive implemented:


 Forward Pass

Act_FGate = ft = sigmoid_(Wgt[0]*(Hin+In)+Bias[0],1,0);        //forget gate
Act_IGate = It = sigmoid_(Wgt[1]*(Hin+In)+Bias[1],1,0);        //Include Gate
Ct_= tanh_(Wgt[2]*(Hin+In)+Bias[2],1,0);
Act_CGate = Ct = ft*Ctin+It*Ct_;
                        //Out gate
Act_OGate = Ot = sigmoid_(Wgt[3]*(Hin+In)+Bias[3],1,0);
Hout = Ot * tanh(Ct);                //Outputs

Backward Pass

***Backprop error:

Hout_Err = Out - Hout

Ctin_Err  = Inv_Tanh(Wgt_O * Hout_Err)

Err_FGate = Inv_Sigmoid(Wgt_F * Hout_Err)

Err_IGate = Inv_Sigmoid(Wgt_I * Hout_Err)

Err_CGate = Inv_Tanh(Wgt_C * Hout_Err)

Hin_Err = Err_CGate + Err_IGate + Err_FGate

Next layer down Hout_Err = Hin_Err

***Update Wgts (For each Wgt_F,Wgt_I,Wgt_C,Wgt_O):

WgDelta = (Hin+In)*Err_Gate*Act_Gate*Lrt + Momentum*PreDelta - Decay*PreWgt

PreWgt = Wgt
PreDelta = WgtDelta
Wgt += WgtDelta

Here is the correct psuedo code. Borrowed / Interpreted from a translation online:

Forward

PreAct_FGate = U_FGate*(Hin+In) + W_FGate*Hout(t-1) + V_FGate*C(t-1)
PreAct_IGate = U_IGate*(Hin+In) + W_IGate*Hout(t-1) + V_IGate*C(t-1)

PreAct_CGate = U_CGate*(Hin+In) + W_CGate*Hout(t-1)

Act_IGate = Sigmoid(PreAct_IGate)
Act_FGate = Sigmoid(PreAct_FGate)
Ct_       = LSigmoid(PreAct_CGate)

Ct = Act_FGate * Ct-1 + Act_IGate * Ct_

PreAct_OGate = U_CGate*(Hin+In) + W_CGate*Hout(t-1) + V_OGate*C(t)

Act_OGate = Sigmoid(Act_OGate)

Hout = Act_OGate *tanh(Ct)



Backpass:


Hin_Err = Sum U_Gate*Err_Gate(t) + Sum W_Gate*Err_Gate(t+1)   <---- For Layer Above

Err_OGate(t) = Inv_Sig(PreAct_OGate(t))*tanh(Ct)*Hout_Err(t)

Ct_Err(t) = Act_OGate*Inv_Tanh(PreAct_CGate(t))*Hout_Err(t)

+ Act_FGate(t+1)*Ct_Err(t+1) + WgtV_I*Err_IGate(t+1)

+ WgtV_F*Err_FGate(t+1) + WgtV_O*Err_OGate(t)

Err_CGate = Inv_LSig(PreAct_CGate(t))*Act_IGate(t)*Ct_Err(t)

Err_FGate = Inv_Sig(PreAct_FGate(t))*Act_CGate(t-1)*Ct_Err(t)

Err_IGate = Inv_Sig(PreAct_IGate(t))*LSig(PreAct_CGate)*Ct_Err(t)


*(Hout_Err(prev) = Hin_Err
  Hout_Err = Out - Hout   )
 
*(Three Activation Functions
Logistic Sigmoid, Tanh, Sigmoid)

*(Three Wgts - U,W,V)

Cannot get this to work yet LSTM!

#include <stdio.h>
#include <stdlib.h>
#include <iostream>
#include <math.h>
#include <vector>
#include <algorithm>    // std::transform
#include <functional>   // std::plus

using namespace std;

double function_x(double sum,int type, double y);
double function_y(double sum,int type, double y);
double sig_vec(double sum);
double tanh_vec(double sum);
double invtanh_vec(double sum);
double invsig_vec(double sum);

class ltsm_module{
public:
double *Wgt;    //4 x Weights for each gate
double Out;    //Finall Output
double Hout;    //Recurrent Output (Hin for next Neuron)
double Hin;
double In;
double Ctin;
double *Err;    //2 x Error from Hout and Out 2 x Error at Hin/X and Ctin
double *Bias;   //Each of the bias for each gate
double Lrt,Momentum,Decay;
//Vector Inputs/Output
int Input_Size;
int Wgt_Size;
int wgt_scalar;
vector<double> Out_Vec;
vector<double> Hout_Vec;
vector<double> Hin_Vec;
vector<double> In_Vec;        //Input Vector
vector<double> Ctin_Vec;
vector<double> Ct_Vec;      //Cell Activation records Ct-1   
//Vector Error
vector<double> Ctin_Err;
vector<double> Hin_Err;
vector<double> Hout_Err;
vector<double> Out_Err;
//Act
vector<double> Act_F;
vector<double> Act_I;
vector<double> Act_C;
vector<double> Act_O;
//Vector Wgts
vector<double> Wgt_F;
vector<double> Wgt_I;
vector<double> Wgt_C;
vector<double> Wgt_O;
//Vector PreDelta
vector<double> PreDlta_F;
vector<double> PreDlta_I;
vector<double> PreDlta_C;
vector<double> PreDlta_O;
//Vector PreWgts
vector<double> PreWgt_F;
vector<double> PreWgt_I;
vector<double> PreWgt_C;
vector<double> PreWgt_O;
//Error Vect
vector<double> Err_FGate;
vector<double> Err_IGate;
vector<double> Err_CGate;

        //Update weights as though they are 3 weights entering
        //a neuron using previous weight values / error and
        //Sub activation of previous unit as
        //Output Wgt uses Hout Error Over InvSigmoid
        //Input Wgt  uses Out Error Over ""
        //Forget Wgt uses Out Error  Over ""
       

double (*m_pointertofunction)(double,int,double);  //Pointer to Func

void init_module(){

Lrt=0.095;
Decay=0.000005;
Momentum=0.005;
wgt_scalar=0;
Wgt_Size = 4;
Wgt = new double[4];
Bias = new double[4];


for(int i=0;i<4;i++){
Wgt[i] = double(rand()/(RAND_MAX + 1.0));
Bias[i] = double(rand()/(RAND_MAX + 1.0));
            }

/*Memory to Module Vector*/
Hin_Vec.resize(Input_Size);
In_Vec.resize(Input_Size);
Ctin_Vec.resize(Input_Size);
Ct_Vec.resize(Input_Size);

Ctin_Err.resize(Input_Size);
Hin_Err.resize(Input_Size);
Hout_Err.resize(Input_Size);
Out_Err.resize(Input_Size);
Wgt_F.resize(Wgt_Size);
Wgt_I.resize(Wgt_Size);
Wgt_C.resize(Wgt_Size);
Wgt_O.resize(Wgt_Size);
Act_F.resize(Wgt_Size);
Act_I.resize(Wgt_Size);
Act_C.resize(Wgt_Size);
Act_O.resize(Wgt_Size);
PreWgt_F.resize(Wgt_Size);
PreWgt_I.resize(Wgt_Size);
PreWgt_C.resize(Wgt_Size);
PreWgt_O.resize(Wgt_Size);
PreDlta_F.resize(Wgt_Size);
PreDlta_I.resize(Wgt_Size);
PreDlta_C.resize(Wgt_Size);
PreDlta_O.resize(Wgt_Size);
Err_IGate.resize(Input_Size);
Err_CGate.resize(Input_Size);
Err_FGate.resize(Input_Size);
Ctin_Err.resize(Input_Size);

Init_Wgts(Wgt_F);
Init_Wgts(Wgt_I);
Init_Wgts(Wgt_C);
Init_Wgts(Wgt_O);                       
}

void update_module(double Hin_,double In_,double Ctin_){
double ft,Ct,It,Ct_,Ot;
double (*sigmoid_)(double,int,double);
double (*tanh_)(double,int,double);
tanh_ = function_y;
sigmoid_ = function_x;
Hin = Hin_;
In = In_;
Ctin= Ctin_;
ft = sigmoid_(Wgt[0]*(Hin+In)+Bias[0],1,0);        //forget gate
It = sigmoid_(Wgt[1]*(Hin+In)+Bias[1],1,0);        //Include Gate
Ct_= tanh_(Wgt[2]*(Hin+In)+Bias[2],1,0);
Ct = ft*Ctin+It*Ct_;
                        //Out gate
Ot = sigmoid_(Wgt[3]*(Hin+In)+Bias[3],1,0);
Hout = Ot * tanh(Ct);                //Outputs
Out = Ct;
}

void update_module_vec(vector<double> Hin_,vector<double> In_,vector<double> Ctin_){
vector<double> ft,Ct,It,Ct_,Ot;

Hin_Vec = Hin_;
In_Vec = In_;
Ctin_Vec= Ctin_;

//Cycle through each Vec apply sigmoid
vector<double>::iterator it;
vector<double> Sum_Vec;
vector<double> Sum_Vec_;
Sum_Vec.resize(In_.size());
Sum_Vec_.resize(In_.size());

//Forget Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt

if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[0]));
          }else{
Sum_Vec_ = Apply_Conv(Wgt_F,Sum_Vec);
             }
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[0]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
ft = Sum_Vec_;
Act_F = Sum_Vec_;

//Include Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[1]));
        }else{
Sum_Vec_ = Apply_Conv(Wgt_I,Sum_Vec);       
        }

//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[1]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
It = Sum_Vec_;
Act_I = Sum_Vec_;

//Out gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[3]));
          }else{
Sum_Vec_ = Apply_Conv(Wgt_C,Sum_Vec);             
          }
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[3]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
Ot = Sum_Vec;
Act_C = Sum_Vec;


//Ct Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt

if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[2]));
        }else{
Sum_Vec_ = Apply_Conv(Wgt_O,Sum_Vec);       
        }

//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[2]));
//Out_Vec =Sum_Vec_;
//Apply Tanh
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),tanh_vec);
Ct_ = Sum_Vec_;   //In online this is At it is the new candidate state
Act_C = Sum_Vec_;

//Multiply ft * Ct-1 //Forget the previous state Ct-1 (Ct_Vec)
transform(Ctin_Vec.begin(),Ctin_Vec.end(),ft.begin(),Sum_Vec.begin(),multiplies<double>());
//Multiply It * Ct_
transform(Ct_.begin(),Ct_.end(),It.begin(),Sum_Vec_.begin(),multiplies<double>());
//Calc Ct
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),plus<double>());
Ct=Sum_Vec;
Ct_Vec=Sum_Vec;

//Apply Tanh for Hout
transform(Ct.begin(),Ct.end(),Sum_Vec.begin(),tanh_vec);
//Calc Hout
transform(Sum_Vec.begin(),Sum_Vec.end(),Ot.begin(),Sum_Vec.begin(),multiplies<double>());


Hout_Vec = Sum_Vec;
Out_Vec  = Ct;
}

///Update_Module_Vec() New
//Backprop_Update_Out() New
//Compute Deltas for each Wgt and Delta for Ct
//Uses Ct-1(Previous Cell state) Ct(Current cell state

void error_module_vec(){  //Calculate Error at Hin/X and Ctin using Error at Out and Hout
//Error at Hout = Err_Ht + Err_Hout (Up and Across)
//Err_Ht = Err_Hin/X (Up)
vector<double> Sum_Vec;
vector<double> Sum_Vec_;

Sum_Vec.resize(Input_Size);
Sum_Vec_.resize(Input_Size);


//Multiply Wgt_Out * Hout_Err
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[3]));
        }else{
Sum_Vec = Apply_Conv(Wgt_O,Hout_Err);       
        }
//Inverse Tanh
transform(Sum_Vec.begin(),Sum_Vec.end(),Ctin_Err.begin(),invtanh_vec);

//Multiply Wgt_f * Out_Err
//Change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[0]));
          }else{
Sum_Vec = Apply_Conv(Wgt_F,Hout_Err);         
          }
//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_FGate.begin(),invsig_vec);

//**change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[1]));
        }else{
Sum_Vec = Apply_Conv(Wgt_I,Hout_Err);           
        }

//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_IGate.begin(),invsig_vec);

//**change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[2]));
        }else{
Sum_Vec = Apply_Conv(Wgt_C,Hout_Err);       
        }
//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_CGate.begin(),invtanh_vec);

//Add All 3 Errors
transform(Err_CGate.begin(),Err_CGate.end(),Err_IGate.begin(),Sum_Vec.begin(),plus<double>());
transform(Err_FGate.begin(),Err_FGate.end(),Sum_Vec.begin(),Sum_Vec_.begin(),plus<double>());
Hin_Err = Sum_Vec_;


}

void Update_Wgts(vector<double> &Wgt,vector<double> &PreDlta,vector<double> &PreWgt, vector<double> &Err,vector<double> &Act){
vector<double> delta(Input_Size);
vector<double> Sum_Vec;
vector<double> Sum_Vec_;
Sum_Vec.resize(Input_Size);
Sum_Vec_.resize(Input_Size);

//Add Hin to In Vector Add changed Hin to Ctin
transform(Hin_Vec.begin(),Hin_Vec.end(),In_Vec.begin(),Sum_Vec.begin(),plus<double>());
//Multiply Act_in * Error
transform(Err.begin(),Err.end(),Act.begin(),Sum_Vec_.begin(),multiplies<double>());
//Lrate
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Lrt));


//Momentum - Convolve PreDelta onto all 1's Sum_Vec **********************
//transform(PreDlta.begin(),PreDlta.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Momentum));

fill(Sum_Vec.begin(),Sum_Vec.end(),1);
Apply_Conv(PreDlta,Sum_Vec);
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Momentum));


//Add Momentum and Lrate
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),plus<double>());

//Decay - Convolve PreWgt onto all 1's Sum_Vec     **********************
fill(Sum_Vec_.begin(),Sum_Vec_.end(),1);
Apply_Conv(PreWgt,Sum_Vec);
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Decay));
//transform(PreWgt.begin(),PreWgt.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Decay));

//Lrate *Err * Actin + Momentum*PreDlta - Decay*PreWgt
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),minus<double>());
PreWgt = Wgt;
PreDlta = Sum_Vec;
//Update Wgt Matrix           
Apply_DeConv(Wgt,Sum_Vec);           
//fill(Wgt.begin(),Wgt.end(),1);           
           
                              }
void Backprop(){
error_module_vec();

Update_Wgts(Wgt_F,PreDlta_F,PreWgt_F,Err_FGate,Act_F);
Update_Wgts(Wgt_I,PreDlta_I,PreWgt_I,Err_IGate,Act_I);
Update_Wgts(Wgt_C,PreDlta_C,PreWgt_C,Err_CGate,Act_C);
Update_Wgts(Wgt_O,PreDlta_O,PreWgt_O,Ctin_Err,Act_O);

}


void Init_Wgts(vector<double> &Wgt){

for(int i=0;i<Wgt_Size;i++){

Wgt[i] = double(rand()/(RAND_MAX+1.0));
}

}

vector<double> Apply_Conv(vector<double> &Wgt_Conv, vector<double> &In_Vec){
//Apply Wgt Convolution Vector to an Input Vector or Err_Vec
vector<double> Out_Vec(Input_Size);

for(int i=0;i<In_Vec.size();i++){
for(int j=0;j<Wgt_Conv.size();j++){
Out_Vec[i]=Wgt_Conv[j]*In_Vec[i];
}
return Out_Vec;
}
                                }
void Apply_DeConv(vector<double> &Wgt_Conv,vector<double> &Err_Vec){
//Change Wgt Convolution with Err_Vec use to compute Wgt Update
int i=0;
int stoch=1;

while(i<Err_Vec.size()){

for(int j=0;j<Wgt_Conv.size();j++){

Wgt_Conv[j] += Err_Vec[i]/10;

//if(stoch==1&&rand()%2==1){        //Add a stochastic element to Weight Update
i++;
//            }
                } }

}

void print_module_Err(){
cout<<"Print Error\n";
cout<<"Hout:\n";

for(vector<double>::iterator it=Hout_Vec.begin(); it!=Hout_Vec.end(); ++it){
cout<<" "<<*it<<"  ";
}

cout<<"Size of FERR = "<<Err_FGate.size();

cout<<"***********HOut_Err Vec***********\n";
for(vector<double>::iterator it=Hout_Err.begin(); it!=Hout_Err.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********FERR************\n";
for(vector<double>::iterator it=Err_FGate.begin(); it!=Err_FGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********IERR********\n";
for(vector<double>::iterator it=Err_IGate.begin(); it!=Err_IGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********CERR************\n";
for(vector<double>::iterator it=Err_CGate.begin(); it!=Err_CGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}

}

void print_module_wgts(){

cout<<"Print Wgts\n";
cout<<"***********FWgts************\n";
for(vector<double>::iterator it=Wgt_F.begin(); it!=Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********IWgts************\n";
for(vector<double>::iterator it=Wgt_I.begin(); it!=Wgt_I.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********CWgts************\n";
for(vector<double>::iterator it=Wgt_C.begin(); it!=Wgt_C.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********OWgts************\n";
for(vector<double>::iterator it=Wgt_O.begin(); it!=Wgt_O.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}

}




};


int main(){
cout<<"Hello!\n";

vector<double> Hin_={0,0,0,0};
vector<double> Ctin_={0,0,0,0};
vector<double> In_ = {.1,.004,.6555,6.55};

vector<vector<double>> In_Vec(20,vector<double>(20));

In_Vec[0] = {10};//{.1,.004,.6555,6.55};
In_Vec[1] = {11};//{.7774,.3956,1.76,.006};
In_Vec[2] = {12};//{9.111,.12,.0102,2.96};
In_Vec[3] = {13};//{5.99,.204,6.0001,3.094};
In_Vec[4] = {14};//{2.4965,.694,0.5,22.003};

In_Vec[5] = {15};//{.1,.004,.6555,6.55};
In_Vec[6] = {16};//{.7774,.3956,1.76,.006};
In_Vec[7] = {17};//{9.111,.12,.0102,2.96};
In_Vec[8] = {18};//{5.99,.204,6.0001,3.094};
In_Vec[9] = {19};//{2.4965,.694,0.5,22.003};

//vector<double> Hin_={0};
//vector<double> In_ = {9.55};

ltsm_module** mymod_;
mymod_ = new ltsm_module*[28];


for(int i=0;i<28;i++){
mymod_[i] = new ltsm_module;
mymod_[i]->Input_Size=1;
mymod_[i]->init_module();

        }
int kint=0;   
int vint=0;   
int vint_=0;
int ahead=0;


for(int i=0;i<10;i++){
cout<<"***********Old Wgts************\n";
for(vector<double>::iterator it=mymod_[i]->Wgt_F.begin(); it!=mymod_[i]->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"*******\n";            }





for(int iter=0;iter<5000;iter++){
kint++;
if(kint==9){kint=0;}

mymod_[0]->Hin_Vec = {0};
mymod_[0]->Ctin_Vec= {0};
mymod_[10]->Hin_Vec = {0};
mymod_[10]->Ctin_Vec= {0};



//Feedforward one row of inputs
for(int i=0;i<9;i++){
vint=kint+i;
if(vint>=9){vint=0+vint-9;}

mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,In_Vec[vint],mymod_[i]->Ctin_Vec);

//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;

            }
           
//Add another layer
for(int i=9;i<18;i++){

mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,mymod_[i-9]->Hout_Vec,mymod_[i]->Ctin_Vec);

//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;


            }
           
           
           
           
//Feedback Create Error from identity for test

for(int i=18;i>9;--i){
vint=kint+i-9;
if(vint>=9){vint=0+vint-9;}
vint_=vint+ahead;
if(vint_>=9){vint_=0+vint_-9;}

if(i>9){     //Output Neurons Err
transform(mymod_[i]->Hout_Vec.begin(),mymod_[i]->Hout_Vec.end(),In_Vec[vint_].begin(),mymod_[i]->Hout_Err.begin(),minus<double>());
    }
//mymod_[i]->print_module_Err();

mymod_[i]->Backprop();        //Create Hin_Err and change weights
            }
           

for(int i=9;i>-1;--i){
vint=kint+i;
if(vint>=9){vint=0+vint-9;}
vint_=vint+5;
if(vint_>=9){vint_=0+vint_-9;}

if(i>-1){     //Output Neurons Err
mymod_[i]->Hout_Err = mymod_[i+10]->Hin_Err;

}
//mymod_[i]->print_module_Err();

mymod_[i]->Backprop();        //Create Hin_Err and change weights
            }

}               


//Identity

ltsm_module* mymod;
mymod = new ltsm_module;

mymod->Input_Size=4;
mymod->init_module();

mymod->Hin_Vec={0,0,0,0};
mymod->Ctin_Vec={0,0,0,0};

mymod->update_module_vec(mymod->Hin_Vec,In_,mymod->Ctin_Vec);

//Create Error from identity for test
transform(mymod->Out_Vec.begin(),mymod->Out_Vec.end(),In_.begin(),mymod->Out_Err.begin(),minus<double>());
transform(mymod->Hout_Vec.begin(),mymod->Hout_Vec.end(),In_.begin(),mymod->Hout_Err.begin(),minus<double>());

//mymod->Hout_Err = {.99,.837,.455,1.22};
//mymod->Out_Err  = {.99,.837,.455,1.22};

for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}

cout<<"*********************************************************\n";

cout<<"**********Out Vec************\n";
for(vector<double>::iterator it=mymod->Out_Vec.begin(); it!=mymod->Out_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}

cout<<"***********HOut Vec***********\n";
for(vector<double>::iterator it=mymod->Hout_Vec.begin(); it!=mymod->Hout_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"Training*************************\n";


for(int i=0;i<100;i++){

//Create Error from identity for test
transform(mymod->Out_Vec.begin(),mymod->Out_Vec.end(),In_.begin(),mymod->Out_Err.begin(),minus<double>());
transform(mymod->Hout_Vec.begin(),mymod->Hout_Vec.end(),In_.begin(),mymod->Hout_Err.begin(),minus<double>());

//mymod->print_module_Err();

mymod->Ctin_Vec = mymod->Out_Vec;    //t+1
mymod->Hin_Vec  = mymod->Hout_Vec;

mymod->update_module_vec(mymod->Hin_Vec,In_,mymod->Ctin_Vec);

mymod->Backprop();

            }
           
           
           
           
           
cout<<"**********Out Vec************\n";
for(vector<double>::iterator it=mymod->Out_Vec.begin(); it!=mymod->Out_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}

cout<<"***********HOut Vec***********\n";
for(vector<double>::iterator it=mymod->Hout_Vec.begin(); it!=mymod->Hout_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}


cout<<"***********HOut Vec***********\n";
mymod_[0]->Hin_Vec = {0};
mymod_[0]->Ctin_Vec= {0};
mymod_[10]->Hin_Vec = {0};
mymod_[10]->Ctin_Vec= {0};


//Feedforward input layer
for(int i=0;i<9;i++){
vint=i;

mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,In_Vec[vint],mymod_[i]->Ctin_Vec);

mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;

}


//Feedforward Output layer
for(int i=9;i<18;i++){
vint=i-9+ahead;
if(vint>=9){vint=0+vint-9;}

mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,mymod_[i-9]->Hout_Vec,mymod_[i]->Ctin_Vec);

transform(mymod_[i]->Hout_Vec.begin(),mymod_[i]->Hout_Vec.end(),In_Vec[vint].begin(),mymod_[i]->Hout_Err.begin(),minus<double>());

//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;

for(vector<double>::iterator it=mymod_[i]->Hout_Vec.begin(); it!=mymod_[i]->Hout_Vec.end(); ++it){
cout<<" "<<*it<<"  ";
}

for(vector<double>::iterator it=In_Vec[vint].begin(); it!=In_Vec[vint].end(); ++it){
cout<<" "<<*it<<"  ";
}

for(vector<double>::iterator it=mymod_[i]->Out_Vec.begin(); it!=mymod_[i]->Out_Vec.end(); ++it){
cout<<" "<<*it<<"  ";
}
for(vector<double>::iterator it=mymod_[i]->Hout_Err.begin(); it!=mymod_[i]->Hout_Err.end(); ++it){
cout<<" "<<*it<<"  ";
}

cout<<"******\n";            }

for(int i=0;i<10;i++){
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod_[i]->Wgt_F.begin(); it!=mymod_[i]->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"*******\n";            }


}









double invtanh_vec(double sum){

return function_y(sum,-1,0);

}

double invsig_vec(double sum){

return function_x(sum,-1,0);
}

double tanh_vec(double sum){

//double sum = Vec1+Vec2;

return function_y(sum,1,0);

}

double sig_vec(double sum){

//double sum = Vec1+Vec2;

return function_x(sum,1,0);

}

double function_y(double sum,int type, double y){  //input function
double rand_=0;
//sum=sum/1000;

if(type==1){
rand_ = ((double)(rand()%1000))/10000;

sum = tanh(sum)+rand_;
//sum = (2/(1+exp(-2*sum)))-1;        //Kick Ass


}
if(type==-1){

//sum = (1-pow(tanh(sum),2))*sum;
sum = 1-(sum*sum)/2;

}

return sum;

}

double function_x(double sum,int type,double y){  //activation function for all hidden layer neurons

double sigmoid,temp;
double rand_=0;

if(type==1){
//rand_ = ((double)(rand()%1000))/1000000;

if(sum>0.5){                    //this is actually hardlim not sigmoid
 sigmoid =  0.5;}else{
 if(sum<-0.5){
 sigmoid = 0;}
 else{
 sigmoid = 1/(1+exp((double) -sum));
     }
}
 sigmoid = sigmoid+rand_;

        }
if(type==-1){


temp = 1/(1+exp((double) -y));

//sigmoid = (0.25-(temp*temp) )* sum;
//sigmoid =  y*(1-y) * sum;        //derivative sigmoid
sigmoid = sum*(1-sum);

//sigmoid = (1-pow(tanh(y),2))*sum;


        }
return sigmoid;


}

Long Term Short Memory LTSM or FICO for short.

The anacronym FICO stands for Forget Include Copy Out these are the gates for the LTSM module based on the infamous LSTM module.

These are neurons with gates controlled by weights with sigmoid activation functions. An Input(Xt) is combined with the previous output of the last cell (Hin) or Hout at t-1. The gates determine whether the previous cells Activation (Ctin) is Forgotten or not (F - Forget Gate). They also determine whether the Input combine with Hin is included in computing the Activation(I - Include gate) of the unit and whether the Activation is passed to Hout (O - Output gate).

1. Include Previous Output?
ht-1 + Xt -> sigmoid -> forget 1/0 -> Adds PreOutput to Output(Ct)

ft = sigmoid(Wf.[ht-1,xt] + bf

2. Include new Input?
ht-1 + xt -> sigmoid -> include 1/0 -> Adds tanh(Input+PreHidden) to Output(Ct)

It = sigmoid(Wi.[ht-1,xt] + bi
~Ct = tanh(Wc.[ht-1,xt] +bc)
Ct = ft*Ct-1 + it*~Ct

3. Include Output as Hidden?
ht-1 + xt -> sigmoid -> include 1/0 -> Adds tanh(Output) to Hidden(ht)

Ot = sigmoid(Wo.[ht-1,xt] + bo
ht = Ot * tanh(Ct)

Four Weights: Wf  Wi  Wc  Wo  or F.I.C.O for short!

I connected everything up in this way and got it to learn an Identity function for one module and I am poised to use it on a time series which I have read is a good application for the LSTM.

I have found however that for Identity - a learnt mapping from input to output sigmoids insteads of the tanh functions work better.

Also for the Identity mapping test I fed back the previous Outputs to their Inputs Cout -> Cin  Hout -> Hin for Xt = X.

We shall see if it works out of the box for time series will it learn how to count and will it predict a sequence of numbers.

I have written the whole unit using vectors making it suitable for taking inputs and producing outputs for a convolutional process. Making it perhaps suitable for image processing.

The std::vector object is simply a faster and easier way of processing 2d arrays.

Also I have made the Weight arrays flexible to being 1d or 2d.  So its all go for Long Term Short Memeory!!

CNN trained on practical surveilance targets.

A stationary surveilance camera is position opposite the exit to a busy station its goal is to locate and track individuals matching a target group and behaving suspicously.

Each individual is given the task of leaving the station carrying a concealed item without being seen by a a set of cameras placed throughout the station.

Using an advanced form of deep learning a Convolutional Neural Network dynamically picks out those from the group who are trying to evade detection.

The group is instructed not to act in anyway to betray their location to surveilance.

The CNN correclty identify's isolates and tracks each individual using advance image processing and deep learning.

The algorithm first works by reading the body language of any individual moving differently to the other members of the public. It then uses a rule of thumb based on its previous training to determine whether they are a valid target.

Using a purpose built database it then matches the individual using rudimentary face recognition.

Source : Reuters

Will the Convolutor be a GAN - Generative Adversarial Network

My CNN Condor has been coming on leaps and bounds trained on MINST data performs admireably only after 205 iterations of training.

Fig 1 These are the Minst images after successive discriminating filters with preloaded weights.

However I cannot get it to perform as a GAN - Generative Adversarial Network.

I have created purpose built functions that assemble the GAN's elements -

Generator - A series of RGB Convolution layers in the literature they are of varying size and donot have a Pool layer or an MLP layer.

Discriminator - Well this is just the CNN pretrained with its preloaded weights which takes as its input the output of the generator - and feeds back the error.

I have tested it with an MLP layer and without but I have yet to vary the size of each layer.

What I get with training is simply an image that looks like an untuned TV with the contrast on high. Its dark mush but it may yet deliver some results after I vary the

size of the layers and perhaps play around with their transfer functions.

Interestingly the MLP included produces the same result even after training.

Random Code -> MLP1 -> Layer 1...3 -> Output Image 100x100

->Input to CNN -> MLP2 Generate Error -> Feedback Error to MLP1 and Layer 1..3

Nice try.

Making some Noise

A new and exciting project is now underway. Negative noise.

I had some time back been experimenting with combining oscilators to produce a chaotic signal and recalled being able to produce a signal that had a negative voltage.

I am not sure how I did it and frustrated having dismantled and forgotten how the original circuit worked. All I recall was that there were two oscillators and the output was passed through my sound card to the oscilloscope.

The soundcard (not being designed for this purpose) although revealling some interesting waveforms eventually caused the soudcard to develop a glitch as the current was too high even though it had passed through a filter.

I then went and brought an oscilloscope which uses the ps2302 usb to serial this works lovely.

I then spent some time trying to build a circuit that could replicate these previous attempts and after many failures I suddenly hit upon the right combination of components. Now I can use a variety of different transistor types to create a noise signal that has both a positive and a negative component. After passing it through another filter I can reveal more of the signal and change its underlying form as a high pass filter its really interesting that you can do so much with just a few components and this circuit design is flexible to use different  transistor types in order to create the noise and then amplify the signal using an op-amp.

I would now like to take it to the next stage and make the signal
audio with more amplification and create an A2D convertor so that the noisy signal can be read by a really LITE version of my Mlp-on-a-chip.

I'd like the Mlp-on-a-chip to read the noise and perform a time series prediction. It may be a lot to ask and it would be enough to get it to distinguish between the different noise types produced by each transistor. An ambitious project that I will hopefully begin with making the noise audible through a pair of headphones and an amplifier circuit addon that I must build.

I shall upload the circuit for my Negative Noise here for anyone interested.

For ages I could only build a working Multivibrator and still cannot get a good Sine Wave. But now I have Negative Noise!


Notes to self use Noisey Hyperparameters as-per Noisey transfer function.

Links within Links - Pointers the journey continues. (Wiring an RBM)

Having a renewed interest in using pointers for other than passing arrays and functions between functions. I am now using them to create a Link matrix that preserves symetry and equality between weights from i to j and from j to i.

I first decied to have each of my weights for each neuron pointing to a weight matrix. Having setup my array of pointers for each neuron I then had to award it the right amount of memory as * **Array needs *** much memory! and not this much * **!

I then had the problem of pointing each Weight to the same Weight as its symetric opposite and I found the simplest way was to turn the Weight matrix into another array of pointers that point to their own symetric weight values.

Now I am all sorted and everything points to something else and I feel complete I feel whole!

Its all connected and I feel great!

Neuron -> Weight Pointer[n] -> Weight Matrix [n][m] -> Weight Matrix [m][n]

Easy!

Now I need only write to and read from Each Neuron for updates and learning etc. In fact I can just isolate one neuron and run everything on that one neuron.

Another triumpth for the Neuron Centered Algo. [WARNING Weight Vectors STUNT GROWTH!]

The Wonder of Pointers and their use in Building Neuron Centered Algo's.

Well I have been looking at the wiring on one of my nets and I found it just wasnt up to the requirement of forward and backward updates. I need it to produce values at the input and the output nodes - this is for CD - contrastive divergence. A clever learning algorithm that uses samples taken at various phases :

Phase 1 clamp inputs generate hidden values (Sample h given v)
Phase 2 unclamp inputs and generate inputs  (Sample v' given h)
Phase 3 clamp to v' and generate h'

This process is repeated and the values are used to compute weight changes as with backpropogation.

My neuron centered design needed to update itself using weights that had formally been assigned to the neurons in successive layers.

The answer to this was the use of pointers as weights. I had tried using a function to equate the symetrical links between neurons but ofcourse all I needed ws these magic pointers. It was as though they were designed for this purpose.

Each neuron has an array of pointers to a weight matrix that holds all the weights of the network. For a restricted boltzman the pointers will point to the same weights allowing for both the forward and backward propogation and for the weight updates to affect both the forward and backward weights that belong to neurons in adjacent layers.

 If I create two pointers to the same weight value and I change either one of them then this single value is changed. Fantasticaly simple and to think I had only been using pinters to pass arrays between functions. They are so much more powerful for this purpose and fully compatable with the ANSI c Open-CL.

Here is my little test program to prove the point:

#include <stdio.h>
#include <stdlib.h>
#include <iostream>

using namespace std;

class neuron{


public:

double  *Wgt;
double* *LWgt;

void init(){
Wgt   = new double[26];
LWgt = new double*[26];
}

};

int main(){

neuron *node;
node = new neuron[10];

for(int i=0;i<10;i++){
node[i].init();
}

for(int i=0;i<5;i++){
node[i].LWgt[23] = &node[i+5].Wgt[12];
             }

      
node[6].Wgt[12]=50.6987;   //These are the initial values in the Weight matrix

node[8].Wgt[12]=0.999923;

cout<<*node[1].LWgt[23]<<*node[3].LWgt[23]<<"--\n";

*node[1].LWgt[23]=33.234; //Here is the pointer to this matrix see it change
                                              //the value on the Weight matrix like magic

cout<<node[6].Wgt[12]<<"---\n";

    }

Wiring a Neuron Centered Network just became a whole lot easier!

Secrets of Parallelisation using OpenCL - Intels SDK

Well this is exciting with my newly installed Scientific Fedora running on the Acer Aspire One Cloudbook (Low cost parallel computing) I have discovered how to use the N3050 Parallel features of this Intel chipset.

If you'd read my previous post you would know that although having a chipset capable of Paralell processing there are limitations on number precision solveable by using BOOST multiprecision. There are librarys developed to use OpenCL - Intels SDK for writing Parallel Programs. But I prefer not to use Boost - I like to keep things inhouse!

The main problem I had initially was that all my Algo's are written in C++ and use Objects with functions, I needed a way to get the Arrays contained in each Object or across many Objects processed by OpenCL to make each function parallel.

A good example is the Neuron Centered Algo that has each Neuron as a instantiation of the Neuron Class Object. So I did some testing and discovered that OpenCL uses C and will accept Structs which are the C equilivant of C++ Objects or Classes. A simple conversion between an Array of Objects to an Array of Structs allows you to do this easily and pass Object Variables  and Arrays of Object to be processed in Parallel using OpenCL.

A copy of the functions used by each Object in the Hierachy of Objects in your program can then be rewritten in C and listed within a seperate file with the *.cl prefix. This is then loaded into the Kernel together with a definition of each struct for each Object and its Variables.

After condensing some of the routines required to do this I then had a program that could update each Objects variables in parallel using a function listed in this file.

With this technique under my belt I can now examine my Algo and find all the functions and processes within functions that I can list and run in parallel. OpenCL simply needs to know the Struct type declaration at the start.

In the adjoining C++ coded Algorithm a Struct is declared for each object like so

class myclass{

public:

int valuex;
int valuey;
int valuez;

void function_x_y(){         //This is the function that is rewritten externally in functions.cl

valuez = valuex * valuey;

}

};

typedef struct{              //Here is the Struct that defines the Object
int valuex;
int valuey;
int valuez;

}myclass_;

int main(){

myclass* thisclass;                     //Create and Array of Objects
thisclass = new myclass[100]; 

myclass_* p_struct;                   //Create a matching Array of Structs
p_struct = new myclass_[100];

p_struct = (myclass_*)thisclass; //Type Convert Object to Struct


so that when you update in parallel the Struct you also update the Object in Parallel. Fantastico!

The definition of the Struct is made in the Open_CL.cc file and the adjoining C standard functions.cl file

 //Declar Structs for Classes
typedef struct{
int valuex;
int valuey;
int valuez;

}myclass_;

//Example function

__kernel void function_x_y_(__global myclass_* input,long n)
{

int gid = get_global_id(0);
if(gid<n)
input[gid].valuez = input[gid].valuey * input[gid].valuex * 8736;

}

Its early days and I have yet to bring all my Algo's over begining with the ANt and the fast CNN Condor.

Running Fedora Core on Acer Aspire One Cloudbook 11

Its new its swish and it costs less than a new bike. Yes thats right the Acer Aspire One Cloudbook was made for Linux.

Heres how to do it.

1. Burn a copy of Fedora Core to a USB using RUFUS.
2. Boot from F12 the USB.
3. Erase the original Disk and install a fresh OS.
4. Set Password in BIOS and add Grub.iefi file to Database

Reboot

Only downside is as I now discovered it has limitations on floating point precision. So very small numbers that work on my Toshiba dont work on the netbook. Possible workaround is to use BOOST.

Other than that everything else works very well and it is way swish and fast as it should be running the N3050 chipset it is able to do parallel processing - (The whole point of me getting this netbook).

I installed OpenCL with little difficulty. Intels SDK for building Parallel processing into your programs.

1. Download OpenCL SDK for Linux
2. Extract the CPU Librarys from the RPM using RPM2CPIO
3. Install the Librarys to your machine and include them in the /etc/opencl/vendors list.
4. Install the headers and the base via dnf
5. Run the examples

Now all I need to do is figure out how it works!

PS : Be warned though it does require the most recent Fedora as Older versions wont recognise the Solid State disc at boot time.

The quest for a Pure Virtual economy?

A virtual economy based entirely in cyberspace bears a striking resemblence to how we are living today.

People perhaps without knowing are infact already completely emmersed in an artifical world of ecommerce connected and interconnected by highspeed information networks.

It wouldnt take much to turn this world entirely virtual. And this was infact what we did with the first space habitat - alpha 1.

Those living and working in space occupied their time within a virtual world of hyper-commerce, trade and advertising, sales and negiotiations including recruitment and personal management was all incorporated into the avatars that already existed within the internet - now a precursor to this newly evolved form of society.

AI could replace some of the more administrative roles and when possible and when trained correctly could mimick many of the functions that required a human personality.

Those no longer required onboard Alpha one could enjoy a lifestyle afforded by their AI counterparts or work shadows whilst the many newly recruited VR entrepreneurs and socialites would continue to push the envolope in what could not be acheived by machines. They negotiated transactions for mineral resource aquasitions the transport of food, air and materials aswell as distribution agents, manufacturers and the like some of which where entirely robotic some were entire industry's with millions of workers all performing technical functions throughout the new worlds.

It is a starting point that those that can communicate effectively and manage resources and people remotely aswell as those that can buy sell and serve to a community living entirely in a virtual reality would be the first to enjoy a life aboard a new age city in space.

However those that would build it and maintain it and push the limits of space dwelling and exploring to its limits would necessarily be chosen carefully and not become the concern of these virtual lifers.

Space dwelling is in fact a perfect place for virtual lifes to prosper without limits. Technology now possible to allow freedom of movement and the interconnectivity of a virtual world is perfect for those that are in a majority living entirely within a virtual machine laden world.

From Bed to tablet
From Sofa to TV
From house to gym
From house to car
From Car to plane
From Everywhere to phone Everyone   --> Entirely virtual
From Person to Person only a matter of time before this can become Virtual

But ofcourse our working world neednt be about face to face contact and for the most part it isnt.

Intimacy of personal contact is essential to most business and life as we know it. However much of our face to face business is a rare custom not always honoured at the lower levels.

In a space world meeting others would be easy to acheive with the inclusion of communal spaces. The bucky ball has the largest known communal space in which all denisens of the ball can meet and converge within this space and leave in groups via a space elevator to earth or around the station using a magnetic lift service to the available ports and onboard a space vessel bound for any particular point within our planetary system.

Each space habitat or space station can be conjoined via gravity synced walkways - rather like arriving at a door that moves to sync with the rotation of the station.

The question is whether or not it is possible to have a completely virtual economy if it was then how would it work.

Already many of our jobs do not require any physical action towards constructing or manufacturing commercially viable products and in fact much of the manufacturing industry is operated by machines albeit with human operators one could indeed invisage there being as there is already a dominant cast of workers that could perform their function without need to ever physically change their environment except to  recieve and implement personal aquisitions.

This entirely consumer based society could function extremely well in space creating commerce for goods manufactured using automated technology as it is already the case that most people do not create the products they consume.

In a purely abstract sense these consumers could exist within a purely virtual economy one which would bare some resemblence to a online computer game. Within this game there are winners and losers. Successful players would be those most capable with the technology to impress upon others the need to consume their products or services.

Credit would therefore be issued in this virtual economy and advertising would be a key ingrediant in a virtual economy that sues social networking to acheive all of its goals.

However the danger is that without regulation consumer choice would be stymied by the restrictions based on the real products that are available to these space farers.

In a purely virtual economy these restrictions would be kept constant as the technology to create an abundance of variety and form would be limited to what is space worthy.

There may be new products that could fill this gap and ofcourse survival in space requires innovation.

However at present the choices and limitations in space prevent a space economy from emmerging first there must be a suitable habitation similar to earth and second a supply and demand chain that makes such requirements for life a matter of commerce.

A successfull station would be one that doesnt only rely on the universal provisions awarded it to remain alive.

For example an overabunance of sea food or a plentiful supply of space grain or water storage capacity. All these features of a space habitat could be varied depending on the requirements of its patrons.

A bare bines station would therefore evolve the capacity for these variations that would stimulated trade and value stations according to their resources and proximity to mining stations on nearbouring planets.

The virtual economy could theoretical start at any time in this evolutionary course ....

Experiment test the validity of a purely Virtual Economy :

Construct a realistic Model of the social economic environment surrounding several business such that their communications are all limited to this model and those that are inside it.

There must be hundreds of examples of how this could be acheived. Realistic data is feed into the model e.g stock prices,product prices, product varients, consumer demand and a customer database linked to turing models of consumers or if necessary real life consumers.

Now we have a purely virtual economy. Like a sandpit for these businesses who are completely unaware of the fact that it is all a simulation.

And watch it tick over. Will it generate an income without anything real being produced. Would the generated interests in these virtual products be useful for something would this demand for a product that doesnt exist convievably be a confidense in a market that would yield an income to those that are working within it.

Say the product is entirely plausable like a way of prolonging the life of orange juice.

One company sells this product to another who in turn aquires an amount of oj from a simulated source and sells it for a profit to another source - translated as a real trade in say currency.

Company B beleives confidently that it can make money from these Oranges that are preserved for longer.

Company A beleives it has sold a product that doesnt exist. Would this transaction be required or could the value of this confidence be enough to generate an income for company A.

Company B pays Company A and Company A pays its sales team a bonus. Its workers are paid in virtual money but they dont know the difference.

They can spend this money its real and comes from a trade being done outside of the simulation being a trade in some randomly chosen stock predicted to generate an income from this interest.

Unfortunately you cannot seperate the world in which they spend this money and the virtual world.

If you did then the value of this money would be self-contained within the virtual economy as would the amount of profit from this original transaction.

Could the simulation acheive this and sustain a pure virtual economy one in which the players are all duped by a machine that provides for each player differentially depending on his/her success.

As we know there are many jobs that command incomes disproportionate to the actual real value of the role.

Interesting. Does an economy demand something to be real or could we be living in a world of confidence?

Podwell : Doing-the-Math

To convert say a london property with average 1,200 sqr meter space costing £500,000 with mortgage monthly repayments of £2.617 (3.92 int) for 25 years.

Size of Podbay capsule = 2x1x1.25 m = 2.5 sqr meter

Upper floor is given to Podbays so thats 1,200 /2 =600 / 2.5 = 240 podbay - 40 podbay for access

Lower floor 1/3 of which has food vendors + eating area 1/3 showers men and women and toilets leaves 1/3 of 600 for access and podbay = 200 / 2.5 =  80 podbay

~= 280-300 podbays for an average London Terraced house worth 300,000

At a rent of £2 per night and an average use of 85%

85/100 * 300 * 2 * 7days in week * 4 weeks in month = £14,280

Utilitys = heating central 50 per week
           water           35-50 per week
           electric        50
           internet        25 (High speed broad band multi-user)
       staff           300*4 per week
       mortgage        2,617 per month = 654.25 per week

Gtotal Running costs = £2029.25 per week *4 = £8,117 per month

Leaves profit of £14,280 - £8,117 = £6163 per month with 85% usage

Part 3 A Paradise in Space

Opening at the top of the ball allows the entry of space vehicles to land and refuel or unload supplies / cargoe or receive electricity.

Space travellers wearing contact magnetic boots disembark and leave the space hanger via one of several pressure sealed airlocks and move to their designated level via a magnetic lift. Once inside they remove their suits and join the many others that currently reside on the station.

The park located along the equator is the most pleasurable place to stay and has abundant wildlife tree's, animals and a lake. It receives all of the sunlight radiated through the solar array and refelected through each of the translucent carbon fibre polymer windows that run tangentially across the Balls surface. The properties of later Balls allowed light filtered to only 85% of earhlight to pass throughout the whole station equilivent to a bright but overcast day. The light is significantly brighter and must be filtered of harmful radiation first and yet when it hits and illuminates the opace semi translucent hul it conveys light throughout.

The views are stunning and the best place is right at the equator. Other stunning views include the low-g viewing locations adjacent the Space Port and the more quiet aquas filled regions of the lower polar region in which resides most of the marine life onboard aswell as the microwave projector.

After the shell is complete a level 1 atmos is applied to the outer shell. This minimizes the pressure exerted on the outer shell whilst retaining its structure and function to keep the vessel pressurised at that layer. A second layer built directly onto the buckyball frame work supports the remaining pressure of the interior once filled with earth level atmos.


The end result is a structure that can hold a variety of life sustaining ecosystems including a marine system, a forest system and ofcourse a hybrid hydropnics system for growing and recycling food aswell as an insectarium to provide a plentiful supply of nutrients to all onboard. Each ecosystem will inhabitat parts of the vessel with a lower gravity that suits the tolerances for the species it will sustain.

For example the Marine system will exist at the polar regions where there is close to zero G as fish and other marine species will quickly adapt to the absense of gravity.

Insects on the other hand do not far aswell in Micro-G and will however prosper in a fraction of earth gravity through selective breeding and adaption a process that takes less time then you'd think.

Other land dwelling mammals tree's and plants that have yet to be hybridized for low G environments will prosper in the forest system which will have earth's gravity whilst sustaining a variety of animal and plant life it will also provide important sensory sustanance to those onboard no matter on which level theyre sleeping quarters reside.

Its a hotel a power station a world unto itself and above all an institution of life.

As the giant spinning buck ball rotates and positions itself with its axis facing the sun it receives continous daylight moderated and filtered through the external electrically controlled materials that let varying amounts of light through this can be varied and controlled to give the appearance of day's and nights whilst staying onboard. The solar collectors distribute the power throughout the ship powering its many onboard systems and sending the remaining power back to earth via the microwave transmitter to a fixed collector on the earths surface or to robot energy carrying rockets that return their energy back to earth or to be used to power deep space flight.

Imagine a sculpted valley running along the central equator with plants animals and grass lands a fully self contained world the size of two small villages.

Part 2 Building the Bucky Ball for Earth Pressure (Untested)

After the shell is complete a level 0.1 atmos is applied to the outer shell. This minimizes the pressure exerted on the outer shell whilst retaining its structure and function to keep the vessel pressurised at that layer. A second layer built directly onto the buckyball frame work supports the remaining pressure of the interior once filled with earth level atmos.

The second interior layer is fully load bearing and must support the forces exerted by the earth level pressures against the .01 atmos in the outer shell.

This double layering performs the function of keeping everything in place as all of the forces are distributed to the buckyball frame and not the non load bearing protective shell.

The second interior is constructed from carbon fibre with titanium struts connecting it to the load bearing Bucky ball.

Each carbon fibre structure must be brought onboard after completion of the primary shell and sealed in place to prevent any possibility of leaks entering the primary shell which is held at 1 atmos.

This layer is then sealed once again using the spray on plastic sealent that covers all surfaces the gaps between each carbon fibre layer is also sealed before hand using carbon fibre resin making an impervious seal.

All sealants are electrically conductive and are connected to sensors throughout that monitor both changes in pressure onboard and any slight changes electrical conductivity of the hull that would indicate a fracture or a warping of the strucure that may be due to an external or internal cause.

Once this is complete the entirior of the Ball can be filled with air and everyone can remove their pressure helmets but not their suits and instead wear low pressure masks and low pressure suits and continue work.

The titanium struts all point into the structure towards the center and are used to carry furth load bearing structures throughout the ship such that all forces are carried evenly throughout the Bucky Ball. As forces are applied to the Ball in all directions they are spread out evenly therefore the Ball's interiiors load bearing structure must refelect the exterior symetry of the Ball or risk an uneven distribution of forces thereby resulting in a weakness.

Other equally strong and stable structures that can acheive artifical gravity in this way and deal with the massive forces exert on them by a pressurisd vessel in space includes the Hexadome.

In order to compensate for the forces of gravity and the rotaion forces of the Ball the Struts would need to radiate out from the equator where the centrifugal forces are greatest towards the upper and lower hemispheres. Thereby ensuring an equal distribution of force throughout. The calculations and designs are in PDF format for download here. With a slow and gradual accelleration to the correct rotation the initial forces due to accelleration of the Ball are minimised until equilibrium is reached and the correct rotational velocity is aquired. A fast accelleration to this velocity could result in unacceptable forces being applied to the structure hence a slow gradual rotation is applied using the Intelligent Thrusters working in unison.

Building the Ball - Part 1

Once the Buckyball structure has been built using 5 to 6 space construction workers armed with basic tools - rachets to securing bolts between struts, grabbers for moving struts into place and EVA packs with the provision of emmergency EVA replenishers and a Dragon capsule to supply all the needs of the workers.

Hardy individuals can last upto two weeks working with food and provisions before the effects of zero-G becomes difficult to work under.

The 'Canvas' Layer is attached carefully rather like the canvas on a tent and is rolled out in one huge peice. Coordination is essential as each corner must be moved over the structure and afixed in exactly the right position. After it has be attached then each worker sets about attaching it to every vertex using the presupplied clips each clip is then tightened until the 'Canvas' is taught for all sections. Because of its unique energy absorbing properties once taught and in place it will absorb incoming collisionsjust like a trampoline.

Once everything is secured and fasten down the workers then commit themselves to the enterior layers. First the carbon fibre messhing panels are unfolded and attached to the entior via another set of clips and some heavey duty bolts giving the structure added protection and rigidity each panel is shaped to unfold into the space between each strut from the inside leaving a minimal gap between the strut and the panel.

Now the extrusion process takes place. Usually this is done by a second or third team. The first team constructs the Bucky Ball frame the second attaches the exerior canvas and the third - the extrusion team then pumps the extruded plastic foam behind the carbon fibre panels. Each Panels caveitys are filled entirely before the entire interior surface is covered in the expanding foam. Designed to work at extremely low temperatures the foam remains liquid for longer and yet is chesive enough not to break apart in zero g.

Ofcourse not forgeting one important stage and that is normally done during the attachment of the carbon fibre panels and is the attachment of pipes, cavitys, and electrical cabels.

Most of the heavey power capables are not placed in the shell of the Ball.

Care is taken to include the bare minimum of pipes and wiring in these walls as any cavity produced could potentialy weaken the wall and lead to a breach. All cavitys are checked and rechecked using its conductive sensitivity. The inspector who is also quite often the 'layer' will test each cavity with a probe during the extrusion process.

Ideally all cabling and pipes could be placed internally - it is an option for the construction company. There have as yet been no breaches due to the effects of cavity's weakening the surrounding strength of the strucure to the high pressures, g-forces, radiation and collisions. The company produced a test ball in order test all of these conditions after building the first Ball in order to prove that it was safe.

After the shell is finished the Ball is pressurised to correct atmos and the remaining workers enter the Ball and wear a minimal pressure suite with a safety mask, an escape tank is provided with room for all workers.

The rest of the construction continues with the assembly of the microwave transmitter - the largest machine part of the structure, position along the axis, which then requires cabling. The flooring is then attached in a rotory fashion around this machine which is activated on ijnstallation to provide power to the station as the work continues. The solar collectors together with the ion thrusters are attached and plugged into the cable ports along the surface of the Ball.

A coms link is made via a wifi to the self powered ion thrusters and their power source the solar panels. Each thruster is controlled by its own sensor before the stations control network is created. The workers have access to a switch which can turn of and isolate individual not working thrusters and a master switch for all thrusters.

If working first time the station finds gravity in 2 hours and 30 minutes depending on the size of the ball this time is for a 1.2 km ball.

The reason it takes so long is that it must take hundreds of minor adjustments in order to reach the correct velocity using the minimum of thrust.

The Gravity drive works using a Hive form of Artifical Intelligence or swarm behaviour in which each thruster operates independantly yet resulting in a collectively optimal performance using minimal power to acheive the correct and stable rotation of the Ball.

The flooring is layerd and has signposts to indicate which deck your on and the gravity level that is on that level at that position within the spehere. There is less decking towards the polar regions and this is usually used for storage of materials and equipment aswell as the emmergency escape tank. Food and water is also kept in this zero-g location.

Part 2 Building the Space port.

Part 3 Installing the Ecolife system.

Is the UK's Space Age yet apon us?

With Britains increased involvement in the Trump administrations goal of protectionism and international security the time is now right for strengthening ties with our overseas cousins and now is no better time than the present.

With Tatar steel now thankfully resurrected the UK's increasing involvement in Aerospace manufacture could potentially eclipse that of our European competitors with the purchase and aquisition plans for a minimal Rocket base and Manufacturing plant based in Wales.

There have been many potential candidates for locating a space ascent site within the UK not least the one used for initial testing of the the Black Knight at High Down. Its first launch was in Woomera Australia. Now with proven technology developed by SpaceX to affordably send payloads in excess of 40 tonnes into lower orbit and beyond the prospects of building a reusable orbital ascent vehical in the UK and actually turning a profit have become viable.

The Falcon 9 reusable rocket's launch station can be situated near to Tatar steels factory capable of producing rocket parts and transferring them to a chosen site within wales that makes use of the original infrastructure that once made the UK and wales the epicenter for the industrialised world. Rail links no longer used within wales were once used to transport coal from the valleys to the sea. One route in particular used a tunnel to transport coal to Port Talbot - the location of Tatar Steels plant.

It is more than likely after the most recent government report on the Aerospace industry that Wales will indeed be chosen for the location of several launch sites perhaps making the UK once more the driving force for the industrialised world.

The land albeit vertical has an abundance of features making it most suitable for heavey excavation and the conveyance of fuel and parts via its coastal ports and prexisting rail infrastructure.

Right now the UK is investing heavily in its rail infrastructure focusing on commutors travelling from the north and midlands to london to work. There already exists plans to revitalise Wales with the inclusion of Tatar steel and the creation of more jobs in construction and the Aerospace industry.

It is most likely that the UK government will start by simply requesting permission from the Welsh Assembly to aquire the neccessary permits and land in order to build projects like this.

Already land has been aquired for the construction of wind farms in all of the proposed sites for a rocket base. Therefore it is not a Land issue at present the UK government is currently negotiating with the patent holders in the US to build rocket factorys in the UK to ship parts overseas to regions within Africa.

The space race is finally here but it may be some time before Wales hears once again the resounding blasts of its furnace like engines as they rip appart the night sky and carve out a new destiny for mankind.

Each site would be heavily excavated and the plans being similar to the peenemunde rotating launch facility used to launch the V2. This would aswell as saving on land would also serve to protect the launch from adverse weather conditions. The maximum tolerable wind speed prior to launch is 70 km and as we already know the weather in wales can often be a might difficult. It is also a safety issue an underground launch facility given adequate depth will minimise noise pollution, emmissions and if there should be an accident it would be self contained within the facility.

The valleys are ideal. Seldom visited the barren mountains that once laid bare and encapsulated the center of wales coal industry have now been brought by energy companys wishing to capitalise on the weather producing energy available from the more recent construction of wind farms. Held by many to be a blight on the landscape these monsters of the valleys mountains would not be necessary with the UK's commitment to building an energy farm in space. In this earlier article I outlined the plans for a Bucky Ball structure not disimilar to the "Death Star" orbiting earth at a higher orbit and with the proven microwave transmission of energy from unfiltered sunlight hitting its many sides adorned with solar collecting panels.

In fact if you read here in my earlier blog you will see it is perfectly possible for many of these bucky balls to be built at low cost and self-assembled in space. With microwave transmission beams aimed at every city or region of high populace.

Those of you taken with some fear and trepedation of a microwave beam of energy being sent down from an orbiting space structure neednt worry as the energy of the beam is only imparted to a collector tuned to that beams frequency. Just as when your mobile phone receives a signal.

The energy needs for London for example would require one Buckyball with a radius of 1.2 kilometers and would require 27 trips by reusable rocket into lower earth orbit.

The worlds FIRST - self assembling super structure in space.

A new world awaits us in outer space - entirely man made and self sufficient.

A spining hollow sphere with varying artificial graivty the most being at the equator where there would be communal space such as a park and further in would be lower gravity with zero at the polar extremes.

People would pay more to be housed closer to the equator where they would experience the best gravity.

Or it would be closer towards the center with higher spin rates and a ultra heavey gravity at the equator suitable only for marine life.

Limited to the correct revolutions to prevent motion sickness some regions within the spehere would be offlimits to humans and yet suitable for food production, hybrid crops grown using hydroponics to produce a variety of Geneticly modified foods.

An artificial ecosystem could be created that would allow recycled air and water to be supplied throughout the station.

The structure could be built using a bucky ball repeating framework or lattice easily manufactured and compacted for transport into orbit and covered with a high density metallic polymer which protects against microcollisions by energy absorption. The entiirior would be filled with extruded high density expanding plastic foam and reinforced with carbon fibre fixtures.

Layer 1 titanium composite metallic buckyball frame - Initial transport 635 x 3 kilo struts
Layer 2 high density collision absorbing metallic polymer - canvas / repairable 759,000 km2
Layer 3 expanding plastic foam - insulating and containing conductive additive for electron emmsision cosmic ray deflections / protection 200 tonnes of plastic foam (Electrons deflect cosmic rays!)
Layer 4 Held between the Buckyball frame and surrounding by foam an strengthened layer of carbon fibre meshing for added strength attached directly to the Bucky ball frame.
Inbetween space for wiring pipes and other essentials needed to transport liquid , electricty and heat throughout the station would be layered within the extruded walls.
Layer 5 An air sealed plastic layer would be sprayed across all interior surfaces. This would also be a conductive plastic that would relay information about any hull breach detections to the communications grid. (Using Intelligent materials. Intelligent Materials are able to communicate stress and strain data from throughout the surface the same way you measure resistence along a length of wire.)

The delivery for materials would take approximately 30 trips into space at a cost of 76 million per trip. Each trip would deliver its cargo to a waiting vessel that would refuel and carry its supplies the necessary distance from earths lower orbits and begin construction.

Well lets see how much space junk ahem I mean satellites can you deliver in thirty round trips to make it pay.

Ofcourse once the space station (Death star) is fully operational around its central axis will be housed a mircowave projector aimed at a fixed location on earth (Preferably London or some major city). Solar energy is then collected from across its surface using a giant array of solar panels that recieve continuous unfiltered sunlight and generating approximately 700 GigaWatts of electricty per minute. 90% of which is then transmitted via microwave to the receiver station or city on earth generating a daily profit of approximately 3 million (earth creds) Per day.

Meanwhile the hoy poly of earth's elite will ofcourse enjoy transit to the station and wine and dine in the greatest comfort as they stay at a region along the equator to enjoy the view.

How is it built?

New Project ANt Clusterer - "Droping off or Picking up?"

My latest project is an ANt clusterer that will gather together data with similar properties within a certain range.

It will work by Random ANts running about a Grid picking up Data and carrying it with them until they find a suitable location to drop off.

At the present stage ANt 0.1 it has managed to pickup every peice of Data and has left behind plenty of nice ANt poo's!😊

Fig 1. ANt Poo's

Yeah its not ready yet. It is supposed to move the Food around not eat it all and leave ANt poo's everywhere.

 "Dropping off or picking up?", GhostBusters(1984).

ANt finally got there and clustered the patterned data into bands. Each pattern is simply a binary 2d array with random bits they are represented on the Grid with their summed totals.

To begin with I simply used their Summed integer value for the ANt to perform its two functions Drop off or Pickup.

It initially didnt score well and this was because I was evaluating each ANt's probability against a Random value. Without the Randomised Roll of the Dice it evaluates the Probability of either Dropping Off and Picking Up against a single value.

So technically speaking its not a probability. Anyway it has proven to be deterministic after 50,000 iterations with 15 ANt's it produces Bands of each pattern.

It uses the Binary representation of each pattern instead of its Summed Integer value and compares the Carried Pattern or the Pattern found at its present location to create a Density of Similar Patterns within a fixed Radius of its location. I got the low down from here.

The Density function is simply :

Sum{|PatternX[n]-PatternY(r)[n]|} / NumPatterns

NumPatterns is the Number of Patterns within the Radius of the ANt
PatternX is the Pattern either carried by the ANt for Dropping Off or
the pattern found at the ANt's location for Picking Up

The ANt needs to Pick up a pattern if the Density of simularity around it is low. It needs to drop a Pattern if it finds the Density surrounding its location to be high.

Density is normalised using Exp(-Const*Density) to create a Probability of an action of:

1-Exp(-Const*Density) / 1+Exp(-Const*Density) for Drop Off

and
                                        1 -
1-Exp(-Const*Density) / 1+Exp(-Const*Density) for Pickup

With these Bands I can now classify each pattern as belonging to a Membership denoted by the Band in which it has been placed.

However I was expecting it to create square Clusters for each Pattern. The ANt seems to have a lilt towards Bands. It also seems to have placed the Bands in order of the its Summed value with the highest at the top.

....I don't know why.

I now need to see how I can acheive this Banding faster with fewer ANt's and for more complex patterns. It would be beneficial also if each ANt could learn to put patterns into Bands that are numerically disimlar. As the ANt is improved with learning to classify more complex patterns there will be scope for speciation or casting amongst the ANt's as some ANt's must learn to classify and therefore recognise specific types of Data.

The end result being an Algo which is domain specific and can classify patterns of different types to Groups within Groups.

Fig. 1 Initial Success - ANt Bands (The little hat shows where it has picked up from, the star shows where it has dropped off)

The ANt story continues have now just applied the ANt to an image to see if it can distinguish primitive features and produce the same Bands now for more complex Data. The results look good with little tinkering this helpful buddy partitioned a complex image into three distinct Bands. At this stage the ANt should have performed a rudimentary Histogram on the Image. I have yet to see however because allthough I have a Binary output similar to that above I have not written the function yet that converts the Grid data produced by the ANt back into jpeg format. Stay tuned to see the output...😋

ReadMe


So far so what. I have plugged the JPEG reader into the ANt program with
limited success due mainly to the fact that I have yet to unscramble the image properly. Also not all the ANts have been instructed to drop their loads and thus we are missing data on the Grid for reassembly into the image.

Even with all the data on the grid the image becomes distorted for every iteration. This shouldnt happen we should beable to get a perfect image out
the other end no matter how many iterations we perform as the ANts are merely labelling each block as belonging to a feature set.

Here are some images with no iterations to give you an idea about how things will proceed. Eventually these colour blocks will be pixels as each pixel will become a data element carrying with it its neighbourhood values and being clustered by the ANts accordingly.

To begin with though it is simpler to use blocks and then apply these blocks as neighbourhoods for each pixel. When this happens the grid will be much larger and thus the processing time will increase. But now that I have brought a new machine I hope to reduce this time with parallelization.

Here are the images. Any more iterations and I lose Data and the image gets distorted.

The goal will be image segmentation using a colour index taken from the Grid.

As you can see there is no segmentation as I have merely reproduced the image from the Grid data with coloured labelling applied. The Ants have yet to perform their task of clustering the image.

 

 Before (First Image) Random Initial state. After (Second Image) ANts have
clustered regions belonging to the Leopard.

10,000 Iterations with different Mapping functions.

It delineates features from both the Antelope and the Leopard.

Captures different features of the Lotus relating to orientation and edge
detection. Powerful ANts!

Next - will this be a suitable replacement for pooling on the CNN?

Note also it leaves the background in tact in both the two previous images.