The main reason mankind began living in space was due to congestion on earth slower net speeds meant teleportation or telepresence became almost impossible as the airwaves were
so clogged with traffic so too were the transport networks. Mankind needed space and took it as the satcom network could be used directly and ultra orbit gravity domes offered go anywhere rates for all virtual deck business men and women. They all enjoyed the emmense spaces that had been built by space grunts and telerobotics. A giant space port was first assembled and channeled precious minerals worth trillions back to earth.
In fact by the year 2089 the space race became a matter of survival when the outbreak of an information virus took hold of the entire connected human race. It was seeded two decades earlier with the advent of quantum computing. The profound effects on the pysches of every connected being on the planet due to quantum entanglement between neurons in the brain and their artificial equilivent that used electrons spinning around inside a vacume had meant the days of the super AI's on earth were numbered.
A war raged between those that wanted to surpress this new technology and those that wanted it unleashed. Unfortunately the choice had already been made as it wasnt the benign creations of the super AI's but rather the interconnectedness of man and machine that was the cause of the outbreak and mankinds dependance on these new age machines as an enhancement to his already diminished natural intelligence.
The outbreak started in various locations and spread like a virus inhabiting the minds of its users like a parasite until their brains exploded.
The cdc was called in to isolate those thought to be carriers but this had meant constructing faraday cages to stop it from travelling via the airwaves to infect others. The virus was clever though and it used quantum tunnelling to escape this.
Eventually attention was drawn to those that had remained unaffected - newborns their brains were more adaptive then those that had grown emmersed into this technological dependant world. If they were raised in a low tech environment - no phones, no radios etc then perhaps there was a chance of their brains being unaffected but it was only true for half the rest regardless of origins became infected even whilst living outside of the grid. The virus had literally hijacked our own biology at the subatomic level.
The cdc who by now had taken to wearing emf suits that scrambled the bioemmessions from the infected and that of the airwaves managed to remain unaffected there attentions now drew to the last remaining unaffected - a group of scientists who had been living in space at the time of the outbreak.
Theyd remained unaffected. Meanwhile the entire planet was now populated by the infected information virus casualtys with the countless dead whose brains had exploded due to a build up of phosphenes in the neurotransmitters reacting with an abundance of calcium ions and causing their brains to explode.
Nasa the cdc and various companys and space tech people joined forces to begin construction of a habitable space station for the survivors to escape the mind virus which by now was completely biological and didnt even need a communicatiosn infrastructure. It used the quantum computing power of interconnected minds and quantum
entanglement to spread essentially reducing those still alive to neural processing units, or quantum processors.
Luckily it was confined to earth by the ionosphere and couldnt transmit to space via any of our satellites or space ships. Once you were free of the ionosphere you could start breathing again content that you no longer had to wear emf suits and take the blockers youd been give by the cdc.
"The planet would heal...it might take years but eventually the virus would run out of human brain matter on earth and die out."
"Yes it is true that many from the cdc thought the cause was our super AI's but now we know this not to be true and the last of the remaining AI's have been rescued from earth and are now in a emf sheilded facility close to the core of this very station and their sole purpose is to monitor the earths ionosphere, cosmic radiation entering the station and ofcourse maintain our automatic gravity."
"Mr Peterson has already raised the concern that the infovirus may interact with the gravitons produced by the station and cause weightlessness but with the deadlocks in place there is no chance of this happening without a ship to ship tether being initiated.....and so far we have yet to see the virus being able to control machines."
We have theorised that the virus occured due to the initiation of quantum entanglement devices being used within the communication industry and thereby by every living being connected to the net in someway. Once it had taken hold it no longer required a quantum device to spread and found the most suitable candidate being the human brain.
I will let Dr Andrews continue with the explanation she is the foremost epert within the cdc.
By the control of certain biochemical events the brain was made to explode due to the surplus of calcium and phosphene ions. This build up led a biochemical reaction that seemed a by product of the brain being overused for some purpose by the infovirus. We have yet to determine the purpose and have theorised that it is not soley for its procreation.
Mind fucked theyve all been mind fucked by that ... thing.
Yes leutenant Jefferies the limbic system is thought to be involved - makinds only weakness so it seems. The origins of the virus is unknown it may have been among us for thousands of years only waiting for the opportunity that our quantum meddeling has provided.
I dont think its fair to blame quantum science for this after all hasnt nature been meddling with the quantum world far longer than us.
Yes doctor philips but it was us who have opened pandoras box nature merely takes the opportunitys on offer.
Yes I agree - nature doesnt meddle it evolves it adapts just like this virus for good or for ill. Now I would like to draw our ettention to the diversity program that doctor johnson has agreed to head up.
Dr johanson a tall slim dark haired swedish woman of appealing features which drew the attention of all the male attendee's began to talk, "We have elected a team to return to earth in order to recover the dna from all persons so that they may be cloned later. and also to look for any survivors that may have some immunity to the virus."
Thankyou dr johanson. The cloning ofcourse shall be initiated once our ships arrive on hibli 8 the third planet in the Iridia system. H8 has been selected as a suitable location or hub from which humanity may spread using our dna receiver ships. They work simply by robotic control and receive and grow humans from synthetic dna. Enhanced humans that is the first created will recieve our telepathic teleportations as will our robotic servants all controlled via quantum entanglement.
Wait a minute did you say quantum entanglement - isnt that the primary mode of transmission of the infovirus.
Yes it is correct.
I see then isnt there a risk.
No no theres no risk the infovirus is perfectly contained within the earths ionosphere. No way out. Perfectly safe, nothing to worry about.
The leutenant grimaced as the scientist seemed to be hiding something. He glanced around the room no one else seemed to pick up on this. There was silver glint in the scientist's eyes only just noticeable - maybe it was just in his head. He had been overworked finishing the new wing on the habipods and being drafted to replace those that were infected back on earth.
The controllers, the grafters all of them had been gotten to by the virus their brains were the first to go exploding over their vr decks, or on their way up in the many autoloaders that brought everything up to the station. Now they were self sufficient but only just. Everything had been stopped after the outbreak and all focus was on making sure the station was self sufficient. A few of the grafters had stayed most had returned to earth in free fall expecting to find something left of the world so they could build their own private sanctuary raise a family.
They hadnt believed the cdc their implacable faith in humanity to survive by hard work alone. Those that remained had been drafted and now formed part of an elite group of planet shifting wise guys. All of them ran their own rig's and brought resources from the inner planets back to each station enroute. But now all the focus was on the dna ships they had the best jobs and fortunatley there had been enough teleworkers plugged into the satcom's direct feeds up here to pitch in with deep space colonization. Expendable their brains often got fried in operating human clones and machines on the other side of the galaxy. Most got delinked lost somewhere in the quantum complexity's of space travel. The grafters didnt get involved they preferred their own bodys to some clone. But somebody had to do the grunt work - fetch the water, mine the ore, build the habipods.
Read More Go to juice.extramindcorp.com
Buy the book 'Seed Army' at Amazon.
Thursday, 2 August 2018
Must remember this samba setup on ubuntu (Maybe help you with samba on linux)
samba/smb.conf
interafces = 192.168.1.0/24 wlan0 eth0
bind interfaces only = yes
security = user
;Server side:
[Resource]
comment = user_home's Resource
path = /home/user_home/Resource
available = yes
valid users = user_remote
read only = no
browsable = yes
public = yes
writable = yes
create mask = 0755
directory mask = 0755
(port opens automatically)
sudo adduser -m user_remote -s /bin/bash
sudo passwd user_remote
smbpasswd user_remote
command to mount client side:
sudo mount /media/Resource
sudo mount -t cifs //192.168.1.38/Resource /media/Resource -o rw,user=user_remote,uid=user_remote,gid=user_remote
Install with:
sudo apt-get install samba samba-common system-config-samba
sudo apt-get install nfs-common
sudo apt-get install cifs-utils
interafces = 192.168.1.0/24 wlan0 eth0
bind interfaces only = yes
security = user
;Server side:
[Resource]
comment = user_home's Resource
path = /home/user_home/Resource
available = yes
valid users = user_remote
read only = no
browsable = yes
public = yes
writable = yes
create mask = 0755
directory mask = 0755
(port opens automatically)
sudo adduser -m user_remote -s /bin/bash
sudo passwd user_remote
smbpasswd user_remote
command to mount client side:
sudo mount /media/Resource
sudo mount -t cifs //192.168.1.38/Resource /media/Resource -o rw,user=user_remote,uid=user_remote,gid=user_remote
Install with:
sudo apt-get install samba samba-common system-config-samba
sudo apt-get install nfs-common
sudo apt-get install cifs-utils
Friday, 6 July 2018
Really REALLY simple Neural Network Reinforcement Learning
Well it was always a project I never quite finished and I found a way back in by using some well written basic scripts for AI to mix them together and make a Neural Q learner Recipe as follows:
Take one ANN for this I used the following:
http://mnemstudio.org/neural-networks-backpropagation-xor.htm
add one Q-learner from here:
http://mnemstudio.org/path-finding-q-learning-example-1.htm
and create a Neural Q-learner
1. add a new hidden layer to the ANN
2. add the complete formula for calculating the Bellman Residual to the Q-learner:
3. Create this learning loop:
i. Randomly choose a legal action (Not a wall)
ii. Connect CurrentState and Action to inputs on ANN
iii. Calculate Q using above formula
iv. Calculate Error of Q outputed from the ANN
v. Update all the weights for each layer started with the last and ending with the first - feeding back the error in the same way as the XOR problem.
4. Test the net
I found it worked best with 8 neurons in each hidden layer and the Gamma of the Bellman Equation set to 1.8 instead of 0.8
Here is the net solving the state matrix of the qlearning program:
First state=1,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=1
Found a winner
Q value=0.757337 Action=2
Final winner for state 1 =2
First state=2,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=2
Found a winner
Q value=0.757337 Action=3
Final winner for state 2 =3
First state=3,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=3
Found a winner
Q value=0.757337 Action=4
Final winner for state 3 =4
First state=4,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=4
Found a winner
Q value=0.757337 Action=5
Final winner for state 4 =5
First state=3,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=3
Found a winner
Q value=0.757337 Action=4
Final winner for state 3 =4
First state=4,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=4
Found a winner
Q value=0.757337 Action=5
Final winner for state 4 =5
First state=5,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=5
Final winner for state 5 =5
First state=2,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=2
Found a winner
Q value=0.757337 Action=3
Final winner for state 2 =3
First state=3,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=3
Found a winner
Q value=0.757337 Action=4
Final winner for state 3 =4
First state=4,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=4
Found a winner
Q value=0.757337 Action=5
Final winner for state 4 =5
First state=4,
Found a winner
You can see the different Q-values for each action produced by the output of the Neural Network. (Needs more work though to get it to converge)
It makes mistakes and doesnt converge yet but with more work it might!
If anyone wants the code for this one (Based on the Mnemstudio C++ original) please add a comment.
Take one ANN for this I used the following:
http://mnemstudio.org/neural-networks-backpropagation-xor.htm
add one Q-learner from here:
http://mnemstudio.org/path-finding-q-learning-example-1.htm
and create a Neural Q-learner
1. add a new hidden layer to the ANN
2. add the complete formula for calculating the Bellman Residual to the Q-learner:
i. Randomly choose a legal action (Not a wall)
ii. Connect CurrentState and Action to inputs on ANN
iii. Calculate Q using above formula
iv. Calculate Error of Q outputed from the ANN
v. Update all the weights for each layer started with the last and ending with the first - feeding back the error in the same way as the XOR problem.
4. Test the net
I found it worked best with 8 neurons in each hidden layer and the Gamma of the Bellman Equation set to 1.8 instead of 0.8
Here is the net solving the state matrix of the qlearning program:
First state=1,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=1
Found a winner
Q value=0.757337 Action=2
Final winner for state 1 =2
First state=2,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=2
Found a winner
Q value=0.757337 Action=3
Final winner for state 2 =3
First state=3,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=3
Found a winner
Q value=0.757337 Action=4
Final winner for state 3 =4
First state=4,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=4
Found a winner
Q value=0.757337 Action=5
Final winner for state 4 =5
First state=3,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=3
Found a winner
Q value=0.757337 Action=4
Final winner for state 3 =4
First state=4,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=4
Found a winner
Q value=0.757337 Action=5
Final winner for state 4 =5
First state=5,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=5
Final winner for state 5 =5
First state=2,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=2
Found a winner
Q value=0.757337 Action=3
Final winner for state 2 =3
First state=3,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=3
Found a winner
Q value=0.757337 Action=4
Final winner for state 3 =4
First state=4,
Found a winner
Q value=0.539621 Action=0
Found a winner
Q value=0.542437 Action=4
Found a winner
Q value=0.757337 Action=5
Final winner for state 4 =5
First state=4,
Found a winner
You can see the different Q-values for each action produced by the output of the Neural Network. (Needs more work though to get it to converge)
It makes mistakes and doesnt converge yet but with more work it might!
If anyone wants the code for this one (Based on the Mnemstudio C++ original) please add a comment.
Tuesday, 3 April 2018
ECG on Orange Pi Zero
This project utilizes the MCP3208 A2D convertor to serve analogue data collected from an Instrumental Amplifier circuit that reads my heart beat.
The MCP3208 uses Spi-dev as detailed here which installs on Armbian. You can also install ASCII-Graph via PIP and produce a barchart of your heart beat.
Edit spiread.py as follows:
if __name__ == '__main__':
spi = MCP3208(0)
count = 0
a0 = 0
a1 = 0
a2 = 0
a3 = 0
graph = Pyasciigraph()
while True:
count += 1
a0 += spi.read(4)
a1 += spi.read(5)
a2 += spi.read(6)
a3 += spi.read(7)
data = [('Output',a3/300)]
for line in graph.graph('Output', data):
print(line)
The circuit is taken from here and uses the first stage filter to produce a rudimentary ECG.
The electrodes are here . The signal vary's from 400 to 0 and the gain of the amplifier can be changed using a variable resistor over pin 1,5 of the Instrumental Amplifier. The first stage filter is necessessary to exclude noise from power source.
If you hold both the inputs you will see the voltage difference drop to zero. If you connect the electrodes and place them on your chest (Left side) the signal will vary with your heart rate.
The MCP3208 uses Spi-dev as detailed here which installs on Armbian. You can also install ASCII-Graph via PIP and produce a barchart of your heart beat.
Edit spiread.py as follows:
if __name__ == '__main__':
spi = MCP3208(0)
count = 0
a0 = 0
a1 = 0
a2 = 0
a3 = 0
graph = Pyasciigraph()
while True:
count += 1
a0 += spi.read(4)
a1 += spi.read(5)
a2 += spi.read(6)
a3 += spi.read(7)
data = [('Output',a3/300)]
for line in graph.graph('Output', data):
print(line)
The circuit is taken from here and uses the first stage filter to produce a rudimentary ECG.
The electrodes are here . The signal vary's from 400 to 0 and the gain of the amplifier can be changed using a variable resistor over pin 1,5 of the Instrumental Amplifier. The first stage filter is necessessary to exclude noise from power source.
If you hold both the inputs you will see the voltage difference drop to zero. If you connect the electrodes and place them on your chest (Left side) the signal will vary with your heart rate.
Wednesday, 26 July 2017
LSTM with no learning - recall and count ahead.
Without any training my LSTM remembers for a specified delay and counts ahead it also has a go at predicting the future for some rudimentary time series.
But its not been trained because the wgt updates dont affect this output it does all this off the cuff!
Whats going on. I hope I can fix this.
Oh the memory thing is affected by the training to specify when to recall. I am uploading this doozy for anyone to play with and maybe get it to do the right thing!
This upload remembers stuff you can specify when to recall by seting ahead=1,2,3 steps ahead.
This upload counts ahead.
All with no learning - whats going on!
Here is roughly what Ive implemented:
Forward Pass
Act_FGate = ft = sigmoid_(Wgt[0]*(Hin+In)+Bias[0],1,0); //forget gate
Act_IGate = It = sigmoid_(Wgt[1]*(Hin+In)+Bias[1],1,0); //Include Gate
Ct_= tanh_(Wgt[2]*(Hin+In)+Bias[2],1,0);
Act_CGate = Ct = ft*Ctin+It*Ct_;
//Out gate
Act_OGate = Ot = sigmoid_(Wgt[3]*(Hin+In)+Bias[3],1,0);
Hout = Ot * tanh(Ct); //Outputs
Backward Pass
***Backprop error:
Hout_Err = Out - Hout
Ctin_Err = Inv_Tanh(Wgt_O * Hout_Err)
Err_FGate = Inv_Sigmoid(Wgt_F * Hout_Err)
Err_IGate = Inv_Sigmoid(Wgt_I * Hout_Err)
Err_CGate = Inv_Tanh(Wgt_C * Hout_Err)
Hin_Err = Err_CGate + Err_IGate + Err_FGate
Next layer down Hout_Err = Hin_Err
***Update Wgts (For each Wgt_F,Wgt_I,Wgt_C,Wgt_O):
WgDelta = (Hin+In)*Err_Gate*Act_Gate*Lrt + Momentum*PreDelta - Decay*PreWgt
PreWgt = Wgt
PreDelta = WgtDelta
Wgt += WgtDelta
Here is the correct psuedo code. Borrowed / Interpreted from a translation online:
Forward
PreAct_FGate = U_FGate*(Hin+In) + W_FGate*Hout(t-1) + V_FGate*C(t-1)
PreAct_IGate = U_IGate*(Hin+In) + W_IGate*Hout(t-1) + V_IGate*C(t-1)
PreAct_CGate = U_CGate*(Hin+In) + W_CGate*Hout(t-1)
Act_IGate = Sigmoid(PreAct_IGate)
Act_FGate = Sigmoid(PreAct_FGate)
Ct_ = LSigmoid(PreAct_CGate)
Ct = Act_FGate * Ct-1 + Act_IGate * Ct_
PreAct_OGate = U_CGate*(Hin+In) + W_CGate*Hout(t-1) + V_OGate*C(t)
Act_OGate = Sigmoid(Act_OGate)
Hout = Act_OGate *tanh(Ct)
Backpass:
Hin_Err = Sum U_Gate*Err_Gate(t) + Sum W_Gate*Err_Gate(t+1) <---- For Layer Above
Err_OGate(t) = Inv_Sig(PreAct_OGate(t))*tanh(Ct)*Hout_Err(t)
Ct_Err(t) = Act_OGate*Inv_Tanh(PreAct_CGate(t))*Hout_Err(t)
+ Act_FGate(t+1)*Ct_Err(t+1) + WgtV_I*Err_IGate(t+1)
+ WgtV_F*Err_FGate(t+1) + WgtV_O*Err_OGate(t)
Err_CGate = Inv_LSig(PreAct_CGate(t))*Act_IGate(t)*Ct_Err(t)
Err_FGate = Inv_Sig(PreAct_FGate(t))*Act_CGate(t-1)*Ct_Err(t)
Err_IGate = Inv_Sig(PreAct_IGate(t))*LSig(PreAct_CGate)*Ct_Err(t)
*(Hout_Err(prev) = Hin_Err
Hout_Err = Out - Hout )
*(Three Activation Functions
Logistic Sigmoid, Tanh, Sigmoid)
*(Three Wgts - U,W,V)
But its not been trained because the wgt updates dont affect this output it does all this off the cuff!
Whats going on. I hope I can fix this.
Oh the memory thing is affected by the training to specify when to recall. I am uploading this doozy for anyone to play with and maybe get it to do the right thing!
This upload remembers stuff you can specify when to recall by seting ahead=1,2,3 steps ahead.
This upload counts ahead.
All with no learning - whats going on!
Here is roughly what Ive implemented:
Forward Pass
Act_FGate = ft = sigmoid_(Wgt[0]*(Hin+In)+Bias[0],1,0); //forget gate
Act_IGate = It = sigmoid_(Wgt[1]*(Hin+In)+Bias[1],1,0); //Include Gate
Ct_= tanh_(Wgt[2]*(Hin+In)+Bias[2],1,0);
Act_CGate = Ct = ft*Ctin+It*Ct_;
//Out gate
Act_OGate = Ot = sigmoid_(Wgt[3]*(Hin+In)+Bias[3],1,0);
Hout = Ot * tanh(Ct); //Outputs
Backward Pass
***Backprop error:
Hout_Err = Out - Hout
Ctin_Err = Inv_Tanh(Wgt_O * Hout_Err)
Err_FGate = Inv_Sigmoid(Wgt_F * Hout_Err)
Err_IGate = Inv_Sigmoid(Wgt_I * Hout_Err)
Err_CGate = Inv_Tanh(Wgt_C * Hout_Err)
Hin_Err = Err_CGate + Err_IGate + Err_FGate
Next layer down Hout_Err = Hin_Err
***Update Wgts (For each Wgt_F,Wgt_I,Wgt_C,Wgt_O):
WgDelta = (Hin+In)*Err_Gate*Act_Gate*Lrt + Momentum*PreDelta - Decay*PreWgt
PreWgt = Wgt
PreDelta = WgtDelta
Wgt += WgtDelta
Here is the correct psuedo code. Borrowed / Interpreted from a translation online:
Forward
PreAct_FGate = U_FGate*(Hin+In) + W_FGate*Hout(t-1) + V_FGate*C(t-1)
PreAct_IGate = U_IGate*(Hin+In) + W_IGate*Hout(t-1) + V_IGate*C(t-1)
PreAct_CGate = U_CGate*(Hin+In) + W_CGate*Hout(t-1)
Act_IGate = Sigmoid(PreAct_IGate)
Act_FGate = Sigmoid(PreAct_FGate)
Ct_ = LSigmoid(PreAct_CGate)
Ct = Act_FGate * Ct-1 + Act_IGate * Ct_
PreAct_OGate = U_CGate*(Hin+In) + W_CGate*Hout(t-1) + V_OGate*C(t)
Act_OGate = Sigmoid(Act_OGate)
Hout = Act_OGate *tanh(Ct)
Backpass:
Hin_Err = Sum U_Gate*Err_Gate(t) + Sum W_Gate*Err_Gate(t+1) <---- For Layer Above
Err_OGate(t) = Inv_Sig(PreAct_OGate(t))*tanh(Ct)*Hout_Err(t)
Ct_Err(t) = Act_OGate*Inv_Tanh(PreAct_CGate(t))*Hout_Err(t)
+ Act_FGate(t+1)*Ct_Err(t+1) + WgtV_I*Err_IGate(t+1)
+ WgtV_F*Err_FGate(t+1) + WgtV_O*Err_OGate(t)
Err_CGate = Inv_LSig(PreAct_CGate(t))*Act_IGate(t)*Ct_Err(t)
Err_FGate = Inv_Sig(PreAct_FGate(t))*Act_CGate(t-1)*Ct_Err(t)
Err_IGate = Inv_Sig(PreAct_IGate(t))*LSig(PreAct_CGate)*Ct_Err(t)
*(Hout_Err(prev) = Hin_Err
Hout_Err = Out - Hout )
*(Three Activation Functions
Logistic Sigmoid, Tanh, Sigmoid)
*(Three Wgts - U,W,V)
Monday, 17 July 2017
Cannot get this to work yet LSTM!
#include <stdio.h>
#include <stdlib.h>
#include <iostream>
#include <math.h>
#include <vector>
#include <algorithm> // std::transform
#include <functional> // std::plus
using namespace std;
double function_x(double sum,int type, double y);
double function_y(double sum,int type, double y);
double sig_vec(double sum);
double tanh_vec(double sum);
double invtanh_vec(double sum);
double invsig_vec(double sum);
class ltsm_module{
public:
double *Wgt; //4 x Weights for each gate
double Out; //Finall Output
double Hout; //Recurrent Output (Hin for next Neuron)
double Hin;
double In;
double Ctin;
double *Err; //2 x Error from Hout and Out 2 x Error at Hin/X and Ctin
double *Bias; //Each of the bias for each gate
double Lrt,Momentum,Decay;
//Vector Inputs/Output
int Input_Size;
int Wgt_Size;
int wgt_scalar;
vector<double> Out_Vec;
vector<double> Hout_Vec;
vector<double> Hin_Vec;
vector<double> In_Vec; //Input Vector
vector<double> Ctin_Vec;
vector<double> Ct_Vec; //Cell Activation records Ct-1
//Vector Error
vector<double> Ctin_Err;
vector<double> Hin_Err;
vector<double> Hout_Err;
vector<double> Out_Err;
//Act
vector<double> Act_F;
vector<double> Act_I;
vector<double> Act_C;
vector<double> Act_O;
//Vector Wgts
vector<double> Wgt_F;
vector<double> Wgt_I;
vector<double> Wgt_C;
vector<double> Wgt_O;
//Vector PreDelta
vector<double> PreDlta_F;
vector<double> PreDlta_I;
vector<double> PreDlta_C;
vector<double> PreDlta_O;
//Vector PreWgts
vector<double> PreWgt_F;
vector<double> PreWgt_I;
vector<double> PreWgt_C;
vector<double> PreWgt_O;
//Error Vect
vector<double> Err_FGate;
vector<double> Err_IGate;
vector<double> Err_CGate;
//Update weights as though they are 3 weights entering
//a neuron using previous weight values / error and
//Sub activation of previous unit as
//Output Wgt uses Hout Error Over InvSigmoid
//Input Wgt uses Out Error Over ""
//Forget Wgt uses Out Error Over ""
double (*m_pointertofunction)(double,int,double); //Pointer to Func
void init_module(){
Lrt=0.095;
Decay=0.000005;
Momentum=0.005;
wgt_scalar=0;
Wgt_Size = 4;
Wgt = new double[4];
Bias = new double[4];
for(int i=0;i<4;i++){
Wgt[i] = double(rand()/(RAND_MAX + 1.0));
Bias[i] = double(rand()/(RAND_MAX + 1.0));
}
/*Memory to Module Vector*/
Hin_Vec.resize(Input_Size);
In_Vec.resize(Input_Size);
Ctin_Vec.resize(Input_Size);
Ct_Vec.resize(Input_Size);
Ctin_Err.resize(Input_Size);
Hin_Err.resize(Input_Size);
Hout_Err.resize(Input_Size);
Out_Err.resize(Input_Size);
Wgt_F.resize(Wgt_Size);
Wgt_I.resize(Wgt_Size);
Wgt_C.resize(Wgt_Size);
Wgt_O.resize(Wgt_Size);
Act_F.resize(Wgt_Size);
Act_I.resize(Wgt_Size);
Act_C.resize(Wgt_Size);
Act_O.resize(Wgt_Size);
PreWgt_F.resize(Wgt_Size);
PreWgt_I.resize(Wgt_Size);
PreWgt_C.resize(Wgt_Size);
PreWgt_O.resize(Wgt_Size);
PreDlta_F.resize(Wgt_Size);
PreDlta_I.resize(Wgt_Size);
PreDlta_C.resize(Wgt_Size);
PreDlta_O.resize(Wgt_Size);
Err_IGate.resize(Input_Size);
Err_CGate.resize(Input_Size);
Err_FGate.resize(Input_Size);
Ctin_Err.resize(Input_Size);
Init_Wgts(Wgt_F);
Init_Wgts(Wgt_I);
Init_Wgts(Wgt_C);
Init_Wgts(Wgt_O);
}
void update_module(double Hin_,double In_,double Ctin_){
double ft,Ct,It,Ct_,Ot;
double (*sigmoid_)(double,int,double);
double (*tanh_)(double,int,double);
tanh_ = function_y;
sigmoid_ = function_x;
Hin = Hin_;
In = In_;
Ctin= Ctin_;
ft = sigmoid_(Wgt[0]*(Hin+In)+Bias[0],1,0); //forget gate
It = sigmoid_(Wgt[1]*(Hin+In)+Bias[1],1,0); //Include Gate
Ct_= tanh_(Wgt[2]*(Hin+In)+Bias[2],1,0);
Ct = ft*Ctin+It*Ct_;
//Out gate
Ot = sigmoid_(Wgt[3]*(Hin+In)+Bias[3],1,0);
Hout = Ot * tanh(Ct); //Outputs
Out = Ct;
}
void update_module_vec(vector<double> Hin_,vector<double> In_,vector<double> Ctin_){
vector<double> ft,Ct,It,Ct_,Ot;
Hin_Vec = Hin_;
In_Vec = In_;
Ctin_Vec= Ctin_;
//Cycle through each Vec apply sigmoid
vector<double>::iterator it;
vector<double> Sum_Vec;
vector<double> Sum_Vec_;
Sum_Vec.resize(In_.size());
Sum_Vec_.resize(In_.size());
//Forget Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[0]));
}else{
Sum_Vec_ = Apply_Conv(Wgt_F,Sum_Vec);
}
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[0]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
ft = Sum_Vec_;
Act_F = Sum_Vec_;
//Include Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[1]));
}else{
Sum_Vec_ = Apply_Conv(Wgt_I,Sum_Vec);
}
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[1]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
It = Sum_Vec_;
Act_I = Sum_Vec_;
//Out gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[3]));
}else{
Sum_Vec_ = Apply_Conv(Wgt_C,Sum_Vec);
}
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[3]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
Ot = Sum_Vec;
Act_C = Sum_Vec;
//Ct Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[2]));
}else{
Sum_Vec_ = Apply_Conv(Wgt_O,Sum_Vec);
}
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[2]));
//Out_Vec =Sum_Vec_;
//Apply Tanh
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),tanh_vec);
Ct_ = Sum_Vec_; //In online this is At it is the new candidate state
Act_C = Sum_Vec_;
//Multiply ft * Ct-1 //Forget the previous state Ct-1 (Ct_Vec)
transform(Ctin_Vec.begin(),Ctin_Vec.end(),ft.begin(),Sum_Vec.begin(),multiplies<double>());
//Multiply It * Ct_
transform(Ct_.begin(),Ct_.end(),It.begin(),Sum_Vec_.begin(),multiplies<double>());
//Calc Ct
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),plus<double>());
Ct=Sum_Vec;
Ct_Vec=Sum_Vec;
//Apply Tanh for Hout
transform(Ct.begin(),Ct.end(),Sum_Vec.begin(),tanh_vec);
//Calc Hout
transform(Sum_Vec.begin(),Sum_Vec.end(),Ot.begin(),Sum_Vec.begin(),multiplies<double>());
Hout_Vec = Sum_Vec;
Out_Vec = Ct;
}
///Update_Module_Vec() New
//Backprop_Update_Out() New
//Compute Deltas for each Wgt and Delta for Ct
//Uses Ct-1(Previous Cell state) Ct(Current cell state
void error_module_vec(){ //Calculate Error at Hin/X and Ctin using Error at Out and Hout
//Error at Hout = Err_Ht + Err_Hout (Up and Across)
//Err_Ht = Err_Hin/X (Up)
vector<double> Sum_Vec;
vector<double> Sum_Vec_;
Sum_Vec.resize(Input_Size);
Sum_Vec_.resize(Input_Size);
//Multiply Wgt_Out * Hout_Err
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[3]));
}else{
Sum_Vec = Apply_Conv(Wgt_O,Hout_Err);
}
//Inverse Tanh
transform(Sum_Vec.begin(),Sum_Vec.end(),Ctin_Err.begin(),invtanh_vec);
//Multiply Wgt_f * Out_Err
//Change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[0]));
}else{
Sum_Vec = Apply_Conv(Wgt_F,Hout_Err);
}
//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_FGate.begin(),invsig_vec);
//**change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[1]));
}else{
Sum_Vec = Apply_Conv(Wgt_I,Hout_Err);
}
//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_IGate.begin(),invsig_vec);
//**change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[2]));
}else{
Sum_Vec = Apply_Conv(Wgt_C,Hout_Err);
}
//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_CGate.begin(),invtanh_vec);
//Add All 3 Errors
transform(Err_CGate.begin(),Err_CGate.end(),Err_IGate.begin(),Sum_Vec.begin(),plus<double>());
transform(Err_FGate.begin(),Err_FGate.end(),Sum_Vec.begin(),Sum_Vec_.begin(),plus<double>());
Hin_Err = Sum_Vec_;
}
void Update_Wgts(vector<double> &Wgt,vector<double> &PreDlta,vector<double> &PreWgt, vector<double> &Err,vector<double> &Act){
vector<double> delta(Input_Size);
vector<double> Sum_Vec;
vector<double> Sum_Vec_;
Sum_Vec.resize(Input_Size);
Sum_Vec_.resize(Input_Size);
//Add Hin to In Vector Add changed Hin to Ctin
transform(Hin_Vec.begin(),Hin_Vec.end(),In_Vec.begin(),Sum_Vec.begin(),plus<double>());
//Multiply Act_in * Error
transform(Err.begin(),Err.end(),Act.begin(),Sum_Vec_.begin(),multiplies<double>());
//Lrate
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Lrt));
//Momentum - Convolve PreDelta onto all 1's Sum_Vec **********************
//transform(PreDlta.begin(),PreDlta.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Momentum));
fill(Sum_Vec.begin(),Sum_Vec.end(),1);
Apply_Conv(PreDlta,Sum_Vec);
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Momentum));
//Add Momentum and Lrate
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),plus<double>());
//Decay - Convolve PreWgt onto all 1's Sum_Vec **********************
fill(Sum_Vec_.begin(),Sum_Vec_.end(),1);
Apply_Conv(PreWgt,Sum_Vec);
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Decay));
//transform(PreWgt.begin(),PreWgt.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Decay));
//Lrate *Err * Actin + Momentum*PreDlta - Decay*PreWgt
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),minus<double>());
PreWgt = Wgt;
PreDlta = Sum_Vec;
//Update Wgt Matrix
Apply_DeConv(Wgt,Sum_Vec);
//fill(Wgt.begin(),Wgt.end(),1);
}
void Backprop(){
error_module_vec();
Update_Wgts(Wgt_F,PreDlta_F,PreWgt_F,Err_FGate,Act_F);
Update_Wgts(Wgt_I,PreDlta_I,PreWgt_I,Err_IGate,Act_I);
Update_Wgts(Wgt_C,PreDlta_C,PreWgt_C,Err_CGate,Act_C);
Update_Wgts(Wgt_O,PreDlta_O,PreWgt_O,Ctin_Err,Act_O);
}
void Init_Wgts(vector<double> &Wgt){
for(int i=0;i<Wgt_Size;i++){
Wgt[i] = double(rand()/(RAND_MAX+1.0));
}
}
vector<double> Apply_Conv(vector<double> &Wgt_Conv, vector<double> &In_Vec){
//Apply Wgt Convolution Vector to an Input Vector or Err_Vec
vector<double> Out_Vec(Input_Size);
for(int i=0;i<In_Vec.size();i++){
for(int j=0;j<Wgt_Conv.size();j++){
Out_Vec[i]=Wgt_Conv[j]*In_Vec[i];
}
return Out_Vec;
}
}
void Apply_DeConv(vector<double> &Wgt_Conv,vector<double> &Err_Vec){
//Change Wgt Convolution with Err_Vec use to compute Wgt Update
int i=0;
int stoch=1;
while(i<Err_Vec.size()){
for(int j=0;j<Wgt_Conv.size();j++){
Wgt_Conv[j] += Err_Vec[i]/10;
//if(stoch==1&&rand()%2==1){ //Add a stochastic element to Weight Update
i++;
// }
} }
}
void print_module_Err(){
cout<<"Print Error\n";
cout<<"Hout:\n";
for(vector<double>::iterator it=Hout_Vec.begin(); it!=Hout_Vec.end(); ++it){
cout<<" "<<*it<<" ";
}
cout<<"Size of FERR = "<<Err_FGate.size();
cout<<"***********HOut_Err Vec***********\n";
for(vector<double>::iterator it=Hout_Err.begin(); it!=Hout_Err.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********FERR************\n";
for(vector<double>::iterator it=Err_FGate.begin(); it!=Err_FGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********IERR********\n";
for(vector<double>::iterator it=Err_IGate.begin(); it!=Err_IGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********CERR************\n";
for(vector<double>::iterator it=Err_CGate.begin(); it!=Err_CGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
}
void print_module_wgts(){
cout<<"Print Wgts\n";
cout<<"***********FWgts************\n";
for(vector<double>::iterator it=Wgt_F.begin(); it!=Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********IWgts************\n";
for(vector<double>::iterator it=Wgt_I.begin(); it!=Wgt_I.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********CWgts************\n";
for(vector<double>::iterator it=Wgt_C.begin(); it!=Wgt_C.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********OWgts************\n";
for(vector<double>::iterator it=Wgt_O.begin(); it!=Wgt_O.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
}
};
int main(){
cout<<"Hello!\n";
vector<double> Hin_={0,0,0,0};
vector<double> Ctin_={0,0,0,0};
vector<double> In_ = {.1,.004,.6555,6.55};
vector<vector<double>> In_Vec(20,vector<double>(20));
In_Vec[0] = {10};//{.1,.004,.6555,6.55};
In_Vec[1] = {11};//{.7774,.3956,1.76,.006};
In_Vec[2] = {12};//{9.111,.12,.0102,2.96};
In_Vec[3] = {13};//{5.99,.204,6.0001,3.094};
In_Vec[4] = {14};//{2.4965,.694,0.5,22.003};
In_Vec[5] = {15};//{.1,.004,.6555,6.55};
In_Vec[6] = {16};//{.7774,.3956,1.76,.006};
In_Vec[7] = {17};//{9.111,.12,.0102,2.96};
In_Vec[8] = {18};//{5.99,.204,6.0001,3.094};
In_Vec[9] = {19};//{2.4965,.694,0.5,22.003};
//vector<double> Hin_={0};
//vector<double> In_ = {9.55};
ltsm_module** mymod_;
mymod_ = new ltsm_module*[28];
for(int i=0;i<28;i++){
mymod_[i] = new ltsm_module;
mymod_[i]->Input_Size=1;
mymod_[i]->init_module();
}
int kint=0;
int vint=0;
int vint_=0;
int ahead=0;
for(int i=0;i<10;i++){
cout<<"***********Old Wgts************\n";
for(vector<double>::iterator it=mymod_[i]->Wgt_F.begin(); it!=mymod_[i]->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"*******\n"; }
for(int iter=0;iter<5000;iter++){
kint++;
if(kint==9){kint=0;}
mymod_[0]->Hin_Vec = {0};
mymod_[0]->Ctin_Vec= {0};
mymod_[10]->Hin_Vec = {0};
mymod_[10]->Ctin_Vec= {0};
//Feedforward one row of inputs
for(int i=0;i<9;i++){
vint=kint+i;
if(vint>=9){vint=0+vint-9;}
mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,In_Vec[vint],mymod_[i]->Ctin_Vec);
//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;
}
//Add another layer
for(int i=9;i<18;i++){
mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,mymod_[i-9]->Hout_Vec,mymod_[i]->Ctin_Vec);
//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;
}
//Feedback Create Error from identity for test
for(int i=18;i>9;--i){
vint=kint+i-9;
if(vint>=9){vint=0+vint-9;}
vint_=vint+ahead;
if(vint_>=9){vint_=0+vint_-9;}
if(i>9){ //Output Neurons Err
transform(mymod_[i]->Hout_Vec.begin(),mymod_[i]->Hout_Vec.end(),In_Vec[vint_].begin(),mymod_[i]->Hout_Err.begin(),minus<double>());
}
//mymod_[i]->print_module_Err();
mymod_[i]->Backprop(); //Create Hin_Err and change weights
}
for(int i=9;i>-1;--i){
vint=kint+i;
if(vint>=9){vint=0+vint-9;}
vint_=vint+5;
if(vint_>=9){vint_=0+vint_-9;}
if(i>-1){ //Output Neurons Err
mymod_[i]->Hout_Err = mymod_[i+10]->Hin_Err;
}
//mymod_[i]->print_module_Err();
mymod_[i]->Backprop(); //Create Hin_Err and change weights
}
}
//Identity
ltsm_module* mymod;
mymod = new ltsm_module;
mymod->Input_Size=4;
mymod->init_module();
mymod->Hin_Vec={0,0,0,0};
mymod->Ctin_Vec={0,0,0,0};
mymod->update_module_vec(mymod->Hin_Vec,In_,mymod->Ctin_Vec);
//Create Error from identity for test
transform(mymod->Out_Vec.begin(),mymod->Out_Vec.end(),In_.begin(),mymod->Out_Err.begin(),minus<double>());
transform(mymod->Hout_Vec.begin(),mymod->Hout_Vec.end(),In_.begin(),mymod->Hout_Err.begin(),minus<double>());
//mymod->Hout_Err = {.99,.837,.455,1.22};
//mymod->Out_Err = {.99,.837,.455,1.22};
for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"*********************************************************\n";
cout<<"**********Out Vec************\n";
for(vector<double>::iterator it=mymod->Out_Vec.begin(); it!=mymod->Out_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********HOut Vec***********\n";
for(vector<double>::iterator it=mymod->Hout_Vec.begin(); it!=mymod->Hout_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"Training*************************\n";
for(int i=0;i<100;i++){
//Create Error from identity for test
transform(mymod->Out_Vec.begin(),mymod->Out_Vec.end(),In_.begin(),mymod->Out_Err.begin(),minus<double>());
transform(mymod->Hout_Vec.begin(),mymod->Hout_Vec.end(),In_.begin(),mymod->Hout_Err.begin(),minus<double>());
//mymod->print_module_Err();
mymod->Ctin_Vec = mymod->Out_Vec; //t+1
mymod->Hin_Vec = mymod->Hout_Vec;
mymod->update_module_vec(mymod->Hin_Vec,In_,mymod->Ctin_Vec);
mymod->Backprop();
}
cout<<"**********Out Vec************\n";
for(vector<double>::iterator it=mymod->Out_Vec.begin(); it!=mymod->Out_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********HOut Vec***********\n";
for(vector<double>::iterator it=mymod->Hout_Vec.begin(); it!=mymod->Hout_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********HOut Vec***********\n";
mymod_[0]->Hin_Vec = {0};
mymod_[0]->Ctin_Vec= {0};
mymod_[10]->Hin_Vec = {0};
mymod_[10]->Ctin_Vec= {0};
//Feedforward input layer
for(int i=0;i<9;i++){
vint=i;
mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,In_Vec[vint],mymod_[i]->Ctin_Vec);
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;
}
//Feedforward Output layer
for(int i=9;i<18;i++){
vint=i-9+ahead;
if(vint>=9){vint=0+vint-9;}
mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,mymod_[i-9]->Hout_Vec,mymod_[i]->Ctin_Vec);
transform(mymod_[i]->Hout_Vec.begin(),mymod_[i]->Hout_Vec.end(),In_Vec[vint].begin(),mymod_[i]->Hout_Err.begin(),minus<double>());
//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;
for(vector<double>::iterator it=mymod_[i]->Hout_Vec.begin(); it!=mymod_[i]->Hout_Vec.end(); ++it){
cout<<" "<<*it<<" ";
}
for(vector<double>::iterator it=In_Vec[vint].begin(); it!=In_Vec[vint].end(); ++it){
cout<<" "<<*it<<" ";
}
for(vector<double>::iterator it=mymod_[i]->Out_Vec.begin(); it!=mymod_[i]->Out_Vec.end(); ++it){
cout<<" "<<*it<<" ";
}
for(vector<double>::iterator it=mymod_[i]->Hout_Err.begin(); it!=mymod_[i]->Hout_Err.end(); ++it){
cout<<" "<<*it<<" ";
}
cout<<"******\n"; }
for(int i=0;i<10;i++){
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod_[i]->Wgt_F.begin(); it!=mymod_[i]->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"*******\n"; }
}
double invtanh_vec(double sum){
return function_y(sum,-1,0);
}
double invsig_vec(double sum){
return function_x(sum,-1,0);
}
double tanh_vec(double sum){
//double sum = Vec1+Vec2;
return function_y(sum,1,0);
}
double sig_vec(double sum){
//double sum = Vec1+Vec2;
return function_x(sum,1,0);
}
double function_y(double sum,int type, double y){ //input function
double rand_=0;
//sum=sum/1000;
if(type==1){
rand_ = ((double)(rand()%1000))/10000;
sum = tanh(sum)+rand_;
//sum = (2/(1+exp(-2*sum)))-1; //Kick Ass
}
if(type==-1){
//sum = (1-pow(tanh(sum),2))*sum;
sum = 1-(sum*sum)/2;
}
return sum;
}
double function_x(double sum,int type,double y){ //activation function for all hidden layer neurons
double sigmoid,temp;
double rand_=0;
if(type==1){
//rand_ = ((double)(rand()%1000))/1000000;
if(sum>0.5){ //this is actually hardlim not sigmoid
sigmoid = 0.5;}else{
if(sum<-0.5){
sigmoid = 0;}
else{
sigmoid = 1/(1+exp((double) -sum));
}
}
sigmoid = sigmoid+rand_;
}
if(type==-1){
temp = 1/(1+exp((double) -y));
//sigmoid = (0.25-(temp*temp) )* sum;
//sigmoid = y*(1-y) * sum; //derivative sigmoid
sigmoid = sum*(1-sum);
//sigmoid = (1-pow(tanh(y),2))*sum;
}
return sigmoid;
}
#include <stdlib.h>
#include <iostream>
#include <math.h>
#include <vector>
#include <algorithm> // std::transform
#include <functional> // std::plus
using namespace std;
double function_x(double sum,int type, double y);
double function_y(double sum,int type, double y);
double sig_vec(double sum);
double tanh_vec(double sum);
double invtanh_vec(double sum);
double invsig_vec(double sum);
class ltsm_module{
public:
double *Wgt; //4 x Weights for each gate
double Out; //Finall Output
double Hout; //Recurrent Output (Hin for next Neuron)
double Hin;
double In;
double Ctin;
double *Err; //2 x Error from Hout and Out 2 x Error at Hin/X and Ctin
double *Bias; //Each of the bias for each gate
double Lrt,Momentum,Decay;
//Vector Inputs/Output
int Input_Size;
int Wgt_Size;
int wgt_scalar;
vector<double> Out_Vec;
vector<double> Hout_Vec;
vector<double> Hin_Vec;
vector<double> In_Vec; //Input Vector
vector<double> Ctin_Vec;
vector<double> Ct_Vec; //Cell Activation records Ct-1
//Vector Error
vector<double> Ctin_Err;
vector<double> Hin_Err;
vector<double> Hout_Err;
vector<double> Out_Err;
//Act
vector<double> Act_F;
vector<double> Act_I;
vector<double> Act_C;
vector<double> Act_O;
//Vector Wgts
vector<double> Wgt_F;
vector<double> Wgt_I;
vector<double> Wgt_C;
vector<double> Wgt_O;
//Vector PreDelta
vector<double> PreDlta_F;
vector<double> PreDlta_I;
vector<double> PreDlta_C;
vector<double> PreDlta_O;
//Vector PreWgts
vector<double> PreWgt_F;
vector<double> PreWgt_I;
vector<double> PreWgt_C;
vector<double> PreWgt_O;
//Error Vect
vector<double> Err_FGate;
vector<double> Err_IGate;
vector<double> Err_CGate;
//Update weights as though they are 3 weights entering
//a neuron using previous weight values / error and
//Sub activation of previous unit as
//Output Wgt uses Hout Error Over InvSigmoid
//Input Wgt uses Out Error Over ""
//Forget Wgt uses Out Error Over ""
double (*m_pointertofunction)(double,int,double); //Pointer to Func
void init_module(){
Lrt=0.095;
Decay=0.000005;
Momentum=0.005;
wgt_scalar=0;
Wgt_Size = 4;
Wgt = new double[4];
Bias = new double[4];
for(int i=0;i<4;i++){
Wgt[i] = double(rand()/(RAND_MAX + 1.0));
Bias[i] = double(rand()/(RAND_MAX + 1.0));
}
/*Memory to Module Vector*/
Hin_Vec.resize(Input_Size);
In_Vec.resize(Input_Size);
Ctin_Vec.resize(Input_Size);
Ct_Vec.resize(Input_Size);
Ctin_Err.resize(Input_Size);
Hin_Err.resize(Input_Size);
Hout_Err.resize(Input_Size);
Out_Err.resize(Input_Size);
Wgt_F.resize(Wgt_Size);
Wgt_I.resize(Wgt_Size);
Wgt_C.resize(Wgt_Size);
Wgt_O.resize(Wgt_Size);
Act_F.resize(Wgt_Size);
Act_I.resize(Wgt_Size);
Act_C.resize(Wgt_Size);
Act_O.resize(Wgt_Size);
PreWgt_F.resize(Wgt_Size);
PreWgt_I.resize(Wgt_Size);
PreWgt_C.resize(Wgt_Size);
PreWgt_O.resize(Wgt_Size);
PreDlta_F.resize(Wgt_Size);
PreDlta_I.resize(Wgt_Size);
PreDlta_C.resize(Wgt_Size);
PreDlta_O.resize(Wgt_Size);
Err_IGate.resize(Input_Size);
Err_CGate.resize(Input_Size);
Err_FGate.resize(Input_Size);
Ctin_Err.resize(Input_Size);
Init_Wgts(Wgt_F);
Init_Wgts(Wgt_I);
Init_Wgts(Wgt_C);
Init_Wgts(Wgt_O);
}
void update_module(double Hin_,double In_,double Ctin_){
double ft,Ct,It,Ct_,Ot;
double (*sigmoid_)(double,int,double);
double (*tanh_)(double,int,double);
tanh_ = function_y;
sigmoid_ = function_x;
Hin = Hin_;
In = In_;
Ctin= Ctin_;
ft = sigmoid_(Wgt[0]*(Hin+In)+Bias[0],1,0); //forget gate
It = sigmoid_(Wgt[1]*(Hin+In)+Bias[1],1,0); //Include Gate
Ct_= tanh_(Wgt[2]*(Hin+In)+Bias[2],1,0);
Ct = ft*Ctin+It*Ct_;
//Out gate
Ot = sigmoid_(Wgt[3]*(Hin+In)+Bias[3],1,0);
Hout = Ot * tanh(Ct); //Outputs
Out = Ct;
}
void update_module_vec(vector<double> Hin_,vector<double> In_,vector<double> Ctin_){
vector<double> ft,Ct,It,Ct_,Ot;
Hin_Vec = Hin_;
In_Vec = In_;
Ctin_Vec= Ctin_;
//Cycle through each Vec apply sigmoid
vector<double>::iterator it;
vector<double> Sum_Vec;
vector<double> Sum_Vec_;
Sum_Vec.resize(In_.size());
Sum_Vec_.resize(In_.size());
//Forget Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[0]));
}else{
Sum_Vec_ = Apply_Conv(Wgt_F,Sum_Vec);
}
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[0]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
ft = Sum_Vec_;
Act_F = Sum_Vec_;
//Include Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[1]));
}else{
Sum_Vec_ = Apply_Conv(Wgt_I,Sum_Vec);
}
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[1]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
It = Sum_Vec_;
Act_I = Sum_Vec_;
//Out gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[3]));
}else{
Sum_Vec_ = Apply_Conv(Wgt_C,Sum_Vec);
}
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[3]));
//Apply Sigmoid
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),sig_vec);
Ot = Sum_Vec;
Act_C = Sum_Vec;
//Ct Gate
//Add Hin to In Vector Add
transform(Hin_.begin(),Hin_.end(),In_.begin(),Sum_Vec.begin(),plus<double>());
//Multiply by Wgt
if(wgt_scalar==1){
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[2]));
}else{
Sum_Vec_ = Apply_Conv(Wgt_O,Sum_Vec);
}
//Add Bias
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(plus<double>(),Bias[2]));
//Out_Vec =Sum_Vec_;
//Apply Tanh
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),tanh_vec);
Ct_ = Sum_Vec_; //In online this is At it is the new candidate state
Act_C = Sum_Vec_;
//Multiply ft * Ct-1 //Forget the previous state Ct-1 (Ct_Vec)
transform(Ctin_Vec.begin(),Ctin_Vec.end(),ft.begin(),Sum_Vec.begin(),multiplies<double>());
//Multiply It * Ct_
transform(Ct_.begin(),Ct_.end(),It.begin(),Sum_Vec_.begin(),multiplies<double>());
//Calc Ct
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),plus<double>());
Ct=Sum_Vec;
Ct_Vec=Sum_Vec;
//Apply Tanh for Hout
transform(Ct.begin(),Ct.end(),Sum_Vec.begin(),tanh_vec);
//Calc Hout
transform(Sum_Vec.begin(),Sum_Vec.end(),Ot.begin(),Sum_Vec.begin(),multiplies<double>());
Hout_Vec = Sum_Vec;
Out_Vec = Ct;
}
///Update_Module_Vec() New
//Backprop_Update_Out() New
//Compute Deltas for each Wgt and Delta for Ct
//Uses Ct-1(Previous Cell state) Ct(Current cell state
void error_module_vec(){ //Calculate Error at Hin/X and Ctin using Error at Out and Hout
//Error at Hout = Err_Ht + Err_Hout (Up and Across)
//Err_Ht = Err_Hin/X (Up)
vector<double> Sum_Vec;
vector<double> Sum_Vec_;
Sum_Vec.resize(Input_Size);
Sum_Vec_.resize(Input_Size);
//Multiply Wgt_Out * Hout_Err
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[3]));
}else{
Sum_Vec = Apply_Conv(Wgt_O,Hout_Err);
}
//Inverse Tanh
transform(Sum_Vec.begin(),Sum_Vec.end(),Ctin_Err.begin(),invtanh_vec);
//Multiply Wgt_f * Out_Err
//Change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[0]));
}else{
Sum_Vec = Apply_Conv(Wgt_F,Hout_Err);
}
//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_FGate.begin(),invsig_vec);
//**change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[1]));
}else{
Sum_Vec = Apply_Conv(Wgt_I,Hout_Err);
}
//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_IGate.begin(),invsig_vec);
//**change to Hout
if(wgt_scalar==1){
transform(Hout_Err.begin(),Hout_Err.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Wgt[2]));
}else{
Sum_Vec = Apply_Conv(Wgt_C,Hout_Err);
}
//Inverse Sigmoid
transform(Sum_Vec.begin(),Sum_Vec.end(),Err_CGate.begin(),invtanh_vec);
//Add All 3 Errors
transform(Err_CGate.begin(),Err_CGate.end(),Err_IGate.begin(),Sum_Vec.begin(),plus<double>());
transform(Err_FGate.begin(),Err_FGate.end(),Sum_Vec.begin(),Sum_Vec_.begin(),plus<double>());
Hin_Err = Sum_Vec_;
}
void Update_Wgts(vector<double> &Wgt,vector<double> &PreDlta,vector<double> &PreWgt, vector<double> &Err,vector<double> &Act){
vector<double> delta(Input_Size);
vector<double> Sum_Vec;
vector<double> Sum_Vec_;
Sum_Vec.resize(Input_Size);
Sum_Vec_.resize(Input_Size);
//Add Hin to In Vector Add changed Hin to Ctin
transform(Hin_Vec.begin(),Hin_Vec.end(),In_Vec.begin(),Sum_Vec.begin(),plus<double>());
//Multiply Act_in * Error
transform(Err.begin(),Err.end(),Act.begin(),Sum_Vec_.begin(),multiplies<double>());
//Lrate
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Lrt));
//Momentum - Convolve PreDelta onto all 1's Sum_Vec **********************
//transform(PreDlta.begin(),PreDlta.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Momentum));
fill(Sum_Vec.begin(),Sum_Vec.end(),1);
Apply_Conv(PreDlta,Sum_Vec);
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec.begin(),bind1st(multiplies<double>(),Momentum));
//Add Momentum and Lrate
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),plus<double>());
//Decay - Convolve PreWgt onto all 1's Sum_Vec **********************
fill(Sum_Vec_.begin(),Sum_Vec_.end(),1);
Apply_Conv(PreWgt,Sum_Vec);
transform(Sum_Vec_.begin(),Sum_Vec_.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Decay));
//transform(PreWgt.begin(),PreWgt.end(),Sum_Vec_.begin(),bind1st(multiplies<double>(),Decay));
//Lrate *Err * Actin + Momentum*PreDlta - Decay*PreWgt
transform(Sum_Vec.begin(),Sum_Vec.end(),Sum_Vec_.begin(),Sum_Vec.begin(),minus<double>());
PreWgt = Wgt;
PreDlta = Sum_Vec;
//Update Wgt Matrix
Apply_DeConv(Wgt,Sum_Vec);
//fill(Wgt.begin(),Wgt.end(),1);
}
void Backprop(){
error_module_vec();
Update_Wgts(Wgt_F,PreDlta_F,PreWgt_F,Err_FGate,Act_F);
Update_Wgts(Wgt_I,PreDlta_I,PreWgt_I,Err_IGate,Act_I);
Update_Wgts(Wgt_C,PreDlta_C,PreWgt_C,Err_CGate,Act_C);
Update_Wgts(Wgt_O,PreDlta_O,PreWgt_O,Ctin_Err,Act_O);
}
void Init_Wgts(vector<double> &Wgt){
for(int i=0;i<Wgt_Size;i++){
Wgt[i] = double(rand()/(RAND_MAX+1.0));
}
}
vector<double> Apply_Conv(vector<double> &Wgt_Conv, vector<double> &In_Vec){
//Apply Wgt Convolution Vector to an Input Vector or Err_Vec
vector<double> Out_Vec(Input_Size);
for(int i=0;i<In_Vec.size();i++){
for(int j=0;j<Wgt_Conv.size();j++){
Out_Vec[i]=Wgt_Conv[j]*In_Vec[i];
}
return Out_Vec;
}
}
void Apply_DeConv(vector<double> &Wgt_Conv,vector<double> &Err_Vec){
//Change Wgt Convolution with Err_Vec use to compute Wgt Update
int i=0;
int stoch=1;
while(i<Err_Vec.size()){
for(int j=0;j<Wgt_Conv.size();j++){
Wgt_Conv[j] += Err_Vec[i]/10;
//if(stoch==1&&rand()%2==1){ //Add a stochastic element to Weight Update
i++;
// }
} }
}
void print_module_Err(){
cout<<"Print Error\n";
cout<<"Hout:\n";
for(vector<double>::iterator it=Hout_Vec.begin(); it!=Hout_Vec.end(); ++it){
cout<<" "<<*it<<" ";
}
cout<<"Size of FERR = "<<Err_FGate.size();
cout<<"***********HOut_Err Vec***********\n";
for(vector<double>::iterator it=Hout_Err.begin(); it!=Hout_Err.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********FERR************\n";
for(vector<double>::iterator it=Err_FGate.begin(); it!=Err_FGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********IERR********\n";
for(vector<double>::iterator it=Err_IGate.begin(); it!=Err_IGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********CERR************\n";
for(vector<double>::iterator it=Err_CGate.begin(); it!=Err_CGate.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
}
void print_module_wgts(){
cout<<"Print Wgts\n";
cout<<"***********FWgts************\n";
for(vector<double>::iterator it=Wgt_F.begin(); it!=Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********IWgts************\n";
for(vector<double>::iterator it=Wgt_I.begin(); it!=Wgt_I.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********CWgts************\n";
for(vector<double>::iterator it=Wgt_C.begin(); it!=Wgt_C.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}cout<<"***********OWgts************\n";
for(vector<double>::iterator it=Wgt_O.begin(); it!=Wgt_O.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
}
};
int main(){
cout<<"Hello!\n";
vector<double> Hin_={0,0,0,0};
vector<double> Ctin_={0,0,0,0};
vector<double> In_ = {.1,.004,.6555,6.55};
vector<vector<double>> In_Vec(20,vector<double>(20));
In_Vec[0] = {10};//{.1,.004,.6555,6.55};
In_Vec[1] = {11};//{.7774,.3956,1.76,.006};
In_Vec[2] = {12};//{9.111,.12,.0102,2.96};
In_Vec[3] = {13};//{5.99,.204,6.0001,3.094};
In_Vec[4] = {14};//{2.4965,.694,0.5,22.003};
In_Vec[5] = {15};//{.1,.004,.6555,6.55};
In_Vec[6] = {16};//{.7774,.3956,1.76,.006};
In_Vec[7] = {17};//{9.111,.12,.0102,2.96};
In_Vec[8] = {18};//{5.99,.204,6.0001,3.094};
In_Vec[9] = {19};//{2.4965,.694,0.5,22.003};
//vector<double> Hin_={0};
//vector<double> In_ = {9.55};
ltsm_module** mymod_;
mymod_ = new ltsm_module*[28];
for(int i=0;i<28;i++){
mymod_[i] = new ltsm_module;
mymod_[i]->Input_Size=1;
mymod_[i]->init_module();
}
int kint=0;
int vint=0;
int vint_=0;
int ahead=0;
for(int i=0;i<10;i++){
cout<<"***********Old Wgts************\n";
for(vector<double>::iterator it=mymod_[i]->Wgt_F.begin(); it!=mymod_[i]->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"*******\n"; }
for(int iter=0;iter<5000;iter++){
kint++;
if(kint==9){kint=0;}
mymod_[0]->Hin_Vec = {0};
mymod_[0]->Ctin_Vec= {0};
mymod_[10]->Hin_Vec = {0};
mymod_[10]->Ctin_Vec= {0};
//Feedforward one row of inputs
for(int i=0;i<9;i++){
vint=kint+i;
if(vint>=9){vint=0+vint-9;}
mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,In_Vec[vint],mymod_[i]->Ctin_Vec);
//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;
}
//Add another layer
for(int i=9;i<18;i++){
mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,mymod_[i-9]->Hout_Vec,mymod_[i]->Ctin_Vec);
//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;
}
//Feedback Create Error from identity for test
for(int i=18;i>9;--i){
vint=kint+i-9;
if(vint>=9){vint=0+vint-9;}
vint_=vint+ahead;
if(vint_>=9){vint_=0+vint_-9;}
if(i>9){ //Output Neurons Err
transform(mymod_[i]->Hout_Vec.begin(),mymod_[i]->Hout_Vec.end(),In_Vec[vint_].begin(),mymod_[i]->Hout_Err.begin(),minus<double>());
}
//mymod_[i]->print_module_Err();
mymod_[i]->Backprop(); //Create Hin_Err and change weights
}
for(int i=9;i>-1;--i){
vint=kint+i;
if(vint>=9){vint=0+vint-9;}
vint_=vint+5;
if(vint_>=9){vint_=0+vint_-9;}
if(i>-1){ //Output Neurons Err
mymod_[i]->Hout_Err = mymod_[i+10]->Hin_Err;
}
//mymod_[i]->print_module_Err();
mymod_[i]->Backprop(); //Create Hin_Err and change weights
}
}
//Identity
ltsm_module* mymod;
mymod = new ltsm_module;
mymod->Input_Size=4;
mymod->init_module();
mymod->Hin_Vec={0,0,0,0};
mymod->Ctin_Vec={0,0,0,0};
mymod->update_module_vec(mymod->Hin_Vec,In_,mymod->Ctin_Vec);
//Create Error from identity for test
transform(mymod->Out_Vec.begin(),mymod->Out_Vec.end(),In_.begin(),mymod->Out_Err.begin(),minus<double>());
transform(mymod->Hout_Vec.begin(),mymod->Hout_Vec.end(),In_.begin(),mymod->Hout_Err.begin(),minus<double>());
//mymod->Hout_Err = {.99,.837,.455,1.22};
//mymod->Out_Err = {.99,.837,.455,1.22};
for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"*********************************************************\n";
cout<<"**********Out Vec************\n";
for(vector<double>::iterator it=mymod->Out_Vec.begin(); it!=mymod->Out_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********HOut Vec***********\n";
for(vector<double>::iterator it=mymod->Hout_Vec.begin(); it!=mymod->Hout_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"Training*************************\n";
for(int i=0;i<100;i++){
//Create Error from identity for test
transform(mymod->Out_Vec.begin(),mymod->Out_Vec.end(),In_.begin(),mymod->Out_Err.begin(),minus<double>());
transform(mymod->Hout_Vec.begin(),mymod->Hout_Vec.end(),In_.begin(),mymod->Hout_Err.begin(),minus<double>());
//mymod->print_module_Err();
mymod->Ctin_Vec = mymod->Out_Vec; //t+1
mymod->Hin_Vec = mymod->Hout_Vec;
mymod->update_module_vec(mymod->Hin_Vec,In_,mymod->Ctin_Vec);
mymod->Backprop();
}
cout<<"**********Out Vec************\n";
for(vector<double>::iterator it=mymod->Out_Vec.begin(); it!=mymod->Out_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********HOut Vec***********\n";
for(vector<double>::iterator it=mymod->Hout_Vec.begin(); it!=mymod->Hout_Vec.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod->Wgt_F.begin(); it!=mymod->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"***********HOut Vec***********\n";
mymod_[0]->Hin_Vec = {0};
mymod_[0]->Ctin_Vec= {0};
mymod_[10]->Hin_Vec = {0};
mymod_[10]->Ctin_Vec= {0};
//Feedforward input layer
for(int i=0;i<9;i++){
vint=i;
mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,In_Vec[vint],mymod_[i]->Ctin_Vec);
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;
}
//Feedforward Output layer
for(int i=9;i<18;i++){
vint=i-9+ahead;
if(vint>=9){vint=0+vint-9;}
mymod_[i]->update_module_vec(mymod_[i]->Hin_Vec,mymod_[i-9]->Hout_Vec,mymod_[i]->Ctin_Vec);
transform(mymod_[i]->Hout_Vec.begin(),mymod_[i]->Hout_Vec.end(),In_Vec[vint].begin(),mymod_[i]->Hout_Err.begin(),minus<double>());
//mymod_[i+1]->Ctin_Vec = mymod_[i]->Out_Vec;
mymod_[i+1]->Hin_Vec = mymod_[i]->Hout_Vec;
for(vector<double>::iterator it=mymod_[i]->Hout_Vec.begin(); it!=mymod_[i]->Hout_Vec.end(); ++it){
cout<<" "<<*it<<" ";
}
for(vector<double>::iterator it=In_Vec[vint].begin(); it!=In_Vec[vint].end(); ++it){
cout<<" "<<*it<<" ";
}
for(vector<double>::iterator it=mymod_[i]->Out_Vec.begin(); it!=mymod_[i]->Out_Vec.end(); ++it){
cout<<" "<<*it<<" ";
}
for(vector<double>::iterator it=mymod_[i]->Hout_Err.begin(); it!=mymod_[i]->Hout_Err.end(); ++it){
cout<<" "<<*it<<" ";
}
cout<<"******\n"; }
for(int i=0;i<10;i++){
cout<<"***********New Wgts************\n";
for(vector<double>::iterator it=mymod_[i]->Wgt_F.begin(); it!=mymod_[i]->Wgt_F.end(); ++it){
cout<<" "<<*it;
cout<<"\n";
}
cout<<"*******\n"; }
}
double invtanh_vec(double sum){
return function_y(sum,-1,0);
}
double invsig_vec(double sum){
return function_x(sum,-1,0);
}
double tanh_vec(double sum){
//double sum = Vec1+Vec2;
return function_y(sum,1,0);
}
double sig_vec(double sum){
//double sum = Vec1+Vec2;
return function_x(sum,1,0);
}
double function_y(double sum,int type, double y){ //input function
double rand_=0;
//sum=sum/1000;
if(type==1){
rand_ = ((double)(rand()%1000))/10000;
sum = tanh(sum)+rand_;
//sum = (2/(1+exp(-2*sum)))-1; //Kick Ass
}
if(type==-1){
//sum = (1-pow(tanh(sum),2))*sum;
sum = 1-(sum*sum)/2;
}
return sum;
}
double function_x(double sum,int type,double y){ //activation function for all hidden layer neurons
double sigmoid,temp;
double rand_=0;
if(type==1){
//rand_ = ((double)(rand()%1000))/1000000;
if(sum>0.5){ //this is actually hardlim not sigmoid
sigmoid = 0.5;}else{
if(sum<-0.5){
sigmoid = 0;}
else{
sigmoid = 1/(1+exp((double) -sum));
}
}
sigmoid = sigmoid+rand_;
}
if(type==-1){
temp = 1/(1+exp((double) -y));
//sigmoid = (0.25-(temp*temp) )* sum;
//sigmoid = y*(1-y) * sum; //derivative sigmoid
sigmoid = sum*(1-sum);
//sigmoid = (1-pow(tanh(y),2))*sum;
}
return sigmoid;
}
Sunday, 11 June 2017
Long Term Short Memory LTSM or FICO for short.
The anacronym FICO stands for Forget Include Copy Out these are the gates for the LTSM module based on the infamous LSTM module.
These are neurons with gates controlled by weights with sigmoid activation functions. An Input(Xt) is combined with the previous output of the last cell (Hin) or Hout at t-1. The gates determine whether the previous cells Activation (Ctin) is Forgotten or not (F - Forget Gate). They also determine whether the Input combine with Hin is included in computing the Activation(I - Include gate) of the unit and whether the Activation is passed to Hout (O - Output gate).
1. Include Previous Output?
ht-1 + Xt -> sigmoid -> forget 1/0 -> Adds PreOutput to Output(Ct)
ft = sigmoid(Wf.[ht-1,xt] + bf
2. Include new Input?
ht-1 + xt -> sigmoid -> include 1/0 -> Adds tanh(Input+PreHidden) to Output(Ct)
It = sigmoid(Wi.[ht-1,xt] + bi
~Ct = tanh(Wc.[ht-1,xt] +bc)
Ct = ft*Ct-1 + it*~Ct
3. Include Output as Hidden?
ht-1 + xt -> sigmoid -> include 1/0 -> Adds tanh(Output) to Hidden(ht)
Ot = sigmoid(Wo.[ht-1,xt] + bo
ht = Ot * tanh(Ct)
Four Weights: Wf Wi Wc Wo or F.I.C.O for short!
I connected everything up in this way and got it to learn an Identity function for one module and I am poised to use it on a time series which I have read is a good application for the LSTM.
I have found however that for Identity - a learnt mapping from input to output sigmoids insteads of the tanh functions work better.
Also for the Identity mapping test I fed back the previous Outputs to their Inputs Cout -> Cin Hout -> Hin for Xt = X.
We shall see if it works out of the box for time series will it learn how to count and will it predict a sequence of numbers.
I have written the whole unit using vectors making it suitable for taking inputs and producing outputs for a convolutional process. Making it perhaps suitable for image processing.
The std::vector object is simply a faster and easier way of processing 2d arrays.
Also I have made the Weight arrays flexible to being 1d or 2d. So its all go for Long Term Short Memeory!!
These are neurons with gates controlled by weights with sigmoid activation functions. An Input(Xt) is combined with the previous output of the last cell (Hin) or Hout at t-1. The gates determine whether the previous cells Activation (Ctin) is Forgotten or not (F - Forget Gate). They also determine whether the Input combine with Hin is included in computing the Activation(I - Include gate) of the unit and whether the Activation is passed to Hout (O - Output gate).
1. Include Previous Output?
ht-1 + Xt -> sigmoid -> forget 1/0 -> Adds PreOutput to Output(Ct)
ft = sigmoid(Wf.[ht-1,xt] + bf
2. Include new Input?
ht-1 + xt -> sigmoid -> include 1/0 -> Adds tanh(Input+PreHidden) to Output(Ct)
It = sigmoid(Wi.[ht-1,xt] + bi
~Ct = tanh(Wc.[ht-1,xt] +bc)
Ct = ft*Ct-1 + it*~Ct
3. Include Output as Hidden?
ht-1 + xt -> sigmoid -> include 1/0 -> Adds tanh(Output) to Hidden(ht)
Ot = sigmoid(Wo.[ht-1,xt] + bo
ht = Ot * tanh(Ct)
Four Weights: Wf Wi Wc Wo or F.I.C.O for short!
I connected everything up in this way and got it to learn an Identity function for one module and I am poised to use it on a time series which I have read is a good application for the LSTM.
I have found however that for Identity - a learnt mapping from input to output sigmoids insteads of the tanh functions work better.
Also for the Identity mapping test I fed back the previous Outputs to their Inputs Cout -> Cin Hout -> Hin for Xt = X.
We shall see if it works out of the box for time series will it learn how to count and will it predict a sequence of numbers.
I have written the whole unit using vectors making it suitable for taking inputs and producing outputs for a convolutional process. Making it perhaps suitable for image processing.
The std::vector object is simply a faster and easier way of processing 2d arrays.
Also I have made the Weight arrays flexible to being 1d or 2d. So its all go for Long Term Short Memeory!!
Subscribe to:
Posts (Atom)