T&C LAB-AI
Robotics
Neural Network
Lecture 3
Jeong-Yean Yang
2020/10/22
1
T&C LAB-AI
Neural Network
Basic Questions Before Learning it
1
2
T&C LAB-AI
Robotics
We Learn Regression Model
Question: How we do regression for Next Data?
• It is NOT a linear and It is NOT a squared function
• It looks like a sine function but it is NOT.
• How we do?
3
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Multiple Lines?
T&C LAB-AI
Robotics
If we divide Three Ranges,
We use Regression
4
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
R1
R2
R3
( 1)
( 2)
( 3)
2
2
2
2
1
1
2
2
3
3
1
2
3
||
(
) ||
||
(
) ||
||
(
) ||
N R
N R
N R
N
i
i
i
i
i
i
i
i
i R
i R
i R
J
e
y
a x
b
y
a x
b
y
a x
b
Who Determines Three or More Region?
Three Regions are Correct?
???
T&C LAB-AI
Robotics
How can we do it?
5
T&C LAB-AI
Robotics
If we add all lines, is it good?
6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
2
2
2
2
1
1
2
2
3
3
||
(
) ||
||
(
) ||
||
(
) ||
N
i
i
i
i
i
i
i
i
i R
J
e
y
a x
b
y
a x
b
y
a x
b
T&C LAB-AI
Robotics
The Sum of Lines is Always a Line
• Given condition)
– Region,R =R1+R2+R3
– We cannot determine Proper Region, R1, R2, and R3.
– We must use R.
• Then, if we use ax+b for every Region, R,
• Thus, we need another Nonlinear function for
7
( 1)
( 2)
( 3)
2
2
2
2
1
1
2
2
3
3
1
2
3
||
(
) ||
||
(
) ||
||
(
) ||
N R
N R
N R
N
i
i
i
i
i
i
i
i
i R
i R
i R
J
e
y
a x
b
y
a x
b
y
a x
b
2
2
1
1
2
2
3
3
ˆ
||
||
ˆ
'
'
N
i
i
i
i R
J
e
y
y
y
a x b
a x b
a x b
a x b
ˆy
T&C LAB-AI
Robotics
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Some Function shapes like,
• Linear function cannot satisfy specific Region
• We Must use every Region Function must not be Line
Non linear function
8
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
T&C LAB-AI
Robotics
Some function shapes like
9
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
• Non linear functions are the candidate for Neural Network
• Even sine or cosine functions works.
T&C LAB-AI
Robotics
Some function shapes like
10
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
• Triangle curves also work
• It is also non linear Remind linearity condition
– Triangle curve is NOT equal that av1+bv2 = v
T&C LAB-AI
Robotics
Remind Boundary Decision
with Linear Function requires Sign func.
• Sign function is also a nonlinear function Phi,
11
( )
(
)
Y
x
sign ax b
T&C LAB-AI
Robotics
Which Nonlinear Functions are
the Best for Kernel?
• No answer, Definitely
12
-5
-4
-3
-2
-1
0
1
2
3
4
5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
2
1
exp
x
RBF
b
b
Radial Basis Function
-5
-4
-3
-2
-1
0
1
2
3
4
5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1
1
Sigmoidal
x
e
Function
-5
-4
-3
-2
-1
0
1
2
3
4
5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0
0
0
Re
(Rectifier linear Unit)
x
ax x
LU Function
T&C LAB-AI
Robotics
Define Kernel Function
• These functions are called Kernel function, K
• All Input data are thought as the results of
• Is it good for every cases?
– 1. In most cases, the results are bad.
One kernel function cannot satisfy all possible cases
– 2. Thus, we use many kernel functions Modern Learnings
13
K(X, Z)
(
( )
)
(Z)
X
Y
X
:
( ) :
function is a dot product of map data
X Input vector
X
Nonlinear function
Kernel
x
Y
( )
X
linear
T&C LAB-AI
Robotics
Kernel Function K or
• Definition of Kernel function is a dot product of
– But in many books kernel function K and nonlinear function
Pi are in mixed usages.
• Basic of Kernel trick
– K is a dot product of Pi’s, thus K is Scalar function
– Nonlinear function Pi, simplifies high dimensional input
into unknown feature space
– Through Non linear Pi, Kernel can simplifies and linearize a
given problem.
14
1
2
K
Scalar
T&C LAB-AI
Kernel Functions are thought as
Nonlinear Regression
2
15
T&C LAB-AI
Robotics
Kernel Function as Regression
• Use Some function like Gaussian function
• Gaussian function( Probabilistic Density Function)
• Radial-basis function
16
x
Y
2
1
1
( )
exp
2
2
~
( , )
x
pdf x
x
N
2
( )
exp
( )
x
RBF x
x
b
T&C LAB-AI
Robotics
Regression with RBF 1
• Objective Function, J is,
17
x
Y
y
ax b
2
2
2
( )
exp
i
i
i
i
i
i
x
c
J
y
x
y
b
2
2
2
( )
( )
2
( )
4
exp
exp
i
i
i
i
i
i
i
i
i
i
i
i
x
J
y
x
y
x
c
c
c
x
c
x
c
x
c
y
b
b
b
c
T&C LAB-AI
Robotics
Tip for operation like (x-b)^2
• X=linspace(s,t,number)
– Ex) x= linspace(0,5,6) =[ 0,1,2,3,4,5]
• Matrix multiplication has two types.
• Matlab has two operations
– Matrix = A*B Hadarmard Product = A.*B
• loop.sys also has two operations
– Matrix = A*B Hadarmard Product = A.mul(B)
18
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
a
b
a
b
aa
bc
ab
bd
AB
c
d
c
d
ca
dc
cb
dd
a
b
a
b
aa
bb
A B
c
d
c
d
cc
dd
General Matrix
multiplication
Hadamard Product
T&C LAB-AI
Robotics
ex/ml/l6rbf1
19
2
2
2
( )
exp
i
i
i
i
i
i
x
c
J
y
x
y
b
( )
i
i
i
e
y
x
2
2
4
exp
exp
4
( )
i
i
i
i
i
i
i
i
i
x
c
x
c
x
c
J
y
c
b
b
b
x
c
e
x
b
T&C LAB-AI
Robotics
Ex/ml/l6rbf1
20
Blue: y=0.5x+0.3
Red: RBF function
• c is the initial center of RBF.
• The result differs with various Initial guess of C.
– Guess C = 0, 0.5, 1, and so on.
ERRORS!
T&C LAB-AI
Robotics
Regression with RBF 2
• Objective Function, J is,
21
x
Y
y
ax b
2
2
2
( )
exp
i
i
i
i
i
i
x
c
J
y
w
x
y
w
b
2
2
( )
2
( )
( )
( )
4
( )
( )
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
J
y
w
x
y
w
x
x
w
c
x
c
J
y
w
x
y
w
x
w
x
c
c
b
c
w
T&C LAB-AI
Robotics
Ex/ml/l6rbf2
• Result Comparison
– Use guess, c= 0
• Weight, w works for the better estimation!!
22
l6rbf1
l6rbf2
T&C LAB-AI
Robotics
Regression with RBF 3
• Objective Function, J is,
23
x
Y
y
ax b
2
2
2
0
0
( )
exp
i
i
i
i
i
i
x
c
J
y
w
x
w
y
w
w
b
2
0
0
0
2
0
0
2
0
0
( )
2
( )
( )
( )
( )
2
( )
4
( )
(
1
)
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
J
y
w
x
w
y
w
x
w
x
J
y
w
x
w
y
w
x
w
w
c
w
c
x
c
J
y
w
x
w
y
w
x
w w
x
c
c
b
c
0
w
T&C LAB-AI
Robotics
Ex/ml/l6rbf3
24
2
2
2
0
0
( )
exp
i
i
i
i
i
i
x
c
J
y
w
x
w
y
w
w
b
2
0
0
2
2
0
0
0
0
0
( )
2
( )
( )
( )
4
( )
( )
( )
2
( )
1
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
J
y
w
x
w
w
c
y
w
x
w
x
J
y
w
x
w
J
y
w
x
w
w
c
c
c
x
c
y
w
x
w w
x
b
y
w
x
w
T&C LAB-AI
Robotics
Comparison
between ex/ml/l6rbf2 and ex/ml/l6rbf3
• Result Comparison
– Use guess, c= 1
• Both weight, w and bias w0 work for the better
estimation.
25
l6rbf2
l6rbf3
T&C LAB-AI
Robotics
Definition of Weight and Bias Parameter
• Only Kernel function
• Use Weight, w
• Use Bias, w0
26
2
( )
i
i
i
J
y
x
2
( )
i
i
i
J
y
w
x
2
0
( )
i
i
i
J
y
w
x
w
T&C LAB-AI
Neural Network
Equation
3
27
T&C LAB-AI
Robotics
Weight
• Non linear function Phi, strengthens some value
in every input space
• Weight is a linear combination of Phi,
28
( )
X
( )
a
X
( )
b
X
(wX)
linear
linear
nonliear
T&C LAB-AI
Robotics
Weight
• Linear weight
• Nonlinear weight
• Linear weight can be simplified as in Homogeneous
Transform
29
( )
( )
Y
a
X
b
W
X
(
)
Y
wX
( )
( )
( )
1
1
X
X
Y
a
X
b
a
b
W
1
1
1
1
1
1
1
2
2
2
2
2
2
2
(X )
(X )
0
(X )
(X)
(X )
0
1
Y
a
b
a
b
Y
W
Y
a
b
a
b
Example
T&C LAB-AI
Robotics
Define Discrimination Function
• Input X
• Output Y:
– During learning, output of is NOT well learned
Approximation of Y =
• Weight : our goal parameter
– like a and b in linear regression
• Cost function, J is a function of Error
30
2
2
ˆ
||
||
i
i
i
i R
J
e
y
y
( )
Y
W
X
ˆ
( )
Y
W
X
ˆ
Y
'
J
J
W
W
W
J
Minimization
Gradient Descent Method
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
x
y
Data(xi,yi)
ˆ
Y
T&C LAB-AI
Robotics
Neural Network Layer
• Input=X and Output= Y
• More Layers are used for creation of hyper space
31
X
W
Y
1
1
1
1
0
0
2
4
0.2
0.1 1
0.6
3
9
3
1
1
1
1
X
Y
Linear
X
W1
Y
Z
Y
WX
W2
(Z)
1
2
2
1
( )
(
)
Z
W X
Y
W
Z
W
W X
Z: Hidden Layer
2
6 3
W
1
3 6
W
T&C LAB-AI
Robotics
Hidden Layers maps
from Lower to Higher level
• Remind
– Face detection by
Haar feature
• Stage 0 maps
dominant feature
• Stage 21 maps
features in more
detailed ways
• Some stages(or
layers) seem
meaningless
Output is meaningful
32
Raw image
Detailed Feature
T&C LAB-AI
Robotics
Build Space Shuttle
33
Your resources, Money, time, labor, and so on
math
cycle
motor
control
Program
turbine
turbine
Physics
…
Aviation Manufact-
uring
Fever
Space shuttle
Hidden or Middle layers can
be more than input or output
T&C LAB-AI
Robotics
Sigmoidal Function-based Neural Network
34
-5
-4
-3
-2
-1
0
1
2
3
4
5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2
1
Sigmoidal
,
1
1
'
1
1
1
(
)
x
x
x
x
Function
e
e
e
e
I
2
1
2
ˆ
(
)
Y
W
W X
W Z
1
2
1
1
2
2
2
1
2
1
2
1
1
2
1
1
1
2
2
1
2
2
1
1
ˆ
||
||
2
2
(
,
)
ˆ
ˆ
(
) 0
ˆ
(
)
'(
) X
ˆ
ˆ
(
) 0
ˆ
ˆ
(
) (
)
(
)
i
i
i D
i D
w
w
w
i
i D
i
i D
w
w
i
i D
i
i
i D
i D
J
e
y
y
J W W
J dw
J dw
J
y
J
y
y
W
W
y
y W
W X
W
W
J
J
y
J
y
y
W
W
y
y
W X
y
y Z
W
W
2
w
J
T&C LAB-AI
Background of
Vector and matrix Differentiation
4
35
T&C LAB-AI
Robotics
Differentiation of Neural Network
• Remind Differentiation for Gradient Descent Method
36
1
1
2
2
1
1
2
1
1
1
1
1
ˆ
||
||
2
2
ˆ
ˆ
(
) 0
ˆ
(
)
'(
) X
i
i
i D
i D
w
i
i D
i
i D
w
J
e
y
y
J
y
J
y
y
W
W
y
y W
W X
W
W
J
Vector
Matrix
• Question:
“Differentiation Vector with Matrix”, Is it Possible?
It looks like
Matrix Multiplication
T&C LAB-AI
Robotics
Differentiation Scalar, Vector, Matrix
Scalar
Vector
Matrix
Scalar
Vector
Matrix
37
y
x
ˆy
x
Y
x
ˆ
y
x
ˆ
ˆ
y
x
y
X
x
y
ˆy
Y
ˆx
X
• Unfortunately,
Differentiation Matrix by Matrix is Impossible
T&C LAB-AI
Robotics
Differentiation of Matrix 1
• Lemma 1
• Proof
38
1
1
m n n
m
y
A x
y
A
x
1
n
i
ik
k
k
y
a x
1
(0 0 ..
.. 0 0)
n
ij
j
i
ik
k
ij
k
j
j
j
a x
y
a x
a
x
x
x
for all i=1...m and j=1... n
i
ij
j
y
a
x
y
A
x
T&C LAB-AI
Robotics
Differentiation of Matrix 2
• Lemma 2
• Lemma 3
39
1
1
m n n
m
y
A x
y
y x
x
A
z
x z
z
1
1
m n n
m
y
A x
T
T
T
T
T
if c
y Ax
y y
y y
c
c
x A y
1
T
T
T
n
c
y Ax
x
y A
y A
x
x
x
?
T
c
y Ax
y
y
T
T
T
T
T
c
c
x A y
x A
y
y
y
T&C LAB-AI
Robotics
Differentiation of Matrix 3
• Lemma 4
• Proof
40
c
T
if
x Ax
T
T
T
c
x A
x A
x
n
n
ij
i
j
i
j
n
n
kj
j
ik
i
j
i
k
c
a x x
c
a x
a x
x
T&C LAB-AI
Robotics
Differentiation of Matrix 4
• Lemma 5
• Proof
41
( ),
( )
c
T
y
y z x
x z
if
y x
scalar
T
T
c
y
x
x
y
z
z
z
T
T
c
c y
c x
z
y z
x z
y
x
x
y
z
z
c
c
T
T
T
y x
x y
T&C LAB-AI
Robotics
Differentiation of Matrix 5
• Lemma 6
• Lemma7
42
c
T
if
x x
scalar
2
T
c
x
x
z
z
c
T
if
y Ax
T
T
T
T
T
c
c y
c x
z
y z
x z
c
y
x
y
x
y A
x A
y A
y
z
z
z
z
T&C LAB-AI
Robotics
Hadamard Product
• Matrix Multiplication ( what you have learned)
• Hadamard Product (Matlab A.*B)
43
m a a n
m n
AB
A B
AB
11
12
1
11
12
1
21
22
2
21
22
2
1
2
1
2
11 11
12 12
1
1
21 21
22 22
2
2
1
1
2
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
n
n
n
n
m n
m n
m
m
mn
m
m
mn
n
n
n
n
m
m
m
a
a
a
b
b
b
a
a
a
b
b
b
A B
A
B
a
a
a
b
b
b
a b
a b
a b
a b
a b
a b
a b
a b
2
...
m n
m
mn mn
AB
a b
T&C LAB-AI
Robotics
Neural Network Multiplication Problems
• You cannot differentiate NN directly
44
1
2
2
1
1
2
1
1
1
ˆ
||
||
2
2
ˆ
ˆ
(
) 0
ˆ
(
)
'(
) X
i
i
i D
i D
w
i
i D
i
i D
J
e
y
y
J
y
J
y
y
W
W
y
y W
W X
i
j
k
i j
ij
i
j
k
k k
k
C
A B
c
a b
i j
i j
i j
ij
ij ij
C
A B
c
a b
Matrix Multiplication
Hadamard Multiplication
Neural Network
Multiplication
T&C LAB-AI
Neural Network Weight Update
5
45
T&C LAB-AI
Robotics
Weight Update by Gradient Descent Method
is the key for Neural Network
46
But, Differentiation is Complex
We learn Basic Structure first
Extend above Neural Network Structure
T&C LAB-AI
Robotics
47
x
W1
I
Z
I
W2
Y
n x h
(h+1)x 1
1
n
x
2
[
]
n
X
x
I
1
2 h
W
n h
Z
(
1)
[
]
n
h
Z
I
2
(
1) 1
h
W
n 1
Y
1
2
Z
([
]
)
[
]
x
I W
Y
Z
I W
Network Model by Matrix Expression I
Neural Network
X
Y1
Z
W1
W2
1
( )
Z
Y
Y
x y
Data
y
Y
e
T&C LAB-AI
Robotics
48
x
W1
I
Z
I
W2
Y
n x h
(h+1)x 1
1
n
x
2
[
]
n
X
x
I
1
2 h
W
n h
Z
(
1)
[
]
n
h
Z
I
2
(
1) 1
h
W
n 1
Y
Network Model by Matrix Expression I
X
Y1
Z
W1
W2
1
( )
Z
Y
Y
x y
Data
y
Y
e
1
2
3
4
x
1
4
9
16
y
2
y
x
Learn
NN
T&C LAB-AI
Robotics
x
W1
I
Z
I
W2
Y
n x h
(h+1)x 1
1
n
x
2
[
]
n
X
x
I
1
2 h
W
n h
Z
(
1)
[
]
n
h
Z
I
2
(
1) 1
h
W
n 1
Y
Network Model by Matrix Expression I
X
Y1
Z
W1
W2
1
( )
Z
Y
Y
1
2
3
4
[x I]
W1
1
1
1
1
w11
w12
w13
w14
w15
w21
w22
w23
w24
w25
n
h
w11
+w21
w12
+w22
w13
+w23
w14
+w24
w15
+w25
2w11
+w21
2w12
+w22
2w13
+w23
2w14
+w24
2w15
+w25
3w11
+w21
3w12
+w22
3w13
+w23
3w14
+w24
3w15
+w25
4w11
+w21
4w12
+w22
4w13
+w23
4w14
+w24
4w15
+w25
=
T&C LAB-AI
Robotics
x
W1
I
Z
I
W2
Y
n x h
(h+1)x 1
1
n
x
2
[
]
n
X
x
I
1
2 h
W
n h
Z
(
1)
[
]
n
h
Z
I
2
(
1) 1
h
W
n 1
Y
Network Model by Matrix Expression I
X
Y1
Z
W1
W2
1
( )
Z
Y
Y
n
w11
+w21
w12
+w22
w13
+w23
w14
+w24
w15
+w25
2w11
+w21
2w12
+w22
2w13
+w23
2w14
+w24
2w15
+w25
3w11
+w21
3w12
+w22
3w13
+w23
3w14
+w24
3w15
+w25
4w11
+w21
4w12
+w22
4w13
+w23
4w14
+w24
4w15
+w25
h
Z
T&C LAB-AI
Robotics
x
W1
I
Z
I
W2
Y
n x h
(h+1)x 1
1
n
x
2
[
]
n
X
x
I
1
2 h
W
n h
Z
(
1)
[
]
n
h
Z
I
2
(
1) 1
h
W
n 1
Y
Network Model by Matrix Expression I
X
Y1
Z
W1
W2
1
( )
Z
Y
Y
n
w11
+w21
w12
+w22
w13
+w23
w14
+w24
w15
+w25
2w11
+w21
2w12
+w22
2w13
+w23
2w14
+w24
2w15
+w25
3w11
+w21
3w12
+w22
3w13
+w23
3w14
+w24
3w15
+w25
4w11
+w21
4w12
+w22
4w13
+w23
4w14
+w24
4w15
+w25
h
1
1
1
1
w2,1
w2,2
w2,3
w2,4
w2,5
w2,6
T&C LAB-AI
Robotics
x
W1
I
Z
I
W2
Y
n x h
(h+1)x 1
1
n
x
2
[
]
n
X
x
I
1
2 h
W
n h
Z
(
1)
[
]
n
h
Z
I
2
(
1) 1
h
W
n 1
Y
Network Model by Matrix Expression I
X
Y1
Z
W1
W2
1
( )
Z
Y
Y
11
21
2,1
12
22
2,2
13
23
2,3
14
24
2,4
15
25
2,5
2,6
11
21
2,1
12
22
2,2
13
23
2,3
14
24
2,4
15
25
2,5
2,6
11
21
2,1
12
22
2,2
ˆ
(
)
(
)
(
)
(
)
(
)
(2
)
(2
)
(2
)
(2
)
(2
)
(3
)
(3
)
(
y
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
13
23
2,3
14
24
2,4
15
25
2,5
2,6
11
21
2,1
12
22
2,2
13
23
2,3
14
24
2,4
15
25
2,5
2,6
3
)
(3
)
(3
)
(4
)
(4
)
(4
)
(4
)
(4
)
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
T&C LAB-AI
Robotics
Differentiation with W2
53
2
2
2
1
1
2
2
T
k
k
T
J
e
e e
J
e
e
W
W
2
1
(
1)
2
2
2
2
(
1)
1
2
(
1) 1
[
]
[
]
[
]
[
]
T
T
T
T
n
n
h
T
T
T
n
h
n
h
Z
I W
J
e
Y
e
e
e
e
Z
I
W
W
W
W
J
Transpose
Vector
Z
I
e
Z
I
e
W
5.
2
T
T
Lemma
c
x x
c
x
x
z
z
T&C LAB-AI
Robotics
Differentiation with W1
54
2
11
11
11
11
11
21
2,1
12
22
2,2
13
23
2,3
14
24
2,4
15
25
2,5
2,6
1
11
11
21
2,1
12
22
2,2
13
23
2,3
2
1
2
ˆ
ˆ
(
)
(
)
(
)
(
)
(
)
(
)
(2
)
(2
)
(2
)
k
k
k
k
k
k
k
k
k
k
k
k
J
e
e
y
y
y
J
e
e
e
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
e
w
w
w
w
w
w
w
w
w
w
e
14
24
2,4
15
25
2,5
2,6
11
11
21
2,1
12
22
2,2
13
23
2,3
14
24
2,4
15
25
2,5
2,6
3
11
11
21
2,1
12
22
2,2
13
23
2,3
14
24
2,4
4
(2
)
(2
)
(3
)
(3
)
(3
)
(3
)
(3
)
(4
)
(4
)
(4
)
(4
)
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
e
w
w
w
w
w
w
w
w
w
w
w
w
w
e
15
25
2,5
2,6
11
(4
)
w
w
w
w
w
T&C LAB-AI
Robotics
Differentiation with W1
55
2
11
11
11
11
11
21
2,1
11
21
2,1
11
21
2,1
11
21
2,1
1
2
3
4
11
11
11
11
1
11
21
2,1
2
11
21
2,1
3
11
1
2
ˆ
ˆ
(
)
(
)
(2
)
(3
)
(4
)
'(
)
'(2
)2
'(3
k
k
k
k
k
k
k
k
k
k
k
k
J
e
e
y
y
y
J
e
e
e
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
e
e
e
e
w
w
w
w
e
w
w
w
e
w
w
w
e
w
21
2,1
4
11
21
2,1
)3
'(4
)4
w
w
e
w
w
w
Can you find the PATTERN?
T&C LAB-AI
Robotics
Differentiation with W1
56
2
1
1
1
1
2
2, j
1
2
2, j
1
2
2, j
1
2
2, j
1
2
3
4
1
1
1
1
1
1
2
2, j
2
1
2
2, j
3
1
1
1
2
ˆ
ˆ
(
)
(
)
(2
)
(3
)
(4
)
'(
)
'(2
)2
'(3
k
k
k
k
k
k
k
k
k
k
k
k
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
J
e
e
y
y
y
J
e
e
e
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
e
e
e
e
w
w
w
w
e
w
w
w
e
w
w
w
e
w
2
2, j
4
1
2
2, j
2
2
2
2
1
2
2, j
1
2
2, j
1
2
2, j
1
2
2, j
1
2
3
4
2
2
2
2
1
1
2
2, j
2
1
)3
'(4
)4
ˆ
ˆ
(
)
(
)
(2
)
(3
)
(4
)
'(
)
'(2
j
j
j
k
k
k
k
k
k
k
k
k
k
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
j
w
w
e
w
w
w
e
y
y
y
J
e
e
e
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
w
e
e
e
e
w
w
w
w
e
w
w
w
e
w
2
2, j
3
1
2
2, j
4
1
2
2, j
)
'(3
)
'(4
)
j
j
j
j
j
j
w
w
e
w
w
w
e
w
w
w
T&C LAB-AI
Robotics
Differentiation with W1
57
2
1
1
1
1
1
2
2, j
2
1
2
2, j
3
1
2
2, j
4
1
2
2, j
1
2
2, j
2
2
2
1
1
2
ˆ
ˆ
(
)
'(
)
'(2
)2
'(3
)3
'(4
)4
'(
)
ˆ
(
)
k
k
k
k
k
k
k
k
k
k
k
k
j
j
j
j
j
j
j
j
j
j
j
j
k
k
j
j
k
k
k
k
k
k
k
k
k
j
j
j
J
e
e
y
y
y
J
e
e
e
w
w
w
w
e
w
w
w
e
w
w
w
e
w
w
w
e
w
w
w
e
x w
w
x w
e
y
y
J
e
e
w
w
w
2
1
1
2
2, j
2
1
2
2, j
3
1
2
2, j
4
1
2
2, j
1
2
2, j
ˆ
'(
)
'(2
)
'(3
)
'(4
)
'(
)
k
k
k
j
j
j
j
j
j
j
j
j
k
k
j
j
k
y
e
w
e
w
w
w
e
w
w
w
e
w
w
w
e
w
w
w
e
x w
w
w
T&C LAB-AI
Robotics
Differentiation with W1
58
2
1
2
2,
2,
2,
1
1
2
2,
2
1
2
1
1
,
2
2
1
2
[
1]
1
'(
)
'
'
1
'(
)
1
'
'
'
k
k
k
k
k
T
ik
T
k k
k
j
j
j
k k
j
kj
k
h
kj
kj
k
k
k
j
T
k
k
j
j
j
h
kj
kj
k
k
j
T
T
T
T
ik
kj
ik
kj
kj
k
k
ij
T
J
e
X
x
x
X
J
x e
x w
w
w
x e w
x eW
w
J
e
x w
w
w
eW
w
J
X
eW
X
eW
w
J
X
W
2
'
T
eW
Hadamard
multiplication
i
j
k
i j
ij
i
j
k
k k
k
C
A B
c
a b
i j
i j
i j
ij
ij ij
C
A B
c
a b
T&C LAB-AI
Robotics
Neural Network Example
59
2
2
2,
1
1
2
1
1
1
ˆ
2
2
2
:
[
]
'
:
[
]
T
k
k
k
k
k
T
T
h
T
J
y
y
e
e e
J
Matrix
x
I
eW
W
J
Vector
Z
I
e
W
x
W1
I
Z
I
W2
Y
n x h
(h+1)x 1
1
n
x
2
[
]
n
X
x
I
1
2 h
W
n h
Z
(
1)
[
]
n
h
Z
I
2
(
1) 1
h
W
n 1
Y
X
Y1
Z
W1
W2
1
( )
Z
Y
Y
T&C LAB-AI
Robotics
Python Example: l3sig
60
x
W1
I
Z
I
W2
Y
n x h
(h+1)x 1
1
n
x
2
[
]
n
X
x
I
1
2 h
W
n h
Z
(
1)
[
]
n
h
Z
I
2
(
1) 1
h
W
n 1
Y
X
Y1
Z
W1
W2
1
( )
Z
Y
Y
x=
2
[
]
n
X
x
I
Z
I
T&C LAB-AI
Robotics
Python Example: l3sig
61
Blue: y
Red: Y(est)
J during 2000 iterations
Jittering?
Why?
You can see
from l3sig.py
T&C LAB-AI
Robotics
If we increase Hidden Space?
:
Hidden Space Increases Estimation Performance
• h=20 h=200
• What happens?
62
Hidden layer increases the DOF of Estimation Results
