T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Mobile Robot
Probability and Bayesian Classifier
Lecture 5
Jeong-Yean Yang
2020/10/22
1
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Probability
• Probability
– Pr(x) = 0.111…
• Sum of all possibilities.
• Continuous domain
• You already learned about probability..
– Korean education is so tough….T_T….
2
Pr( )
1
x
Pr( )
1
x dx
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Gaussian Probability Generation
3
2
1
1
Pr( )
exp
2
2
x
x
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
With C++ or Python,
How to Generate Gaussian Distribution?
• Rand() returns integer from 0 to RAND_MAX(32767)
– Rand() is NOT Gaussian(Normal) distribution
• Remind the video
4
*Marsaglia polar method
~
(0,1)
r
N
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
N(0,1) returns Gaussian Distribution
randn(1,1000) generates
1000 samples
Question:
How we generate x with
mean and standard
deviation?
5
1000 samples
~
(0,1)
' ~
( , ) ?
x
N
x
N
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Gaussian Generation
• Mean value: is a offset from 0
• Standard deviation
6
' ~
( , )
x
N
~
(0,1)
x
N
' ~
(0,1)
( ,1)
x
N
N
~
(0,1)
x
N
'
4
' ~
(0,1)
4
(4,1)
x
x
x
N
N
~
(0,1)
x
N
' ~
(0,1)
(0, )
x
N
N
-4
-2
0
2
4
0
20
40
60
80
100
0
2
4
6
8
0
20
40
60
80
100
-10
-5
0
5
10
0
20
40
60
80
100
'
3
' ~ 3 (0,1)
(0, 3)
x
x
x
N
N
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Gaussian Distribution or
Normal Distribution(Z)
• We learn it at high school, TT.
• Z is called “Normal Distribution”
• X is normalized with mean and standard deviation
7
z ~
(0,1)
N
z
~
(0,1)
~
(0,1)
( , )
x
N
x
N
N
2
1
1
Pr( )
exp
2
2
x
x
2
1
1
Pr(z)
exp
2
2
z
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Probability in 2D Space
• How to generate 2D Gaussian Prob.?
– Easy. A= randn(1000,2) and plot(A(:,1),A(:,2),’.’)
8
-4
-2
0
2
4
-4
-3
-2
-1
0
1
2
3
4
Plot( A(:,1),A(:,2),’.’)
1
z ~
(0,1)
N
2
0
z
~
,
0
x
N
y
1 DIM
2 DIM
mean
mean
x
y
?
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
9
-4
-2
0
2
4
-4
-3
-2
-1
0
1
2
3
4
Plot( A(:,1),A(:,2),’.’)
-4
-2
0
2
4
-4
-3
-2
-1
0
1
2
3
4
Plot( 2*A(:,1),A(:,2),’.’)
-4
-2
0
2
4
-4
-3
-2
-1
0
1
2
3
4
Plot(A(:,1), 1.5*A(:,2),’.’)
2
z
x
y
2
2
z'
x
y
2
z'
1.5
x
y
-10
-5
0
5
10
-10
-5
0
5
How we make it?
2
0.5
'
0.5 1.5
x
z
y
x
y
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Quiz
10
2
3
'
3
1.5
x
z
y
How it will distribute?
2
3
Hint :
3 3
0
3
1.5
Det
-10
-5
0
5
10
-10
-5
0
5
10
-4
-2
0
2
4
-4
-3
-2
-1
0
1
2
3
4
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Probability in n-dim. Space
• 1Dim
• N-Dim
• Look, Sigma matrix
11
2
1
1
Pr( )
exp
2
2
x
x
~
( , )
x
N
ˆ
ˆ
~
( , )
x
N
1
1
2
1
ˆ
Pr( )
(2
)
exp
2
T
x
Det
x
x
2
0.5
0.5 1.5
2
0
0 1.5
Scale factor for
principal axis
...
0.5
0.5
...
Rotation
Important for
Map
matching
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Two types of Probability
• A Priori Probability
– When you use probability, you use a prior probability
• Posterior Probability (Conditional probability)
– Bayesian probability
– Prob. Of A on condition that B occurs,
• A prior and Posterior probability are very different.
12
Pr(A)
0.6
Pr(A | B)
0.6
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Conditional Probability
• What is Pr(A|B)?
– Probability of A under the Probability of B
– Or Probability of A within the given B
13
A
B
A^B
B
= Pr(A|B)
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Why Posterior Prob. Is very different?
• Rock-Paper-Scissors game.
– Prob(Rock) = 1/3
• When a player did “Rock” before,
– Prob(Rock) is still 1/3? - No, in general.
14
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Posterior Prob.
• When events A and B occur,
• P(A): Probability of A occurrence
• P(B): Probability of B occurrence.
• P(A^B): Probability of Both A and B occurrence
• Definition:
15
( | ) ( )
( ^ )
( | ) ( )
( | ) ( )
Pr( | )
( )
P A B P B
P A B
P B A P A
P B A P A
A B
P B
(A^ B)
Pr( | )
( )
P
A B
P B
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Why Posterior Probability?
It reduces Classification Errors..
• What is Classification?
• When a data x is given, is it a specific class, C?
– It is called, “classification”
16
Is it ‘a’?
if x
C or Not
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Is It Big or Not?
• Normal human can say that..
– Right is bigger.. ^_^..
• When a X is given, can you say that “it is big or not”?
17
<
X
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Posterior Prob.
• When events A and B occur,
• P(A): Probability of A occurrence
• P(B): Probability of B occurrence.
• P(A^B): Probability of Both A and B occurrence
• Definition:
18
( | ) ( )
( ^ )
( | ) ( )
( | ) ( )
Pr( | )
( )
P A B P B
P A B
P B A P A
P B A P A
A B
P B
(A^ B)
Pr( | )
( )
P
A B
P B
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Classification:
Bayesian Classifier
• Random variable, x : probability of event occurrence.
• When x is given, is x involved in class w1 or w2?
– Ex)
– Assume X is height,
– When x = 170, is it tall(w1) or not(w2) ?
19
X
x
X
w1
w2
Who will
determine it?
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Classification:
Bayesian Classifier
• Random variable, x : probability of event occurrence.
• When x is given, is x involved in class w1 or w2?
20
X
x
X
W1
=tall
0
5
10
15
20
25
170
180
190
200
170
175
180
185
190
195
200
0
2
4
6
H=180, I think tall.
Test1.m
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Classification:
Bayesian Classifier
• Random variable, x : probability of event occurrence.
• When x is given, is x involved in class w1 or w2?
21
X
x
X
W2
=~tall
H=160, I think ~tall
0
5
10
15
20
25
30
160
165
170
175
160
162
164
166
168
170
172
0
1
2
3
4
5
Test2.m
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Samples from Surveys.
• Assume that samples have Gaussian distribution.
• (m1,s1) = ( 181.143, 6.54)
• (m2,s2) = ( 165.14, 3.12)
22
160
165
170
175
180
185
190
195
200
0
1
2
3
4
5
6
w1
=tall
w2
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Classification:
Bayesian Classifier
• Random variable, x : probability of event occurrence.
• When x is given, is x involved in class w1 or w2?
23
X
x
X
w2
w1
Pr( )
?
x
2
Pr(
)
?
w
2
Pr(
| )
?
w
x
2
Pr( |
)
x w
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Classification:
Bayesian Classifier
• Random variable, x : probability of event occurrence.
• When x is given, is x involved in class w1 or w2?
24
X
x
X
w2
w1
2
P( |
)
x w
1
P( |
)
x w
In w2 group,
samples xs are
gathered.
In w1 group,
samples xs are
gathered.
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Samples.
25
X
x
X
w1
1
Pr( |
)
x w
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
For Bayesian Classifier,
p(x|w) and p(w) are required.
• How to find P(w)?
26
n(w1)
=21
n(w2)
=28
P(w1)
=21/(21+28)
P(w2)
=28/(21+28)
P(w1)+P(w2)
=1
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Back to Bayesian Probability
• x is given, is it w1 or w2?
x
X
w2
w1
2
P( |
)
x w
1
P( |
)
x w
1
1
1
( |
) (
)
P(
| )
( )
P x w p w
w x
P x
( | ) ( )
( ^ )
( | ) ( )
( | ) ( )
Pr( | )
( )
P A B P B
P A B
P B A P A
P B A P A
A B
P B
(A^ B)
P( | )
( )
P
A B
P B
2
2
2
( |
) (
)
P(
| )
( )
P x w p w
w
x
P x
A w1, Bx
A w2, Bx
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Definition
Bayesian Classifier
28
x
X
w2
w1
2
Pr( |
)
x w
1
Pr( |
)
x w
2
Pr(
)
w
1
Pr(
)
w
1
1
2
2
1
2
( |
) (
)
( |
) (
)
Pr(
| )
Pr(
| )
( )
( )
P x w p w
P x w p w
w x
w
x
P x
P x
1
2
,
,
then
x
w
otherwise
x
w
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Finally, p(x)=?
• Finally,
Posterior Probability
29
1
1
1
( |
) (
)
P(
| )
( )
P x w p w
w x
P x
1
1
2
2
P( )
P
|
P(
)
P( |
) P(
)
x
x w
w
x w
w
1
2
1
P(
)
P(
)
w
w
1
1
1
1
1
2
2
( |
) (
)
P(
| )
( |
) (
)
( |
) (
)
P x w p w
w x
P x w p w
P x w p w
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Posterior Probability in General
30
1
1
2
2
( |
) (
)
(
| )
( )
( |
) (
)
( |
) (
)
i
i
i
P x w p w
P w x
P x
P x w p w
P x w p w
( |
) (
)
P(
| )
( |
) (
)
i
i
i
k
k
k
P x w P w
w x
P x w P w
(
) 1
k
k
when
P w
!...
( |
) (
)
1
k
k
k
Warning
P x w P w
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Engineering Notation
31
(x | w) (w)
P(w | x)
(x)
P
P
P
likelihood
prior
Posterior
Evidence
In engineering, likelihood is one of the popular solution.
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
What is the difference between
Likelihood and Posterior probability?
• likelihood-based classifier
• Posterior probability-based classifier
32
1
1
2
2
( |
) (
)
(
| )
( |
) (
)
( |
) (
)
i
i
i
P x w P w
P w x
P x w P w
P x w P w
1
2
1
( |
)
( |
)
P x w
P x w
then x
w
1
2
1
(
| )
(
| )
,
P w x
P w
x
then
x
w
대충 키 큰사람은
평균이
181,
작은 사람은
165
이니
,
X= 175는 키가 큰쪽에
확률에 가깝다
?
X=175인 경우,
키가 클 확률은 얼마
작을 확률은 얼마이므로
키다 크다 또는 작다
..
(x | w) (w)
P(w | x)
(x)
P
P
P
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Example: Test4.m
• N(m,s) normpdf in Matlab
• x =175 N(m,s,x=175)
• Likelihood prob. classifier
• Posterior prob. classifier
33
2
2
1
(
)
~
( , )
exp
2
2
x
x
N
1
2
( |
)
1
0.0392
( |
)
2
0.0009
P x w
pxw
P x w
pxw
1
2
(
| )
1
0.971
(
| )
2
0.029
P w x
pw x
p w
x
pw x
2
1
1
2
1
1
(175
)
1
(
175 |
)
exp
2
2
p x
w
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Theoretical Interest
• We can think error.
• Thus, from posterior classifier, we define p(e)
34
( )
( | ) ( )
P e
P e x P x
1
2
2
1
(
| ) if we decide w
(
| )
(
| ) if we decide w
p w x
P error x
p w
x
1
2
(
| )
(
| )
: arg max( (
| ))
: arg min( (
| ))
i
i
P w x
P w
x
Bayesian classifier
p w x
Bayesian Error
p w x
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Bayesian Error is,
• Very small.
• In many cases, Bayesian classifier is better than you.
• Most classifiers are compared with Bayesian error.
• If you have success of designing new classifier,
in general, its performance is probably rather better
than Bayesian classifier.
• Mathematically, Bayesian classifier is VERY
STRONG.
– Question: Why Deep Learning is so good?
– Because, DL has the function of finding GOOD Feature.
35
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Example. Test 5.m
Plot everything
• P(x)
36
1
1
2
2
P( )
P
|
P(
)
P( |
) P(
)
x
x w
w
x w
w
150
160
170
180
190
200
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
height
p
(x
)
p(x)
150
160
170
180
190
200
0
0.5
1
1.5
2
2.5
3
height
C
la
s
s
Classification Result
: arg max( (
| ))
i
Bayesian classifier
p w x
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Likelihood Vs. Posterior
37
i
arg max( ( |
))
i
P x w
arg max( (
| ))
i
i
P w x
Vs.
150
160
170
180
190
200
0
0.5
1
1.5
2
2.5
3
height
C
la
s
s
Likelihood
150
160
170
180
190
200
0
0.5
1
1.5
2
2.5
3
height
C
la
s
s
Bayes
Equal?
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Overlapped Area
38
170
170.5
171
171.5
172
0
0.5
1
1.5
2
2.5
3
height
C
la
s
s
Likelihood
170
170.5
171
171.5
172
0
0.5
1
1.5
2
2.5
3
height
C
la
s
s
Bayes
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
The Most Important Factor of Classifier.
Minimize Error on Overlapped data
• New Data in Test 6.m
• w1~N(30,2) w2~N(40,4) p(w1)=p(w2)=0.5
39
0
20
40
60
80
100
0
0.05
0.1
0.15
0.2
blue:w1 red: w2
0
20
40
60
80
100
0
0.02
0.04
0.06
0.08
x
p
(x
)
p(x)
0
20
40
60
80
100
1
1.2
1.4
1.6
1.8
2
x
C
la
s
s
blue:Likelihood, red:Bayesian
0
20
40
60
80
100
0
0.01
0.02
0.03
0.04
x
P
(e
)
P(e), Blue:Likelihood, Red:Posterior
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
0
20
40
60
80
100
0
0.05
0.1
0.15
0.2
blue:w1 red: w2
0
20
40
60
80
100
0
0.02
0.04
0.06
0.08
0.1
x
p
(x
)
p(x)
0
20
40
60
80
100
1
1.2
1.4
1.6
1.8
2
x
C
la
s
s
blue:Likelihood, red:Bayesian
0
20
40
60
80
100
0
0.01
0.02
0.03
0.04
x
P
(e
)
P(e), Blue:Likelihood, Red:Posterior
Problems of Likelihood.
When _________
• Likelihood CANNOT be used for
• Ex) P(w1)=0.1 P(w2)=0.9
40
1
2
P(
)
P(
)
w
w
1
2
P(
)
P(
)
w
w
Difference
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Example of Bayesian Classifier
: Sensor for Something
• Example of PSD (distance sensor)
• If (distance>0.8) then “human exists” else “nothing”.
41
Returns “distance”
Accident
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Who will choose threshold?
• 1. Adhoc
– Well, 0.8 could be the possible value.
– You will go to Jail… T_T..
• 2. Likelihood
– After 100 samples,
– When samples are not balanced… it also fails.
• 3. Bayesian
– After 100 samples
– You did your best except for Deep Learning..^_^…
42
( |
)
( |
)
Human
human
nothing
P x w
P x w
(
| x)
(
| x)
Human
human
nothing
P w
P w
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Specification of Bayesian Classification
• Bayesian classification
– It requires a lot of Samples
– Everything are designed with Probabilistic Distribution
– Therefore, Modeling-based Method( Parametric Method)
• When class is added in online environment, it is
useless.
– But, most classifications are useless, too.
• When new samples cannot be used.
– After sampling, Bayesian classifier is calculated.
• Any method in which New samples are updated,
– Non parametric method( usually, Kernel based method)
43
T&C LAB-AI
Dept. of Intelligent Robot Eng. MU
Robotics
Classification and Features
• x is a random variable.
• But, x is also called as a feature vector.
• In a given problem, you should find a good feature.
• Grade, creativity, moral attention could be features for
recruiting students.
• PSD distance is not enough. Movements could be OK.
• Without GOOD features, classifier cannot work.
44
