Zhankun Luo

PUID: 0031195279

Email: luo333@pnw.edu

Fall-2018-ECE-59500-009

Instructor: Toma Hentea

文章目录

• Homework 5
• • Function
  • • plot_point.m
    • Calc_SwSbSm.m
    • J3.m
    • FDR.m
  • Problem
  • • Problem 5.1
    • • Proof of $E[z] = \lVert \mu \rVert^2$
      • Proof of $\sigma_{z}^2 = \lVert \mu \rVert^2 l$
      • Proof of the probability of error $P_e$
      • • Gram–Schmidt process
        • Preparing work
        • Calculate Probability
        • Calculate $P_e$
  • Computer Experiment
  • • Experiment 5.2
    • • meaning of S_w, S_b, S_m, J3
      • code: experiment5_2.m
      • result
      • conclusion
    • Experiment 5.4
    • • Theory of Fisher’s Linear Discriminant
      • code: experiment5_4.m
      • result
      • conclusion

Homework 5

Function

plot_point.m

function plot_point(X,y) %this function can handle up to 6 different classes [l,N]=size(X); %N=no. of data vectors, l=dimensionality if(l~=2)fprintf('NO PLOT CAN BE GENERATED\n')return elsepale=['ro';'g+';'b.';'y.';'m.';'c.']; %Plot of the data vectorshold onfor i=1:Nplot(X(1,i),X(2,i),pale(y(i),:))end hold off end

Calc_SwSbSm.m

function [ S_w, S_b, S_m ] = Calc_SwSbSm( X, y ) % [ S_w, S_b, S_m ] = Calc_SwSbSm( X, y ) % Calculate S_w, S_b, S_m % OUTPUT: % S_w: the within-class % S_b: the between-class % S_m: the mixture Sm = Sw + Sb c = max(y); % number of classes [l, N] = size(X); % N: number of vectors, l: dimensions mu = zeros(l, c); S_w = zeros(l, l); S_b = zeros(l, l); mu_0 = zeros(l, 1); P = zeros(1, c); for i = 1:cindex_class_i = find(y == i);Mu = sum(X(:, index_class_i), 2) / length(index_class_i);mu(:, i) = Mu; mu_0 = mu_0 + sum(X(:, index_class_i), 2) / N;P(i) = length(index_class_i) / N;X_relative = X(:, index_class_i) - repmat(Mu, 1, length(index_class_i));S_wi = zeros(l, l);for j = 1:length(index_class_i)S_wi = S_wi + X_relative(:, j) * X_relative(:, j)';endS_w = S_w + S_wi / N; end for i = 1:cS_b = S_b + P(i) * (mu(:, i) - mu_0) * (mu(:, i) - mu_0)'; end S_m = S_w + S_b;

J3.m

function J3 = J3(S_w, S_m) J3 = trace(S_w \ S_m); end

FDR.m

function [ FDR, w ] = FDR( X, y , D_y) %function [ FDR, w ] = FDR( X, y , D_y) % Fisher's Discriminant Ratio % INPUT: % X: points % y: y==i ==> belong to Class i % D_y: dimension of w, how many features Z_i = w_i'* X need to classify % OUTPUT: % FDR: trace((w * S_w * w') \ (w * S_b * w')) % w: use w ==> make Z = w'* X => calculate tr(S_w \ S_b) of Z % <=> maximize FDR of X [ S_w, S_b, S_m ] = Calc_SwSbSm( X, y ); [ Vector, Diag ] = eig( S_w \ S_b ); vector = fliplr(Vector); % make highest eig show first w = vector(:, 1:D_y); % select D_y vectors corresponding to D_y highest eig values FDR = trace((w'* S_w * w) \ (w'* S_b * w)); end

Problem

Problem 5.1

Both classes, ${\omega }_1$ , ${\omega }_2$ , are described by Gaussian distributions of
the same covariance matrix $\Sigma =I$ , where $I$ is the identity matrix and mean values
$\mu$ and $-\mu$, respectively, where:
$$\begin{gathered} \mu =[{\mu }_1,...,{\mu }_l{]}^T=[1,\frac{1}{\sqrt{2}},...,\frac{1}{\sqrt{l}}{]}^T \\ {b}_l=\Vert \mu \Vert =\sqrt{\sum _{i=1}^l{\mu }_i^2} \end{gathered}$$
Define:
$$z\equiv x^T\mu ,wherex=[x_1,...,x_l{]}^T$$

Proof of $E[z]=\Vert \mu {\Vert }^2$

$$\begin{gathered} Forx\in {\omega }_1:E[z]=E[x^T\mu ]=E[x^T]\mu ={\mu }^T\mu =\Vert \mu {\Vert }^2 \\ Forx\in {\omega }_2:E[z]=E[x^T\mu ]=E[x^T]\mu =-{\mu }^T\mu =-\Vert \mu {\Vert }^2 \end{gathered}$$

Proof of ${\sigma }_z^2=\Vert \mu {\Vert }^{2}l$

$Forx\in {\omega }_1:$
$$\begin{gathered} {\sigma }_z^2=E[(x-\mu {)}^2\Vert \mu {\Vert }^2] \\ =\Vert \mu {\Vert }^2{\iiint }_{-\infty }^{\infty }(x-\mu {)}^2\frac{1}{(2\pi {)}^{\frac{l}{2}}{\Sigma }^{\frac{1}{2}}}exp(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))dx \\ =\Vert \mu {\Vert }^2{\iiint }_{-\infty }^{\infty }\sum _{i=1}^l(x_i-{\mu }_i{)}^2\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}\sum _{j=1}^l(x_j-{\mu }_j{)}^2)dx \\ =\Vert \mu {\Vert }^2\sum _{i=1}^l{\iiint }_{-\infty }^{\infty }(x_i-{\mu }_i{)}^2\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}\sum _{j=1}^l(x_j-{\mu }_j{)}^2)d{x}_1...d{x}_l \\ =\Vert \mu {\Vert }^2\sum _{i=1}^l{\int }_{-\infty }^{\infty }(x_i-{\mu }_i{)}^2\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}(x_i-{\mu }_i{)}^2)d{x}_i\prod _{j̸=i,j=1}^l{\int }_{-\infty }^{+\infty }\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}(x_j-{\mu }_j{)}^2)d{x}_j \\ =\Vert \mu {\Vert }^2\sum _{i=1}^{l}1\cdot 1^{(l-1)} \\ =\Vert \mu {\Vert }^{2}l \end{gathered}$$
$Forx\in {\omega }_2:$
$${\sigma }_z^2=E[(x+\mu {)}^2\Vert \mu {\Vert }^2]=\Vert \mu {\Vert }^{2}l$$

Proof of the probability of error $P_e$

Gram–Schmidt process

Set:
$$\begin{gathered} {\alpha }_1=\mu ,{\alpha }_2=[0,1,0,...,0{]}^T,{\alpha }_3=[0,0,1,0,...,0{]}^T,{\alpha }_l=[0,...,0,1{]}^T \\ ThesevectorsareLinearlyIndependent \end{gathered}$$
Then:
$$\begin{gathered} {\beta }_1={\alpha }_1=\mu \\ {\beta }_2={\alpha }_1-\frac{<{\alpha }_2,{\beta }_1>}{<{\beta }_1,{\beta }_1>}{\beta }_1 \\ ... \\ {\beta }_i={\alpha }_i-\frac{<{\alpha }_i,{\beta }_1>}{<{\beta }_1,{\beta }_1>}{\beta }_1-...-\frac{<{\alpha }_i,{\beta }_{i-1}>}{<{\beta }_{i-1},{\beta }_{i-1}>}{\beta }_{i-1}(i=2,...l) \end{gathered}$$
Normalize:
$$\begin{gathered} e_1=\frac{{\beta }_1}{\Vert \mu \Vert }=\frac{\mu }{b_l} \\ {e}_i=\frac{{\beta }_i}{\Vert {\beta }_i\Vert }(i=1,...,l) \end{gathered}$$

Preparing work

Set:
$$\begin{gathered} P=[e_1,...,e_l{]}^T \\ <e_i,e_j>=\{\begin{array}{lc}0& ifi̸=j\\ 1& ifi=j\end{array} \\ P{P}^T=[e_1,...,e_l{]}^T[e_1,...,e_l]=I_l \\ Pisanorthogonalmatrix,P^{-1}=P^T,P^{T}P=I_l,|det(P)|=1 \end{gathered}$$
Apply P to $(x-\mu )$
$$\begin{gathered} P(x-\mu )=[e_1,e_2,...,e_l{]}^T(x-\mu )=[e_1^{T}x,...,e_l^{T}x{]}^T-[b_l,0,...,0{]}^T \\ because,e_i^T\mu =e_i^T(b_l{e}_1)=\{\begin{array}{lc}0& ifi̸=1\\ b_l& ifi=1\end{array} \\ definey=[y_1,...,y_l{]}^T\equiv [e_1^{T}x,...,e_l^{T}x{]}^T \\ thusP(x-\mu )=[y_1-b_l,y_2,...,y_l{]}^T \end{gathered}$$

Calculate Probability

The probability density function for $(x_1,...,x_l)$ is given by, ($\Sigma =I_l$):

class ${\omega }_1$:
$$\begin{gathered} p(x;\mu ,\Sigma )=\frac{1}{(2\pi {)}^{\frac{l}{2}}{\Sigma }^{\frac{1}{2}}}exp(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )) \\ =\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}(x-\mu {)}^T(x-\mu )) \\ =\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}(x-\mu {)}^T{P}^{T}P(x-\mu )) \\ =\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}[y_1-b_l,y_2,...,y_l][y_1-b_l,y_2,...,y_l{]}^T) \\ =\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}[(y_1-b_l{)}^2+\sum _{i=2}^l{y}_i^2]) \end{gathered}$$
class ${\omega }_2$:
$$\begin{gathered} p(x;-\mu ,\Sigma )=\frac{1}{(2\pi {)}^{\frac{l}{2}}{\Sigma }^{\frac{1}{2}}}exp(-\frac{1}{2}(x+\mu {)}^T{\Sigma }^{-1}(x+\mu )) \\ =\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}(x+\mu {)}^T(x+\mu )) \\ =\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}[(y_1+b_l{)}^2+\sum _{i=2}^l{y}_i^2]) \end{gathered}$$
Because:
$$\begin{gathered} dy=Pdx \\ {P}^{-1}dy=P^{T}dy=dx \\ |det(P^T)|d{y}_1...d{y}_l=d{y}_1...d{y}_l=d{x}_1...d{x}_l \end{gathered}$$
Then:

For class ${\omega }_1$:
$$\begin{gathered} P(z=x^T\mu <0|{\omega }_1) \\ =P(z=x^T\mu =x^T{b}_l{e}_1=b_l{y}_1<0|{\omega }_1)=P(y_1<0|{\omega }_1) \\ ={\iiint }_{y_1<0}p(x;\mu ,\Sigma )d{x}_1...d{x}_l \\ ={\iiint }_{y_1<0}\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}[(y_1-b_l{)}^2+\sum _{i=2}^l{y}_i^2])d{y}_1...d{y}_l \\ ={\int }_{-\infty }^0\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}(y_1-b_l{)}^2)d{y}_1\prod _{i=2}^l{\int }_{-\infty }^{+\infty }\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}{y}_i^2)d{y}_i \\ ={\int }_{-\infty }^0\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}(y_1-b_l{)}^2)d{y}_1 \\ ={\int }_{-\infty }^{-b_l}\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}{Z}^2)dZ \\ ={\int }_{b_l}^{+\infty }\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}{Z}^2)dZ \end{gathered}$$
For class ${\omega }_2$:
$$\begin{gathered} P(z=x^T\mu >0|{\omega }_2) \\ =P(y_1>0|{\omega }_2) \\ ={\iiint }_{y_1>0}\frac{1}{(2\pi {)}^{\frac{l}{2}}}exp(-\frac{1}{2}[(y_1+b_l{)}^2+\sum _{i=2}^l{y}_i^2])d{y}_1...d{y}_l \\ ={\int }_0^{+\infty }\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}(y_1+b_l{)}^2)d{y}_1 \\ ={\int }_{b_l}^{+\infty }\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}{Z}^2)dZ \end{gathered}$$

Calculate $P_e$

We have $P({\omega }_1)=\frac{1}{2},P({\omega }_2)=\frac{1}{2}$:
$$\begin{gathered} P_e=P({\omega }_1)P(z=x^T\mu <0|{\omega }_1)+P({\omega }_2)P(z=x^T\mu >0|{\omega }_2) \\ =\frac{1}{2}{\int }_{b_l}^{+\infty }\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}{Z}^2)dZ+\frac{1}{2}{\int }_{b_l}^{+\infty }\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}{Z}^2)dZ \\ ={\int }_{b_l}^{+\infty }\frac{1}{(2\pi {)}^{\frac{1}{2}}}exp(-\frac{1}{2}{Z}^2)dZ \end{gathered}$$
Where $b_l=\Vert \mu \Vert$

Computer Experiment

Experiment 5.2

meaning of S_w, S_b, S_m, J3

number of classes: $M$

number of training samples in class ${\omega }_i$: $n_i$

number of features: $N$

dimension of features: $l$

Within Class:
$$\begin{gathered} S_w=\sum _{i=1}^M{P}_i{S}_{wi} \\ where{S}_{wi}=E[(x-{\mu }_i)(x-{\mu }_i{)}^T],P_i=\frac{n_i}{N} \end{gathered}$$
Between Class:
$$\begin{gathered} S_b=\sum _{i=1}^M{P}_i({\mu }_i-{\mu }_0)({\mu }_i-{\mu }_0{)}^T \\ where{\mu }_0=\sum _{i=1}^M{P}_i{\mu }_i \end{gathered}$$
Mixture Scatter:
$$S_m=E[(x-{\mu }_0)(x-{\mu }_0{)}^T]=S_w+S_b$$
Criteria $J_3$:
$$J_3=trace(S_w^{-1}{S}_m)$$

code: experiment5_2.m

%% Computer Experiment 5.2 close('all'); clear; clc; m = [-10 -10 10 10; -10 10 -10 10]; m_test = [-1 -1 1 1;-1 1 -1 1]; s1 = 0.2 * [1 0;0 1]; s2 = 3 * [1 0;0 1]; P = [0.25 0.25 0.25 0.25]'; N = 400; % 100 points each class, 4 class %% the Generated 400 Vectors in X1, X1_test, X2, X2_test randn('seed', 0); % reproducible for i = 1:size(m, 2) if i == 1X1 = mvnrnd(m(:, i), s1, fix(P(i)*N))';X1_test = mvnrnd(m_test(:, i), s1, fix(P(i)*N))';X2 = mvnrnd(m(:, i), s2, fix(P(i)*N))';X2_test = mvnrnd(m_test(:, i), s2, fix(P(i)*N))';elseX1 = [X1, mvnrnd(m(:, i), s1, fix(P(1)*N))'];X1_test = [X1_test, mvnrnd(m_test(:, i), s1, fix(P(1)*N))'];X2 = [X2, mvnrnd(m(:, i), s2, fix(P(1)*N))'];X2_test = [X2_test, mvnrnd(m_test(:, i), s2, fix(P(1)*N))'];end end y1 = [ones(1, fix((P(1)*N))), 2 * ones(1, fix(P(2)*N)), 3 * ones(1, fix((P(3)*N))), 4 * ones(1, fix((P(4)*N)))]; y1_test = y1; y2 = y1; y2_test = y2; %% Plot all Situations for 4 Classes figure; plot_point(X1, y1); title('Classes are Far away, $${\Sigma = 0.2 I}$$','Interpreter','latex'); figure; plot_point(X1_test, y2_test); title('Classes are Close, $${\Sigma = 0.2 I}$$','Interpreter','latex'); figure; plot_point(X2, y2); title('Classes are Far away, $${\Sigma = 3 I}$$','Interpreter','latex'); figure; plot_point(X2_test, y2_test); title('Classes are Close, $${\Sigma = 3 I}$$','Interpreter','latex'); %% Calculate S_w, S_b, S_m, J3 = trace(S_w \ S_m) [ S_w, S_b, S_m ] = Calc_SwSbSm( X1, y1 ) J_3 = J3(S_w, S_m) [ S_w, S_b, S_m ] = Calc_SwSbSm( X1_test, y1_test ) J_3 = J3(S_w, S_m) [ S_w, S_b, S_m ] = Calc_SwSbSm( X2, y2 ) J_3 = J3(S_w, S_m) [ S_w, S_b, S_m ] = Calc_SwSbSm( X2_test, y2_test ) J_3 = J3(S_w, S_m)

result

% m = [-10 -10 10 10; % -10 10 -10 10]; % Sigma = 0.2 I S_w =0.2070 0.00460.0046 0.2145 S_b =99.8278 -0.2653-0.2653 100.0591 S_m =100.0348 -0.2607-0.2607 100.2736 J_3 = 951.3471 % m_test = [-1 -1 1 1; % -1 1 -1 1]; % Sigma = 0.2 I S_w =0.2042 0.00350.0035 0.1999 S_b =1.0225 -0.0537-0.0537 0.9842 S_m =1.2266 -0.0502-0.0502 1.1841 J_3 = 11.9440 % m = [-10 -10 10 10; % -10 10 -10 10]; % Sigma = 3 I S_w =3.0501 -0.0759-0.0759 3.1290 S_b =99.7179 -0.4153-0.4153 100.9610 S_m =102.7680 -0.4912-0.4912 104.0900 J_3 = 66.9920 % m_test = [-1 -1 1 1; % -1 1 -1 1]; % Sigma = 3 I S_w =2.8682 -0.0492-0.0492 2.9545 S_b =1.1316 0.00250.0025 1.1283 S_m =3.9997 -0.0466-0.0466 4.0828 J_3 = 2.7766

conclusion

When $\Sigma$ of classes are the same:
$$S_w\approx \Sigma$$

When classes are Far away$\Rightarrow$ $trace(S_b)$ is big.
$$trace(S_b)=\frac{1}{N}\sum _{i=1}^M{n}_i({\mu }_i-{\mu }_0{)}^2,where{\mu }_0=\frac{1}{N}\sum _{i=1}^M{n}_i{\mu }_i$$

For $J3=trace(S_w^{-1}{S}_m)$:

• Relationship of $\sigma ,trace(s_b)$and $J3$:
  $$\begin{gathered} \Rightarrow \Sigma =\sigma I \\ also{S}_w\approx \Sigma =\sigma I \\ J3=trace(S_w^{-1}{S}_m)\approx trace(\frac{1}{\sigma }I{S}_m)=\frac{trace(S_m)}{\sigma } \\ =\frac{trace(S_w+S_b)}{\sigma }=\frac{trace(S_w)+trace(S_b)}{\sigma }\approx \frac{\sigma l+trace(S_b)}{\sigma }=l+\frac{trace(S_b)}{\sigma } \end{gathered}$$

  $m=10[-1-111;-11-11],\sigma =0.2$$m=[-1-111;-11-11],\sigma =0.2$

  $J3$	951.3471	11.9440
  $l+\frac{trace(S_b)}{\sigma }$	1001.4	12.0335
  	$m=10[-1-111;-11-11],\sigma =3$	$m=[-1-111;-11-11],\sigma =3$
  $J3$	66.9920	2.7766
  $l+\frac{trace(S_b)}{\sigma }$	68.8930	2.7533

• How $J3$ changes:

  Classes are Far away:
  $$\begin{gathered} \Rightarrow ({\mu }_i-{\mu }_0{)}^2\uparrow \\ \Rightarrow trace(S_b)=\frac{1}{N}\sum _{i=1}^M{n}_i({\mu }_i-{\mu }_0{)}^2\uparrow \\ \Rightarrow J3=trace(S_w^{-1}{S}_b)\approx l+\frac{trace(S_b)}{\sigma }\uparrow \end{gathered}$$
  Features within every class are Close:
  $$\begin{gathered} \Rightarrow \sigma \downarrow \\ \Rightarrow J3=trace(S_w^{-1}{S}_b)\approx l+\frac{trace(S_b)}{\sigma }\uparrow \end{gathered}$$
  ​

Experiment 5.4

FDR: Fisher’s discriminant ratio

Theory of Fisher’s Linear Discriminant

Columns of $w$ are corresponding to the Highest Eigen Values of $S_W^{-1}{S}_B$

code: experiment5_4.m

%% Computer Experiment 5.4 close('all'); clear; clc; m = [2 2.5; 4 10]; s1 = [1 0;0 1]; s2 = 0.25 * [1 0;0 1]; P = [0.5 0.5]'; N = 200; % 100 points each class, 2 class %% the Generated 200 Vectors in X1, X1_test, X2, X2_test randn('seed', 0); % reproducible for i = 1:size(m, 2) if i == 1X1 = mvnrnd(m(:, i), s1, fix(P(i)*N))';X2 = mvnrnd(m(:, i), s2, fix(P(i)*N))';elseX1 = [X1, mvnrnd(m(:, i), s1, fix(P(1)*N))'];X2 = [X2, mvnrnd(m(:, i), s2, fix(P(1)*N))'];end end y1 = [ones(1, fix((P(1)*N))), 2 * ones(1, fix(P(2)*N))]; y2 = y1; %% Get FDR value, w (line projected on) of X1, X2 [ S_w, S_b, S_m ] = Calc_SwSbSm( X1, y1 ) [ FDR_1, w_1 ] = FDR( X1, y1 , 1) % set D_y: Dimension of w = 1 S_w \ [m(:, 1)-m(:, 2)] ans = ans / sqrt(sum(ans.^2)) [ S_w, S_b, S_m ] = Calc_SwSbSm( X2, y2 ); [ FDR_2, w_2 ] = FDR( X2, y2 , 1) S_w \ [m(:, 1)-m(:, 2)] ans = ans / sqrt(sum(ans.^2)) %% Plot all Situations for 4 Classes f1 = figure; plot_point(X1, y1); hold on; h1 = ezplot(@(x, y) w_1(1)*x + w_1(2)*y); hold on; H1 = ezplot(@(x, y) w_1(2)*x - w_1(1)*y, [-2 8 0 15]); f2 = figure; plot_point(X2, y2); hold on; h2 = ezplot(@(x, y) w_2(1)*x + w_2(2)*y); hold on; H2 = ezplot(@(x, y) w_2(2)*x - w_2(1)*y, [-4 6 0 12]); figure(f1); title('$${\Sigma = I}$$','Interpreter','latex'); set(h1,'Color','r'); set(H1,'Color','g'); legend([h1 H1], 'y = wTx', 'line to be projected on', 'Location', 'Best') figure(f2); title('$${\Sigma = 0.25 I}$$','Interpreter','latex'); set(h2,'Color','r'); set(H2,'Color','g'); legend([h2 H2], 'y = wTx', 'line to be projected on', 'Location', 'NorthWest')

result

% Sigma = I FDR_1 = 8.1336 w_1 =-0.1118-0.9937 S_w \ [m(:, 1)-m(:, 2)] = % for 2 classes-0.6136-5.5634 normalized S_w \ [m(:, 1)-m(:, 2)] =-0.1096-0.9940 % Sigma = 0.25 I FDR_2 = 38.4050 w_2 =0.0075-1.0000 S_w \ [m(:, 1)-m(:, 2)] = % for 2 classes-0.0019-25.9308 normalized S_w \ [m(:, 1)-m(:, 2)] =-0.0001-1.0000

conclusion

Theoretically, for 2 classes:

When $J(w)$ is maximized,
$$\begin{gathered} DifferentiatingJ(w)=\frac{w^T{S}_{b}w}{w^T{S}_{w}w} \\ Having(w^T{S}_{b}w)S_{w}w=(w^T{S}_{b}w)S_{b}w \\ \Rightarrow w=J(w)(S_w^{-1}{S}_b)w,J(w)=k_1 \\ Also{S}_b=(m_1-m_2)(m_1-m_2{)}^T,S_{b}w=((m_1-m_2{)}^{T}w)(m_1-m_2),((m_1-m_2{)}^{T}w)=k_2 \\ \Rightarrow w=k_1{k}_2{S}_w^{-1}(m_1-m_2) \end{gathered}$$
So $w$ has the same direction with $S_w^{-1}(m_1-m_2)$

Actually:

When $\Sigma =I$, covariance is big enough, $w$ and $S_w^{-1}(m_1-m_2)$ are in the same direction.

When $\Sigma =0.25I$, covariance is too small, there is big difference between directions of $w$ and $S_w^{-1}(m_1-m_2)$.

$w$$S_w^{-1}(m_1-m_2)$normalized $S_w^{-1}(m_1-m_2)$$J(w)$

$\Sigma =I$	-0.1118; -0.9937	-0.6136; -5.5634	-0.1096; -0.9940	8.1336
$\Sigma =0.25I$	0.0075; -1.0000	-0.0019; -25.9308	-0.0001; -1.0000	38.4050

Theoretically:
$$\begin{gathered} Having{S}_w\approx \Sigma =\sigma I,Restraintcondition:w^{T}w=1,Maxeigenvalueof{S}_b:{\Lambda }_{max} \\ DifferentiatingJ(w)=\frac{w^T{S}_{b}w}{w^T{S}_{w}w} \\ Having(w^T{S}_{b}w)S_{w}w=(w^T{S}_{b}w)S_{b}w \\ \Rightarrow eigenvalueproblem:J(w)w=S_w^{-1}{S}_{b}w\approx \frac{S_b}{\sigma }w \\ For2classes:J(w)\approx {\lambda }_{max-\sigma }maxeigenvalueof\frac{S_b}{\sigma } \\ J(w)\approx {\lambda }_{max-\sigma }=\frac{{\Lambda }_{max}}{\sigma } \end{gathered}$$
When Distance between centers is not changed:
$$\begin{gathered} \Rightarrow ({\mu }_i-{\mu }_0)({\mu }_i-{\mu }_0{)}^T=constmatrix \\ \Rightarrow S_b=\frac{1}{N}\sum _{i=1}^M{n}_i({\mu }_i-{\mu }_0)({\mu }_i-{\mu }_0{)}^T=constmatrix \\ \Rightarrow {\Lambda }_{max}=const \\ \Rightarrow J(w)\sigma \approx {\Lambda }_{max}=const \end{gathered}$$
$\sigma$ of classes is big $\Rightarrow $ FDR: Fisher’s discriminant ratio is small.
$$\begin{gathered} \sigma \uparrow \\ \Rightarrow J(w)\approx {\lambda }_{max-\sigma }=\frac{{\Lambda }_{max}}{\sigma }\downarrow \end{gathered}$$
$\sigma$ of classes is small $\Rightarrow $ FDR: Fisher’s discriminant ratio is big.
$$\begin{gathered} \sigma \downarrow \\ \Rightarrow J(w)\approx {\lambda }_{max-\sigma }=\frac{{\Lambda }_{max}}{\sigma }\uparrow \end{gathered}$$
Actually:

$\sigma$$J(w)$$\sigma J(w)$${\Lambda }_{max}$$\frac{\sigma J(w)}{{\Lambda }_{max}}\times 100\%$

$\Sigma =I$	1	8.1336	8.1336	8.7515	92.94%
$\Sigma =0.25I$	0.25	38.4050	9.6013	8.9344	107.46%

​

总结

以上是生活随笔为你收集整理的homework5_ZhankunLuo的全部内容，希望文章能够帮你解决所遇到的问题。

如果觉得生活随笔网站内容还不错，欢迎将生活随笔推荐给好友。