> For the complete documentation index, see [llms.txt](https://ramsane.gitbook.io/deep-learning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ramsane.gitbook.io/deep-learning/multi-layered-perceptron-1/dnn-from-scratch/forward-propagation.md). # Forward propagation Before jumping into the implementation, I would suggest you to watch this video, just to refresh your memory on how Neural Network works. This is part of a playlist. And I will refer all those videos wherever necessary. For now, you can watch this video and move on to the article. {% embed url="" %} Basics of Neural networks and forward propagation. {% endembed %} Assuming that you understood the video, let's take a closer look at the network once again with annotated parameters for the input layer, first hidden layer and the weights that connecting them. ![](https://3125871907-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LskDNqFNx04llzI1sLA%2F-M0vpidVzVM7jNHPQ6-T%2F-M0wCKNY-Vd2EY_UqtBF%2Fimage.png?alt=media\&token=614c88d0-bb39-4c1a-849c-1a590e716878) ### Understanding Notations We will consider$$\boldsymbol H\_1$$as our current layer and when we talk w\.r.t $$\boldsymbol H\_1$$, then$$\boldsymbol I$$would become the previous layer. {% hint style="info" %} In this notation, superscript always denotes the layer number and the subscript always related to the node in a particular layer. {% endhint %} From the above diagram, $$w\_{11}^{1} \ \rightarrow$$ weight of an edge that connects the $$1^{st}$$ neuron in the previous layer$$(l-1)$$to the $$1^{st}$$ neuron in the present layer$$(1)$$.\ $$w\_{21}^{1} \ \rightarrow$$ weight of an edge that connects the $$2^{nd}$$ neuron in the previous layer$$(l-1)$$to the $$1^{st}$$ neuron in the present layer$$(1)$$.\ \ $$w\_{12}^{1} \rightarrow$$ weight of an edge that connects $$1^{st}$$ neuron from the previous layer$$(l-1)$$to the $$2^{nd}$$ neuron in the present layer $$(1)$$ .\ $$w\_{22}^{1} \rightarrow$$ weight of an edge that connects $$2^{nd}$$ neuron from the previous layer$$(l-1)$$to the $$2^{nd}$$ neuron in the current layer$$(1)$$.\ ...\ ..\ .\ $$z^1\_1, a^1\_1 \rightarrow$$ pre-activation and post-activation of $$1^{st}$$layer's $$1^{st}$$ neuron\ $$z^1\_2, a^1\_2 \rightarrow$$ pre-activation and post-activation of $$1^{st}$$ layer's $$2^{nd}$$ neuron\ $$z^1\_3, a^1\_3 \rightarrow$$ pre-activation and post-activation of $$1^{st}$$ layer's $$3^{rd}$$ neuron In general,\ \ $$\boldsymbol {w\_{i,j}^{l}} \rightarrow \text {weight between } \boldsymbol {(l-1)^{th}} \text { layer's } \boldsymbol{(i)^{th}} \text { node } \text {and } \boldsymbol {(l)^{th}} \text { layer's } \boldsymbol {(j)^{th}} \text {node.}$$$$\boldsymbol {b^l\_i} \rightarrow \text {bias for } \boldsymbol {(l)^{th}} \text { layer's } \boldsymbol {(i)^{th}} \text {node. }$$\ $$\boldsymbol {z^l\_i} \rightarrow \text {pre-activation of } \boldsymbol {(l)^{th}} \text { layer's } \boldsymbol {(i)^{th}} \text {node. }$$ $$\boldsymbol {a^l\_i} \rightarrow \text {post-activation of } \boldsymbol {(l)^{th}} \text { layer's } \boldsymbol {(i)^{th}} \text {node. } \ \hspace3em \text {This is also an input to the next layer.}$$ ## Layer - 1 ( First hidden layer ) ![](https://3125871907-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LskDNqFNx04llzI1sLA%2F-M1VAzNsAVKe8VOuYF_I%2F-M1VBD_xW7OFgeY6yWrJ%2Fimage.png?alt=media\&token=8d88dd37-58a0-4cfa-bc56-49727f6d7e0f) ### Non-vectorized $$\longrightarrow$$ Vectorized : The pre-activations for the first layer can be computed as follows (assuming all the $$ \gdef\w#1#2#3{w^#1\_{#2#3}} \gdef\a#1#2{a^#1\_#2} \gdef\b#1#2{b^#1\_#2} \gdef\z#1#2{z^#1\_#2} \gdef\redhash{#D0021B} \gdef\red{\color{\redhash}} \gdef\brownhash{#8B572A} \gdef\brown{\color{\brownhash}} \gdef\greenhash{#7ED321} \gdef\green{\color{\greenhash}} \gdef\bluehash{#4A90E2} \gdef\blue{\color{\bluehash}} \large \begin{aligned} %================================ &{\red \w111}{\brown x\_1} + {\red \w121}{\brown x\_2} + {\red \b11} = {\red \z11}\\\\ % ================================ &{\green\w112}{\brown x\_1} + {\green \w122}{\brown x\_2} + {\green \b12} = {\green \z12}\\\\ %================================ &{\blue \w113}{\brown x\_1} + {\blue \w123}{\brown x\_2} + {\blue \b13} = {\blue \z13} %================================ \end{aligned} \gdef\w#1#2#3{w^#1\_{#2#3}} \gdef\a#1#2{a^#1\_#2} \gdef\b#1#2{b^#1\_#2} \gdef\z#1#2{z^#1\_#2} \gdef\redhash{#D0021B} \gdef\red{\color{\redhash}} \gdef\brownhash{#8B572A} \gdef\brown{\color{\brownhash}} \gdef\greenhash{#7ED321} \gdef\green{\color{\greenhash}} \gdef\bluehash{#4A90E2} \gdef\blue{\color{\bluehash}} \longrightarrow \large %============ \begin{pmatrix} \red \w111 \enspace \w121 \\\\ \green \w112 \enspace \w122 \\\\ \blue \w113 \enspace \w123 \end{pmatrix} %============ \begin{pmatrix} \brown x\_1 \\\\ \brown x\_2 \end{pmatrix}\ %============ \+ %============ \begin{pmatrix} \red \b11\\\\ \green \b12\\\\ \blue \b13 \end{pmatrix} %============ \= %============ \begin{pmatrix} \red \z11 \\\\ \green \z12\\\\ \blue \z13 \end{pmatrix} %============ $$ We can replace the $$z's$$ vector with $$\boldsymbol {Z\_1}$$ (1 in the denominator indicates the layer number ie., 1 in this case), weight matrix with $$\boldsymbol {W\_1}$$, input with $$\boldsymbol {A\_0}$$and finally bias vector for this layer as $$\boldsymbol {W\_1}$$**.**\ The denominator indicates the layer number ie., $$1$$ in this case {% hint style="info" %} #### why $$\boldsymbol A\_0$$instead of $$\boldsymbol X$$for the input? During forward propagation in further layers, the input would be the the output from previous layers which would be denoted as $$\boldsymbol A\_{(l-1)}$$ , where $$(l-1)$$ is the previous layer index. ie.,$$\boldsymbol A\_1$$is the input for second hidden layer, and for 3rd hidden layer, the input is $$\boldsymbol A\_2$$and so on. Just to make the notation consistent across the network, we consider$$\boldsymbol A\_0$$instead of$$\boldsymbol X$$ {% endhint %} Finally, we can replace the above equations with the below vector notation. $$ \large \begin{aligned} \&W\_{1\_{\enspace(3,2)}} \&A\_0{*{\enspace(2,1)}} &+ B*{1\_{\enspace(3,1)}} &= Z\_{1\_{\enspace(3,1)}}\\ &\[{W\_1}&{A\_0}]*{*{\enspace(3,1)}} &+ B\_{1\_{\enspace(3,1)}} &= Z\_{1\_{\enspace(3,1)}} \end{aligned} $$ where, $$ \gdef\w#1#2#3{w^#1\_{#2#3}} \gdef\a#1#2{a^#1\_#2} \gdef\b#1#2{b^#1\_#2} \gdef\z#1#2{z^#1\_#2} \gdef\redhash{#D0021B} \gdef\red{\color{\redhash}} \gdef\brownhash{#8B572A} \gdef\brown{\color{\brownhash}} \gdef\greenhash{#7ED321} \gdef\green{\color{\greenhash}} \gdef\bluehash{#4A90E2} \gdef\blue{\color{\bluehash}} \large W\_1 = \begin{pmatrix} \red \w111 \enspace \w121 \\\\ \green \w112 \enspace \w122 \\\\ \blue \w113 \enspace \w123 \end{pmatrix}, A\_0 = \begin{pmatrix} \brown x\_1 \\\\ \brown x\_2 \end{pmatrix} , B\_1 = \begin{pmatrix} \red \b11\\\\ \green \b12\\\\ \blue \b13 \end{pmatrix}, Z\_1 = \begin{pmatrix} \red \z11 \\\\ \green \z12\\\\ \blue \z13 \end{pmatrix} $$ Now, post activation$$\large \boldsymbol {A\_1}$$is just a function of $$\large \boldsymbol {Z\_1}$$ , which can be of any non-linear function such as $$\text {Sigmoid}$$ , $$\text {Tanh}$$, $$\text {ReLU}$$, $$\text {Leaky ReLU}$$. In this case, I would be sticking with $$\text {Sigmoid}$$. $$ \large \begin{aligned} A\_1 &= Sigmoid(Z\_1)\ &= \sigma(Z\_1)\ &=\frac{1 }{1+e^{- Z\_1}} \end{aligned} $$ ## Layer 2 ( Second Hidden layer ) We will look at the second layer, But as we have already discussed on how to vectorize the operations, we will mention the matrices directly. ![](https://3125871907-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LskDNqFNx04llzI1sLA%2F-M108qfmn6Qr71VQhrT9%2F-M10Lh2g0AP9wRNhFBaD%2Fimage.png?alt=media\&token=1366645d-afbf-4e88-892f-9adb6c889b69) ### Non-vectorized $$\longrightarrow$$ Vectorized: $$ \gdef\w#1#2#3{w^#1\_{#2#3}} \gdef\a#1#2{a^#1\_#2} \gdef\b#1#2{b^#1\_#2} \gdef\z#1#2{z^#1\_#2} \gdef\layer{2} \gdef\redhash{#D0021B} \gdef\red{\color{\redhash}} \gdef\brownhash{#8B572A} \gdef\brown{\color{\brownhash}} \gdef\greenhash{#7ED321} \gdef\green{\color{\greenhash}} \gdef\bluehash{#4A90E2} \gdef\blue{\color{\bluehash}} \large {\begin{aligned} %================================ &{\red \w211}{\brown \a11} + {\red \w221}{\brown \a12} + {\red \w231}{\brown \a13} + {\red \b21} = {\red \z21}\\\\ &{\green \w212}{\brown \a11} + {\green \w222}{\brown \a12} + {\green \w232}{\brown \a13} + {\green \b22} = {\green \z22} \\\\ &{\blue \w213}{\brown \a11} + {\blue \w223}{\brown \a12} + {\blue \w233}{\brown \a13} + {\green \b23} = {\green \z23} %================================ \end{aligned}} \longrightarrow {%============ \begin{pmatrix} \red \w\layer11 \enspace \w\layer21 \enspace \w\layer31 \\\\ \green \w\layer12 \enspace \w\layer22 \enspace \w\layer32 \\\\ \blue \w\layer13 \enspace \w\layer23 \enspace \w\layer33 \end{pmatrix} %============ \begin{pmatrix} \brown \a11 \\\\ \brown \a12 \\\\ \brown \a13 \end{pmatrix}\ %============ \+ %============ \begin{pmatrix} \red \b\layer1\\\\ \green \b\layer2\\\\ \blue \b\layer3 \end{pmatrix} %============ \= %============ \begin{pmatrix} \red \z21 \\\\ \green \z22\\\\ \blue \z23 \end{pmatrix}} %============ $$ For this layer, the inputs are, $$\large \boldsymbol {A\_1}$$(output from the previous layer ), the weight matrix $$\large \boldsymbol {W\_2}$$and bias vector $$\large \boldsymbol {B\_2}$$. The equations would be as follows. $$ \large \begin{aligned} Z\_{2\_{\enspace(3,1)}} &= W\_{2\_{\enspace(3,3)}} A\_{1\_{\enspace(3,1)}} &+B\_{2\_{\enspace(3,1)}} \\ Z\_{2\_{\enspace(3,1)}} &= \[{W\_2A\_1}]*{*{\enspace(3,1)}} &+ B\_{2\_{\enspace(3,1)}} \\ \\ A\_{2\_{\enspace(3,1)}} &=Sigmoid(Z\_{2{\_{\enspace(3,1)}}}) \end{aligned} $$ where, $$ \gdef\layer{2} \gdef\w#1#2#3{w^#1\_{#2#3}} \gdef\a#1#2{a^#1\_#2} \gdef\b#1#2{b^#1\_#2} \gdef\z#1#2{z^#1\_#2} \gdef\redhash{#D0021B} \gdef\red{\color{\redhash}} \gdef\brownhash{#8B572A} \gdef\brown{\color{\brownhash}} \gdef\greenhash{#7ED321} \gdef\green{\color{\greenhash}} \gdef\bluehash{#4A90E2} \gdef\blue{\color{\bluehash}} \large W\_\layer = \begin{bmatrix} \red \w\layer11 \enspace \w\layer21 \enspace \w\layer31 \\\\ \green \w\layer12 \enspace \w\layer22 \enspace \w\layer32 \\\\ \blue \w\layer13 \enspace \w\layer23 \enspace \w\layer33 \end{bmatrix} , A\_1 = \begin{bmatrix} \brown \a11 \\\\ \brown \a12 \\\\ \brown \a13 \end{bmatrix} , B\_2 = \begin{bmatrix} \red \b\layer1\\\\ \green \b\layer2\\\\ \blue \b\layer3 \end{bmatrix} , Z\_2 = \begin{bmatrix} \red \z21 \\\\ \green \z22\\\\ \blue \z23 \end{bmatrix} $$ ## Output Layer
### Non-vectorized $$\longrightarrow$$ vectorized: $$ \gdef\layer{3} \gdef\w#1#2#3{w^#1\_{#2#3}} \gdef\a#1#2{a^#1\_#2} \gdef\b#1#2{b^#1\_#2} \gdef\z#1#2{z^#1\_#2} \gdef\redhash{#D0021B} \gdef\red{\color{\redhash}} \gdef\brownhash{#8B572A} \gdef\brown{\color{\brownhash}} \large \begin{aligned} %================================ {\red \w\layer11}{\brown \a21} + {\red \w\layer21}{\brown \a22} + {\red \w\layer31}{\brown \a23} + {\red \b\layer1} = {\red \z\layer1} %================================ \end{aligned} \gdef\layer{3} \gdef\w#1#2#3{w^#1\_{#2#3}} \gdef\a#1#2{a^#1\_#2} \gdef\b#1#2{b^#1\_#2} \gdef\z#1#2{z^#1\_#2} \gdef\redhash{#D0021B} \gdef\red{\color{\redhash}} \gdef\brownhash{#8B572A} \gdef\brown{\color{\brownhash}} \longrightarrow %============ \begin{bmatrix} \red \w\layer11 \enspace \w\layer21 \enspace \w\layer31 \\ \end{bmatrix} %============ \begin{bmatrix} \brown \a21 \\\\ \brown \a22 \\\\ \brown \a23 \end{bmatrix}\ %============ \+ %============ \begin{bmatrix} \red \b\layer1\\ \end{bmatrix} %============ \= %============ \begin{bmatrix} \red \z\layer1 \end{bmatrix} %============ $$ For this layer, the inputs are, $$\large \boldsymbol {A\_2}$$(output from the previous layer ), the weight matrix $$\large \boldsymbol {W\_3}$$and bias vector $$\large \boldsymbol {B\_3}$$. The equations would be as follows. $$ \gdef\layer{3} \gdef\prevlayer{2} \large \begin{aligned} Z\_{\layer\_{\enspace(1,1)}} &= W\_{\layer\_{\enspace(1,3)}} A\_{\prevlayer\_{\enspace(3,1)}} &+B\_{\layer\_{\enspace(1,1)}} \\ Z\_{\layer\_{\enspace(1,1)}} &= \[{{W\_\layer}{A\_\prevlayer}}]*{*{\enspace(1,1)}} &+ B\_{\layer\_{\enspace(1,1)}} \end{aligned} $$ where, $$ \gdef\layer{3} \gdef\w#1#2#3{w^#1\_{#2#3}} \gdef\a#1#2{a^#1\_#2} \gdef\b#1#2{b^#1\_#2} \gdef\z#1#2{z^#1\_#2} \gdef\redhash{#D0021B} \gdef\red{\color{\redhash}} \gdef\brownhash{#8B572A} \gdef\brown{\color{\brownhash}} \large W\_\layer = \begin{bmatrix} \red \w\layer11 \enspace \w\layer21 \enspace \w\layer31 \\ \end{bmatrix} , A\_2 = \begin{bmatrix} \brown \a21 \\\\ \brown \a22 \\\\ \brown \a23 \end{bmatrix} , B\_\layer = \begin{bmatrix} \red \b\layer1 \end{bmatrix} , Z\_\layer = \begin{bmatrix} \red \z\layer1 \end{bmatrix} $$ As it is the $$Output\ layer$$ , the output from this layer is nothing but the predicted value(s) for the given input example. As we are dealing with binary classification, we have just one neuron with $$Sigmoid$$ activation function. $$ \gdef\layer{3} \gdef\prevlayer{2} \large A\_{\layer\_{\enspace(1,1)}} = Sigmoid(Z\_{\layer{\_{\enspace(1,1)}}}) $$ The output value is a single value between 0 and 1. We can use this as a probability score to predict the given label for an appropriate threshold during inference (test time). While training, we will calculate the $$Cross\ Entropy$$ for all the input examples.
Output layer with loss function
$$ \begin{aligned} Cross \ Entropy, \ \mathit L &= -\sum\_{i=1}^{m} y\_i\log{\hat {y\_i}} \\ &= -\sum\_{i=1}^{m} y\_i\log{(a^3\_1)} \end{aligned} $$ But current example, we have only one input example, so $$m=1$$. The whole process with multiple batch would be mentioned at the end, even though it doesn't change much of the equations around here except the shape of input at each layer. ( More would be discussed later. ) ## Summary ![](https://3125871907-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LskDNqFNx04llzI1sLA%2F-M13qzxtrVpHzd-4O4vL%2F-M168zVTVK4lLEU2_1Av%2Fimage.png?alt=media\&token=bc2f2cf4-6d19-4029-a40a-300ba9814a3d) ### Is that Loss function correct ? For people, who know about `log loss` and `cross entropy`, might get confused with this. As I've used `cross-entropy` even when we have one neuron. This is just to keep the notation constant. All you have to do to make it work for `cross-entropy` is to create probability for the other class, by subtracting 1 from the output. And for the loss function also holds same for `log loss`and `cross-entropy`. Please do little research on this, you will come to know that both are same. ## Task: * Write a class that takes #input\_dimensions, #hidden\_layers, #hidden\_layer\_size(number of neurons in the hidden layer), #output\_dimensions ( 1 in this example with sigmoid activation ). * Initialize all of them as class variables. * As you already know the dimensions of each weight and bias matrices, initialize them randomly. * Write a function `call()` that takes the input of any batch\_size, and return the output. * You can check the shape of the output to cross-verify your code.