0

I'm working through Andrej Karpathy's awesome introduction to neural networks and the backpropagation algorithm, and am trying to differentiate the sigmoid function:

$$ \sigma(x) = \frac{1}{1+e^{-x}} $$

My understanding is the quotient rule holds that given an equation $y=\frac{t}{b}$ the derivative of the function should be:

$$y' = \frac{t'b - tb'}{b^{2}}$$

After applying the quotient rule to the sigmoid function, I thought the unsimplified result would be:

$$ \frac{-e^{-x}}{(1 + e^{-x})^{2}} $$

Because $t'$ is 0, which I thought would yield $ 0 - tb' $ in the numerator, or $-e^{-x}$. However, Karpathy's unsimplified derivative looks like:

$$ \frac{e^{-x}}{(1 + e^{-x})^{2}} $$

Does anyone know why the negative sign in the numerator gets dropped? I'd be very grateful for any help others can offer with this question!

duhaime
  • 207
  • 2
  • 10
  • the derivative of $e^{-x}$ is $-e^{-x}$, together with the minus sign from your formula, they cancel out – barmanthewise Dec 24 '17 at 18:08
  • @barmanthewise I thought the derivative of $e^{-x}$ is just $e^{-x}$, but is $e^{-x}$ really $-e^{-x}$? – duhaime Dec 24 '17 at 18:13
  • no, thats the derivative if $e^x$, look up a thing called 'chain rule' – barmanthewise Dec 24 '17 at 18:14
  • Ah yes, you're quite right, I was using the exponent rule to handle $e^{-x}$ and my book didn't show negative exponent examples. This is exactly what I was overlooking. If you make this an answer I'll mark it right. – duhaime Dec 24 '17 at 18:16
  • Related: https://math.stackexchange.com/questions/78575/derivative-of-sigmoid-function-sigma-x-frac11e-x – Hans Lundmark Dec 24 '17 at 20:38

2 Answers2

1

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Use the quotient rule, don't forget about the MINUS sign from the rule, and the MINUS sign due to the derivative of $e^{-x}$.

$$\sigma'(x) = \frac{-(-e^{-x})}{(1 + e^{-x})^2} = \frac{e^{-x}}{(1+e^{-x})^2}$$

Laplacian
  • 23,646
  • 9
  • 34
  • 53
0

better you write $$\sigma(x)=\frac{e^x}{e^x+1}$$ and then the first derivative is given by $$\sigma'(x)=\frac{e^x\cdot (e^x+1)-e^x\cdot e^x}{(e^x+1)^2}$$

duhaime
  • 207
  • 2
  • 10