Derivative of sigmoid function $\sigma (x) = \frac{1}{1+e^{-x}}$

Question

In my AI textbook there is this paragraph, without any explanation.

The sigmoid function is defined as follows

$$\sigma (x) = \frac{1}{1+e^{-x}}.$$

This function is easy to differentiate because

$$\frac{d\sigma (x)}{d(x)} = \sigma (x)\cdot (1-\sigma(x)).$$

It has been a long time since I've taken differential equations, so could anyone tell me how they got from the first equation to the second?

@frog1944: It seems to be *Artificial Intelligence Illuminated* by Ben Coppin, page 302 ([Google Books link](https://books.google.com/books?id=LcOLqodW28EC&pg=PA302&lpg=PA302)). — Hans Lundmark, Nov 06 '17 at 11:18
Any book on neural networks will deal with the sigmoid function. It is useful because of the simple way backpropagation works; a lot of computing work is saved when training a network from a set of results. In nature, other functions are possible, like arctan, rational functions, and more. — richard1941, Jan 18 '18 at 02:02
One of the reasons they use the sigmoid is that it is easy to differentiate and facilitates backpropagation. Not so for other candidates like sign(x), arctangent(x), sinh(x), etc. — richard1941, Dec 21 '18 at 18:22

Michael Percy · Accepted Answer · 2017-09-01T04:25:39.863

403

Let's denote the sigmoid function as $\sigma(x) = \dfrac{1}{1 + e^{-x}}$.

The derivative of the sigmoid is $\dfrac{d}{dx}\sigma(x) = \sigma(x)(1 - \sigma(x))$.

Here's a detailed derivation:

$$ \begin{align} \dfrac{d}{dx} \sigma(x) &= \dfrac{d}{dx} \left[ \dfrac{1}{1 + e^{-x}} \right] \\ &= \dfrac{d}{dx} \left( 1 + \mathrm{e}^{-x} \right)^{-1} \\ &= -(1 + e^{-x})^{-2}(-e^{-x}) \\ &= \dfrac{e^{-x}}{\left(1 + e^{-x}\right)^2} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{e^{-x}}{1 + e^{-x}} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \dfrac{(1 + e^{-x}) - 1}{1 + e^{-x}} \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \left( \dfrac{1 + e^{-x}}{1 + e^{-x}} - \dfrac{1}{1 + e^{-x}} \right) \\ &= \dfrac{1}{1 + e^{-x}\ } \cdot \left( 1 - \dfrac{1}{1 + e^{-x}} \right) \\ &= \sigma(x) \cdot (1 - \sigma(x)) \end{align} $$

edited Sep 01 '17 at 04:25

answered Apr 08 '15 at 07:10

Michael Percy

4,146
2
10
5

5

Sir, Is d(e^x)=e^x? – Ravinder Payal Mar 06 '17 at 11:47
6

@RavinderPayal: Yes, d/dx(e^x) = e^x; If you want a proof of that see: https://www.khanacademy.org/math/ap-calculus-ab/advanced-differentiation-ab/proofs-for-derivatives-of-ex-and-lnx-ab/v/proof-d-dx-e-x-e-x – Michael Percy Mar 06 '17 at 23:02
1

Where does the (1 + e^-x) - 1 suddenly come from in third row up from bottom? – Jarad Jul 11 '17 at 05:26
for my untrained eyes this explanation runs rather smoothly except pre-last => last line. What happens there? – Vladimir Ignatov Aug 07 '17 at 13:51
5

@Jarad: e^-x == 1 + e^-x - 1; we are just adding 1 and subtracting 1 from the same term, which changes nothing. – Michael Percy Aug 17 '17 at 09:27
@VladimirIgnatov: Sorry, I'm not sure which line you are referring to. – Michael Percy Aug 17 '17 at 09:29
15

Thank you. Your explanation is much better than the answer. – Dmitriy Dokshin Sep 21 '17 at 19:03
It has been a long time since I've taken differential equations, so Its little complex to understand for me. I understand everything in line 1,2,4-9 except 3. From where there is an extra (-e^-x) ? An explanation would be very helpful. – Choudhury A. M. Jul 15 '18 at 15:13
@ChoudhuryA.M. The reason why d/dx[(1+e^-x)^-1] = −(1+e^−x)^−2(−e^−x) has to do with the chain rule for derivatives. See https://www.khanacademy.org/math/differential-calculus/dc-chain – Michael Percy Jul 18 '18 at 22:10
This really help on Andrew's DL week2 – Aung Zan Baw Apr 01 '21 at 07:29
Thank you. This is very helpful. – Avv Nov 23 '21 at 03:37

Hans Lundmark · Answer 2 · 2020-12-06T14:41:12.290

117

Consider $$ f(x)=\dfrac{1}{\sigma(x)} = 1+e^{-x} . $$ Then, on the one hand, the chain rule gives $$ f'(x) = \frac{d}{dx} \biggl( \frac{1}{\sigma(x)} \biggr) = -\frac{\sigma'(x)}{\sigma(x)^2} , $$ and on the other hand, $$ f'(x) = \frac{d}{dx} \bigl( 1+e^{-x} \bigr) = -e^{-x} = 1-f(x) = 1 - \frac{1}{\sigma(x)} = \frac{\sigma(x)-1}{\sigma(x)} . $$ Equate the two expressions, and voilà!

(Cf. also this answer.)

edited Dec 06 '20 at 14:41

answered Nov 03 '11 at 14:12

Hans Lundmark

48,535
7
82
143

How do you derive 1 + e^-x as -e^-x? (Update: I think it's because the derivative of e^x = e^x) https://en.wikipedia.org/wiki/Derivative#Rules_for_basic_functions – Adam Grant Aug 04 '17 at 23:36
2

@AdamGrant: Yes, since then the chain rule gives $e^{kx}=k e^{kx}$ for any constant $k$. (In this case, we have $k=-1$.) – Hans Lundmark Aug 05 '17 at 06:51
Correction to silly typo in the previous comment: it should be $\frac{d}{dx} e^{kx} = k e^{kx}$, of course. – Hans Lundmark Jan 24 '19 at 07:18

score 22 · Answer 3 · edited Sep 29 '15 at 14:10

22

Note that from your given equation,

$(1+e^{-x})\sigma=1$

$\Rightarrow -e^{-x}\sigma+(1+e^{-x})\frac{d\sigma}{dx}=0$ (differentiating using product rule)

$\Rightarrow \frac{d\sigma}{dx}=\sigma.\frac{e^{-x}}{(1+e^{-x})}=\sigma.\frac{(1+e^{-x})-1}{(1+e^{-x})}=\sigma.\left[1-\frac{1}{(1+e^{-x})}\right]=\sigma.(1-\sigma)$

edited Sep 29 '15 at 14:10

vonjd

8,348
10
47
74

answered Nov 03 '11 at 14:22

Tapu

3,446
11
19

score 8 · Answer 4 · edited May 06 '19 at 00:16

Since $\sigma(x)$ is a composite function, firstly we need to use chain rule to dig down to the x term, then we can factor back to the $\sigma(x)$ fuction: $$ \begin{align} \frac{d}{dx}\sigma(x) &= (\frac{1}{1+e^{-x}})' \\ &= -\frac{1}{(1+e^{-x})^{2}} \cdot (1) \cdot -e^{-x} \\ &= \frac{e^{-x}}{(1+e^{-x})^{2}}, \\ \because \sigma(x) &= \frac{1}{1+e^{-x}}, \\ e^{-x} &= \frac{1 - \sigma(x)}{\sigma(x)}, \\ 1+e^{-x} &= \frac{1}{\sigma(x)}; \\ \therefore \frac{d}{dx}\sigma(x) &= \frac{\frac{1 - \sigma(x)}{\sigma(x)}}{(\frac{1}{\sigma(x)})^{2}} \\ &= (1 - \sigma(x)) \cdot \sigma(x) \end{align}$$

Hugh Perkins · Answer 5 · 2017-09-29T09:16:22.633

Let's say we want to find the derivative of $y=σ(x)=(1+\exp(−x))^{−1}$. So we have:

$$ \begin{align} \frac{dy}{dx} & = (-1)(1 + \exp(-x))^{-2} \frac{d}{dx}(1 + \exp(-x)) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(0 + \frac{d}{dx}\exp(-x)) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(\exp(-x)) \frac{d}{dx}(-x) \\ \\ & = (-1)(1 + \exp(-x))^{-2}(\exp(-x))(-1) \\ \\ & = \frac{\exp(-x)} {(1 + \exp(-x))^2} \\ \\ & = \frac{1 + \exp(-x) -1} {(1 + \exp(-x))^2} \\ \\ & = \frac{1 + \exp(-x)} {(1 + \exp(-x))^2} - \frac{1} {(1 + \exp(-x))^2} \\ \\ & = \sigma(x) - (\sigma(x))^2 \\ \\ & = \sigma(x) \cdot (1 - \sigma(x)) \end{align} $$

Narasimham · Answer 6 · 2020-02-20T09:33:14.793

By directly differenting:

$$ \sigma^{'} (x)= \frac{1. e^{-x}}{(1+e^{-x})^2} $$

Separately compute, multiply:

$${\sigma(x)}.{(1-\sigma(x))} =\frac{ e^{-x}}{(1+e^{-x}) } . \frac{ 1}{(1+e^{-x}) }$$

The RHSs agree.

EDIT1:

In general a solution of differential equation

$$ \frac{dy}{dx}=y(1-y) $$

can be seen to be

$$\frac{1}{1+c e^{-x}} \rightarrow \frac{1}{1+ e^{-x}} $$

with center point integration constant evaluated at $x=0, y=\frac12;\, c=1. $

score 0 · Answer 7 · answered Feb 12 '21 at 14:44

No answer yet involves the logarithm. If $\log$ denotes the natural logarithm then by the chain rule $$\frac{d}{dx} \log(\sigma(x)) = \frac{1}{\sigma(x)}\frac{d \sigma(x)}{dx}. $$ Furthermore $\log(\sigma(x))=\log(e^x) -\log(1+e^x)=x - \log(1+e^x)$ so that $$\frac{d}{dx} \log(\sigma(x)) = 1 - \frac{e^x}{1+e^x} = 1-\sigma(x).$$ Equaling the two displays gives $(d/dx)\sigma(x) = \sigma(x)(1-\sigma(x))$ as desired.

score 0 · Answer 8 · answered Feb 26 '21 at 06:18

Another approach using the quotient rule is as follows:

let $\sigma=\frac{1}{1+e^-x}$ upon rearranging ,

$\sigma=\frac{e^x}{1+e^x}$

using the quotient rule

$\frac{d}{{dx}}\left( {\frac{{f\left( x \right)}}{{g\left( x \right)}}} \right) = \frac{{\frac{d}{{dx}}f\left( x \right)g\left( x \right) - f\left( x \right)\frac{d}{{dx}}g\left( x \right)}}{{g^2 \left( x \right)}}$,

let $f(x)=e^x$ and $g(x)=1+e^x$,

we get $\frac{d\sigma}{dx}=\frac{(e^x)(1+e^x)-e^xe^x}{(1+e^x)^2}$

upon rearraning, we get :

$\frac{d\sigma}{dx}=\frac{e^x}{1+e^x}\frac{(1+e^x)-e^x}{1+e^x}$

upon further rearrangement:

$\frac{d\sigma}{dx}=\frac{e^x}{1+e^x}(1-\frac{e^x}{1+e^x})$

$\frac{d\sigma}{dx}=\sigma(1-\sigma)$

score -1 · Answer 9 · edited Nov 11 '21 at 09:10

-1

$\exp(-x) = \frac{1}{\sigma} -1 $ (By definition). Take the derivative of both sides:

$-\exp(-x)= -\frac{\sigma'}{\sigma^2}$

Add the two to get: $0 = \frac{1}{\sigma} -1 -\frac{\sigma'}{\sigma^2}$

and solve for $\sigma'=\sigma(1-\sigma)$ qed

edited Nov 11 '21 at 09:10

Gary

19,889
3
17
36

answered Nov 11 '21 at 08:26

Art Gagao

1

score -3 · Answer 10 · answered Mar 26 '20 at 01:17

Using my HP Prime, I differentiated 1/(1+exp(-x)) to get exp(-x)/(1+exp(-x))^2. Factor out 1/(1+exp(-x), which is sigma(x), and the rest is 1-sigma(x). That is proof by calculator. Beware! That machine can become addictive because of the way it amplifies your capability.

Derivative of sigmoid function $\sigma (x) = \frac{1}{1+e^{-x}}$

10 Answers10

Linked

Related