Questions tagged [scalar-fields]

A scalar field is a function of the type $X\to \Bbb R$, where $X$ may be an open set in $\mathbb R^n$ or more generally a smooth manifold.

286 questions
139
votes
13 answers

Why is gradient the direction of steepest ascent?

$$f(x_1,x_2,\dots, x_n):\mathbb{R}^n \to \mathbb{R}$$ The definition of the gradient is $$ \frac{\partial f}{\partial x_1}\hat{e}_1 +\ \cdots +\frac{\partial f}{\partial x_n}\hat{e}_n$$ which is a vector. Reading this definition makes me consider…
Jing
  • 1,999
  • 4
  • 17
  • 27
93
votes
5 answers

Difference between gradient and Jacobian

Could anyone explain in simple words (and maybe with an example) what the difference between the gradient and the Jacobian is? The gradient is a vector with the partial derivatives, right?
67
votes
3 answers

Gradient of 2-norm squared

Could someone please provide a proof for why the gradient of the squared $2$-norm of $x$ is equal to $2x$? $$\nabla\|x\|_2^2 = 2x$$
64
votes
5 answers

How to take the gradient of the quadratic form?

It's stated that the gradient of: $$\frac{1}{2}x^TAx - b^Tx +c$$ is $$\frac{1}{2}A^Tx + \frac{1}{2}Ax - b$$ How do you grind out this equation? Or specifically, how do you get from $x^TAx$ to $A^Tx + Ax$?
38
votes
2 answers

Taking a derivative with respect to a matrix

I'm studying about EM-algorithm and on one point in my reference the author is taking a derivative of a function with respect to a matrix. Could someone explain how does one take the derivative of a function with respect to a matrix...I don't…
31
votes
4 answers

How to calculate the gradient of log det matrix inverse?

How to calculate the gradient with respect to $X$ of: $$ \log \mathrm{det}\, X^{-1} $$ here $X$ is a positive definite matrix, and det is the determinant of a matrix. How to calculate this? Or what's the result? Thanks!
pluskid
  • 1,059
  • 2
  • 9
  • 16
27
votes
4 answers

Derivative of squared Frobenius norm of a matrix

In linear regression, the loss function is expressed as $$\frac1N \left\|XW-Y\right\|_{\text{F}}^2$$ where $X, W, Y$ are matrices. Taking derivative w.r.t $W$ yields $$\frac 2N \, X^T(XW-Y)$$ Why is this so?
25
votes
2 answers

The gradient as a row versus column vector

Kaplan's Advanced Calculus defines the gradient of a function $f : \mathbb{R^n} \to \mathbb{R}$ as the $1 \times n$ row vector whose entries respectively contain the $n$ partial derivatives of $f$. By this definition then, the gradient is just the…
ItsNotObvious
  • 10,045
  • 11
  • 64
  • 96
19
votes
5 answers

Differentiate $f(x)=x^TAx$

Calculate the differential of the function $f: \Bbb R^n \to \Bbb R$ given by $$f(x) = x^T A x$$ with $A$ symmetric. Also, differentiate this function with respect to $x^T$. How exactly does this work in the case of vectors and matrices? Could…
12
votes
5 answers

Proof for the funky trace derivative : $d (\operatorname{trace} (ABA'C))$?

Given three matrices $A$, $B$ and $C$ such that $ABA^T C$ is a square matrix, the derivative of the trace with respect to $A$ is: $$ \nabla_A \operatorname{trace}( ABA^{T}C ) = CAB + C^T AB^T $$ There is a proof here, page 4 (PDF file). However, I…
Harold
  • 187
  • 1
  • 6
11
votes
3 answers

What is the derivative of $\log \det X$ when $X$ is symmetric?

According to Appendix A.4.1 of Boyd & Vandenberghe's Convex Optimization, the gradient of $f(X):=\log \det X$ is $$\nabla f(X) = X^{-1}$$ The domain of the $f$ here is the set of symmetric matrices $\mathbf S^n$. However, according to the book…
11
votes
2 answers

Derivative of quadratic matrix form with respect to the matrix

Suppose we have the following quadratic form $$ f(M)=x^TMx $$ where $f: \mathbb{R}^{n \times m} \rightarrow \mathbb{R}$, and $x \in \mathbb{R}^n$. As it is obvious, this function is linear in $M$. What is the derivative of $f(M)$ with respect to…
Saeed
  • 4,031
  • 1
  • 8
  • 23
11
votes
1 answer

Prove that $\nabla_{\mathrm X} \mbox{tr} (\mathrm A \mathrm X^{-1} \mathrm B) = - \mathrm X^{-\top} \mathrm A^\top \mathrm B^\top \mathrm X^{-\top}$

Prove that $$\nabla_{\mathrm X} \mbox{tr} (\mathrm A \mathrm X^{-1} \mathrm B) = - \mathrm X^{-\top} \mathrm A^\top \mathrm B^\top \mathrm X^{-\top}$$ My proof is below. I am interested in other proofs. My proof Let $$f (\mathrm X) := \mbox{tr}…
10
votes
2 answers

Gradient of squared Frobenius norm

I would like to find the gradient of $\frac{1}{2} \big \| X A^T \big \|_F^2$ with respect to $X_{ij}$. Going by the chain rule in the Matrix Cookbook (eqn 126), it's something like $$\frac{\partial}{\partial X_{ij}} \Big[\frac{1}{2} \big\| X A…
8
votes
4 answers

How to differentiate product of vectors (that gives scalar) by vector?

I'm trying to understand derivation of the least squares method in matrices terms: $$S(\beta) = y^Ty - 2 \beta X^Ty + \beta ^ T X^TX \beta$$ Where $\beta$ is $m \times 1$ vertical vector, $X$ is $n \times m$ matrix and $y$ is $n \times 1$…
1
2 3
19 20