15

Let $f:\mathbb{R}^m \to \mathbb{R}^n$. Is it possible to do a Taylor expansion of $f$ around $\theta\in\mathbb{R}^m$? I am hoping for something like

$$f\left(\theta\right) = f\left(\theta_0\right) + A \left(\theta - \theta_0\right) + \left(\theta - \theta'\right)^T \text{something} \left(\theta - \theta'\right)$$

where $A$ is a $n\times m$ matrix, and its rows are gradient of $f_i$ ($i$-the entry of vector $f$) with respect to vector $\theta$. But what should "something" be?

For example, let us consider the simple case where $f :\mathbb{R}^m \to \mathbb{R}^2$. Denote $f = \left(f_1,f_2\right)$. Then

$$f_1\left(\theta_n\right) = f_1(\theta_0) + \nabla f_1\left(\theta_0\right)(\theta_n - \theta_0) + (\theta_n - \theta_0)^T H_1(\theta') (\theta_n - \theta_0) $$

$$f_2\left(\theta_n\right) = f_2(\theta_0) + \nabla f_2\left(\theta_0\right)(\theta_n - \theta_0) + (\theta_n - \theta_0)^T H_2(\theta') (\theta_n - \theta_0) $$

If I try to put this in matrix form, I will get something like

$$f\left(\theta_0\right) = f(\theta_0) + \begin{pmatrix} \nabla f_1(\theta_0)\\ \nabla f_2(\theta_0)\end{pmatrix} (\theta_n - \theta_0) + ???$$

Yuki Kawabata
  • 649
  • 1
  • 4
  • 11

3 Answers3

8

As you seem to have worked out for yourself, you can just write the Taylor series for each component of $f$ separately; so I guess the remaining issue is how to write this down neatly.

Your "something" here is the second derivative of $f$, which is the third-order tensor comprised of the Hessian matrices $H_1,\ldots,H_n$; so you need to use a notation that can deal with this kind of object. If you're comfortable with the Einstein summation convention, then you can write $$f_i(\theta) = f_i(\theta_0) + A_{ij} (\theta_0)(\theta - \theta_0)_j + \frac 1 2 H_{ijk}(\theta')(\theta - \theta_0)_j(\theta-\theta_0)_k,$$ where $H_{ijk} = \frac{\partial^2 f_i}{\partial x_k \partial x_j}.$

Alternatively, you could go with something like $$f(\theta) = f(\theta_0) + A(\theta_0)(\theta - \theta_0) + \frac 1 2 H (\theta')(\theta-\theta_0,\theta-\theta_0)$$where $H$ is interpreted as a bilinear form on $\mathbb R^m$ taking values in $\mathbb R^n.$ You could write this using matrix multiplication as in Mostafa's answer, but this is slightly nonstandard and you should be clear that your "vector" $A$ and "matrix" $H$ are actually $\mathbb R^n$-valued.

Anthony Carapetis
  • 32,851
  • 3
  • 41
  • 89
  • Are we writing $f$ component-wise here? I hope to find a vector-valued Taylor expansion because, as you can see, if we apply Taylor expansion to $f_i$'s, the $\theta'$ we get in the second-order remainder might be different for different $f_i$'s... – Yuki Kawabata Feb 13 '18 at 18:14
  • @YukiKawabata: you're right, I glossed over this point - using the Lagrange form of the remainder is awkward in this case. I don't think you can always choose $\theta'$ independent of $i$. – Anthony Carapetis Feb 14 '18 at 22:53
  • @AnthonyCarapetis I'm pretty sure $A_{ij}$ in the first expansion is the Jacobian matrix. If I fix the $i$-index (i.e. focus on vector component), is the first expansion shown that for a multivariable scalar value function (https://en.wikipedia.org/wiki/Taylor_series#Taylor_series_in_several_variables)? In the second expansion (after "Alternatively..."), are you expressing a vector valued function? Is $\textbf{A}$ the Jacobian matrix? I'm confused because at the end you say "vector $\textbf{A}$", which makes me think it's not the Jacobian matrix and I'm missing something. – ThatsRightJack Apr 29 '21 at 12:02
4

the $something$ is ,roughly say Hessian matrix. In fact according to Taylor's theorem we have:$$f\left(\theta\right) = f\left(\theta_0\right) + A \left(\theta - \theta_0\right) + \dfrac{1}{2}\left(\theta - \theta'\right)^T H(\zeta) \left(\theta - \theta'\right)$$where $\zeta$ is between $\theta$ and $\theta_0$. Also we can write$$f\left(\theta\right) = f\left(\theta_0\right) + A \left(\theta - \theta_0\right) +\dfrac{1}{2} \left(\theta - \theta'\right)^T H(\theta_0) \left(\theta - \theta'\right)+o(||\theta-\theta_0||^2)$$where $o(.)$ indicates on little-o notation.

Mostafa Ayaz
  • 28,959
  • 5
  • 20
  • 56
3

Just for the sake of completeness, Wikipedia knows such expansions under the name jet. The form of the remainder has also been discussed on this site.