The Minimalist Calculus Cheat Sheet

We do not know closed form solutions for all optimization problems, even when they are somewhat innoccent-looking. One of the many possible methods in such as case is to use (stochastic) gradient descent to iteratively refine the solution to the problem. This involves the computation of… yes, the gradient.

In its simplest form, the gradient descent algorithm computes the gradient of an objective function relative to the parameters, evaluated on all training examples, and uses that gradient to adjust the model’s parameters. The gradient of a (not necessarily objective) function $f$ has the general form

$\displaystyle\nabla f=\left(\frac{\partial f}{\partial \theta_1},\frac{\partial f}{\partial \theta_2},\ldots,\frac{\partial f}{\partial \theta_n}\right)$ .

but to compute the partial derivative $\displaystyle\frac{\partial f}{\partial \theta_i}$ , you must know, or at least remember how to compute the derivative, and it’s not always trivial. But, as I was trying to remember a specific rule (involving the friendly hyperbolic tangent), I decided that I might as well make a list of useful derivative and integral rules and share it.

Of course, it is possible to have an infinite number of rules, if only by mere combination, but let us restrain ourself to the basic rules, the ones we’re most likely to encounter in (simple) optimization problems and basic geometry.

I would like this list to eventually grow, but, unlike most compendia, not to merely include more rules, but by adding actual derivations of the rules from the most basic ones. For example, the Leibniz’s integral rule clearly make sense, but what’s the simplest demonstration one can have?

*
* *

for $a$ constant, $\displaystyle\frac{\partial a}{\partial x}=0$

$\displaystyle\frac{\partial x}{\partial x}=1$

$\displaystyle\frac{\partial }{\partial x}au=a\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}(u+v)=\frac{\partial u}{\partial x}+\frac{\partial v}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}uv=u\frac{\partial v}{\partial x}+v\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}uvw=uv\frac{\partial w}{\partial x}+uw\frac{\partial v}{\partial x}+vw\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}\frac{u}{v}=\frac{v\frac{\partial u}{\partial x}-u\frac{\partial v}{\partial x}}{v^2}=\frac{1}{\partial v}\frac{\partial u}{\partial x}-\frac{u}{v^2}\frac{\partial v}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}u^n=nu^{n-1}\frac{\partial u}{\partial x}$ , $\displaystyle\frac{\partial }{\partial x}\frac{1}{n}=\frac{\partial }{\partial x}u^{-n}=\frac{-n}{u^{n+1}}\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}\sqrt{u}=\frac{1}{2\sqrt{u}}\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}\frac{1}{u}=-\frac{1}{u^2}\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}f(u)=\left(\frac{\partial }{\partial u}f(u)\right)\frac{\partial u}{\partial x}$ , the “chain rule.”

$\displaystyle\frac{\partial ^2}{\partial x^2}f(u)=\frac{\partial }{\partial u}f(u)\frac{\partial ^2u}{\partial x^2}+\frac{\partial ^2}{\partial u^2}f(u)\left(\frac{\partial u}{\partial x}\right)^2$

$\displaystyle\frac{\partial }{\partial x}\log_au=\ln a\frac{1}{u}\frac{\partial u}{\partial x}$ , $\displaystyle\frac{\partial }{\partial x}\ln u=\frac{1}{u}\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}a^u=a^u\ln a\frac{\partial u}{\partial x}$ , $\displaystyle\frac{\partial }{\partial x}e^u=e^u\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}u^v=v u^{v-1}\frac{\partial u}{\partial x}+u^v\ln u\frac{\partial v}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}\tanh u=(\mathrm{sech}^2 u)\frac{\partial u}{\partial x}$

$\displaystyle\frac{\partial }{\partial x}\tanh^{-1} u=\frac{1}{1-u^2}\frac{\partial u}{\partial x}$ , for $u^2<1$

$\displaystyle\frac{\partial }{\partial b}\int_a^b f(x)\partial x=f(b)$ , with $a$ constant

$\displaystyle\frac{\partial }{\partial a}\int_a^b f(x)\partial x=-f(a)$ , with $b$ constant

$\displaystyle\frac{\partial }{a\partial }\int_b^c f(x,a)\partial x=\int_b^c \frac{\partial }{\partial a}f(x,a)\partial x-f(b,a)\frac{\partial b}{\partial a}+f(c,a)\frac{\partial c}{\partial a}$

*
* *

$\displaystyle\int a \partial x = a\int \partial x = ax$

$\displaystyle\int a f(x) \partial x = a\int f(x) \partial x$

$\displaystyle\int (u+v) \partial x= \int u \partial x + \int v \partial x$

$\displaystyle\int u \partial v = u\int \partial v - \int v \partial u = uv - \int v \partial u$ , with variant $\displaystyle\int u \frac{\partial v}{\partial x}\partial x=uv-\int v\frac{\partial u}{\partial x}\partial x$

$\displaystyle \int x^n \partial x=\frac{1}{n+1}x^{n+1}$ , if $n\neq -1$ . If $n=-1$ , $\displaystyle\int \frac{1}{x}\partial x=\ln x$

$\displaystyle \int \frac{\frac{\partial}{\partial x}f(x)}{f(x)}\partial x=\ln f(x)$

, with special cases
1. $b=e$ , $\displaystyle \int e^{ax}\partial x=\frac{1}{a}e^{ax}$ ,
2. and $b=e$ , $a=1$ , $\displaystyle \int e^x \partial x = e^x$

$\displaystyle \int \log_a x \partial x = \frac{1}{\ln a}\int \ln x \partial x = \frac{x \ln x - x}{\ln a}$

$\displaystyle \int a^x \ln a \partial x=\ln a\int a^x\partial x=a^x$

$\displaystyle \int (a+bx)^n\partial x = \frac{(a+bx)^{n+1}}{(n+1)b}$ , if $n\neq-1$

$\displaystyle \int x^m(a+bx)^n \partial x=$ $\displaystyle\frac{x^{m+1}(a+bx)^n}{m+n+1}+\frac{an}{m+n-1}\int x^m(a+bx)^{n-1}\partial x$

*
* *

You can download the list as a pdf.

I am pretty sure the list is accurate (in the PDF version at least), but if you spot an error, please let me know.

This entry was posted on Tuesday, February 14th, 2012 at 10:29 am and is filed under algorithms, Mathematics, machine learning. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Harder, Better, Faster, Stronger