This is what helped me understand it most:

from here

primarily the picture on the right side.

Let the pink left side of the rectangle be a vector v you're projecting onto.

Let the black line labeled with a '1' be a vector u that you're projecting onto v.

The bit of pink on the top left is the orthogonal "line of projection" from the tip of u to the span of v.

The angle $\alpha - \beta$ in the pink triangle is *the angle between v and u*.

Imagine you're tilting the diagram a bit clockwise so that the long side ($cos\beta$) of the orange triangle is horizontal.

Now the $cos\beta$ side of the orange triangle is the x component of u in an imagined cartesian coordinate system, and the $sin\beta$ side of the orange triangle is the y component.

The blue side on the right is equal in length to the pink side on the left since this is a rectangle. On the blue side we can see the length is made up of $cos\beta \cdot cos\alpha + sin\beta \cdot sin\alpha$, which, if u and v are unit length, is the dot product.

Here's another diagram of the same scenario with a "normal" axis, and then also a rotated grid corresponding to v, the orange vector being projected onto

$cos\beta$ is the length of the x component of u, a unit vector $\beta$ counter-clockwise from the x axis.

$cos\alpha$ is the same for v. It's also the factor by which a vector pointing along the x axis (such as the x part of u) is *scaled* when it's projected onto v. This is because: say you have a right triangle with hypotenuse of length 1 lying on the x axis, with another side aligned with v. That side will have angle $\alpha$ with the hypotenuse (because v is $\alpha$ counter-clockwise from the x axis) and so its length will be $cos\alpha$.

So the x component of u is of length $cos\beta$, then when being "scaled" by projecting onto $v$, it becomes $cos\beta \cdot cos\alpha$.

The case with y components and $sin$ is analogous.