Here is an "illustrative example":

Show that the set $\Bbb Q-\{-1\}$ (all the rationals except $-1$) forms a group under the operation $\ast$, where $a\ast b = a + b + ab$.

First, we need to know $\ast$ is actually a binary operation on said set. It is clear that the domain of $(\Bbb Q-\{-1\})\times (\Bbb Q-\{-1\})$ is perfectly acceptable, but it is not immediately evident that the range of $\ast$ is wholly contained in $\Bbb Q-\{-1\}$. This is an important condition, called *closure*, which often needs to be verified.

So let's see if this can happen: suppose $a\ast b = -1$. Since we are dealing with rational numbers, we can freely use facts about rational numbers we already know. Thus:

$a + b + ab = -1 \implies a(b + 1) + b + 1 = 0 \implies (a + 1)(b + 1) = 0$

Since this can only happen if either $a$ or $b$ (or both) is $-1$, and neither is, we are assured of closure. Verifying associativity is, in this case, straightforward (but a little tedious):

$(a\ast b)\ast c = (a + b + ab)\ast c = a + b + ab + c + (a + b + ab)c$

$= a + b + c + ab + ac + bc + abc$

$= a + (b + c + bc) + ab + ac + abc = a + (b + c + bc) + a(b + c + bc)$

$= a \ast (b + c + bc) = a \ast(b \ast c)$

Now rather than assuming an identity exists (because we have no idea of knowing IF it does), we instead look for which rational numbers might be possible candidates. Suppose that $e$ is one such candidate, then for it actually to be an identity we need for any given $a$:

$a\ast e = a$, that is:

$a + e + ae = a \implies e + ae = 0 \implies e(1 + a) = 0$.

Since $1 + a \neq 0$ (because $a \neq -1$), the only viable candidate is $e = 0$. Now it is straightforward to show $0$ is indeed a two-sided identity for $\ast$ (alternatively, we could show that $\ast$ is commutative, and show $0$ is a one-sided identity).

Similarly, we have no idea if inverses exist under $\ast$. If $a$ has an inverse, say $b$ (bearing in mind such an $a$ may not exist, or that only SOME $a$'s may have such an inverse $b$), we have that:

$a \ast b = 0 \implies a + b + ab = 0 \implies b + ab = -a \implies b = \dfrac{-a}{1 + a}$.

Note that the only rational number $a$ for which $b$ is undefined is $a = -1$, which is not an element of our set. One more caveat, we have to show $b$ is always in our set, as well, which in this example boils down to showing $b$ cannot be $-1$ (it is clearly rational). But that leads to:

$-1 = \dfrac{-a}{1 + a} \implies \dfrac{a}{1 + a} = 1 \implies a = 1 + a \implies 0 = 1$, a contradiction. So it is clear this does not happen, and now we can readily verify that given $a \in \Bbb Q - \{-1\}$, that we have the unique inverse:

$a^{-1} = \dfrac{-a}{1+a}$, and we indeed have a group.

This procedure "doesn't always work", for example, when trying to determine if all 2x2 real matrices have inverses, one has to solve a system of two simultaneous linear equations:

$\begin{bmatrix}a&b\\c&d\end{bmatrix}$ has an inverse if the system:

$ax + by = 0$

$cx + dy = 0$

has only the unique solution $x = 0, y = 0$. When one tries to use elimination and substitution on this system of equations, to ensure uniqueness one must make the assumption the quantity $ad - bc \neq 0$. However, this cannot be guaranteed to be the case, and it turns out it is precisely these 2x2 matrices that fail to have inverses.