Vector spaces are not just a set! They are an abstract concept, involving a set $V$, a field $\mathbb{F}$, and operations
\begin{align*}
+ &: V \times V \rightarrow V \\
\cdot &: \mathbb{F} \times V \rightarrow V,
\end{align*}
addition and scalar multiplication respectively, satisfying a bunch of axioms. There's a lot more at play here than the set $V$ itself. Sets that can be made into vector spaces with the right field and operations are extremely common, but it's much rarer to be a vector space if the set already comes with the field and operations.

For example, the set of positive numbers $(0, \infty)$ doesn't seem like it's a vector space, but with scalar field $\mathbb{R}$ and with the (non-standard) operations,

\begin{align*}
\oplus &: (0, \infty) \times (0, \infty) \rightarrow (0, \infty) : (x, y) \mapsto xy \\
\odot &: \mathbb{R} \times (0, \infty) \rightarrow (0, \infty), (\lambda, x) \mapsto x^\lambda
\end{align*}

it forms a vector space. Even the natural numbers could be defined to be a vector space over a finite field, or a countable field like $\mathbb{Q}$ (although the operations would look a little funky).

So, to answer your question, can I come up with a set that is definitely not a vector space? Yes. As it turns out, all finite fields have a cardinality that takes the form $q = p^m$, where $p$ is prime and $m \in \mathbb{N}$. As such, finite vector spaces over such a finite field, which must have some finite dimension $n \ge 0$, must have cardinality $q^n$. Therefore, a set with a number of elements not equal to a prime power $p^{mn}$ must not be a finite vector space under any operations. For example, a set with $6$ elements is definitely not a vector space!