## Answer:

the most mathematical thing would be to view them just as part of the axioms. One of the interpretations of a physical field is an operator-valued distribution meaning that in more analytical settings a physical field $\phi$ acts on the Hilbert space only after smearing with a test function $f$:
$$
\phi(f) = \int \phi(x) f(x) dx, \quad \phi(f) \in \mathrm{End}\, \mathcal{H},
$$
where $\mathcal{H}$ is a Hilbert space.
Usual test function spaces on $\mathbb{R}^n$ are the Schwartz space of rapidly decreasing functions or the bump functions. Without smearing, most of the fields are very badly behaved and singular. If we were to abuse the notation and write $\phi(x)$ instead of $\phi(f)$ and moreover assume that $\phi(f)$ has some kind of a Fourier series, then
$$
\phi(x)= \sum_{n\in\mathbb{Z}} a_{(n)} x^{-n-1},\quad a_{(n)}\in\mathrm{End}\,\mathcal{H}
$$
and hence
$$
\phi(x)\in \mathrm{End}\,\mathcal{H}[[x^{\pm 1}]],
$$
if we also forget that our $x$ was initially in $\mathbb{R}^n$. Finally, we can also forget the Hilbert space structure to end up with a vector space $\mathrm{H}$ as you have correctly observed. Also, CFT is rather special and we have state-field correspondence meaning that to each state of the physical system (e.g. $|p\rangle\in\mathcal{H}$ is a state of particle having momentum exactly $p$) there exists a field and vice versa. This is included in the axioms as
$$ Y(a, z)|0\rangle|_{z=0} = a \quad\text{(field to state)} \quad\text{and}\quad Y(\cdot,z)\quad \text{(state to field)}.$$

Note, however, that a Segal CFT can encode both: the full CFT and the chiral parts, whereas a VOA encodes just a chiral part and one has to patch-up two VOAs to produce a full field algebra.

## More precise answers:

Just add some formal variable $z$. Formal means that nothing about $z$ is implied, it's just there for bookkeeping purposes. You can of course say that $z\in \mathbb{C}$, but you don't have to. The extra $z$ as opposed to pair of pants, has to do with Schroedinger and Heisenberg approaches to evolution in quantum mechanics: whereas Segal CFT is a functorial QFT (FQFT), meaning that states of the system (vectors in the Hilbert/vector space) evolve and the operators acting on them don't, VOAs are in algebraic QFT (AQFT)- the states don't change, but the operators do.

Moreover, according to
nLab:

due to (Huang 91) it is now known that vertex operator algebras have equivalently an FQFT-type characterization which manifestly captures this relation to $n$-point functions in the 2d CFT:

There is a monoidal category or operad whose morphisms are conformal spheres with $n$-punctures marked as incoming and one puncture marked as outgoing (each puncture equipped with a conformally parametrized annular neighborhood). Composition of such spheres is by gluing along punctures. This can be regarded as a category $2Cob_0^{conf}$ of 2-dimensional genus-0 conformal cobordisms.

As shown by theorems by Yi-Zhi Huang and Liang Kong, a vertex operator algebra is precisely a holomorphic representation of this category, or algebra over an operad for this operad, i.e. a genus-0 conformal FQFT, hence a monoidal functor

$$ V:2Cob_0^{conf}\to Vect $$
such that its component $V_1$ is a holomorphic function on the moduli space of conformal punctured spheres.

Due to the above nLab reference I think that we cannot.

Certainly not :)

## Remarks

Simplest pair of pants explanation,

not all VOAs are rational, e.g. from Abe, Buhl, Dong "Rationality, regularity, and $C2$-cofiniteness":

Definition 2.6. A vertex operator algebra is called rational if every admissible
module is a direct sum of simple admissible modules.

Intro to "On Axiomatic Approaches
to Vertex Operator Algebras
and Modules" of Frenkel, Huang, Lepowsky mentions VOA similarities to Lie algebras and commutative associative algebras. Also, see Schottenloher "A Mathematical Intro to CFT" pp. 191-192 for connections with commutative associative algebras or Section 1.4 of Kac "Vertex algebras for beginners".