The protocol behind the "Serial Port" is RS-232.
As every communication protocol it deals with symbols and frames and it is agnostic of the payload it is transmitting.
This means that the payload format is totally up to you.
RS-232 can handle 8-bit payload so you don't need any fancy bit manipulation even when sending binary data (I assume, of course, your machine has 8 bit bytes).
So, long story short, you can send whatever you want.
Writing to Serial Port, server side
I have absolutely no idea why you want to use node.js to do this since JS has no concept of integers (I don't know Node.js however, it may have. In that case ignore last sentence).
Anyway in order to write to the serial port you need help from your OS.
If you are using Linux, you can start from here.
Basically writing to serial port is like writing to a file.
On Windows, you do the same.
All of this boils down to writing a string to a file, I assume you know how to do it in Node.js.
Don't forget to properly set the settings (Bound rate, parity, start/stop) of the RS-232. Seriously, don't.
Reading from Serial Port, AVR side
This is very easy, it just boils down to handling a bunch of register.
I'm not gonna explain how to do it in details as you can reach the AVR datasheet for more information.
Anyway here a page which shows how to read the serial port avoiding polling.
Payload format
Since on the XMega side you will receive the payload one octet at a time, having variable length message, with integer coded as string would be just a pain the on back.
This however depends on which language you are more comfortable with.
If I were you I would choose a simple payload format:
op immediate
Where op is a 8 bit opcode, something like
00h = Nop
01h = Command 1
02h = Command 2
...
and immediate is a two octets field of data for the command.
This way every command is three octets long, you collect these three octets and then start the processing.
One problem you should be aware of is synchronization.
If for same reason you server and the XMega software go out of sync, no future command will be correctly processed.
You can use an autosync technique, like UTF-8 do: reserve the MSb of each octet for sync.
0ooo oooo 10ii iiii 11ii iiii
ooo oooo = 7 bit opcode
ii iiii ii iiii = 12 bit immediate
Where the digit bits are fixed and the letter bits are the actual payload: o are the bits of the (7-bit) opcode and i are the bit of the (12-bit) payload).
This give you 128 possible commands, each one with a number up to 4095 as operand and you can easily tell which position in the command a byte is from, allowing you to recover from loss of synchronization.
You can further shrink this down to two octets if you need no more that 32 commands
0ooo ooii 1iii iiii
This gives 5 bit opcode and 9 bit (Up to 512) operands. Auto synchronized.
