1

When writing network code we often find ourselves populating structs from byte slices to access the data in form of an object.

Let's take this struct

type PACKETHEAD struct {
    Type uint16
    Size uint16
    Hash uint32
}

and a byte slice that has been somehow populated with data

data := make([]byte, 1024)

My solution would be to

var pkthead PACKETHEAD
pktsiz := unsafe.Sizeof(pkthead)
pktbuf := bytes.NewReader(buf[:pktsiz])
err = binary.Read(pktbuf, binary.BigEndian, &pkthead)
if err != nil {
    // handle it
}

But

  • It uses unsafe

  • Requires ~7 lines of code for every cast (what if we had hundreds of different packets)

  • Can't be trivially packed into a Cast(*struct, data) function

  • No control over struct padding, what If go's compiler decides to add extra bytes in between members on one end of a network?

  • binary.Read performs a data copy if I'm not mistaken (this isn't necessarily a con)


In C one would just #pragma pack(1) on both network ends, agree on one type of endianess

and finally PACKETHEAD* pkt = (PACKETHEAD*)dataptr;

How can we achieve the same thing with Go?


Have a nice day, Kris

I .
  • 73
  • 1
  • 8
  • Possible duplicate of [Converting several bytes in an array to another type in Go](http://stackoverflow.com/questions/4308385/converting-several-bytes-in-an-array-to-another-type-in-go) – Denzel Feb 18 '16 at 03:38
  • @Denzel It's not a duplicate, the answer from the link you posted is literally my suggestion. – I . Feb 18 '16 at 03:41
  • Exactly, that's the correct way. Doing it the C way would break Go's type safety, hence it's not offered. – Denzel Feb 18 '16 at 03:42
  • @Denzel Well, I hope that's not the only way. That's why I asked this question to begin with. – I . Feb 18 '16 at 03:48
  • @I . Please refer to the [Conversions](https://golang.org/ref/spec#Conversions) section of the Go Programming Language specification. It explains in detail what conversions (commonly referred to as casts) are possible in Go. – Denzel Feb 18 '16 at 04:03
  • 1
    How about using a serializer like protobuf? That seems a lot safer. – Andy Schweig Feb 18 '16 at 04:05
  • Or json is another nice choice. – Justlike Feb 18 '16 at 05:24
  • Is this a good use case for the gob encoding? https://golang.org/pkg/encoding/gob/ – foo Feb 18 '16 at 05:57

1 Answers1

3

Shameless plug for gopack, a library I (and others) wrote to support bitpacking in Go. Note: it uses unsafe operations under the hood, if that's a problem.

joshlf
  • 16,739
  • 9
  • 55
  • 78
  • +1 This is actually gold. Go devs should consider integrating your work into the standard package. One question, how does your library handle padding and alignment? I'm asking because the server is written in Go and the client in C. – I . Feb 18 '16 at 06:04
  • It ignores alignment, and only packs the data that the programmer is aware of. So, for example, if you had `struct{ a, b uint32 }`, but it was aligned with 4 bytes between `a` and `b` (ie, effectively `struct{ a, _, b uint32 }`), and you packed into an 8-byte slice, everything would fit. – joshlf Feb 18 '16 at 07:24
  • Relatedly, I'm not even sure what it would mean to pay attention to alignment when you ask for fewer bits than the size of the actual field (for example, what would it mean to pack 7 bits of an 8 bit field followed by 4 bytes of padding followed by 3 bits of a 16 bit field?). – joshlf Feb 18 '16 at 07:25
  • Also, as for adding it to the stdlib, I appreciate the suggestion, but it needs a lot of work before that'd be a reasonable suggestion. That said, I'm always open to pull requests ;) – joshlf Feb 18 '16 at 07:26