How to convert boost multiprecision integers into big endian from little endian?

Question

I am trying to fix this part of an abandonware program because I failed to find an alternative program.

As you can see the data of PUSH instructions are in the wrong order whereas Ethereum is a big endian machine (address are correctly represented because they use a smaller type).
An alternative is to run porosity.exe --code '0x61004b60026319e44e32' --disassm

Theu256 type is defined as

using u256 = boost::multiprecision::number<boost::multiprecision::cpp_int_backend<256, 256, boost::multiprecision::unsigned_magnitude, boost::multiprecision::unchecked, void>>;

Here’s a minimal example to reproduce the bug:

#include <sstream>
#include <iostream>
#include <iomanip>
#include <boost/multiprecision/cpp_int.hpp>

using u256 = boost::multiprecision::number<boost::multiprecision::cpp_int_backend<256, 256, boost::multiprecision::unsigned_magnitude, boost::multiprecision::unchecked, void>>;

int main() {
    std::stringstream stream;
    u256 data=0xFEDEFA;
    for (int i = 0; i<5; ++i) { // print only the first 5 digits
        uint8_t dataByte = int(data & 0xFF);
        data >>= 8;
        stream << std::setfill('0') << std::setw(sizeof(char) * 2) << std::hex << int(dataByte) << "  ";
    }
    std::cout << stream.str();
}

So numbers are converted to string with a space between each byte (and only the first bytes).

But then I ran into an endianness problem: bytes were printed in the reverse order. I mean for example, 31722 is written 8a 02 02 on my machine and 02 02 8a when compiled for a big endian target.

So as I don’t which boost function to call, I modified the code:

#include <sstream>
#include <iostream>
#include <iomanip>
#include <boost/multiprecision/cpp_int.hpp>

using u256 = boost::multiprecision::number<boost::multiprecision::cpp_int_backend<256, 256, boost::multiprecision::unsigned_magnitude, boost::multiprecision::unchecked, void>>;

int main() {
    std::stringstream stream;
    u256 data=0xFEDEFA;
    for (int i = 0; i<5; ++i) {
        uint8_t dataByte = int(data >> ((32 - i - 1) * 8));
        stream << std::setfill('0') << std::setw(sizeof(char) * 2) << std::hex << int(dataByte) << "  ";
    }
    std::cout << stream.str();
}

Now, why are my 256 bits integers printed mostly as series of 00 00 00 00 00?

Answer 1

BTW, this is not an endianness issue; you aren't doing byte accesses to the object-representation. You're operating on it as a 256-bit integer and simply asking for the low 8 bits at a time with data & 0xFF.

If you did know the endianness of the target C implementation, and the data layout of the boost object, you could efficiently loop over it in descending address order with unsigned char*.

You're introducing the idea of endianness only because it's associated with byte-reversal, which is what you're trying to do. But that's really inefficient, just loop over the bytes of your bigint the other way.

---

I'm hesitant to recommend a specific solution because I don't know what will compile efficiently. But you might want something like this instead of byte-reversing ahead of time:

for (outer loop) {
    uint64_t chunk = data >> (64*3);  // grab the highest 64-bit chunk
    data <<= 64;   // and shift everything up
    // alternative: maybe keep a shift-count in a variable instead of modifying `data`

    // Then pick apart the chunk into its component bytes, in MSB first order
    for (int = 0 ; i<8 ; i++) {
        unsigned tmp = (chunk >> 56) & 0xFF;
        // do something with it
        chunk <<= 8;                   // bring the next byte to the top
    }
}

In the inner loop, more efficient than using two shifts can be using a rotate to bring the high byte to the bottom (for & 0xFF) at the same time as shifting lower bytes upward. Best practices for circular shift (rotate) operations in C++

In the outer loop, IDK if boost::multiprecision::number has any APIs for efficient indexing of chunks built in; if so using that is probably more efficient.

I used nested loops because I assume data <<= 8 doesn't compile particularly efficiently, and neither would (data >> (256-8)) & 0xFF. But that's how you'd grab bytes from the top instead of the bottom.

---

Another option is the standard trick for converting numbers to strings: store characters into a buffer in descending order. A 256-bit (32-byte) number will take 64 hex digits, and you want another 32 bytes of spaces between them.

For example:

  // 97 = 32 * 2 + 32, plus 1 byte for an implicit-length C string terminator
  // plus another 1 for an extra space
  char buf[98];            // small enough to use automatic storage
  char *outp = buf+96;     // pointer to the end
  *outp = 0;               // terminator
  const char *hex_lut = "0123456789abcdef";

  for (int i=0 ; i<32 ; i++) {
      uint8_t byte = data & 0xFF;
      *--outp = hex_lut[byte >> 4];
      *--outp = hex_lut[byte & 0xF];
      *--outp = ' ';
      data >>= 8;
  }
  // outp points at an extra ' '
  outp++;
  // outp points at the first byte of a string like  "12 ab cd"
  stream << outp;

If you want to break that up into chunks to put a line break in there, you can do that too.

---

If you're interested in efficient conversion to hex for 8, 16 or 32 bytes of data at once, see How to convert a number to hex? for some x86 SIMD ways. The asm should port easily to C++ intrinsics. (You can use SIMD shuffles to handle putting bytes into MSB-first printing order after loading from little-endian integers.)

You could also use a SIMD shuffle to space-separate your pairs of hex digits before storing to memory like you apparently want here.

---

Bug in the code you added:

So I added this code before the loop above:

  for(unsigned int i=0,data,_data;i<33;++i)

unsigned i, data, _data declares new variables of type unsigned int that shadow the previous declarations of data and _data. That loop has zero effect on data or _data outside the scope of the loop. (And contains UB because you read _data and data without initializing them.)

If those vars are actually both still the u256 vars of the outer scope, I don't see an obvious problem other than efficiency, but maybe I'm missing the obvious too. I didn't look very hard because using 64x 256-bit shifts and 32x ORs seems like a horrible idea. It's possible it could optimize away completely, or into bswap byte-reverse instructions on ISAs that have them, but I doubt it. Especially not through the extra complication of the boost::multiprecision::number wrapper functions.