How to include data object files (images, etc.) in program and access the symbols?

Question

I've turned a couple of resource files into .obj files using objcopy and i link them with my programs source code. I can very well access the symbols inside the object file in my program with the following code, but only with GCC/G++ (Cygwin):

extern uint8_t data[]   asm("_binary_Resources_0_png_start");
extern uint8_t size[]   asm("_binary_Resources_0_png_size");
extern uint8_t end[]    asm("_binary_Resources_0_png_end");

The code doesn't work in Visual Studio, probably because VS has it's own __asm command. I want to include my programs resources (Images, Shaders, etc.) in my final executable's .data section through linking them.

But how can i access the symbols defined in the object file in VC++? I tried extern uint8_t _binary_Resources_0_png_start[] or extern "C" uint8_t _binary_Resources_0_png_start[] without the assembly command, but i get unresolved symbol link errors.

Answer 1

The trick with objcopy isn't meant as a full-featured way to embed resources and isn't portable at all, as you have seen.

Microsoft has its own mechanism for resources, so if you're specifically targeting windows, you could use a windows resource file and the RCDATA resource.

If you want something completely portable, your only option is to format the file as C sourcecode like e.g.

const uint8_t my_binary[] = { 0x00, 0x01, ... }

It's straight forward to write your own conversion tool for that.

Answer 2

It is a may be completely different approach but it provides a rather simple but portable solution:

We use a small tool to load a binary file and output it as C (or C++ source). Actually, I saw things like this in XPM and GIMP but it can be used for rather any binary data.

To include such tool in the build chain is not difficult in VS, even more simple in make and cmake also.

Such a tool could look like this:

#include <fstream>
#include <iostream>
#include <string>

using namespace std;

int main(int argc, char **argv)
{
  if (argc < 2) {
    cerr << "Usage: " << argv[0] << " FILE [FILE...]" << endl;
    return -1;
  }
  for (size_t i = 1; i < argc; ++i) {
    fstream fIn(argv[i], ios::in | ios::binary);
    if (!fIn.good()) {
      cerr << "ERROR: Cannot open '" << argv[i] << "'!" << endl;
      continue;
    }
    // make name
    string name = argv[i];
    name = name.substr(0, name.find('.'));
    /// @todo more sophisticated name mangling?
    // print preface
    cout << "struct { const char *data; size_t size; } " << name << " = {" << endl
      << "  \"";
    // print data
    const char hex[] = "0123456789abcdef";
    unsigned char byte;
    enum { BytesPerLine = 16 };
    size_t n = 0;
    for (unsigned char byte; fIn.get((char&)byte); ++n) {
      if (n && !(n % BytesPerLine)) cout << "\"\n  \"";
      cout << "\\x" << hex[byte / 16] << hex[byte % 16];
    }
    // print size
    cout << "\",\n"
      "  " << n << "\n"
      "};" << endl;
  }
  return 0;
}

Compiling and test:

$ g++ -std=c++11 -o binToC binToC.cc

$ ./binToC
Usage: ./binToC FILE [FILE...]

More testing with fluffy_cat.png :

$ ./binToC fluffy_cat.png > fluffy_cat.inc

$ cat >fluffy_cat_test.cc <<'EOF'
> #include <fstream>
> 
> using namespace std;
> 
> #include "fluffy_cat.inc"
> 
> int main()
> {
>   ofstream fOut("fluffy_cat_test.png", ios::out | ios::binary);
>   fOut.write(fluffy_cat.data, fluffy_cat.size);
>   fOut.close();
>   return 0;
> }
> EOF

$ g++ -std=c++11 -o fluffy_cat_test fluffy_cat_test.cc

$ ./fluffy_cat_test

$ diff fluffy_cat.png fluffy_cat_test.png

$

As the diff shows – the C source reproduces the original exactly.

Btw. I used the same technique (in similar form) in my answer to SO: Paint a rect on qglwidget at specifit times.

Answer 3

Your question originally didn't state whether this is for 64-bit Cygwin G++/MSVC++ or 32-bit. There is a subtle difference when it comes to name decorations.

---

x86 (32-bit Windows PE) solution with OBJCOPY

I'll assume you had a resource file called Resources_0.png. You can generate a 32-bit Windows PE object file with:

objcopy --prefix-symbol=_ --input-target binary --output-target \
    pe-i386 --binary-architecture i386 Resources_0.png Resources_0.obj

The --prefix-symbol=_ appends an additional underscore (_) to each label. Name decorating with an additional _ is standard for Win32/PE external object. The resulting file would have produced an object with these labels:

__binary_Resources_0_png_start
__binary_Resources_0_png_end
__binary_Resources_0_png_size

MSVC++ and Cygwin G++ targeting 32-bit executables can reference these labels as:

extern "C" uint8_t _binary_Resources_0_png_start[];
extern "C" uint8_t _binary_Resources_0_png_end[];
extern "C" uint8_t _binary_Resources_0_png_size[];

---

x86-64 (64-bit Windows PE) solution with OBJCOPY

You can generate a 64-bit Windows PE object file with:

objcopy --input-target binary --output-target pe-x86-64 --binary-architecture i386 \
    Resources_0.png Resources_0.obj

This is similar to the 32-bit however we no longer add an additional underscore (_) before each label. That is because in 64-bit PE code the names aren't decorated with an additional underscore.

The resulting file would have produced an object with these labels:

_binary_Resources_0_png_start
_binary_Resources_0_png_end
_binary_Resources_0_png_size

MSVC++ and Cygwin G++ targeting 64-bit Windows PE executables can reference these labels the exact same was as the 32-bit Windows PE version above:

extern "C" uint8_t _binary_Resources_0_png_start[];
extern "C" uint8_t _binary_Resources_0_png_end[];
extern "C" uint8_t _binary_Resources_0_png_size[];

Special note: When compiling with MSVC++ as 64-bit code you may end up with this linking error when using the size label:

absolute symbol '_binary_Resources_0_png_size' used as target of REL32 relocation in section 4

With 64-bit code you can avoid this by computing the size in your C++ code by using the difference between the start and end labels like this:

size_t binary_Resources_0_png_size = _binary_Resources_0_png_end - \
                                     _binary_Resources_0_png_start;

---

Other Observations

Even if using G++/GCC this is bad form:

extern uint8_t data[]   asm("_binary_Resources_0_png_start");
extern uint8_t size[]   asm("_binary_Resources_0_png_size");
extern uint8_t end[]    asm("_binary_Resources_0_png_end");

There is little need for doing this and it is less portable. See the solutions above that don't use asm directive on variables for G++ code.

---

The question is tagged both C and C++ and the question contains code with extern "C". The answer above assumes you are compiling .cpp files with G++/MSVC++. If compiling .c files with GCC/MSVC then change extern "C" to extern

---

If you want to generate Windows PE objects with OBJCOPY where the data is placed in the read-only .rdata section rather than .data section, you can add this option to the OBJCOPY commands above:

--rename-section .data=.rdata,CONTENTS,ALLOC,LOAD,READONLY,DATA

I discuss this option in this Stackoverflow answer. The difference being that in Windows PE the read-only section is usually called .rdata where as with ELF objects it is .rodata

Answer 4

After working around and testing different things, i came back to my original approach (linking) and it worked like magic, here is the details:

In order to include data in the final executable's .data section, you need to first turn that data files (which could be an arbitrary binary file (anything!)) into a linkable file format, also known as an object file.

The tool objcopy which is included in GNU Binutils and is accessible in windows through Cygwin or MinGW, takes a file and produces an object file. objcopy requires two things to know before generating the object file, the output file format and the output architecture. In order to determine these two things, i check a valid linkable object file with the tool objdump:

objdump -f main.o

This gives me the following information:

main.o:     file format pe-x86-64
architecture: i386:x86-64, flags 0x00000039:
HAS_RELOC, HAS_DEBUG, HAS_SYMS, HAS_LOCALS
start address 0x0000000000000000

With this knowledge now i can create the object file:

objcopy -I binary -O pe-x86-64 -B i386 data_file.data data_file_data.o

In order to handle large number of files, batch files could come in handy.

I then simply link the produced object file(s) together with my programs source and dereference the pointers that objcopy generated, through the symbols, whose names could easily be queried with:

objdump -t data_file_data.o

Which results in:

data_file_data.o:     file format pe-x86-64

SYMBOL TABLE:
[  0](sec  1)(fl 0x00)(ty  0)(scl  2) (nx 0) 0x0000000000000000 _binary_data_file_data_start
[  1](sec  1)(fl 0x00)(ty  0)(scl  2) (nx 0) 0x0000000000000006 _binary_data_file_data_end
[  2](sec -1)(fl 0x00)(ty  0)(scl  2) (nx 0) 0x0000000000000006 _binary_data_file_data_size

Practically speaking, the following code works with GCC/G++:

extern uint8_t data[]   asm("_binary_data_file_data_start");
extern uint8_t end[]    asm("_binary_data_file_data_end");

And the following with MSVC++:

extern "C" uint8_t _binary_data_file_data_start[]; // Same name as symbol
extern "C" uint8_t _binary_data_file_data_end[];   // Same name as symbol

The size of each each file is calculated with:

_binary_data_file_data_end - _binary_data_file_data_start

You could for example write the data back into a file:

FILE* file;

file = fopen("data_file_reproduced.data", "wb");
fwrite(_binary_data_file_data_start,                               //Pointer to data
       1,                                                          //Write block size
       _binary_data_file_data_end - _binary_data_file_data_start,  //Data size
       file);

fclose(file);