Segmentation fault when popping x86 stack

Question

I'm trying to link x86 assembly and C.

My C program:

extern int plus_10(int);

# include <stdio.h>

int main() {
    int x = plus_10(40);
    printf("%d\n", x);
    return 0;
}

My assembly program:

[bits 32]

section .text

global plus_10
plus_10:
    pop edx
    mov eax, 10
    add eax, edx
    ret

I compile and link the two as follows:

gcc -c prog.c -o prog_c.o -m32
nasm -f elf32 prog.asm -o prog_asm.o
gcc prog_c.o prog_asm.o -m32

However, when I run the resulting file, I get a segmentation fault.

But when I replace

pop edx

with

mov edx, [esp+4]

the program works fine. Can someone please explain why this happens?

`pop edx` moves the stack pointer, `mov edx, [esp+4]` doesn't. Normally in C it's up to the caller to clean the stack. — Jabberwocky, May 13 '19 at 13:52
@Jabberwocky But why would that cause a segmentation fault? The stack is common for both functions, right? — Susmit Agrawal, May 13 '19 at 13:53
Because you popped the return address not the argument. You can't use pop like this. — R.. GitHub STOP HELPING ICE, May 13 '19 at 13:55
@SusmitAgrawal because the return address is on the stack. Your `pop edx` actually pops the return adress from the stack and when the `ret` is executed the processor jumps to whatever address is on the stack — Jabberwocky, May 13 '19 at 13:55

score 31 · Accepted Answer · edited May 14 '19 at 02:20

This is a possible assembly code of int x = plus_10(40);

        push    40                      ; push argument
        call    plus_10                 ; call function
retadd: add     esp, 4                  ; clean up stack (dummy pop)
        ; result of the function call is in EAX, per the calling convention

        ; if compiled without optimization, the caller might just store it:
        mov     DWORD PTR [ebp-x], eax  ; store return value
                                        ; (in eax) in x

Now when you call plus_10, the address retadd is pushed on the stack by the call instruction. It's effectively a push+jmp, and ret is effectively pop eip.

So your stack looks like this in the plus_10 function:

|  ...   |
+--------+
|   40   |  <- ESP+4 points here (the function argument)
+--------+
| retadd |  <- ESP points here
+--------+

ESP points to a memory location that contains the return address.

Now if you use pop edx the return address goes into edx and the stack looks like this:

|  ...   |
+--------+
|   40   |  <- ESP points here
+--------+

Now if you execute ret at this point, the program will actually jump to address 40 and most likely segfault or behave in some other unpredictable way.

The actual assembly code generated by the compiler may be different, but this illustrates the problem.

BTW, a more efficient way to write your function is this: it's what most compilers would do with optimization enabled, for a non-inline version of this tiny function.

global plus_10
plus_10:
    mov   eax,  [esp+4]    ; retval = first arg
    add   eax,  10         ; retval += 10
    ret

This is smaller and slightly more efficient than

    mov   eax,  10
    add   eax,  [esp+4]        ; decode to a load + add.
    ret

The cdecl calling convention will expect the value to get returned through eax though. So you can't just write the asm function the way you like, it has to be compatible with the compiler-generated C. — Lundin, May 13 '19 at 14:12
@Lundin apparently his platform uses the cdecl convention. I also wrote _it's possible assembly code_, so depending on the platform it might be somewhat different. Edited and clarified. Thanks. — Jabberwocky, May 13 '19 at 14:14

Segmentation fault when popping x86 stack

1 Answers1