x86 assembly jl doesn't work

Question

I'm coding in x86 assembly (AT&T syntax) on 64-bit Ubuntu (so I'm using as --32 and ld -melf_i386, which has been working fine for other exercises so far).

The jl instruction is working opposite to what I expected. I can actually get the code to work properly with jg, which would basically solve my problem, but I'd like to find out the underlying issue here.

The code snippet is the following:

   # Open file for reading
   movl $SYS_OPEN, %eax         # prepare syscall 5 
   movl $input_file_name, %ebx      # move input file name into %ebx
   movl $0, %ecx                    # open in read-only mode
   movl $0666, %edx     
   int $LINUX_SYSCALL               # execute system call,
                                   # sys_open returns file descriptor in %eax

   movl %eax, ST_INPUT_DESCRIPTOR(%ebp) # store the input file descriptor away

   # This will test and see if %eax is negative. 
   # If it is not negative, it will jump to continue_processing.
   # Otherwise it will handle the error condition that the negative number represents.
   cmpl $0, %eax
   jl continue_processing

   pushl $no_open_file_msg
   pushl $no_open_file_code
   call error_exit

continue_processing:

   # Open file for writing
   movl $SYS_OPEN, %eax         # prepare syscall 5

...and the program continues, though the rest should be irrelevant for this issue.

Debugging with gdbgui, I see the open sys call returns the input file descriptor (eax = 3) without a problem. Then you compare 0 to 3. If I'm not an idiot, 0 < 3 so the jl instruction should take you to continue_processing, but it does not.

However, if I instead use jg, it works. The open sys call returns 3 in eax and jg properly jumps to continue_processing.

I've read that the order of the operands for jumps may depend on the assembler. Could that be the case here? Compiling with gcc -m32 -nostdlib has the same behavior. I also tried swapping the order to cmpl %eax, $0 but I get Error: operand type mismatch for 'cmp'.

Or could it just be a quirk of the fact that this 32-bit code is being run on 64-bit Ubuntu?

I am reading the book Programming From the Ground Up. On page 125 the book example interjects a .section .data right after the jl continue_processing, inserts some labels and .ascii commands, and then resumes the code with a .section .text right before the pushl $no_open_file_msg. To clean up the code, I've consolidated the .section .data up on top and so do not require a second .section .text. It doesn't seem to affect the jl issue but I thought I'd mention it in case the problem does actually lie there.

The book you are reading is Programming From the Ground Up (I added a reference in your question). The code in question is on page 125. This is in fact an error in the book and it should be `JGE` and not `JL`. This error has been reported on the bug tracking system for the book. That bug report is here: https://savannah.nongnu.org/bugs/?17654 . Unfortunately the book hasn't been updated in a decade. You can find a list of all the reported problems here: https://savannah.nongnu.org/bugs/?group=pgubook — Michael Petch, Dec 11 '17 at 13:31

fuz · Accepted Answer · 2017-12-11T13:49:16.023

1

In AT&T syntax, the order of operands is exchanged compared to Intel syntax. For this reason,

cmpl $0, %eax

actually computes eax − 0 and sets flags instead of computing 0 − eax as you might expect. Thus, all the flags are set the other way round than you expect initially, causing the issue you observe. There is not really a way to remedy this problem as it isn't really wrong. I recommend you to get used to this quirk.

As a general note, it's usually a good idea to use the testl instruction instead of comparing values with zero. testl %eax,%eax is more compact than cmp $0,%eax in the machine code and faster on some microarchitectures, too. testl sets flags according to the bitwise and of its operands, so if eax is negative, SF is set after testl %eax,%eax which you can check like this:

testl %eax,%eax
jns continue_processing

edited Dec 11 '17 at 13:49

answered Dec 11 '17 at 12:49

fuz

76,641
24
165
316

CF=0 after `and` **always**. But he can test SF `js negative_eax_detected`. ... and remedy is to use Intel syntax. – Ped7g Dec 11 '17 at 12:57
1

@Ped7g: It's more efficient to use JL instead of JS. Test (and cmp against zero) always clear OF as well, so it's equivalent to testing SF directly, but can macro-fuse on more CPUs. i.e.`test %eax,%eax` / `jl negative_eax_detected`. [TEST same,same and CMP-with-zero set all flags identically](https://stackoverflow.com/questions/33721204/test-whether-a-register-is-zero-with-cmp-reg-0-vs-or-reg-reg/33724806#33724806), except AF. – Peter Cordes Dec 11 '17 at 13:01
Thanks for the confirmation, it's odd because the book is all x86 AT&T assembly and the author used jl when I suppose he meant jg. But this is good enough, I have no problem getting used to the quirk. – Jaime Salazar Dec 11 '17 at 13:15
@JaimeSalazar Ah the book is Programming from the Ground Up page 125 – Michael Petch Dec 11 '17 at 13:22
@Ped7g Thank you, I confused CF and SF. – fuz Dec 11 '17 at 13:49
`testl` is only faster than `cmp` if you use `jge` instead of `jns`. You can think of `test same,same` as a peephole optimization for comparing against zero, because it sets flags identically. – Peter Cordes Dec 11 '17 at 14:40

x86 assembly jl doesn't work

1 Answers1