Check that register is a negative 4 bytes value in NASM

Question

i would like to check that the value I put into the rax register is a negative or null 8 bytes value (a negative long int in C).

It leads me to check if the 64 bits into the rax register correspond to a signed bits value.

After research, i found that if we reverse each bits of the value like -86 in base10 and add 1, we obtain the invert value 86.

Based on this, an negative inverted value will be lesser than the negative value in term of bits.

I am building and running a code in NASM on a x86_64 Linux.

I am applying the following code that display a message when I is negative :

section .data
msg db "I is negative", 0

section .text
global main
extern printf, exit

%define I 9

main:
mov rax, I
; Invert the bits into rax 
xor rax, 0xFFFFFF
inc rax
mov rbx, I
cmp rax, rbx
jl  EXIT
; Display message when I is negative
lea rdi, [msg]
xor rax, rax
call printf

EXIT:
call exit
ret

Here is how i compile the NASM code :

nasm -f elf64 Program.s -o Program.o -Werror 
gcc Program.o -o a.out

But this program is wrong because it does not work correctly.

It seems that i am misunderstanding the way i can check if a register contains a negative integer. Any help ?

Answer 1

EAX is the low 4 bytes of RAX.

test eax,eax               ; sets flags the same as cmp eax,0, like from eax-0
jnl  I_was_non_negative    ; jumps if EAX was *not* less-than 0.  Synonym for jge

Test whether a register is zero with CMP reg,0 vs OR reg,reg? explains why test is slightly more efficient than cmp for comparing against zero. (Mostly the 1 byte of code-size).

The less-than condition tests SF and OF, but compare against zero can't overflow, so it's equivalent to just testing SF (the sign flag, set from the MSB of the result). In this case you have the choice of jnl or jns (not less or not signed), or jge. Pick whichever one has the semantic meaning you like best.

---

int in C is 4 bytes, not 8. (In all the standard 32 and 64-bit x86 ABIs, including x86-64 System V that you'll find on Linux).

---

Problems with your attempt to implement -I < I (I think) using the 2's complement identity
-x = ~x + 1

xor rax, 0xFFFFFF only flips the low 24 bits (that's 6 F digits, not 8).

But xor rax, 0xFFFFFFFF is not encodable, because it doesn't fit in a 32-bit sign-extended immediate. x86-64 still uses 8 or 32 bit immediates even for 64-bit operand-size, not 8 or 64, because that would be very bloated code size. See https://felixcloutier.com/x86/XOR.html for what's encodable. (There is a mov r64, imm64, but that's the only instruction that takes a 64-bit immediate.)

So if you'd used xor eax, -1 (or not eax) instead of insisting on using 64-bit operand-size, then maybe your weird code would have worked for comparing -I < I. But flipping the low 32 or 24 bits of a 64 bit register, and then doing a 64-bit compare, isn't helpful. The upper bits will always be zero.

If you had used a 32-bit compare, then you'd have a problem for the most-negative number. 0x80000000 is its own 2's complement inverse. i.e. neg eax would leave it unchanged (and set OF because 0 - 0x80000000 causes signed overflow back to negative).

If you had sign-extended your 4-byte input in EAX to 8 bytes in RAX before negating, then it could have worked.

    mov     eax, I
    movsxd  rax, eax     ; or cdqe
   ; not     rax
   ; inc     rax
    neg     rax
    cmp     rax, I        ; use I as a sign-extended 32-bit immediate
    jl     I_is_positive  ; won't be taken for 0 either.

Note the different between positive and non-negative. -I < I is false for I=0, but you asked about checking for negative (the opposite of non-negative, not the opposite of positive).

---

I is an assemble-time constant

You can use the NASM preprocessor to test it.

default rel        ; always a good idea to use RIP-relative for static data

; %define I 9

extern puts
global main
main:
%if  I < 0
    lea rdi, [rel msg]
    xor eax, eax
    call puts              ; puts appends a newline
%endif

   xor  eax,eax           ; return 0. Otherwise we might as well jmp puts to tailcall it
   ret

section .rodata    ; read-only data can go here

msg:  db "I is negative", 0     ; colon after labels is always good style in NASM

I commented out the %define so I could pass it on the command line:

$ nasm -felf64 -DI=0 nasm.asm  && gcc -no-pie nasm.o && ./a.out   # no output for I=0
$ nasm -felf64 -DI=9 nasm.asm  && gcc -no-pie nasm.o && ./a.out   # or I=9
$ nasm -felf64 -DI=-9 nasm.asm  && gcc -no-pie nasm.o && ./a.out 
I is negative

$ nasm -felf64 -DI=0x80000000 nasm.asm  && gcc -no-pie nasm.o && ./a.out  # NASM doesn't truncate to 32-bit 2's complement though.
$ nasm -felf64 -DI=0x8000000000000000 nasm.asm  && gcc -no-pie nasm.o && ./a.out # apparently it's 64-bit.

I had to use -no-pie because I used call puts instead of call puts@plt or any PIE-compatible thing. For some reason the linker doesn't rewrite direct calls to use the PLT for you when making a PIE instead of a position-dependent executable.

Answer 2

It's way easier than that. Do a test rax, rax for 64-bit or a test eax, eax for 32-bit.
Then check the Sign Flag (SF) with a conditional instruction. If it's set, the number is negative.