computer science, math, programming and other stuff
a blog by Christopher Swenson

Assembly language programming under OS X with NASM

One of my favorite passions from my teenage days was assembly language programming. Don't laugh.

It embodies a lot of my favorite things about programming: I have total control, it is clean and simple, and it is just and fast and functional as I am capable of making it. The only thing standing in the way of me and world domination is how well I can program.

Well, with some minor exceptions. Having a decent assembler is really key, and for x86 architectures, there are many choices. If you are hacking a small function together to support some higher-level language, then maybe you can get by using MASM (for Windows/DOS), or, if you really hate yourself, gas (for every platform).

But, if you intend to spend some time programming in assembly and not hating every minute of it, then you need to use NASM, probably the best assembler for the x86 family, ever (or possibly Yasm, a NASM clone that I have no experience with).

So, this past week I was interested in playing some more with NASM, and so I thought that I would see how what I could do under OS X (I previously worked primarily in DOS). Unfortunately, assembly language support in OS X is fairly hampered if you want follow its standard calling conventions for 32-bit x86 code. The innocuous-looking statement "The stack is 16-byte aligned at the point of function calls" seems innocent, but is a nightmare if you using external calls.

Basically, this means that you have to keep very close track of your stack size when calling functions. And even worse is that your stack never enters your function correctly aligned: the return address is always 4 bytes long, meaning you are always 12 bytes off when you start.

What this means is that I may have to use a VM with Linux to have any fun with assembly language programming again.

If anyone is interested, here is a quick NASM file I threw together that demonstrates how to use NASM on OS X to call glibc functions. I tested it with the latest NASM (2.07) on Snow Leopard (you'll probably need XCode installed to get this to work). This program prints "Hello World", allocates some memory using malloc, uses that memory to write 10 letters of the alphabet on the screen (using printf), frees the memory, and returns.

;
; Basic OS X calls to glibc
;
; compile with:
; nasm -g -f macho malloc.asm
; gcc -o a.out malloc.o
;

; glibc stuff
extern _puts, _printf, _malloc, _free

; static data
segment .data

hello_world_str db "Hello world!", 10, 0
int_str db "Address %x", 10, 0

; code
segment .text

global _main

_main:
push ebp ; setup the frame
mov  ebp, esp

sub  esp, 4 ; align the stack
push dword hello_world_str
call _puts

; malloc 16 bytes
push  dword 16
call  _malloc

; check if the malloc failed
test  eax, eax
jz    fail_exit

sub   esp, 0xC ; align the stack
mov   ebx, eax
push  ebx
push  dword int_str
call  _printf

; print "A\nB\n..."
mov   [ebx], dword 0xD41 ; 'A\n'

mov   edi, 10
push  ebx
_loop:
call  _puts
inc  dword [ebx]
dec  edi
jnz  _loop

; free the malloc'd memory
push  ebx
call  _free
add  esp, 4 ; cleanup the stack
pop  ebp
ret

fail_exit:
mov  eax, 1
pop  ebp
ret


The output should look something like this:

Hello world!