Lua is global by default. Declare all the variables as local and you'll see significant improvement. Also, there is a boolean type so you can use true and false directly instead of comparing numbers.

I tried using locals, and there was no change to the time. Using a boolean return value for isPrime shaved off two seconds.

if you run it with luajit -jv primes.lua you'll see NYIs about math.mod not implemented.

Replacing math.mod(n, i) with (n % i) gives roughly 9.4x performance.

EDIT: luajit version was LuaJIT 2.0.4 on Mac OSX

Thanks, using % did the trick.