I was just asked this question and referred to Phoronix: Ubuntu 12.04 LTS: 32-bit vs. 64-bit Performance. Phoronix is this excellent web-site run by Michael Larabel.
Below is a chart from 2012. Not all charts look this astonishing!
This was also true in 2006, see below compilation times.
So the answer generally is 64-bit. There are of course some shortcomings, like possibly slightly larger binaries, or some binaries being slower, but in general 64-bit is the way to go, regardless of the memory size in your computer. If you want to address memory above 2 GB, then of course you have no real choice.
- The number of registers is doubled. This makes it possible to store intermediate data and local variables in registers rather than in memory.
- Function parameters are transferred in registers rather than on the stack. This makes function calls more efficient.
- The allocation and deallocation of big memory blocks is more efficient.
- The SSE2 instruction set is supported on all 64-bit CPUs and operating systems.
- The 64 bit instruction set supports self-relative addressing of data. This make positions independent code more efficient.
Added 11-Mar-2016: 64-bit can be more power efficient, see phoronix: Using 64-bit Linux Software To Reduce Power Consumption?