; cpu time (non-gc) 60,202 msec (00:01:00.202) user, 421 msec system
; cpu time (gc)     33,623 msec user, 10 msec system
; cpu time (total)  93,825 msec (00:01:33.825) user, 431 msec system
; real time  105,632 msec (00:01:45.632)
; space allocation:
;  967,240 cons cells, 3,719,472,272 other bytes, 4,472 static bytes

But converting all "single-float"  to double-float:

; cpu time (non-gc) 4,446 msec user, 120 msec system
; cpu time (gc)     2,524 msec user, 0 msec system
; cpu time (total)  6,970 msec user, 120 msec system
; real time  7,961 msec
; space allocation:
;  418,422 cons cells, 292,782,544 other bytes, 0 static bytes
The compiler complained about the "shadow" statement
warning: compile-file found "SHADOW" at the top-level --  see the
          documentation for
          comp:*cltl1-compile-file-toplevel-compatibility-p*

My guess is that much of the verbosity for the sbcl version could be struck
out of the allegro cl version without any loss in speed, and that careful attention
to other potential optimizations/ declarations could squeeze out better performance.

It could be that this single/double issue relates to your AMD64 timings.
Most of it may be converting single to double and back.