All your nested classes are non-static. That means that they all have
a reference to the enclosing *object*. That reference is a field, and
it's not used, so it wastes space and therefore time.

Make all your nested classes static. I'm certain that will give
you a boost.

Fiddling with the code, I see that you only ever use scaling in
conjunction with addition, i.e., a+s*b. I changed addition to taking
an extra scaling argument, avoiding one intermediate vector value.

<URL:http://www.infimum.dk/privat/RayTracing.java>

Generally, I made everything static that could be made static, and
everything final, that could be made final. The latter probably
doesn't affect performance much in the long run, but it should allow
the JVM to do some optimizations sooner.

I tried not to change the actual algorithm at all.