Hey guys

If you've been reading other threads i replied to, i probably
mentioned I've been working on my own language. And i am not sure
whats the best way to compare my language as in benchmarking it
against other languages.

I mean what kind of algorithm's should one implement to test a
language and its features against others. To test for speed,
efficiency and memory usage throughout the runtime; I do a Mathematics
degree, and so i started working on doing some numerical analysis
algorithms like Rhomberg Integration, Taylor Series and Lagrange
Interpolation, because its' quite number crunching so hopefully that
is a decent measure in speed possibly.

But its not really enough to test features I don't think. I know its
probably not a great way of testing languages by just implementing
algorithms and comparing them to similarly implemented algorithms in
other languages because there can be very different styles of
implementation in different languages which have side effects on how
things are evaluated, but it would be nice to have a general feel to
see how its coming along.

Also I May Be Doing A Release Of My Language Soon, Though I Doubt
There Will Be Much Documentation Of The Language Itself For Sometime
As The Implementation Wont Be Language Complete But I Am Aiming To
Have All The Primitive Features Working Pretty Well, I Am Working
Re-Implementing How Conditionals Are Evaluated And Loops Are Executed
So I Want To Finish That Before I Release. And When I Do A Release Is
It Ok To Send A Small Announce Mail To Here?

Thanks guys!

- --Phil
http://redbrain.co.uk
[It's hard to imagine a meaningful way to benchmark one language
against another, since the criteria are so subjective.  You can show
that one program runs faster than another, but that's comparing
implementations, not languages.  But yes, you can send an announcement
here to tell people where to find it. -John]

Philip Herron schrieb:

IMO algorithms are so language independent, that they may boil down to
the very same machine code, regardless of the used language and compiler.

DoDi

Comparing between programming languages is like comparing between
natural human languages: it is a complex mixture of culture, history,
economical power, trading protocols, and so on.

If you are forced to provide a quantitative comparison between two
languages, you will highly be in troubles !
- measure expressiveness of a language: depends on the application domain
- measure the performance of the code: depends on the compiler, on the
machine, on the code (implementation) and on the experimental protocol
- measure the popularity: depends on the users (on their educational level)
- measure the profitability: how much do we gain if we go from my actual
language to your language - how much we loose ?
- measure the simplicity: what are the tools that are developed for the
language, what is the education level needed for using your language, etc.
- portability : what can I do with my all previous codes ?

And so on.

Probably the best way that's still doable for a single person is to
implement the shootout benchmarks <http://shootout.alioth.debian.org/>
in your language and compare them to the solutions in other languages
in the various metrics.

- anton
--
M. Anton Ertl
an...@mips.complang.tuwien.ac.at
http://www.complang.tuwien.ac.at/anton/

On Mon, Oct 19, 2009 at 12:12 AM, Hans-Peter Diettrich

This is only true if you take a very narrow view of language
implementations. Of the "most popular" languages today, lets suppose
C, C++, Java, PHP, C#, VB, and Javascript, I would say that there
exist only C and C++ implementations which could be expected to
execute the very same machine code at run-time, even for simple
algorithms. It would be tough to expect different interpreters, JITs,
managed languages and virtual machines to execute a program similarly.

Paul

--
Paul Biggar
paul.big...@gmail.com

On Oct 19, 5:56 am, an...@mips.complang.tuwien.ac.at (Anton Ertl)
wrote:

And if that's what you choose to do -

http://shootout.alioth.debian.org/u32q/faq.php#measurementscripts

Isn't that the point of big-O/Landau notation in computational
complexity theory - to abstract away the machine/environment?

I know this only provides a worst/average case of usage of computational
resources, but anything more fine-grained for strict comparison purposes
would most probably be skewed by other factors?

Tom
[At the level of C and C++, if the various languages don't produce
programs that are all O(the same) something is seriously peculiar.  I
could imagine, e.g., perl and awk might be different if you use their
built in hhash tables and your data hashes a lot better in one than in
the other, but if all your loops and such are explicit, how could the
performance be different by more than a constant factor? -John]

Hey guys,

Thanks for all the replies everyone! Comp.Compilers has been really
useful for me last few months to understand the conventions people have
in talking about this stuff instead of me being in my own world ;).

Thanks for the link (looks like the ideas i am hitting on) i remember
seeing some presentation maybe something related to ParrotVM about
some Debian people benchmarking languages and their virtual machines
or interpreters but not quite sure it was some time ago, just reminded
me since that was a Debian sub-domain it must be related i am
guessing.

I see everyone picked me up on that this is such a subjective topic,
in that there isn't really a perfect way to do it since every language
is different but i already said that in my mail. But i don't think its
totally impossible for example say you were to write some program X in
C++ and do the same algorithm in Java i am fairly certain the
implementations could nearly be word for word if you write it in a
certain way. As in avoid pointer syntax so Java can be happy :) but
then it isn't a fair representation of C. Some people are getting
confused on what i am benchmarking or comparing as in worrying that
the execution of the syntax will be different in the back-ends or the
runtime or virtual machine. This is of course going to be different
were you go, but thats the interesting part if you have to
implementations of the same idea there are definitely going to be
different areas the developers focused on so you can see areas to
improve in either implementation. So in my case i have my interpreter
and i comparing usually against Java and python I am sure both of
those languages and their implementations will have every area working
better than mine ;) but at least maybe with the behavior of the
executions i will see areas where i can improve. By area i mean as in
garbage collection throughout the runtime of the program so it should
be more memory conservation i guess or maybe it being nice to the cpu
and not eating language unseen cpu as in the implementation isn't
doing some random stuff in between the execution of each language
expression, that kind of stuff.

Something i am starting to play with is, a runner script to execute my
interpreter on my program, measure the runtime length, the memory
usage at specified intervals throughout, as well as cpu time etc and
trying to learn how to use gnuplot to graph it. Its seems like the
only way to do and then try and implement the program as similar as i
can in language x. Its not that i want to publish some kind of very
formal benchmarks but its give me quite good insight in the past to
see where applications can improve.

Thanks again everyone. I think someone said something about 'Big O
notation' I am not sure if that really helps me its more useful at a
very high abstract language and algorithm design level, rather than
the actual execution on a computer. Although it would make a nice
solution for that side of things to measure the growth rates of
functions and execution i guess well thats what i call it anyways not
sure growth is the right word. Feel free to fix me on that ;).

--Phil

If the programs allocate and release memory frequently, the
implementation of these operations can dominate the time.  And (in spite
of many people's naive expectations), neither allocation of nor freeing
memory are necessarily constant-time operations.  Different
implementations of malloc() and free() may have different tradeoffs
(without either being a "bad" implementation), so a particular program
may well have a different big-O complexity with different
implementations of malloc() and free().

You could say that this is not a question of language but of library
implementation, but the distinction between what is a language feature
and what is a library feature is strained at best if all implementations
of a language are required to implement certain standard functions.

        Torben
[Good point, libraries can make a huge difference in performance. -John]

On Tue, 20 Oct 2009 11:30:08 +0100, Paul Biggar

Indeed.  The same algorithm will look very different compiled in C
than when compiled in a language that has, e.g., arbitrary precision
arithmetic, string ropes, immutable data, etc.

Heck, in most languages the algorithm will compile very differently
when implemented (equivalently) using recursion vs iteration.  I've
only ever seen a Lisp compiler reduce an iterative loop and a self
recursion to the same (iterative) machine code - and then only for
very simple loops.

George

Some folk take a very liberal view of what "Optimizing" means, and
do things like memoizing the results of functions which can be proven
side-effect free or lifting iterative additions or multiplications
out of a loop and replacing them with partial evaluation forms,
sigma forms, equation products, multiplications or exponentiations.

In cases where memoization cuts duplicated branching recursive
calls, or the last operation gets lifted out of a loop and the
loop itself disappears (both of which can happen to, eg,
naive implementations of exponentiation), this changes not just
the constant factors, but also the order of the computation.

I know of at least a couple of compilers where you can write
programs that look like they ought to take exponential time and
discover that they run in constant time or linear time due to
such tricks.

                       Bear

I like to benchmark using low-level features, eg. integer operations, or
sometimes a bit of string processing.

This gives an quick idea of how quick a language (or implementation) really
is. This might be important if someone wants to add functionality which
isn't built-in and they don't particularly want to write it as a separate C
module.

For example I came across this benchmark which at first made me wonder why
they bothered as there seems hardly anything here to get your teeth into. As
Python code this is:

i=0
while i<=100000000:
     i=i+1
print (i)

Both Ruby and Python seemed to have considerable trouble with running this
at any sort of speed. (I just tried this and Python took some 17000 to 25000
msec (2.6/3.1), compared with 70 to 450ms in C (optimised/unoptimised) (and
700ms in my latest interpreted language)).

Higher-level features are more difficult to compare, as one language may
implement them more efficiently compared to another, which may not even have
the feature. But this does allow inefficient languages to still be of some
practical use...

OK, so we benchmark implementations instead! Since that is what will be run.

But the language can still influence the speed by making the implementation
akward due to it's requirements.

For example if all basic objects must be strings, then that's going to
influence the maximum speed of arithmetic operations, even with a clever
implementation that minimises string/number conversions.

--
Bartc