If you've been reading other threads i replied to, i probably mentioned I've been working on my own language. And i am not sure whats the best way to compare my language as in benchmarking it against other languages.
I mean what kind of algorithm's should one implement to test a language and its features against others. To test for speed, efficiency and memory usage throughout the runtime; I do a Mathematics degree, and so i started working on doing some numerical analysis algorithms like Rhomberg Integration, Taylor Series and Lagrange Interpolation, because its' quite number crunching so hopefully that is a decent measure in speed possibly.
But its not really enough to test features I don't think. I know its probably not a great way of testing languages by just implementing algorithms and comparing them to similarly implemented algorithms in other languages because there can be very different styles of implementation in different languages which have side effects on how things are evaluated, but it would be nice to have a general feel to see how its coming along.
Also I May Be Doing A Release Of My Language Soon, Though I Doubt There Will Be Much Documentation Of The Language Itself For Sometime As The Implementation Wont Be Language Complete But I Am Aiming To Have All The Primitive Features Working Pretty Well, I Am Working Re-Implementing How Conditionals Are Evaluated And Loops Are Executed So I Want To Finish That Before I Release. And When I Do A Release Is It Ok To Send A Small Announce Mail To Here?
Thanks guys!
- --Phil http://redbrain.co.uk [It's hard to imagine a meaningful way to benchmark one language against another, since the criteria are so subjective. You can show that one program runs faster than another, but that's comparing implementations, not languages. But yes, you can send an announcement here to tell people where to find it. -John]
Comparing between programming languages is like comparing between natural human languages: it is a complex mixture of culture, history, economical power, trading protocols, and so on.
If you are forced to provide a quantitative comparison between two languages, you will highly be in troubles ! - measure expressiveness of a language: depends on the application domain - measure the performance of the code: depends on the compiler, on the machine, on the code (implementation) and on the experimental protocol - measure the popularity: depends on the users (on their educational level) - measure the profitability: how much do we gain if we go from my actual language to your language - how much we loose ? - measure the simplicity: what are the tools that are developed for the language, what is the education level needed for using your language, etc. - portability : what can I do with my all previous codes ?
Philip Herron <herron.phi...@googlemail.com> writes: >And i am not sure >whats the best way to compare my language as in benchmarking it >against other languages.
Probably the best way that's still doable for a single person is to implement the shootout benchmarks <http://shootout.alioth.debian.org/> in your language and compare them to the solutions in other languages in the various metrics.
On Mon, Oct 19, 2009 at 12:12 AM, Hans-Peter Diettrich
<DrDiettri...@aol.com> wrote: > IMO algorithms are so language independent, that they may boil down to > the very same machine code, regardless of the used language and compiler.
This is only true if you take a very narrow view of language implementations. Of the "most popular" languages today, lets suppose C, C++, Java, PHP, C#, VB, and Javascript, I would say that there exist only C and C++ implementations which could be expected to execute the very same machine code at run-time, even for simple algorithms. It would be tough to expect different interpreters, JITs, managed languages and virtual machines to execute a program similarly.
On Oct 19, 5:56 am, an...@mips.complang.tuwien.ac.at (Anton Ertl) wrote:
> Philip Herron <herron.phi...@googlemail.com> writes: > >And i am not sure > >whats the best way to compare my language as in benchmarking it > >against other languages.
> Probably the best way that's still doable for a single person is to > implement theshootoutbenchmarks <http://shootout.alioth.debian.org/> > in your language and compare them to the solutions in other languages > in the various metrics.
DoDi wrote: > Philip Herron schrieb: >> I mean what kind of algorithm's should one implement to test a >> language and its features against others.
> IMO algorithms are so language independent, that they may boil down to > the very same machine code, regardless of the used language and compiler.
Isn't that the point of big-O/Landau notation in computational complexity theory - to abstract away the machine/environment?
I know this only provides a worst/average case of usage of computational resources, but anything more fine-grained for strict comparison purposes would most probably be skewed by other factors?
Tom [At the level of C and C++, if the various languages don't produce programs that are all O(the same) something is seriously peculiar. I could imagine, e.g., perl and awk might be different if you use their built in hhash tables and your data hashes a lot better in one than in the other, but if all your loops and such are explicit, how could the performance be different by more than a constant factor? -John]
Thanks for all the replies everyone! Comp.Compilers has been really useful for me last few months to understand the conventions people have in talking about this stuff instead of me being in my own world ;).
> Probably the best way that's still doable for a single person is to > implement the shootout benchmarks <http://shootout.alioth.debian.org/> > in your language and compare them to the solutions in other languages > in the various metrics.
Thanks for the link (looks like the ideas i am hitting on) i remember seeing some presentation maybe something related to ParrotVM about some Debian people benchmarking languages and their virtual machines or interpreters but not quite sure it was some time ago, just reminded me since that was a Debian sub-domain it must be related i am guessing.
I see everyone picked me up on that this is such a subjective topic, in that there isn't really a perfect way to do it since every language is different but i already said that in my mail. But i don't think its totally impossible for example say you were to write some program X in C++ and do the same algorithm in Java i am fairly certain the implementations could nearly be word for word if you write it in a certain way. As in avoid pointer syntax so Java can be happy :) but then it isn't a fair representation of C. Some people are getting confused on what i am benchmarking or comparing as in worrying that the execution of the syntax will be different in the back-ends or the runtime or virtual machine. This is of course going to be different were you go, but thats the interesting part if you have to implementations of the same idea there are definitely going to be different areas the developers focused on so you can see areas to improve in either implementation. So in my case i have my interpreter and i comparing usually against Java and python I am sure both of those languages and their implementations will have every area working better than mine ;) but at least maybe with the behavior of the executions i will see areas where i can improve. By area i mean as in garbage collection throughout the runtime of the program so it should be more memory conservation i guess or maybe it being nice to the cpu and not eating language unseen cpu as in the implementation isn't doing some random stuff in between the execution of each language expression, that kind of stuff.
Something i am starting to play with is, a runner script to execute my interpreter on my program, measure the runtime length, the memory usage at specified intervals throughout, as well as cpu time etc and trying to learn how to use gnuplot to graph it. Its seems like the only way to do and then try and implement the program as similar as i can in language x. Its not that i want to publish some kind of very formal benchmarks but its give me quite good insight in the past to see where applications can improve.
Thanks again everyone. I think someone said something about 'Big O notation' I am not sure if that really helps me its more useful at a very high abstract language and algorithm design level, rather than the actual execution on a computer. Although it would make a nice solution for that side of things to measure the growth rates of functions and execution i guess well thats what i call it anyways not sure growth is the right word. Feel free to fix me on that ;).
Tom Crick <t...@cs.bath.ac.uk> writes: > [At the level of C and C++, if the various languages don't produce > programs that are all O(the same) something is seriously peculiar. I > could imagine, e.g., perl and awk might be different if you use their > built in hhash tables and your data hashes a lot better in one than in > the other, but if all your loops and such are explicit, how could the > performance be different by more than a constant factor? -John]
If the programs allocate and release memory frequently, the implementation of these operations can dominate the time. And (in spite of many people's naive expectations), neither allocation of nor freeing memory are necessarily constant-time operations. Different implementations of malloc() and free() may have different tradeoffs (without either being a "bad" implementation), so a particular program may well have a different big-O complexity with different implementations of malloc() and free().
You could say that this is not a question of language but of library implementation, but the distinction between what is a language feature and what is a library feature is strained at best if all implementations of a language are required to implement certain standard functions.
Torben [Good point, libraries can make a huge difference in performance. -John]
<paul.big...@gmail.com> wrote: >On Mon, Oct 19, 2009 at 12:12 AM, Hans-Peter Diettrich ><DrDiettri...@aol.com> wrote: >> IMO algorithms are so language independent, that they may boil down to >> the very same machine code, regardless of the used language and compiler.
>This is only true if you take a very narrow view of language >implementations. Of the "most popular" languages today, lets suppose >C, C++, Java, PHP, C#, VB, and Javascript, ...
Indeed. The same algorithm will look very different compiled in C than when compiled in a language that has, e.g., arbitrary precision arithmetic, string ropes, immutable data, etc.
Heck, in most languages the algorithm will compile very differently when implemented (equivalently) using recursion vs iteration. I've only ever seen a Lisp compiler reduce an iterative loop and a self recursion to the same (iterative) machine code - and then only for very simple loops.
> [At the level of C and C++, if the various languages don't produce > programs that are all O(the same) something is seriously peculiar. I > could imagine, e.g., perl and awk might be different if you use their > built in hhash tables and your data hashes a lot better in one than in > the other, but if all your loops and such are explicit, how could the > performance be different by more than a constant factor? -John]
Some folk take a very liberal view of what "Optimizing" means, and do things like memoizing the results of functions which can be proven side-effect free or lifting iterative additions or multiplications out of a loop and replacing them with partial evaluation forms, sigma forms, equation products, multiplications or exponentiations.
In cases where memoization cuts duplicated branching recursive calls, or the last operation gets lifted out of a loop and the loop itself disappears (both of which can happen to, eg, naive implementations of exponentiation), this changes not just the constant factors, but also the order of the computation.
I know of at least a couple of compilers where you can write programs that look like they ought to take exponential time and discover that they run in constant time or linear time due to such tricks.
"Philip Herron" <herron.phi...@googlemail.com> wrote in message > If you've been reading other threads i replied to, i probably > mentioned I've been working on my own language. And i am not sure > whats the best way to compare my language as in benchmarking it > against other languages.
> I mean what kind of algorithm's should one implement to test a > language and its features against others. To test for speed, > efficiency and memory usage throughout the runtime; I do a Mathematics > degree, and so i started working on doing some numerical analysis > algorithms like Rhomberg Integration, Taylor Series and Lagrange > Interpolation, because its' quite number crunching so hopefully that > is a decent measure in speed possibly.
I like to benchmark using low-level features, eg. integer operations, or sometimes a bit of string processing.
This gives an quick idea of how quick a language (or implementation) really is. This might be important if someone wants to add functionality which isn't built-in and they don't particularly want to write it as a separate C module.
For example I came across this benchmark which at first made me wonder why they bothered as there seems hardly anything here to get your teeth into. As Python code this is:
i=0 while i<=100000000: i=i+1 print (i)
Both Ruby and Python seemed to have considerable trouble with running this at any sort of speed. (I just tried this and Python took some 17000 to 25000 msec (2.6/3.1), compared with 70 to 450ms in C (optimised/unoptimised) (and 700ms in my latest interpreted language)).
Higher-level features are more difficult to compare, as one language may implement them more efficiently compared to another, which may not even have the feature. But this does allow inefficient languages to still be of some practical use...
> [It's hard to imagine a meaningful way to benchmark one language > against another, since the criteria are so subjective. You can show > that one program runs faster than another, but that's comparing > implementations, not languages. ... -John]
OK, so we benchmark implementations instead! Since that is what will be run.
But the language can still influence the speed by making the implementation akward due to it's requirements.
For example if all basic objects must be strings, then that's going to influence the maximum speed of arithmetic operations, even with a clever implementation that minimises string/number conversions.
|
