Exclude coments when comparing size in Benchmarker.go

[ReneSac]

Source code comments shouldn't be counted on the program size. Of course, the benchmark would need to parse each file knowing the language's comment syntax to remove them. But it is better than to force people to remove the comments of the source files themselves.

Other indicators like LOC or number of characters would be interesting, but maybe overwhelming. I'm just afraid that things like "public static final int, public static final int, public static final int" will compress too well. But of course, if I were to chose only one metric, the compressed size is better.

Not directly related, but it would also be interesting to know the executable file size, with notes if it requires external VMs and such or is standalone.

[logicchains]

I've tried to avoid this issue by not commenting any of the implementations apart from the Go one, and then when I do the final measurement I'll do it on a version of Go.go with the comments stripped out. There don't appear to be too many comments on the Nimrod one so I could just strip those out manually when measuring.

Lines of Code is something I could easily add. I'd prefer SLOC, like Github shows, but I'm not sure exactly how that's calculated. Number of characters should simply be a matter of reading the file in, converting it to a string and counting its length.

I'm already planning to add executable file size measurements, but I don't think it's necessary to mention whether a VM is required or not, as the only compiled languages used that run on VMs are Java and C#, and I think most people are aware that they run on virtual machines.

[ReneSac]

For nimrod, a simple \s*\#.* regular expression can find all the comments to be deleted w/o collateral damage. Commenting other implementations besides Go make it easier to read them. But now, as you said, every other implementation is w/o them, and I can easily submit a new version w/o them to avoid any mistakes (already here in my HD).

And about whether a language has a VM or not, I was more concerned about the unknown languages, that people may not be aware that they do not run on VMs. But a quick mention on the text like this is probably sufficient. Oh, and add python to VM list too (if you update the python version to be benchmarked with the others).

[logicchains]

You're right that I could remove comments with regex, the only issue is that would require me to parse each file first and then create a stripped temporary to compile, rather than compiling the original. I'm planning on doing that however for another tiny benchmark I'm working on in the meantime, so I suppose I'll bring that over to this Benchmarker.go.

I'm just worried about keeping the results table tidy; it's already crowded as it is. I think I'll highlight the ones running on a VM in a different color and note it in a legend; that should be sufficient, and saves space. Probably not much of a chance of me updating the Python version however, as the problem seems fundamentally beyond my ability to fix (genBuffer crashes the python interpreter, yet immediate mode drawing works fine).

[ReneSac]

Indeed, too much information is also bad. If you don't want that work removing comments now, I will remove them for the nimrod version. I already have a pull request ready with other changes, and it is a bit unlikely that other languages will add them (and some things like /* */ comments aren't easy to remove with only regex). It is your call.

[logicchains]

How about just having a separate ..comments file for every language, that's just the same file with comments? It'd clutter the repo a bit, but I imagine most people would either only be looking to run a single language or they'd use Benchmarker.go, so it shouldn't matter too much, and I imagine most implementations will remain uncommented.

[ReneSac]

A new folder would help with the cluttering. But it will be extra work to keep the two versions syncronized (or four versions, where there are already two for the language...). I will just strip the comments from the nimrod version... you may revert that if you will.

[logicchains]

Okay then. I'll make a note when I post it that if anyone wants a commented version other than Go then they can let me know and I'll make one available.

[logicchains]

If you're interested, while I'm waiting for the Rust glfw library to again
reach a stage where I can get it to compile I've been working on an
extremely simple benchmark inspired by this
onehttp://mechanical-sympathy.blogspot.com.au/2012/10/compact-off-heap-structurestuples-in.html,
which can be found at https://github.com/logicchains/ArrayAccessBench. I
wasn't planning on including Nimrod, C++, Rust or D as it's trivial for
them to reach C speed for this simple task (it's more focused on testing
languages without stack allocation or pointers), but there's no harm in
submitting one if you want. It should only take around half an hour to
write.

On 16 February 2014 11:16, ReneSac notifications@github.com wrote:

A new folder would help with the cluttering. But it will be extra work to
keep the two versions syncronized (or four versions, where there are
already two for the language...). I will just strip the comments from the
nimrod version... you may revert that if you will.

Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-35176111
.