Grand Unified Python Benchmarks evaluation

[catalin-manciu]

Hello,

This is Catalin from the Server Scripting Languages Optimization Team at Intel Corporation. I tried doing an initial evaluation of FatPython's current performance on the Grand Unified Python Benchmarks (located at https://hg.python.org/benchmarks/). I encountered the following issues:

• I had to patch fatoptimizer because it failed when running any benchmark (I'm attaching the patch, the error was that callsite doesn't have a member called starargs or kwargs in some cases : fatoptimizer_gupb_compat.patch.zip)
• I had to patch perf.py from the benchmarks repository because not all benchmarks will work (currently 37/43, patch is also attached :
  gupb_fatpython_compat.patch.zip).
• Benchmarks not working:
  • 2to3
  • chameleon_v2
  • tornado_http
  • django_v3
  • normal_startup
  • startup_nosite

Revisions

• GUPB: 9923b81a1d34 + patch
• Baseline CPython 3: 0f46c9a5735f

Hardware and OS Configuration

• Platform: Intel XEON (Haswell-EP) 18 Cores
• BIOS settings:
  • Intel Turbo Boost Technology: false
  • Hyper-Threading: false
• OS: CentOS 7.1.1503
• OS configuration:
  • Address Space Layout Randomization (ASLR) disabled to reduce run to run variation by echo 0 > /proc/sys/kernel/randomize_va_space
  • CPU frequency set fixed at 2.3GHz
• GCC version: 4.8.3

Running GUBP

The command used for running GUBP was:
python perf.py /path/to/baseline/python /path/to/fatpython/fatpython -b all -r --csv outfile.csv --affinity 6
fatpython is a shell script that contains the following two lines:
#!/bin/bash
/absolute/path/to/fatpython/python -X fat "$@"

Results

Benchmark	Improvement
spectral_norm	23.57%
nqueens	4.57%
unpack_sequence	2.22%
nbody	2.08%
unpickle_list	2.02%
meteor_contest	1.87%
fastpickle	1.38%
regex_effbot	0.77%
chaos	0.20%
json_load	0.00%
etree_iterparse	-0.48%
hexiom2	-0.48%
pidigits	-0.78%
pickle_dict	-0.80%
etree_parse	-1.38%
mako_v2	-1.50%
fannkuch	-1.63%
fastunpickle	-1.73%
etree_generate	-2.15%
regex_v8	-2.15%
regex_compile	-2.50%
pickle_list	-2.81%
richards	-2.90%
pathlib	-3.39%
etree_process	-3.57%
raytrace	-3.77%
silent_logging	-4.13%
go	-4.22%
telco	-4.23%
call_simple	-4.49%
json_dump_v2	-4.49%
formatted_logging	-4.54%
call_method	-4.86%
call_method_slots	-5.63%
float	-5.92%
simple_logging	-6.12%
call_method_unknown	-8.34%