Profiling Python C extensions

During my collaboration with Upinder S. Bhalla's lab, I made possible to profile MOOSE C++ code through Python scripting, which means one can profile the native functions of MOOSE without writing any C++ code.

For profiling I used gperftools: I wrapped three of its functions, particularly ProfilerStart, ProfilerStop and ProfilerFlush in python functions using Cython. The recompilation of MOOSE is also needed with the -lprofiler flag. After that, one can just call the wrapper functions for ProfilerStart and ProfilerStop before and after the python code that calls the C extensions one is interested in to profile. Then pprof can be used to investigate the results of profiling.

Howto

To profile Python C extensions, first Cython, gperftools and libc6-prof packages need to be installed. If you'd like a visual representation of the results of your profiling, better install kcachegrind as well.

The simplest way to get the wrapper done is to write a cython script wrapping the gperftools functions and a python script that compiles the wrapped functions and link them to the gperftools library.

Let's call the Cython script gperftools_wrapped.pyx:

1:  cdef extern from "gperftools/profiler.h":  
2:      int ProfilerStart(char* fname)  
3:      void ProfilerStop()  
4:      void ProfilerFlush()  
5:    
6:  def ProfStart(fname):  
7:      return ProfilerStart(fname)  
8:    
9:  def ProfStop():  
10:      ProfilerStop()  
11:    
12:  def ProfFlush():  
13:      ProfilerFlush()  
14:    

Here we define a python function for each function of gperftools that we wrap. More functions can be wrapped for more custom profiling (see ProfilerStartWithOptions()).

The python compiler script may look something like this (setup.py):

1:  from distutils.core import setup  
2:  from Cython.Build import cythonize  
3:    
4:  setup(  
5:      name = 'gperftools_wrapped',  
6:      ext_modules = cythonize("gperftools_wrapped.pyx"),  
7:  )  

Now setup.py may be run with the following manner, adding the -lprofiler flag:

~$ python setup.py build_ext --inplace -lprofiler

If everything went right, now you should have gperftools_wrapped.c, gperftools_wrapped.so, and a build directory as result of the compilation.

Put gperftools_wrapped.so nearby your python testbed and import as gperftools_wrapped, so you can profile python C extensions. Here's an example:

1:  ProfStart('output.prof')  
2:  # ... Python code that calls C extension you intend to profile ...  
3:  ProfFlush()  
4:  ProfStop()  

You can check the results by:

~$ pprof --text /path_to_C_extension_so/C_extension.so output.prof > log
~$ less log

If you have kcachegrind installed, then by:

~$ pprof --callgrind /path_to_C_extension_so/C_extension.so output.prof > output.callgrind
~$ kcachegrind output.callgrind

Happy optimizing!