6938

Replacing parts of the function code on-the-fly

Question:

<a href="https://stackoverflow.com/a/12227731/862380" rel="nofollow">Here</a> I came up with the solution to the other question asked by me on how to remove all costly calling to debug output function scattered over the function code (slowdown was 25 times with using empty function lambda *p: None).

The solution is to edit function code dynamically and prepend all function calls with comment sign #.

from __future__ import print_function DEBUG = False def dprint(*args,**kwargs): '''Debug print''' print(*args,**kwargs) def debug(on=False,string='dprint'): '''Decorator to comment all the lines of the function code starting with string''' def helper(f): if not on: import inspect source = inspect.getsource(f) source = source.replace(string, '#'+string) #Beware! Swithces off the whole line after dprint statement with open('temp_f.py','w') as file: file.write(source) from temp_f import f as f_new return f_new else: return f #return f intact return helper def f(): dprint('f() started') print('Important output') dprint('f() ended') f = debug(DEBUG,'dprint')(f) #If decorator @debug(True) is used above f(), inspect.getsource somehow includes @debug(True) inside the code. f()

The problems I see now are these:

<ul><li># commets all line to the end; but there may be other statements separated by ;. This may be addressed by deleting all pprint calls in f, not commenting, still it may be not that trivial, as there may be nested parantheses.</li> <li>temp_f.py is created, and then new f code is loaded from it. There should be a better way to do this without writing to hard drive. I found <a href="http://code.activestate.com/recipes/550804-create-a-restricted-python-function-from-a-string/" rel="nofollow">this recipe</a>, but haven't managed to make it work.</li> <li>if decorator is applied with special syntax used @debug, then inspect.getsource includes the line with decorator to the function code. This line can be manually removed from string, but it may lead to bugs if there are more than one decorator applied to f. I solved it with resorting to old-style decorator application f=decorator(f).</li> </ul>

What other problems do you see here?

How can all these problems be solved?

What are upsides and downsides of this approach?

What can be improved here?

Is there any better way to do what I try to achieve with this code?

<hr />

I think it's a very interesting and contentious technique to preprocess function code before compilation to byte-code. Strange though that nobody got interested in it. I think the code I gave may have a lot of shaky points.

Answer1:

A decorator can return either a wrapper, or the decorated function unaltered. Use it to create a better debugger:

from functools import wraps def debug(enabled=False): if not enabled: return lambda x: x # Noop, returns decorated function unaltered def debug_decorator(f): @wraps(f) def print_start(*args, **kw): print('{0}() started'.format(f.__name__)) try: return f(*args, **kw) finally: print('{0}() completed'.format(f.__name__)) return print_start return debug_decorator

The debug function is a decorator <em>factory</em>, when called it produces a decorator function. If debugging is disabled, it simply returns a lambda that returns it argument unchanged, a no-op decorator. When debugging is enabled, it returns a debugging decorator that prints when a decorated function has started and prints again when it returns.

The returned decorator is then applied to the decorated function.

Usage:

DEBUG = True @debug(DEBUG) def my_function_to_be_tested(): print('Hello world!')

To reiterate: when DEBUG is set to false, the my_function_to_be_tested remains unaltered, so runtime performance is not affected at all.

Answer2:

Here is the solution I came up with after composing answers from another questions asked by me here on StackOverflow.

This solution don't comment anything and just deletes standalone dprint statements. It uses <a href="http://docs.python.org/library/ast.html" rel="nofollow">ast</a> module and works with Abstract Syntax Tree, it lets us avoid parsing source code. This idea was written in the comment <a href="https://stackoverflow.com/questions/12238511/preprocessing-function-text-in-runtime-bofore-compilation#comment16402435_12238511" rel="nofollow">here</a>.

Writing to temp_f.py is replaced with execution f in necessary environment. This solution was offered <a href="https://stackoverflow.com/a/12238725/862380" rel="nofollow">here</a>.

Also, the last solution addresses the problem of decorator recursive application. It's solved by using _blocked global variable.

This code solves the problem asked to be solved in the question. But still, it's <a href="https://stackoverflow.com/questions/12238511/preprocessing-function-text-in-runtime-bofore-compilation#comment16403281_12238725" rel="nofollow">suggested not to be used in real projects</a>:

<blockquote>

You are correct, you should never resort to this, there are so many ways it can go wrong. First, Python is not a language designed for source-level transformations, and it's hard to write it a transformer such as comment_1 without gratuitously breaking valid code. Second, this hack would break in all kinds of circumstances - for example, when defining methods, when defining nested functions, when used in Cython, when inspect.getsource fails for whatever reason. Python is dynamic enough that you really don't need this kind of hack to customize its behavior.

</blockquote> from __future__ import print_function DEBUG = False def dprint(*args,**kwargs): '''Debug print''' print(*args,**kwargs) _blocked = False def nodebug(name='dprint'): '''Decorator to remove all functions with name 'name' being a separate expressions''' def helper(f): global _blocked if _blocked: return f import inspect, ast, sys source = inspect.getsource(f) a = ast.parse(source) #get ast tree of f class Transformer(ast.NodeTransformer): '''Will delete all expressions containing 'name' functions at the top level''' def visit_Expr(self, node): #visit all expressions try: if node.value.func.id == name: #if expression consists of function with name a return None #delete it except(ValueError): pass return node #return node unchanged transformer = Transformer() a_new = transformer.visit(a) f_new_compiled = compile(a_new,'<string>','exec') env = sys.modules[f.__module__].__dict__ _blocked = True try: exec(f_new_compiled,env) finally: _blocked = False return env[f.__name__] return helper @nodebug('dprint') def f(): dprint('f() started') print('Important output') dprint('f() ended') print('Important output2') f() <h2>Other relevant links:</h2> <ul><li><a href="https://stackoverflow.com/questions/12223204/switching-off-debug-prints" rel="nofollow">Switching off debug prints</a></li> </ul>

Recommend

  • H2 tag auto ID in php string
  • Sorting a vector multiple times
  • Mybatis Annotations in Complex Applications
  • How to build several targets by invoking only one command?
  • Pickle both class variables and instance variables?
  • Regex takes a long time to complete
  • How to read a certificate from a USB token in C#
  • Client-Side: Accessing Windows Azure Drive?
  • How can I sum two different columns at once where one contains Decimal objects in pandas?
  • Capturing HTML Text Input Key press after key has been applied?
  • How to get ID of changed file on Google Drive
  • How to have a website splash screen (web app)
  • Multiple flexboxes with margin-right, except the last one in the row? Without JS?
  • Creating a C++ function that calls other Lua function
  • How gzip file gets stored in HDFS
  • Wrapping a c#/WPF GUI around c++/cli around native c++
  • Why isn't my “Fizz Buzz” test in R working?
  • ZipList with Scalaz
  • Alamofire and Reachability.swift not working on xCode8-beta5
  • Validate jQuery plugin, field not required
  • How do I display a dialog that asks the user multi-choice questıon using tkInter?
  • Question about instantiating object
  • how does System.Web.HttpRequest::PathInfo work?
  • How can I enlarge video fullscreen without the affected interface project in as3?
  • x64 applications using gdi+: what are the consequences on performance?
  • Test if a set exists before trying to drop it
  • Email verification using google app script and google forms
  • How do I alternate colors in Flat List (React Native)
  • Listbox within Listbox and scrolling trouble in Windows Phone 7 Silverlight
  • Chrome doesn't support silverlight anymore? How to solve this?
  • Projection media query: browser support and workarounds?
  • Knitr HTML Loop - Some HTML output, some R output
  • jquery mobile loadPage not working
  • R: gsub and capture
  • jqPlot EnhancedLegendRenderer plugin does not toggle series for Pie charts
  • Unanticipated behavior
  • Comma separated Values
  • File upload with ng-file-upload throwing error
  • Benchmarking RAM performance - UWP and C#
  • How to load view controller without button in storyboard?