RedBaron
a bottom up approach to refactoring in python

Presenter Notes

Me

  • Laurent Peuch
  • Bram

Verbose version: http://worlddomination.be/about/about.html

Presenter Notes

Plan

  • Why?
  • Solution for the first problem (baron)
  • Solution for the second problem (RedBaron)
  • Conclusion

Presenter Notes

Before starting: some reminders

Presenter Notes

Refactoring (eclipse)

refactoring.png

Presenter Notes

Abstract Syntax Tree (AST)

ast.png

Presenter Notes

Why?

Presenter Notes

Custom refactoring

  • I always wanted to be able to write code to modify source code
  • Extremely hard: working on a huge string without any sens, analysing it, moving stuff around, syntax, too many possibilities
  • Frustrating, so many situation where I though "Ah! If I could do that!"
  • Always bothering me
  • Code generation too
  • Only a bunch of people in the world actually do that
  • Extremely hard: I'm at (x,y) in one file, what is the meaning of my surroundings?

Presenter Notes

Ast.py

ast.py.png

Presenter Notes

Ast.py

Not lossless !

ast_to_code(code_to_ast(code_source)) != source_code

(Comments, formatting)

(And ast_to_code doesn't exist officially).

Presenter Notes

Ast.py

API Sax like

1 class KeyAttributesFinder(ast.NodeVisitor):
2     def visit_Assign(self, assign_node):
3         # ...
4 
5     def visit_FunctionDef(self, function_node):
6         # ...
7 
8     # visit_...

Terribly boring and unfunny to use (and unusable in a shell like IPython.)

Presenter Notes

pythonfmt

Source code auto formater

Presenter Notes

Code generation

Django (opendata projects like memopol):

donnees.json -> models.py + import.py

Presenter Notes

Refactoring: top to bottom

refactoring1.png

Presenter Notes

Refactoring en python

  • BycleRepairMan
  • Rope (ast.py + regexs)
  • PyCharm

Presenter Notes

Conclusion: 2 problems

  • We miss a good abstraction
  • We miss a good API

Presenter Notes

Solution 1: abstraction -> Baron

Presenter Notes

Baron

  • lossless ast!
  • source == ast_to_code(code_to_ast(source))
  • from an analysis problem to a graph modification problem
  • output json for maximal interoperability (and data structure > objects in simplicity)

Presenter Notes

Example

 1 from baron.helpers import show
 2 
 3 print show("1 + 2")
 4 
 5 [
 6     {
 7         "first_formatting": [
 8             {
 9                 "type": "space",
10                 "value": " "
11             }
12         ],
13         "value": "+",
14         "second_formatting": [
15             {
16                 "type": "space",
17                 "value": " "
18             }
19         ],
20         "second": {
21             "section": "number",
22             "type": "int",
23             "value": "2"
24         },
25         "type": "binary_operator",
26         "first": {
27             "section": "number",
28             "type": "int",
29             "value": "1"
30         }
31     }
32 ]

Presenter Notes

refactoring2.png

Presenter Notes

State of the project

~1 year of work (needed to learn)

  • +1000 unit tests (TDD)
  • works on the top 100 of pypi
  • utilities: position_to_path, position_to_node, bounding_box, walker etc...
  • fully documented

Presenter Notes

Solution 2: API -> RedBaron

Presenter Notes

Plan

  • principle
  • exploration (query)
  • graph modification
  • lists modification

Presenter Notes

RedBaron

  • API on top of Baron
  • Like BeautifulSoup/JQuery: mapping from a datastructure to objects
  • For the human: user friendly as much as possible
  • try to do all the boring low level stuff for you
  • Designed to be used in IPython (but not only)

Presenter Notes

RedBaron

Very simple API:

1 from redbaron import RedBaron
2 
3 red = RedBaron("some source code as a string")
4 # ...
5 red.dumps()  # source code

Presenter Notes

As intuitive as possible

Overloading of __repr__:

BeautifulSoup:

soup.png

Presenter Notes

As intuitive as possible

Overloading of __repr__:

BeautifulSoup:

soup.png

RedBaron:

repr.png

Presenter Notes

Self descriptive

RedBaron:

at_0.png

Presenter Notes

Self descriptive

RedBaron:

at_0.png

".help()"

help.png

Presenter Notes

Exploration

Like BeautifulSoup:

 1 red = RedBaron("a = 42\ndef test_chocolate(): pass")
 2 red.find("name")
 3 red.find("int", value=42)
 4 red.find("def", name="g:test_*")
 5 red.find("def", name="re:test_*")
 6 red.find("assignment", lambda x: x.target.dumps() == "INSTALLED_APPS")
 7 
 8 red.find_all("name")
 9 red.find_all(("name", "int"))
10 red.find_all("def", arguments=lambda x: len(x) == 3)
11 red.find_all("def", recursive=False)

Presenter Notes

Exploration

Like BeautifulSoup:

 1 red = RedBaron("a = 42\ndef test_chocolate(): pass")
 2 red.find("name")
 3 red.find("int", value=42)
 4 red.find("def", name="g:test_*")
 5 red.find("def", name="re:test_*")
 6 red.find("assignment", lambda x: x.target.dumps() == "INSTALLED_APPS")
 7 
 8 red.find_all("name")
 9 red.find_all(("name", "int"))
10 red.find_all("def", arguments=lambda x: len(x) == 3)
11 red.find_all("def", recursive=False)

Shortcuts (like BeautifulSoup):

1 red = RedBaron("a = 42\ndef test_chocolate(): pass")
2 red.name
3 red.int
4 red.else_
5 
6 red("name")
7 red(("name", "int"))
8 red("def", arguments=lambda x: len(x) == 3)

Presenter Notes

Modification

How to modify a node?

1 from redbaron import RedBaron, BinaryOperatorNode
2 
3 red = RedBaron("a = 'plop'")
4 red[O].value  # 'plop'
5 red[0].value = BinaryOperatorNode({'first_formatting':[{'type': 'space',
6 'value': ' '}], 'value': '+', 'second_formatting': [{'type': 'space',
7 'value': ' '}], 'second': {'section': 'number', 'type': 'int', 'value':
8 '1'}, 'type': 'binary_operator', 'first': {'section': 'number', 'type':
9 'int', 'value': '1'}})

Pretty much horrible.

Presenter Notes

Magic with __setattr__

1 from redbaron import RedBaron, BinaryOperatorNode
2 
3 red = RedBaron("a = 'plop'")
4 red[0].value = "1 + 1"
5 
6 # works also with: redbaron nodes and json ast

Works for every nodes.

Presenter Notes

Advanced modifications:

Another problem: what is the body of the bar function?

1 class Foo():
2     def bar(self):
3         pass
4 
5     def baz(self):
6         pass

Presenter Notes

Advanced modifications:

Another problem: what is the body of the bar function?

1 class Foo():
2     def bar(self):
3         pass
4 
5     def baz(self):
6         pass

Expected:

expected.png

Presenter Notes

Advanced modifications:

Another problem: what is the body of the bar function?

1 class Foo():
2     def bar(self):
3         pass
4 
5     def baz(self):
6         pass

Expected:

expected.png

Reality:

reality.png

Presenter Notes

Solution: magic!

magic.gif

1 red.find("def", name="bar").value = "pass"
2 red.find("def", name="bar").value = "pass\n"
3 red.find("def", name="bar").value = "    pass\n"
4 red.find("def", name="bar").value = "    pass\n    "
5 red.find("def", name="bar").value = "        pass\n        "
6 red.find("def", name="bar").value = "\n    pass\n    "
7 # etc ..

Works for every node with a body: else, exceptions, finally, elif etc...

Presenter Notes

Lists

Problem: how many elements are in this list? ['a', 'b', 'c']

Presenter Notes

Lists

Problem: how many elements are in this list?

Expected:

list_expected.png

Presenter Notes

Lists

Problem: how many elements are in this list?

Expected:

list_expected.png

Reality:

list_reality.png

Presenter Notes

Lists: solution

Solution: "proxy lists", gives you the same API than python list and handle formatting for you.

Reality again:

list_expected.png

Works for:

  • "," (with and without indentations)
  • ".", for example: a.b.c().pouet[stuff]
  • endl separated lines (function body, "blocks")

Presenter Notes

Helpers

  • .map
  • .apply
  • .filter
  • .next, .previous, .parent
  • .replace
  • .insert_before .insert_after

Presenter Notes

Some examples

1 # rename a 'name' (warning: won't rename everything)
2 for i in red('name', value='pouet'): i.value = 'plop'

Presenter Notes

Some examples

1 # install a django app
2 red.find("assign", target=lambda x: x.dumps() == 'INSTALLED_APPS').\
3     value.append("'debug_toolbar.apps.DebugToolbarConfig'")

Presenter Notes

Some examples

1 # lines_profiler
2 red('def', recursive=False).\
3     map(lambda x: x.decorators.insert(0, '@profile'))

Presenter Notes

Some examples

1 # lines_profiler
2 red('def', recursive=False).\
3     map(lambda x: x.decorators.insert(0, '@profile'))
4 
5 # remove them
6 red("decorator", lambda x: x.dumps() == "@decorator").\
7     map(lambda x: x.parent.parent.decorators.remove(x))

Presenter Notes

Some examples

1 # print a -> logger.debug(a)
2 red('print', value=lambda x: len(x) == 1).\
3     map(lambda x: x.replace('logger.debug(%s)' % x.value.dumps())
4 
5 # print a, b, c -> logger.debug("%s %s %s" % (a, b, c))
6 red('print', value=lambda x: len(x) == 1).\
7     map(lambda x: x.replace('logger.debug("%s" % (%s))' %
8                                 (" ".join('%s' * len(x.value)))

Presenter Notes

refactoring3.png

Presenter Notes

Project state

  • +1200 tests
  • fully documented (example for everything
  • tutorial
  • reference lib to use works with baron
  • still in alpha, still some a bit annoying bugs
  • should be working for 80% of the cases
  • No static analysis (later with astroid/jedi?)

Presenter Notes

Documentation

Every example are run at documentation compile time:

documentation.png

Presenter Notes

Conclusion

Presenter Notes

kent.png

Presenter Notes

"Dude, you are coding the new 'ed' of the 21th century with 4 more level of abstractions!"
a friend, drunk

Presenter Notes

Infos

RedBaron:

Baron:

Contacts:

  • Me: cortex@worlddomination.be
  • Irc: irc.freenode.net#baron

Presenter Notes

Sister project: PyFmt

Usage:

1 from pyfmt import format_code
2 
3 format_code(source_code)

Presenter Notes

Sister project: RedFlyingBaron

1 red *.py  # in a shell bash/zsh/autre

It launchs a Ipython shell with:

 1 red
 2 red[0]
 3 red["./test_redflyingbaron.py"]
 4 red["test_redflyingbaron.py"]
 5 red["test_redflyingbaron"]
 6 red[1:]
 7 
 8 red.display()
 9 
10 red[0].save()
11 red.save()
12 red[0].reload()
13 red.reload()
14 
15 red["f:redflyingbaron"]
16 red[re.compile(r'[^_]+')]
17 red["re:[^_]+"]
18 red[lambda key, value: "red" in key]
19 
20 red.find("stuff")
21 red.find_all("stuff")
22 
23 red.add("/path/to/file", "/path/to/another/file", "again.py")

Presenter Notes