[TOKEEP ?] autojsdoc extension

This commit is contained in:
Victor Feyens 2020-09-24 16:11:55 +02:00
parent e218ae4991
commit fcec9a9d7f
20 changed files with 3559 additions and 0 deletions

View File

@ -0,0 +1,87 @@
:orphan:
======================================
JSDoc parser/Sphinx extension for Odoo
======================================
Why?
====
Spent about a week trying to coerce "standard" javascript tools (jsdoc_ with
the hope of using sphinx-js_ for integration or `documentation.js`_) and
failed to ever get a sensible result: failed to get any result with the
current state of the documentation, significant changes/additions/fixes to
docstrings brought this up to "garbage output" level.
Bug reports and mailing list posts didn't show any path to improvement on the
ES5 codebase (if we ever go whole-hog on ES6 modules and classes things could
be different, in fact most of JSDoc's current effort seem focused on
ES6/ES2015 features) but both experience and looking at the mailing lists
told me that spending more time would be wasted.
Even more so as I was writing visitors/rewriters to generate documentation
from our existing structure, which broadly speaking is relatively strict, and
thus
What?
=====
If it were possible to generate JSDoc annotations from our relatively
well-defined code structures, it was obviously possible to extract documentary
information directly from it, hence this Odoo-specific package/extension
trying to do exactly that.
This package should eventually provide:
* a command-line interface which can be invoked via ``-m autojsdoc`` (assuming
your ``PYTHONPATH`` can find it) which should allow dumping the parsed AST
in a convenient-ish form, possibly doing searches through the AST, a
dependency graph extractor/analysis and a text dumper for the
documentation.
* a sphinx extension (``autojsdoc.sphinx``) which can be used to integrate the
parsed JSDoc information into the Sphinx doc.
How?
====
Sphinx-aside, the package relies on 3 libraries:
* pyjsparser_, an Esprima-compliant ES5.1 parser (with bits of ES6 support),
sadly it does not support comments in its current form so I had to fork it.
Fed a javascript source file, pyjsparser_ simply generates a bunch of nested
dicts representing an Esprima ast, ast-types_ does a reasonably good job of
describing it once you understand that "bases" are basically just structural
mixins.
Because the original does not, this package provides a ``visitor`` module
for pyjsparser_ ASTs.
* pyjsdoc_, a one-file "port" of jsdoc, can actually do much of the JS parsing
(using string munging) but its core semantics don't fit our needs so I'm
only using it to parse the actual JSDoc content, and the ``jsdoc`` module
contains some replacement classes, extensions & monkey patches for things
`pyjsdoc`_ itself does not support, at the time of this writing:
- a bug in FunctionDoc.return_val
- a type on FunctionDoc so it's compatible with ParamDoc
- a more reliable comments-parsing function
- a replacement ModuleDoc as the original does not materialise AMD modules
- a ClassDoc extension to support mixins
- two additional CommentDoc extensions for "namespaces" objects (bag of
attributes without any more information) and mixin objects
* pytest_ to configure and run the test suite, which you can run by invoking
``pytest doc/_extensions`` from the project top-level, the tests represent
both "happy path" things we want to parse and various code patterns which
tripped the happy path because e.g. they were matched and should not have,
they were not matched and should have, or they were more complex than the
happy path had expected
.. _ast-types: _https://github.com/benjamn/ast-types/blob/master/def/core.js
.. _documentation.js: http://documentation.js.org
.. _jsdoc: http://usejsdoc.org
.. _pyjsdoc: https://github.com/nostrademons/pyjsdoc
.. _pyjsparser: https://github.com/PiotrDabkowski/pyjsparser
.. _pytest: https://pytest.org/
.. _sphinx-js: https://sphinx-js-howto.readthedocs.io

View File

@ -0,0 +1 @@
# -*- coding: utf-8 -*-

View File

@ -0,0 +1,154 @@
# -*- coding: utf-8 -*-
from __future__ import print_function
import cgitb
import fnmatch
import io
import logging
import click
import pyjsparser
import sys
from .parser.parser import ModuleMatcher
from .parser.visitor import Visitor, SKIP
from .parser import jsdoc
class Printer(Visitor):
def __init__(self, level=0):
super(Printer, self).__init__()
self.level = level
def _print(self, text):
print(' ' * self.level, text)
def enter_generic(self, node):
self._print(node['type'])
self.level += 1
def exit_generic(self, node):
self.level -= 1
def enter_Identifier(self, node):
self._print(node['name'])
return SKIP
def enter_Literal(self, node):
self._print(node['value'])
return SKIP
def enter_BinaryExpression(self, node):
self._print(node['operator'])
self.level += 1
def visit_files(files, visitor, ctx):
for name in files:
with io.open(name) as f:
ctx.logger.info("%s", name)
try:
yield visitor().visit(pyjsparser.parse(f.read()))
except Exception as e:
if ctx.logger.isEnabledFor(logging.DEBUG):
ctx.logger.exception("while visiting %s", name)
else:
ctx.logger.error("%s while visiting %s", e, name)
# bunch of modules various bits depend on which are not statically defined
# (or are outside the scope of the system)
ABSTRACT_MODULES = [
jsdoc.ModuleDoc({
'module': 'web.web_client',
'dependency': {'web.AbstractWebClient'},
'exports': jsdoc.NSDoc({
'name': 'web_client',
'doc': 'instance of AbstractWebClient',
}),
}),
jsdoc.ModuleDoc({
'module': 'web.Tour',
'dependency': {'web_tour.TourManager'},
'exports': jsdoc.NSDoc({
'name': 'Tour',
'doc': 'maybe tourmanager instance?',
}),
}),
# OH FOR FUCK'S SAKE
jsdoc.ModuleDoc({
'module': 'summernote/summernote',
'exports': jsdoc.NSDoc({'doc': "totally real summernote"}),
})
]
@click.group(context_settings={'help_option_names': ['-h', '--help']})
@click.option('-v', '--verbose', count=True)
@click.option('-q', '--quiet', count=True)
@click.pass_context
def autojsdoc(ctx, verbose, quiet):
logging.basicConfig(
level=logging.INFO + (quiet - verbose) * 10,
format="[%(levelname)s %(created)f] %(message)s",
)
ctx.logger = logging.getLogger('autojsdoc')
ctx.visitor = None
ctx.files = []
ctx.kw = {}
@autojsdoc.command()
@click.argument('files', type=click.Path(exists=True), nargs=-1)
@click.pass_context
def ast(ctx, files):
""" Prints a structure tree of the provided files
"""
if not files:
print(ctx.get_help())
visit_files(files, lambda: Printer(level=1), ctx.parent)
@autojsdoc.command()
@click.option('-m', '--module', multiple=True, help="Only shows dependencies matching any of the patterns")
@click.argument('files', type=click.Path(exists=True), nargs=-1)
@click.pass_context
def dependencies(ctx, module, files):
""" Prints a dot file of all modules to stdout
"""
if not files:
print(ctx.get_help())
byname = {
mod.name: mod.dependencies
for mod in ABSTRACT_MODULES
}
for modules in visit_files(files, ModuleMatcher, ctx.parent):
for mod in modules:
byname[mod.name] = mod.dependencies
print('digraph dependencies {')
todo = set()
# if module filters, roots are only matching modules
if module:
for f in module:
todo.update(fnmatch.filter(byname.keys(), f))
for m in todo:
# set a different box for selected roots
print(' "%s" [color=orangered]' % m)
else:
# otherwise check all modules
todo.update(byname)
done = set()
while todo:
node = todo.pop()
if node in done:
continue
done.add(node)
deps = byname[node]
todo.update(deps - done)
for dep in deps:
print(' "%s" -> "%s";' % (node, dep))
print('}')
try:
autojsdoc.main(prog_name='autojsdoc')
except Exception:
print(cgitb.text(sys.exc_info()))

View File

@ -0,0 +1,13 @@
# -*- coding: utf-8 -*-
from .directives import automodule_bound, autodirective_bound
from .extractor import _get_roots
def setup(app):
app.add_config_value('js_roots', _get_roots, 'env')
modules = {}
app.add_directive_to_domain('js', 'automodule', automodule_bound(app, modules)
)
autodirective = autodirective_bound(app, modules)
for n in ['autonamespace', 'automixin', 'autoclass', 'autofunction']:
app.add_directive_to_domain('js', n, autodirective)

View File

@ -0,0 +1,719 @@
# -*- coding: utf-8 -*-
import abc
import collections
import contextlib
import fnmatch
import io
import re
from docutils import nodes
from docutils.parsers.rst import Directive
from docutils.statemachine import StringList
from sphinx import addnodes
from sphinx.ext.autodoc import members_set_option, bool_option, ALL
from autojsdoc.ext.extractor import read_js
from ..parser import jsdoc, types
class DocumenterError(Exception): pass
@contextlib.contextmanager
def addto(parent, newnode):
assert isinstance(newnode, nodes.Node), \
"Expected newnode to be a Node, got %s" % (type(newnode))
yield newnode
parent.append(newnode)
def documenter_for(directive, doc):
if isinstance(doc, jsdoc.FunctionDoc):
return FunctionDocumenter(directive, doc)
if isinstance(doc, jsdoc.ClassDoc):
return ClassDocumenter(directive, doc)
if isinstance(doc, jsdoc.MixinDoc):
return MixinDocumenter(directive, doc)
if isinstance(doc, (jsdoc.PropertyDoc, jsdoc.LiteralDoc)):
return PropertyDocumenter(directive, doc)
if isinstance(doc, jsdoc.InstanceDoc):
return InstanceDocumenter(directive, doc)
if isinstance(doc, jsdoc.Unknown):
return UnknownDocumenter(directive, doc)
if isinstance(doc, jsdoc.NSDoc):
return NSDocumenter(directive, doc)
raise TypeError("No documenter for %s" % type(doc))
def to_list(doc, source=None):
return StringList([
line.rstrip('\n')
for line in io.StringIO(doc)
], source=source)
DIRECTIVE_OPTIONS = {
'members': members_set_option,
'undoc-members': bool_option,
'private-members': bool_option,
'undoc-matches': bool_option,
}
def automodule_bound(app, modules):
class AutoModuleDirective(Directive):
required_arguments = 1
has_content = True
option_spec = DIRECTIVE_OPTIONS
# self.state.nested_parse(string, offset, node) => parse context for sub-content (body which can contain RST data)
# => needed for doc (converted?) and for actual directive body
def run(self):
self.env = self.state.document.settings.env
modname = self.arguments[0].strip()
# TODO: cache/memoize modules & symbols?
if not modules:
read_js(app, modules)
mods = [
(name, mod)
for name, mod in modules.items()
if fnmatch.fnmatch(name, modname)
]
ret = []
for name, mod in mods:
if mod.is_private:
continue
if name != modname and not (mod.doc or mod.exports):
# this module has no documentation, no exports and was
# not specifically requested through automodule -> skip
# unless requested
if not self.options.get('undoc-matches'):
continue
modsource = mod['sourcefile']
if modsource:
self.env.note_dependency(modsource)
# not sure what that's used for as normal xrefs are resolved using the id directly
target = nodes.target('', '', ids=['module-' + name], ismod=True)
self.state.document.note_explicit_target(target)
documenter = ModuleDocumenter(self, mod)
ret.append(target)
ret.extend(documenter.generate())
return ret
return AutoModuleDirective
def autodirective_bound(app, modules):
documenters = {
'js:autoclass': ClassDocumenter,
'js:autonamespace': NSDocumenter,
'js:autofunction': FunctionDocumenter,
'js:automixin': MixinDocumenter,
}
class AutoDirective(Directive):
required_arguments = 1
has_content = True
option_spec = DIRECTIVE_OPTIONS
def run(self):
self.env = self.state.document.settings.env
objname = self.arguments[0].strip()
if not modules:
read_js(app, modules)
# build complete path to object
path = self.env.temp_data.get('autojs:prefix', []) + objname.split('.')
# look for module/object split
for i in range(1, len(path)):
modname, objpath = '.'.join(path[:-i]), path[-i:]
module = modules.get(modname)
if module:
break
else:
raise Exception("Found no valid module in " + '.'.join(path))
item = module
# deref' namespaces until we reach the object we're looking for
for k in objpath:
item = item.get_property(k)
docclass = documenters[self.name]
return docclass(self, item).generate()
return AutoDirective
class Documenter(object):
objtype = None
def __init__(self, directive, doc):
self.directive = directive
self.env = directive.env
self.item = doc
@property
def modname(self):
return self.env.temp_data.get('autojs:module', '')
@property
def classname(self):
return self.env.temp_data.get('autojs:class', '')
def generate(self, all_members=False):
"""
:rtype: List[nodes.Node]
"""
try:
return self._generate(all_members=all_members)
except Exception as e:
raise DocumenterError("Failed to document %s" % self.item) from e
def _generate(self, all_members=False):
objname = self.item.name
prefixed = (self.item['sourcemodule'].name + '.' + objname) if self.item['sourcemodule'] else None
objtype = self.objtype
assert objtype, '%s has no objtype' % type(self)
root = addnodes.desc(domain='js', desctype=objtype, objtype=objtype)
with addto(root, addnodes.desc_signature(
module=self.modname or '',
fullname=objname,
)) as s:
s['class'] = self.classname
s['ids'] = []
if objname:
s['ids'].append(objname)
if prefixed:
s['ids'].append(prefixed)
if objtype:
s += addnodes.desc_annotation(
objtype, objtype,
nodes.Text(' '),
)
env = self.env
if objname:
env.domaindata['js']['objects'][objname] = (env.docname, objtype)
if prefixed:
env.domaindata['js']['objects'][prefixed] = (env.docname, objtype)
# TODO: linkcode_resolve
s += self.make_signature()
with addto(root, addnodes.desc_content()) as c:
# must be here otherwise nested_parse(self.content) will not have
# the prefix set
self.env.temp_data.setdefault('autojs:prefix', []).append(self.item.name)
c += self.make_content(all_members=all_members)
self.env.temp_data['autojs:prefix'].pop()
return [root]
def make_signature(self):
"""
:rtype: List[nodes.Node]
"""
return [addnodes.desc_name(self.item.name, self.item.name)]
@abc.abstractmethod
def make_content(self, all_members):
"""
:rtype: List[nodes.Node]
"""
def document_subtypes(self, subtypes):
docs = []
with with_mapping_value(self.directive.options, 'undoc-members', True):
for cls in subtypes:
docs += ClassDocumenter(self.directive, cls).generate(all_members=True)
return docs
class NSDocumenter(Documenter):
objtype = 'namespace'
def make_content(self, all_members):
doc = self.item
ret = nodes.section()
if doc.doc:
self.directive.state.nested_parse(to_list(doc.doc), 0, ret)
self.directive.state.nested_parse(self.directive.content, 0, ret)
ret += self.document_properties(all_members)
return ret.children
def should_document(self, member, name, all_members):
"""
:type member: jsdoc.CommentDoc
:type name: str
:type all_members: bool
:rtype: bool
"""
options = self.directive.options
members = options.get('members') or []
if not (all_members or members is ALL):
# if a member is requested by name, it's always documented
return name in members
# ctor params are merged into the class doc
if member.is_constructor:
return False
# only document "private" members if option is set
if self.is_private(member, name) and not options.get('private-members'):
return False
# TODO: what if member doesn't have a description but has non-desc tags set?
# TODO: add @public to force documenting symbol? => useful for implicit typedef
return bool(member.doc or options.get('undoc-members'))
def is_private(self, member, name):
return member.is_private
def document_properties(self, all_members):
ret = nodes.section()
# TODO: :member-order: [alphabetical | groupwise | bysource]
for (n, p) in self.item.properties:
if not self.should_document(p, n, all_members):
continue
# FIXME: maybe should use property name as name inside?
ret += documenter_for(self.directive, p).generate(all_members=True)
return ret.children
_NONE = object()
@contextlib.contextmanager
def with_mapping_value(mapping, key, value, restore_to=_NONE):
""" Sets ``key`` to ``value`` for the duration of the context.
If ``restore_to`` is not provided, restores ``key``'s old value
afterwards, removes it entirely if there was no value for ``key`` in the
mapping.
.. warning:: for defaultdict & similar mappings, may restore the default
value (depends how the collections' .get behaves)
"""
if restore_to is _NONE:
restore_to = mapping.get(key, _NONE)
mapping[key] = value
try:
yield
finally:
if restore_to is _NONE:
del mapping[key]
else:
mapping[key] = restore_to
class ModuleDocumenter(NSDocumenter):
objtype = 'module'
def document_properties(self, all_members):
with with_mapping_value(self.env.temp_data, 'autojs:module', self.item.name, ''):
return super(ModuleDocumenter, self).document_properties(all_members)
def make_content(self, all_members):
doc = self.item
content = addnodes.desc_content()
if doc.exports or doc.dependencies:
with addto(content, nodes.field_list()) as fields:
if doc.exports:
with addto(fields, nodes.field()) as field:
field += nodes.field_name('Exports', 'Exports')
with addto(field, nodes.field_body()) as body:
ref = doc['exports'] # warning: not the same as doc.exports
label = ref or '<anonymous>'
link = addnodes.pending_xref(
ref, nodes.paragraph(ref, label),
refdomain='js',
reftype='any',
reftarget=ref,
)
link['js:module'] = doc.name
body += link
if doc.dependencies:
with addto(fields, nodes.field()) as field:
self.make_dependencies(field, doc)
if doc.doc:
# FIXME: source offset
self.directive.state.nested_parse(to_list(doc.doc, source=doc['sourcefile']), 0, content)
self.directive.state.nested_parse(self.directive.content, 0, content)
content += self.document_properties(all_members)
return content
def make_dependencies(self, field, doc):
field += nodes.field_name("Depends On", "Depends On")
with addto(field, nodes.field_body()) as body:
with addto(body, nodes.bullet_list()) as deps:
for dep in sorted(doc.dependencies):
ref = addnodes.pending_xref(
dep, nodes.paragraph(dep, dep),
refdomain='js',
reftype='module',
reftarget=dep,
)
deps += nodes.list_item(dep, ref)
def should_document(self, member, name, all_members):
# member can be Nothing?
if not member:
return False
modname = getattr(member['sourcemodule'], 'name', None)
doc = self.item
# always document exported symbol (regardless undoc, private, ...)
# otherwise things become... somewhat odd
if name == doc['exports']:
return True
# if doc['exports'] the module is exporting a "named" item which
# does not need to be documented twice, if not doc['exports'] it's
# exporting an anonymous item (e.g. object literal) which needs to
# be documented on its own
if name == '<exports>' and not doc['exports']:
return True
# TODO: :imported-members:
# FIXME: *directly* re-exported "foreign symbols"?
return (not modname or modname == doc.name) \
and super(ModuleDocumenter, self).should_document(member, name, all_members)
class ClassDocumenter(NSDocumenter):
objtype = 'class'
def document_properties(self, all_members):
with with_mapping_value(self.env.temp_data, 'autojs:class', self.item.name, ''):
return super(ClassDocumenter, self).document_properties(all_members)
def make_signature(self):
sig = super(ClassDocumenter, self).make_signature()
sig.append(self.make_parameters())
return sig
def make_parameters(self):
params = addnodes.desc_parameterlist('', '')
ctor = self.item.constructor
if ctor:
params += make_desc_parameters(ctor.params)
return params
def make_content(self, all_members):
doc = self.item
ret = nodes.section()
ctor = self.item.constructor
params = subtypes = []
if ctor:
check_parameters(self, ctor)
params, subtypes = extract_subtypes(doc.name, ctor)
fields = nodes.field_list()
fields += self.make_super()
fields += self.make_mixins()
fields += self.make_params(params)
if fields.children:
ret += fields
if doc.doc:
self.directive.state.nested_parse(to_list(doc.doc), 0, ret)
self.directive.state.nested_parse(self.directive.content, 0, ret)
ret += self.document_properties(all_members)
ret += self.document_subtypes(subtypes)
return ret.children
def is_private(self, member, name):
return name.startswith('_') or super(ClassDocumenter, self).is_private(member, name)
def make_super(self):
doc = self.item
if not doc.superclass:
return []
sup_link = addnodes.pending_xref(
doc.superclass.name, nodes.paragraph(doc.superclass.name, doc.superclass.name),
refdomain='js', reftype='class', reftarget=doc.superclass.name,
)
sup_link['js:module'] = doc.superclass['sourcemodule'].name
return nodes.field(
'',
nodes.field_name("Extends", "Extends"),
nodes.field_body(doc.superclass.name, sup_link),
)
def make_mixins(self):
doc = self.item
if not doc.mixins:
return []
ret = nodes.field('', nodes.field_name("Mixes", "Mixes"))
with addto(ret, nodes.field_body()) as body:
with addto(body, nodes.bullet_list()) as mixins:
for mixin in sorted(doc.mixins, key=lambda m: m.name):
mixin_link = addnodes.pending_xref(
mixin.name, nodes.paragraph(mixin.name, mixin.name),
refdomain='js', reftype='mixin', reftarget=mixin.name
)
mixin_link['js:module'] = mixin['sourcemodule'].name
mixins += nodes.list_item('', mixin_link)
return ret
def make_params(self, params):
if not params:
return []
ret = nodes.field('', nodes.field_name('Parameters', 'Parameters'))
with addto(ret, nodes.field_body()) as body,\
addto(body, nodes.bullet_list()) as holder:
holder += make_parameters(params, mod=self.modname)
return ret
class InstanceDocumenter(Documenter):
objtype = 'object'
def make_signature(self):
cls = self.item.cls
ret = super(InstanceDocumenter, self).make_signature()
if cls:
super_ref = addnodes.pending_xref(
cls.name, nodes.Text(cls.name, cls.name),
refdomain='js', reftype='class', reftarget=cls.name
)
super_ref['js:module'] = cls['sourcemodule'].name
ret.append(addnodes.desc_annotation(' instance of ', ' instance of '))
ret.append(addnodes.desc_type(cls.name, '', super_ref))
if not ret:
return [addnodes.desc_name('???', '???')]
return ret
def make_content(self, all_members):
ret = nodes.section()
if self.item.doc:
self.directive.state.nested_parse(to_list(self.item.doc), 0, ret)
self.directive.state.nested_parse(self.directive.content, 0, ret)
return ret.children
class FunctionDocumenter(Documenter):
@property
def objtype(self):
return 'method' if self.classname else 'function'
def make_signature(self):
ret = super(FunctionDocumenter, self).make_signature()
with addto(ret, addnodes.desc_parameterlist()) as params:
params += make_desc_parameters(self.item.params)
retval = self.item.return_val
if retval.type or retval.doc:
ret.append(addnodes.desc_returns(retval.type or '*', retval.type or '*'))
return ret
def make_content(self, all_members):
ret = nodes.section()
doc = self.item
if doc.doc:
self.directive.state.nested_parse(to_list(doc.doc), 0, ret)
self.directive.state.nested_parse(self.directive.content, 0, ret)
check_parameters(self, doc)
params, subtypes = extract_subtypes(self.item.name, self.item)
rdoc = doc.return_val.doc
rtype = doc.return_val.type
if params or rtype or rdoc:
with addto(ret, nodes.field_list()) as fields:
if params:
with addto(fields, nodes.field()) as field:
field += nodes.field_name('Parameters', 'Parameters')
with addto(field, nodes.field_body()) as body,\
addto(body, nodes.bullet_list()) as holder:
holder.extend(make_parameters(params, mod=doc['sourcemodule'].name))
if rdoc:
with addto(fields, nodes.field()) as field:
field += nodes.field_name("Returns", "Returns")
with addto(field, nodes.field_body()) as body,\
addto(body, nodes.paragraph()) as p:
p += nodes.inline(rdoc, rdoc)
if rtype:
with addto(fields, nodes.field()) as field:
field += nodes.field_name("Return Type", "Return Type")
with addto(field, nodes.field_body()) as body, \
addto(body, nodes.paragraph()) as p:
p += make_types(rtype, mod=doc['sourcemodule'].name)
ret += self.document_subtypes(subtypes)
return ret.children
def pascal_case_ify(name):
"""
Uppercase first letter of ``name``, or any letter following an ``_``. In
the latter case, also strips out the ``_``.
=> key_for becomes KeyFor
=> options becomes Options
"""
return re.sub(r'(^|_)\w', lambda m: m.group(0)[-1].upper(), name)
def extract_subtypes(parent_name, doc):
""" Extracts composite parameters (a.b) into sub-types for the parent
parameter, swaps the parent's type from whatever it is to the extracted
one, and returns the extracted type for inclusion into the parent.
:arg parent_name: name of the containing symbol (function, class), will
be used to compose subtype names
:type parent_name: str
:type doc: FunctionDoc
:rtype: (List[ParamDoc], List[ClassDoc])
"""
# map of {param_name: [ParamDoc]} (from complete doc)
subparams = collections.defaultdict(list)
for p in map(jsdoc.ParamDoc, doc.get_as_list('param')):
pair = p.name.split('.', 1)
if len(pair) == 2:
k, p.name = pair # remove prefix from param name
subparams[k].append(p)
# keep original params order as that's the order of formal parameters in
# the function signature
params = collections.OrderedDict((p.name, p) for p in doc.params)
subtypes = []
# now we can use the subparams map to extract "compound" parameter types
# and swap the new type for the original param's type
for param_name, subs in subparams.items():
typename = '%s%s' % (
pascal_case_ify(parent_name),
pascal_case_ify(param_name),
)
param = params[param_name]
param.type = typename
subtypes.append(jsdoc.ClassDoc({
'name': typename,
'doc': param.doc,
'_members': [
# TODO: add default value
(sub.name, jsdoc.PropertyDoc(dict(sub.to_dict(), sourcemodule=doc['sourcemodule'])))
for sub in subs
],
'sourcemodule': doc['sourcemodule'],
}))
return params.values(), subtypes
def check_parameters(documenter, doc):
"""
Check that all documented parameters match a formal parameter for the
function. Documented params which don't match the actual function may be
typos.
"""
guessed = set(doc['guessed_params'] or [])
if not guessed:
return
documented = {
# param name can be of the form [foo.bar.baz=default]\ndescription
jsdoc.ParamDoc(text).name.split('.')[0]
for text in doc.get_as_list('param')
}
odd = documented - guessed
if not odd:
return
app = documenter.directive.env.app
app.warn("Found documented params %s not in formal parameter list "
"of function %s in module %s (%s)" % (
', '.join(odd),
doc.name,
documenter.modname,
doc['sourcemodule']['sourcefile'],
))
def make_desc_parameters(params):
for p in params:
if '.' in p.name:
continue
node = addnodes.desc_parameter(p.name, p.name)
if p.optional:
node = addnodes.desc_optional('', '', node)
yield node
def make_parameters(params, mod=None):
for param in params:
p = nodes.paragraph('', '', nodes.strong(param.name, param.name))
if param.default is not None:
p += nodes.Text('=', '=')
p += nodes.emphasis(param.default, param.default)
if param.type:
p += nodes.Text(' (')
p += make_types(param.type, mod=mod)
p += nodes.Text(')')
if param.doc:
p += [
nodes.Text(' -- '),
nodes.inline(param.doc, param.doc)
]
yield p
def _format_value(v):
if v == '|':
return nodes.emphasis(' or ', ' or ')
if v == ',':
return nodes.Text(', ', ', ')
return nodes.Text(v, v)
def make_types(typespec, mod=None):
# TODO: in closure notation {type=} => optional, do we care?
def format_type(t):
ref = addnodes.pending_xref(
t, addnodes.literal_emphasis(t, t),
refdomain='js', reftype='class', reftarget=t,
)
if mod:
ref['js:module'] = mod
return ref
try:
return types.iterate(
types.parse(typespec),
format_type,
_format_value
)
except ValueError as e:
raise ValueError("%s in '%s'" % (e, typespec))
class MixinDocumenter(NSDocumenter):
objtype = 'mixin'
class PropertyDocumenter(Documenter):
objtype = 'attribute'
def make_signature(self):
ret = super(PropertyDocumenter, self).make_signature()
proptype = self.item.type
if proptype:
typeref = addnodes.pending_xref(
proptype, nodes.Text(proptype, proptype),
refdomain='js', reftype='class', reftarget=proptype
)
typeref['js:module'] = self.item['sourcemodule'].name
ret.append(nodes.Text(' '))
ret.append(typeref)
return ret
def make_content(self, all_members):
doc = self.item
ret = nodes.section()
self.directive.state.nested_parse(self.directive.content, 0, ret)
if doc.doc:
self.directive.state.nested_parse(to_list(doc.doc), 0, ret)
return ret.children
class UnknownDocumenter(Documenter):
objtype = 'unknown'
def make_content(self, all_members):
return []

View File

@ -0,0 +1,87 @@
# -*- coding: utf-8 -*-
import io
import os
import pyjsdoc
import pyjsparser
try:
from sphinx.util import status_iterator
def it(app): return status_iterator
except ImportError:
# 1.2: Builder.status_iterator
# 1.3: Application.status_iterator (with alias on Builder)
# 1.6: sphinx.util.status_iterator (with *deprecated* aliases on Application and Builder)
# 1.7: removed Application and Builder aliases
# => if no sphinx.util.status_iterator, fallback onto the builder one for
# 1.2 compatibility, remove this entirely if we ever require 1.6+
status_iterator = None
def it(app): return app.builder.status_iterator
from ..parser import jsdoc, parser
def _get_roots(conf):
env_roots = os.environ.get('AUTOJSDOC_ROOTS_PATH')
if env_roots:
return env_roots.split(':')
return []
def read_js(app, modules):
"""
:type app: sphinx.application.Sphinx
:type modules: Dict[str, jsdoc.ModuleDoc]
"""
roots = map(os.path.normpath, app.config.js_roots or [os.path.join(app.confdir, '..')])
files = [
os.path.join(r, f)
for root in roots
for r, _, fs in os.walk(root)
if 'static/src/js' in r
for f in fs
if f.endswith('.js')
]
modules.update((mod.name, mod) for mod in ABSTRACT_MODULES)
for name in it(app)(files, "Parsing javascript files...", length=len(files)):
with io.open(name) as f:
ast = pyjsparser.parse(f.read())
modules.update(
(mod.name, mod)
for mod in parser.ModuleMatcher(name).visit(ast)
)
_resolve_references(modules)
def _resolve_references(byname):
# must be done in topological order otherwise the dependent can't
# resolve non-trivial references to a dependency properly
for name in pyjsdoc.topological_sort(
*pyjsdoc.build_dependency_graph(
list(byname.keys()),
byname
)
):
byname[name].post_process(byname)
ABSTRACT_MODULES = [
jsdoc.ModuleDoc({
'module': u'web.web_client',
'dependency': {u'web.AbstractWebClient'},
'exports': jsdoc.NSDoc({
'name': u'web_client',
'doc': u'instance of AbstractWebClient',
}),
}),
jsdoc.ModuleDoc({
'module': u'web.Tour',
'dependency': {u'web_tour.TourManager'},
'exports': jsdoc.NSDoc({
'name': u'Tour',
'doc': u'maybe tourmanager instance?',
}),
}),
jsdoc.ModuleDoc({
'module': u'summernote/summernote',
'exports': jsdoc.NSDoc({'doc': u"totally real summernote"}),
})
]

View File

@ -0,0 +1 @@
# -*- coding: utf-8 -*-

View File

@ -0,0 +1,357 @@
# -*- coding: utf-8 -*-
import re
import collections
import pyjsdoc
def strip_stars(doc_comment):
"""
Version of jsdoc.strip_stars which always removes 1 space after * if
one is available.
"""
return re.sub('\n\s*?\*[\t ]?', '\n', doc_comment[3:-2]).strip()
class ParamDoc(pyjsdoc.ParamDoc):
"""
Replace ParamDoc because FunctionDoc doesn't properly handle optional
params or default values (TODO: or compounds) if guessed_params is used
=> augment paramdoc with "required" and "default" items to clean up name
"""
def __init__(self, text):
super(ParamDoc, self).__init__(text)
# param name and doc can be separated by - or :, strip it
self.doc = self.doc.strip().lstrip('-:').lstrip()
self.optional = False
self.default = None
# there may not be a space between the param name and the :, in which
# case the : gets attached to the name, strip *again*
# TODO: formal @param/@property parser to handle this crap properly once and for all
self.name = self.name.strip().rstrip(':')
if self.name.startswith('['):
self.name = self.name.strip('[]')
self.optional = True
if '=' in self.name:
self.name, self.default = self.name.rsplit('=', 1)
def to_dict(self):
d = super(ParamDoc, self).to_dict()
d['optional'] = self.optional
d['default'] = self.default
return d
pyjsdoc.ParamDoc = ParamDoc
class CommentDoc(pyjsdoc.CommentDoc):
namekey = object()
is_constructor = False
@property
def name(self):
return self[self.namekey] or self['name'] or self['guessed_name']
def set_name(self, name):
# not great...
if name != '<exports>':
self.parsed['guessed_name'] = name
@property
def is_private(self):
return 'private' in self.parsed
def to_dict(self):
d = super(CommentDoc, self).to_dict()
d['name'] = self.name
return d
# don't resolve already resolved docs (e.g. a literal dict being
# include-ed in two different classes because I don't even care anymore
def become(self, modules):
return self
class PropertyDoc(CommentDoc):
@classmethod
def from_param(cls, s, sourcemodule=None):
parsed = ParamDoc(s).to_dict()
parsed['sourcemodule'] = sourcemodule
return cls(parsed)
@property
def type(self):
return self['type'].strip('{}')
def to_dict(self):
d = super(PropertyDoc, self).to_dict()
d['type'] = self.type
d['is_private'] = self.is_private
return d
class InstanceDoc(CommentDoc):
@property
def cls(self):
return self['cls']
def to_dict(self):
return dict(super(InstanceDoc, self).to_dict(), cls=self.cls)
class LiteralDoc(CommentDoc):
@property
def type(self):
if self['type']:
return self['type']
valtype = type(self['value'])
if valtype is bool:
return 'Boolean'
elif valtype is float:
return 'Number'
elif valtype is type(u''):
return 'String'
return ''
@property
def value(self):
return self['value']
def to_dict(self):
d = super(LiteralDoc, self).to_dict()
d['type'] = self.type
d['value'] = self.value
return d
class FunctionDoc(CommentDoc):
type = 'Function'
namekey = 'function'
@property
def is_constructor(self):
return self.name == 'init'
@property
def params(self):
tag_texts = self.get_as_list('param')
# turns out guessed_params is *almost* (?) always set to a list,
# if empty list of guessed params fall back to @params
if not self['guessed_params']:
# only get "primary" params (no "." in name)
return [
p for p in map(ParamDoc, tag_texts)
if '.' not in p.name
]
else:
param_dict = {}
for text in tag_texts:
param = ParamDoc(text)
param_dict[param.name] = param
return [param_dict.get(name) or ParamDoc('{} ' + name)
for name in self.get('guessed_params')]
@property
def return_val(self):
ret = self.get('return') or self.get('returns')
type = self.get('type')
if '{' in ret and '}' in ret:
if not '} ' in ret:
# Ensure that name is empty
ret = re.sub(r'\}\s*', '} ', ret)
return ParamDoc(ret)
if ret and type:
return ParamDoc('{%s} %s' % (type, ret))
return ParamDoc(ret)
def to_dict(self):
d = super(FunctionDoc, self).to_dict()
d['name'] = self.name
d['params'] = [param.to_dict() for param in self.params]
d['return_val']= self.return_val.to_dict()
return d
class NSDoc(CommentDoc):
namekey = 'namespace'
def __init__(self, parsed_comment):
super(NSDoc, self).__init__(parsed_comment)
self.members = collections.OrderedDict()
def add_member(self, name, member):
"""
:type name: str
:type member: CommentDoc
"""
member.set_name(name)
self.members[name] = member
@property
def properties(self):
if self.get('property'):
return [
(p.name, p)
for p in (
PropertyDoc.from_param(p, self['sourcemodule'])
for p in self.get_as_list('property')
)
]
return list(self.members.items()) or self['_members'] or []
def has_property(self, name):
return self.get_property(name) is not None
def get_property(self, name):
return next((p for n, p in self.properties if n == name), None)
def to_dict(self):
d = super(NSDoc, self).to_dict()
d['properties'] = [(n, p.to_dict()) for n, p in self.properties]
return d
class MixinDoc(NSDoc):
namekey = 'mixin'
class ModuleDoc(NSDoc):
namekey = 'module'
def __init__(self, parsed_comment):
super(ModuleDoc, self).__init__(parsed_comment)
#: callbacks to run with the modules mapping once every module is resolved
self._post_process = []
def post_process(self, modules):
for callback in self._post_process:
callback(modules)
@property
def module(self):
return self # lol
@property
def dependencies(self):
"""
Returns the immediate dependencies of a module (only those explicitly
declared/used).
"""
return self.get('dependency', None) or set()
@property
def exports(self):
"""
Returns the actual item exported from the AMD module, can be a
namespace, a class, a function, an instance, ...
"""
return self.get_property('<exports>')
def to_dict(self):
vars = super(ModuleDoc, self).to_dict()
vars['dependencies'] = self.dependencies
vars['exports'] = self.exports
return vars
def __str__(self):
s = super().__str__()
if self['sourcefile']:
s += " in file " + self['sourcefile']
return s
class ClassDoc(NSDoc):
namekey = 'class'
@property
def constructor(self):
return self.get_property('init')
@property
def superclass(self):
return self['extends'] or self['base']
def get_property(self, method_name):
if method_name == 'extend':
return FunctionDoc({
'doc': 'Create subclass for %s' % self.name,
'guessed_function': 'extend',
})
# FIXME: should ideally be a proxy namespace
if method_name == 'prototype':
return self
return super(ClassDoc, self).get_property(method_name)\
or (self.superclass and self.superclass.get_property(method_name))
@property
def mixins(self):
return self.get_as_list('mixes')
def to_dict(self):
d = super(ClassDoc, self).to_dict()
d['mixins'] = self.mixins
return d
DEFAULT = object()
class UnknownNS(NSDoc):
params = () # TODO: log warning when (somehow) trying to access / document an unknown object as ctor?
def get_property(self, name):
return super(UnknownNS, self).get_property(name) or \
UnknownNS({'name': '{}.{}'.format(self.name, name)})
def __getitem__(self, item):
if self._probably_not_property(item):
return super().__getitem__(item)
return self.get_property(item)
def _probably_not_property(self, item):
return (
not isinstance(item, str)
or item in (self.namekey, 'name', 'params')
or item.startswith(('_', 'guessed_'))
or item in self.parsed
)
class Unknown(CommentDoc):
@classmethod
def from_(cls, source):
def builder(parsed):
inst = cls(parsed)
inst.parsed['source'] = source
return inst
return builder
@property
def name(self):
return self['name'] + ' ' + self['source']
@property
def type(self):
return "Unknown"
def get_property(self, p):
return Unknown(dict(self.parsed, source=self.name, name=p + '<'))
def parse_comments(comments, doctype=None):
# find last comment which starts with a *
docstring = next((
c['value']
for c in reversed(comments or [])
if c['value'].startswith(u'*')
), None) or u""
# \n prefix necessary otherwise parse_comment fails to take first
# block comment parser strips delimiters, but strip_stars fails without
# them
extract = '\n' + strip_stars('/*' + docstring + '\n*/')
parsed = pyjsdoc.parse_comment(extract, u'')
if doctype == 'FunctionExpression':
doctype = FunctionDoc
elif doctype == 'ObjectExpression' or doctype is None:
doctype = guess
if doctype is guess:
return doctype(parsed)
# in case a specific doctype is given, allow overriding it anyway
return guess(parsed, default=doctype)
def guess(parsed, default=UnknownNS):
if 'class' in parsed:
return ClassDoc(parsed)
if 'function' in parsed:
return FunctionDoc(parsed)
if 'mixin' in parsed:
return MixinDoc(parsed)
if 'namespace' in parsed:
return NSDoc(parsed)
if 'module' in parsed:
return ModuleDoc(parsed)
if 'type' in parsed:
return PropertyDoc(parsed)
return default(parsed)

View File

@ -0,0 +1,572 @@
# -*- coding: utf-8 -*-
import collections
import pyjsdoc
from . import jsdoc
from . import utils
from .visitor import Visitor, SKIP
DECLARATOR_INIT_TO_REF = ('Literal', 'Identifier', 'MemberExpression')
class ModuleMatcher(Visitor):
"""Looks for structures of the form::
odoo.define($string, function ($name) {
These are *Odoo module definitions*, upon encountering one the
matcher:
* creates a module entry, optionally associated with the module comment
* spawns off a :class:`ModuleBodyMatcher` on the function's body with the
module name and the $name as "require" function
"""
def __init__(self, filename):
super(ModuleMatcher, self).__init__()
self.filename = filename
self.result = []
def enter_Program(self, node):
pass # allows visiting toplevel
def enter_ExpressionStatement(self, node):
# we're interested in expression statements (toplevel call)
if utils.match(node, {'expression': {
'callee': {
'object': {'name': 'odoo'},
'property': {'name': 'define'},
},
}}):
[module, *_, func] = node['expression']['arguments']
mod = jsdoc.parse_comments(node.get('comments'), jsdoc.ModuleDoc)
# set module name
mod.set_name(module['value'])
mod.parsed['sourcefile'] = self.filename
self.result.append(mod)
# get name of require parameter
require = None # a module can have no dependencies
if func['params']:
require = func['params'][0]['name']
mod.parsed['dependency'], post = ModuleExtractor(mod, require).visit(func['body'])
mod._post_process.extend(post)
# don't recurse since we've fired off a sub-visitor for the
# bits we're interested in
return SKIP
def enter_generic(self, node):
# skip all other toplevel statements
return SKIP
ref = collections.namedtuple('Ref', 'object property')
def _name(r):
bits = []
while isinstance(r, ref):
bits.append(str(r.property))
r = r.object
return '.'.join(reversed(bits))
def deref(item, prop=None):
assert isinstance(item, ref)
while isinstance(item, ref):
obj = item.object
if isinstance(obj, ref):
obj = deref(obj)
if isinstance(obj, (jsdoc.NSDoc, jsdoc.Unknown)):
item = obj.get_property(item.property)
elif isinstance(obj, dict):
item = obj[item.property]
elif isinstance(obj, jsdoc.PropertyDoc):
# f'n dynamic crap
item = jsdoc.Unknown(obj.to_dict()).get_property(item.property)
else:
raise ValueError("%r (%s) should be a dict or namespace" % (obj, type(obj)))
return item
def m2r(me, scope):
# create a ref to the scope in case it's a hoisted function declaration,
# this may false-positive on other hoisting but w/e
if me['type'] == 'Literal':
return jsdoc.LiteralDoc({'value': me['value']})
if me['type'] == 'Identifier':
return ref(scope, me['name'])
if me['type'] != 'MemberExpression':
raise ValueError(me)
return ref(
m2r(me['object'], scope),
utils._value(me['property'], strict=True)
)
NOTHING = object()
class Declaration(object):
__slots__ = ['id', 'comments']
def __init__(self, id=None, comments=NOTHING):
self.id = id
self.comments = [] if comments is NOTHING else comments
class ModuleContent(object):
__slots__ = ['dependencies', 'post']
def __init__(self, dependencies=NOTHING, post=NOTHING):
self.dependencies = set() if dependencies is NOTHING else dependencies
self.post = [] if post is NOTHING else post
def __iter__(self):
yield self.dependencies
yield self.post
class Nothing(object):
def __init__(self, name):
self.name = name
def __bool__(self):
return False
__nonzero__ = __bool__
class RefProxy(object):
def __init__(self, r):
self._ref = r
def become(self, modules):
s = self
other = deref(self._ref) or Nothing(_name(self._ref))
# ??? shouldn't all previous refs have been resolved?
if isinstance(other, (RefProxy, ModuleProxy)):
other = other.become(modules)
self.__class__ = other.__class__
self.__dict__ = other.__dict__
return self
def set_name(self, name):
pass # ???
class ModuleProxy(object):
def __init__(self, name):
self._name = name
# replace the ModuleProxy by the module's exports
def become(self, modules):
s = self
m = modules[self._name].get_property('<exports>') or Nothing(self._name)
self.__class__ = m.__class__
self.__dict__ = m.__dict__
return self
def set_name(self, name):
pass # FIXME: ???
jq = jsdoc.UnknownNS({
'name': u'jQuery',
'doc': u'<jQuery>',
})
window = jsdoc.UnknownNS({
'doc': '<window>',
'name': 'window',
})
class BaseScope(collections.defaultdict):
""" The base scope assumes anything it's asked for is just an unknown
(global) namespace of some sort. Can hold a bunch of predefined params but
avoids the variables inference system blowing up when new (browser)
globals get used in module bodies.
"""
def __missing__(self, key):
it = jsdoc.UnknownNS({
'name': key,
'doc': u'<%s>' % key,
})
self[key] = it
return it
BASE_SCOPE = BaseScope(None, {
'_': jsdoc.UnknownNS({'doc': u'<underscore.js>', 'name': u'_'}),
'$': jq, 'jQuery': jq,
'window': window,
'document': window.get_property('document'),
'Date': jsdoc.ClassDoc({
'name': u'Date',
'doc': u'',
}),
'Backbone': jsdoc.UnknownNS({
'_members': [
('Model', jsdoc.ClassDoc({
'name': u'Model',
'doc': u'',
})),
('Collection', jsdoc.ClassDoc({
'name': u'Collection',
'doc': u'',
})),
]
}),
'odoo': jsdoc.UnknownNS({
'name': u'odoo',
'doc': u"Odoo",
'_members': [
('name', jsdoc.PropertyDoc({'name': u'csrf_token', 'type': u'{String}'})),
]
}),
'undefined': jsdoc.LiteralDoc({'name': u'undefined', 'value': None}),
})
class Scope(object):
"""
Add hoc scope versioning/SSA such that rebinding a symbol in a module
scope does not screw everything up e.g. "Foo = Foo.extend({})" should not
have the final Foo extending itself...
"""
def __init__(self, mapping):
self._namemap = self._empty(mapping)
self._targets = []
for k, v in mapping.items():
self[k] = v
@staticmethod
def _empty(mapping):
m = mapping.copy()
m.clear()
return m
def __setitem__(self, k, v):
self._namemap[k] = len(self._targets)
self._targets.append(v)
def freeze(self):
d = self._empty(self._namemap)
for k, v in self._namemap.items():
d[k] = self._targets[v]
return d
class ModuleExtractor(Visitor):
def __init__(self, module, requirefunc):
super(ModuleExtractor, self).__init__()
self.module = module
self.requirefunc = requirefunc
self.result = ModuleContent()
self.scope = Scope(BASE_SCOPE)
self.declaration = None
def enter_BlockStatement(self, node):
Hoistifier(self).visit(node)
def exit_BlockStatement(self, node):
for k, v in self.scope._namemap.items():
if k not in BASE_SCOPE:
self.module.add_member(k, self.scope._targets[v])
for t in TypedefMatcher(self).visit(node):
self.module.add_member(t.name, t)
def enter_VariableDeclaration(self, node):
self.declaration = Declaration(comments=node.get('comments'))
def enter_VariableDeclarator(self, node):
# otherwise we've already hoisted the declaration so the variable
# already exist initialised to undefined, and ValueExtractor does not
# handle a None input node so it returns None which makes no sense
if node['init']:
self.declaration.id = node['id']['name']
self.scope[self.declaration.id] = ValueExtractor(
self, self.declaration
).visit(node['init'] or [])
self.declaration.id = None
return SKIP
def exit_VariableDeclaration(self, node):
self.declaration = None
# as the name denotes, AssignmentExpression is an *expression*, which
# means at the module toplevel it is wrapped into an ExpressionStatement
# which is where the comments get attached
def enter_ExpressionStatement(self, node):
self.declaration = Declaration(comments=node.get('comments'))
def exit_ExpressionStatement(self, node):
self.declaration = None
def enter_AssignmentExpression(self, node):
target = node['left']
if target['type'] == 'Identifier':
self.declaration.id = target['name']
self.scope[self.declaration.id] = ValueExtractor(
self, self.declaration
).visit(node['right'])
self.declaration.id = None
return SKIP
if target['type'] != 'MemberExpression':
raise ValueError("Unhandled assign to %s" % target['type'])
# only assign to straight a.b.c patterns (OK and trivial literals)
if target['computed'] and target['property']['type'] != 'Literal':
return SKIP
name = utils._value(target['property'], strict=True)
if isinstance(name, type(u'')) and name.endswith('.extend'):
return SKIP # ignore overwrite of .extend (WTF)
self.declaration.id = name
it = ValueExtractor(
self, self.declaration
).visit(node['right'])
self.declaration.id = None
assert it, "assigned a non-value from %s to %s" % (node['right'], name)
@self.result.post.append
def _augment_module(modules):
try:
t = deref(m2r(target['object'], self.scope.freeze()))
except ValueError:
return # f'n extension of global libraries garbage
if not isinstance(t, jsdoc.NSDoc):
# function Foo(){}; Foo.prototype = bar
# fuck that yo
return
# TODO: note which module added this
m = it
if isinstance(m, jsdoc.LiteralDoc):
m = jsdoc.PropertyDoc(m.to_dict())
t.add_member(name, m)
return SKIP
def enter_FunctionDeclaration(self, node):
""" Already processed by hoistitifier
"""
return SKIP
def enter_ReturnStatement(self, node):
self.declaration = Declaration(comments=node.get('comments'))
if node['argument']:
export = ValueExtractor(self, self.declaration).visit(node['argument'])
if isinstance(export, RefProxy):
self.module.parsed['exports'] = _name(export._ref)
self.scope['<exports>'] = export
self.declaration = None
return SKIP
def enter_CallExpression(self, node):
if utils.match(node, {
'callee': {
'type': 'MemberExpression',
'object': lambda n: (
n['type'] in ('Identifier', 'MemberExpression')
# _.str.include
and not utils._name(n).startswith('_')
),
'property': {'name': 'include'},
}
}):
target = RefProxy(m2r(node['callee']['object'], self.scope.freeze()))
target_name = utils._name(node['callee']['object'])
items = ClassProcessor(self).visit(node['arguments'])
@self.result.post.append
def resolve_extension(modules):
t = target.become(modules)
if not isinstance(t, jsdoc.ClassDoc):
return # FIXME: log warning
# raise ValueError("include() subjects should be classes, %s is %s" % (target_name, type(t)))
# TODO: note which module added these
for it in items:
if isinstance(it, dict):
for n, member in it.items():
t.add_member(n, member)
else:
t.parsed.setdefault('mixes', []).append(it.become(modules))
return SKIP
def refify(self, node, also=None):
it = m2r(node, self.scope.freeze())
assert isinstance(it, ref), "Expected ref, got {}".format(it)
px = RefProxy(it)
@self.result.post.append
def resolve(modules):
p = px.become(modules)
if also: also(p)
return px
class ValueExtractor(Visitor):
def __init__(self, parent, declaration=None):
super(ValueExtractor, self).__init__()
self.parent = parent
self.declaration = declaration or Declaration()
def enter_generic(self, node):
self.result = jsdoc.parse_comments(
self.declaration.comments,
jsdoc.Unknown.from_(node['type'])
)
self._update_result_meta()
return SKIP
def _update_result_meta(self, name=None):
self.result.parsed['sourcemodule'] = self.parent.module
n = name or self.declaration.id
if n:
self.result.set_name(n)
def enter_Literal(self, node):
self.result = jsdoc.parse_comments(
self.declaration.comments, jsdoc.LiteralDoc)
self._update_result_meta()
self.result.parsed['value'] = node['value']
return SKIP
def enter_Identifier(self, node):
self.result = self.parent.refify(node)
return SKIP
def enter_MemberExpression(self, node):
self.result = RefProxy(ref(
ValueExtractor(self.parent).visit(node['object']),
utils._value(node['property'], strict=True)
))
self.parent.result.post.append(self.result.become)
return SKIP
def enter_FunctionExpression(self, node):
name, comments = (self.declaration.id, self.declaration.comments)
self.result = jsdoc.parse_comments(comments, jsdoc.FunctionDoc)
self.result.parsed['name'] = node['id'] and node['id']['name']
self._update_result_meta()
self.result.parsed['guessed_params'] = [p['name'] for p in node['params']]
return SKIP
def enter_NewExpression(self, node):
comments = self.declaration.comments if self.declaration else node.get('comments')
self.result = ns = jsdoc.parse_comments(comments, jsdoc.InstanceDoc)
self._update_result_meta()
def _update_contents(cls):
if not isinstance(cls, jsdoc.ClassDoc):
return
ns.parsed['cls'] = cls
self.parent.refify(node['callee'], also=_update_contents)
return SKIP
def enter_ObjectExpression(self, node):
self.result = obj = jsdoc.parse_comments(self.declaration.comments)
self._update_result_meta()
for n, p in MemberExtractor(parent=self.parent).visit(node['properties']).items():
obj.add_member(n, p)
return SKIP
def enter_CallExpression(self, node):
# require(a_module_name)
if utils.match(node, {'callee': {'type': 'Identifier', 'name': self.parent.requirefunc}}):
depname = node['arguments'][0]['value']
# TODO: clean this up
self.parent.result.dependencies.add(depname)
self.result = ModuleProxy(name=depname)
self.parent.result.post.append(self.result.become)
# Class.extend(..mixins, {})
elif utils.match(node, {
'callee': {
'type': 'MemberExpression',
'object': lambda n: (
n['type'] in ('Identifier', 'MemberExpression')
# $.extend, $.fn.extend, _.extend
and not utils._name(n).startswith(('_', '$'))
),
'property': {'name': 'extend'},
},
}): # creates a new class, but may not actually return it
obj = node['callee']['object']
comments = self.declaration.comments
self.result = cls = jsdoc.parse_comments(comments, jsdoc.ClassDoc)
cls.parsed['extends'] = self.parent.refify(obj)
self._update_result_meta()
items = ClassProcessor(self.parent).visit(node['arguments'])
@self.parent.result.post.append
def add_to_class(modules):
for item in items:
if isinstance(item, dict):
# methods/attributes
for n, method in item.items():
cls.add_member(n, method)
else:
cls.parsed.setdefault('mixes', []).append(item)
# other function calls
else:
self.result = jsdoc.parse_comments(self.declaration.comments, jsdoc.guess)
self._update_result_meta()
return SKIP
class ClassProcessor(Visitor):
def __init__(self, parent):
super(ClassProcessor, self).__init__()
self.result = []
self.parent = parent
def enter_generic(self, node):
self.result.append(self.parent.refify(node))
return SKIP
def enter_ObjectExpression(self, node):
self.result.append(MemberExtractor(parent=self.parent, for_class=True).visit(node['properties']))
return SKIP
String = type(u'')
class MemberExtractor(Visitor):
def __init__(self, parent, for_class=False):
super(MemberExtractor, self).__init__()
self.result = collections.OrderedDict()
self.parent = parent
self.for_class = for_class
def enter_Property(self, node):
name = utils._value(node['key'])
prop = ValueExtractor(
self.parent,
Declaration(id=name, comments=node.get('comments'))
).visit(node['value'])
if isinstance(prop, jsdoc.LiteralDoc):
prop = jsdoc.PropertyDoc(prop.to_dict())
# ValueExtractor can return a Ref, maybe this should be sent as part
# of the decl/comments?
if name.startswith('_') and hasattr(prop, 'parsed'):
prop.parsed['private'] = True
self.result[name] = prop
return SKIP
class Hoistifier(Visitor):
"""
Processor for variable and function declarations properly hoisting them
to the "top" of a module such that they are available with the relevant
value afterwards.
"""
def __init__(self, parent):
super(Hoistifier, self).__init__()
self.parent = parent
def enter_generic(self, node):
return SKIP
# nodes to straight recurse into, others are just skipped
enter_BlockStatement = enter_VariableDeclaration = lambda self, node: None
def enter_VariableDeclarator(self, node):
self.parent.scope[node['id']['name']] = BASE_SCOPE['undefined']
def enter_FunctionDeclaration(self, node):
funcname = node['id']['name']
self.parent.scope[funcname] = fn = jsdoc.parse_comments(
node.get('comments'),
jsdoc.FunctionDoc,
)
fn.parsed['sourcemodule'] = self.parent.module
fn.parsed['name'] = funcname
fn.parsed['guessed_params'] = [p['name'] for p in node['params']]
return SKIP
class TypedefMatcher(Visitor):
def __init__(self, parent):
super(TypedefMatcher, self).__init__()
self.parent = parent
self.result = []
enter_BlockStatement = lambda self, node: None
def enter_generic(self, node):
# just traverse all top-level statements, check their comments, and
# bail
for comment in node.get('comments') or []:
if '@typedef' in comment['value']:
extract = '\n' + jsdoc.strip_stars('/*' + comment['value'] + '\n*/')
parsed = pyjsdoc.parse_comment(extract, u'')
p = jsdoc.ParamDoc(parsed['typedef'])
parsed['name'] = p.name
parsed['sourcemodule'] = self.parent.module
# TODO: add p.type as superclass somehow? Builtin types not in scope :(
self.result.append(jsdoc.ClassDoc(parsed))
return SKIP

View File

@ -0,0 +1,6 @@
:orphan:
These files should be run via pytest_, simply install pytest and from the top
of the Odoo project run ``pytest doc/_extensions``.
.. _pytest: https://pytest.org/

View File

@ -0,0 +1,73 @@
# -*- coding: utf-8 -*-
import operator
import pyjsparser
from autojsdoc.parser import jsdoc, parser
params = operator.attrgetter('name', 'type', 'doc')
def parse(s, source=None):
tree = pyjsparser.parse(s)
mods = parser.ModuleMatcher(source).visit(tree)
post(mods)
return mods
def post(mods):
modules = dict(BASE_MODULES)
modules.update((m.name, m) for m in mods)
for mod in mods:
mod.post_process(modules)
BASE_MODULES = {
'other': jsdoc.ModuleDoc({
'module': 'other',
'_members': [
('<exports>', jsdoc.LiteralDoc({'name': 'value', 'value': "ok"})),
],
}),
'dep2': jsdoc.ModuleDoc({
'module': 'dep2',
'_members': [
('<exports>', jsdoc.LiteralDoc({'value': 42.})),
],
}),
'dep3': jsdoc.ModuleDoc({
'module': 'dep3',
'_members': [
('<exports>', jsdoc.LiteralDoc({'value': 56.})),
],
}),
'Class': jsdoc.ModuleDoc({
'module': 'Class',
'_members': [
('<exports>', jsdoc.ClassDoc({
'name': 'Class',
'doc': "Base Class"
})),
],
}),
'mixins': jsdoc.ModuleDoc({
'module': 'mixins',
'_members': [
('<exports>', jsdoc.NSDoc({
'name': 'mixins',
'_members': [
('Bob', jsdoc.ClassDoc({'class': "Bob"})),
]
})),
],
}),
'Mixin': jsdoc.ModuleDoc({
'module': 'Mixin',
'_members': [
('<exports>', jsdoc.MixinDoc({
'_members': [
('a', jsdoc.FunctionDoc({'function': 'a'})),
]
})),
],
})
}

View File

@ -0,0 +1,227 @@
# -*- coding: utf-8 -*-
import pytest
from autojsdoc.parser import jsdoc
from support import params, parse
def test_classvar():
[mod] = parse("""
odoo.define('a.A', function(require) {
var Class = require('Class');
/**
* This is my class-kai
*/
var A = Class.extend({});
return A;
});
""")
cls = mod.exports
assert type(cls) == jsdoc.ClassDoc
assert type(cls.superclass) == jsdoc.ClassDoc
assert cls.superclass.name == 'Class'
assert cls.name == 'A'
assert cls.constructor is None
assert cls.properties == []
assert cls['doc'] == 'This is my class-kai'
def test_classret():
[mod] = parse("""
odoo.define('a.A', function(require) {
var Class = require('Class');
/**
* This is my class-kai
*/
return Class.extend({});
});
""")
cls = mod.exports
assert type(cls) == jsdoc.ClassDoc
assert cls.name == ''
assert cls.constructor is None
assert cls.properties == []
assert cls['doc'] == 'This is my class-kai'
def test_methods():
[mod] = parse("""
odoo.define('a.A', function(require) {
var Class = require('Class');
return Class.extend({
/**
* @param {Widget} parent
*/
init: function (parent) {},
/**
* @returns {Widget}
*/
itself: function () { return this; },
/**
* @param {MouseEvent} e
*/
_onValidate: function (e) {},
});
});
""")
cls = mod.exports
assert len(cls.properties) == 3
assert cls.constructor
# assume methods are in source order
[_, init] = cls.properties[0]
assert init == cls.constructor
assert init.name == 'init'
assert not init.is_private
assert init.is_constructor
[param] = init.params
assert params(param) == ('parent', 'Widget', '')
[_, itself] = cls.properties[1]
assert itself.name == 'itself'
assert not itself.is_private
assert not itself.is_constructor
assert not itself.params
assert params(itself.return_val) == ('', 'Widget', '')
[_, _on] = cls.properties[2]
assert _on.name == '_onValidate'
assert _on.is_private
assert not _on.is_constructor
[param] = _on.params
assert params(param) == ('e', 'MouseEvent', '')
def test_mixin_explicit():
[mod] = parse("""
odoo.define('a.A', function (require) {
var Class = require('Class');
var mixins = require('mixins');
/**
* This is my class-kai
* @mixes mixins.Bob
*/
return Class.extend({});
});
""")
cls = mod.exports
# FIXME: ClassDoc may want to m2r(mixin, scope)?
assert cls.mixins == ['mixins.Bob']
def test_mixin_implicit():
[mod] = parse("""
odoo.define('a.A', function(require) {
var Class = require('Class');
var Mixin = require('Mixin');
/**
* This is my class-kai
*/
return Class.extend(Mixin, { foo: function() {} });
});
""")
cls = mod.exports
[mixin] = cls.mixins
assert type(mixin) == jsdoc.MixinDoc
assert params(mixin.properties[0][1]) == ('a', 'Function', '')
assert params(mixin.get_property('a')) == ('a', 'Function', '')
assert params(cls.get_property('foo')) == ('foo', 'Function', '')
def test_instanciation():
[A, a] = parse("""
odoo.define('A', function (r) {
var Class = r('Class');
/**
* @class A
*/
return Class.extend({
foo: function () {}
});
});
odoo.define('a', function (r) {
var A = r('A');
var a = new A;
return a;
});
""")
assert type(a.exports) == jsdoc.InstanceDoc
assert a.exports.cls.name == A.exports.name
def test_non_function_properties():
[A] = parse("""
odoo.define('A', function (r) {
var Class = r('Class');
return Class.extend({
template: 'thing',
a_prop: [1, 2, 3],
'other': {a: 7}
});
});
""")
t = A.exports.get_property('template')
assert type(t) == jsdoc.PropertyDoc
assert params(t) == ('template', 'String', '')
assert not t.is_private
def test_non_extend_classes():
[mod] = parse("""
odoo.define('A', function () {
/**
* @class
*/
var Class = function () {}
return Class;
});
""")
assert type(mod.exports) == jsdoc.ClassDoc
def test_extend():
[a, _] = parse("""
odoo.define('A', function (require) {
var Class = require('Class');
return Class.extend({});
});
odoo.define('B', function (require) {
var A = require('A');
A.include({
/** A property */
a: 3,
/** A method */
b: function () {}
});
});
""")
cls = a.exports
assert type(cls) == jsdoc.ClassDoc
a = cls.get_property('a')
assert type(a) == jsdoc.PropertyDoc
assert params(a) == ('a', 'Number', 'A property')
b = cls.get_property('b')
assert type(b) == jsdoc.FunctionDoc
assert params(b) == ('b', 'Function', 'A method')
# TODO: also support virtual members?
# TODO: computed properties?
@pytest.mark.skip(reason="Need to implement member/var-parsing?")
def test_members():
[mod] = parse("""
odoo.define('A', function (r) {
var Class = r('Class');
return Class.extend({
init: function () {
/**
* This is bob
* @var {Foo}
*/
this.foo = 3;
this.bar = 42;
/**
* @member {Baz}
*/
this.baz = null;
}
});
});
""")
cls = mod.exports
assert params(cls.members[0]) == ('foo', 'Foo', 'This is bob')
assert params(cls.members[1]) == ('bar', '', '')
assert params(cls.members[2]) == ('baz', 'Baz', '')

View File

@ -0,0 +1,75 @@
# -*- coding: utf-8 -*-
"""
Test various crap patterns found in Odoo code to ensure they don't blow up
the parser thingie
"""
from autojsdoc.parser import jsdoc
from support import parse
def test_export_external():
[mod] = parse("""
odoo.define('module', function () {
return $.Deferred().reject();
});
""")
assert isinstance(mod.exports, jsdoc.CommentDoc)
assert mod.exports.doc == ''
def test_extend_jq():
parse("""
odoo.define('a', function (r) {
$.extend($.expr[':'], { a: function () {} });
$.fn.extend({ a: function () {} });
});
""")
def test_extend_dynamic():
parse("""
odoo.define('a', function () {
foo.bar.baz[qux + '_external'] = function () {};
});
""")
def test_extend_deep():
parse("""
odoo.define('a', function () {
var eventHandler = $.summernote.eventHandler;
var dom = $.summernote.core.dom;
dom.thing = function () {};
var fn_editor_currentstyle = eventHandler.modules.editor.currentStyle;
eventHandler.modules.editor.currentStyle = function () {}
});
""")
def test_arbitrary():
parse("""
odoo.define('bob', function () {
var page = window.location.href.replace(/^.*\/\/[^\/]+/, '');
var mailWidgets = ['mail_followers', 'mail_thread', 'mail_activity', 'kanban_activity'];
var bob;
var fldj = foo.getTemplate().baz;
});
""")
def test_prototype():
[A, B] = parse("""
odoo.define('mod1', function () {
var exports = {};
exports.Foo = Backbone.Model.extend({});
exports.Bar = Backbone.Model.extend({});
var BarCollection = Backbone.Collection.extend({
model: exports.Bar,
});
exports.Baz = Backbone.Model.extend({});
return exports;
});
odoo.define('mod2', function (require) {
var models = require('mod1');
var _super_orderline = models.Bar.prototype;
models.Foo = models.Bar.extend({});
var _super_order = models.Baz.prototype;
models.Bar = models.Baz.extend({});
});
""")

View File

@ -0,0 +1,173 @@
# -*- coding: utf-8 -*-
from autojsdoc.parser import jsdoc
from support import params, parse, BASE_MODULES
def test_single():
[mod] = parse("""
/**
* This is a super module!
*/
odoo.define('supermodule', function (req) {
var other = req('other');
});
""")
assert mod.name == 'supermodule'
assert mod.dependencies == {'other'}
assert mod.exports is None
assert mod.doc == "This is a super module!"
m = mod.get_property('other')
assert m.name == 'value', "property is exported value not imported module"
def test_multiple():
[mod1, mod2, mod3] = parse("""
odoo.define('module1', function (req) {
return 1;
});
odoo.define('module2', function (req) {
return req('dep2');
});
odoo.define('module3', function (req) {
var r = req('dep3');
return r;
});
""")
assert mod1.name == 'module1'
assert mod1.dependencies == set()
assert isinstance(mod1.exports, jsdoc.LiteralDoc)
assert mod1.exports.value == 1.0
assert mod1.exports['sourcemodule'] is mod1
assert mod1.doc == ""
assert mod2.name == 'module2'
assert mod2.dependencies == {'dep2'}
assert isinstance(mod2.exports, jsdoc.LiteralDoc)
assert mod2.exports.value == 42.0
assert mod2.doc == ""
assert mod3.name == 'module3'
assert mod3.dependencies == {'dep3'}
assert isinstance(mod3.exports, jsdoc.LiteralDoc)
assert mod3.exports.value == 56.0
assert mod3.doc == ''
def test_func():
[mod] = parse("""
odoo.define('module', function (d) {
/**
* @param {Foo} bar this is a bar
* @param {Baz} qux this is a qux
*/
return function (bar, qux) {
return 42;
}
});
""")
exports = mod.exports
assert type(exports) == jsdoc.FunctionDoc
assert exports['sourcemodule'] is mod
assert exports.name == ''
assert exports.is_constructor == False
assert exports.is_private == False
assert params(exports.params[0]) == ('bar', 'Foo', 'this is a bar')
assert params(exports.params[1]) == ('qux', 'Baz', 'this is a qux')
assert params(exports.return_val) == ('', '', '')
def test_hoist():
[mod] = parse("""
odoo.define('module', function() {
return foo;
/**
* @param a_thing
*/
function foo(a_thing) {
return 42;
}
});
""")
actual = mod.exports
assert type(actual) == jsdoc.FunctionDoc
[param] = actual.params
assert params(param) == ('a_thing', '', '')
def test_export_instance():
[mod] = parse("""
odoo.define('module', function (require) {
var Class = require('Class');
/**
* Provides an instance of Class
*/
return new Class();
});
""")
assert type(mod.exports) == jsdoc.InstanceDoc
assert mod.exports.doc == 'Provides an instance of Class'
assert mod.exports['sourcemodule'] is mod
def test_bounce():
[m2, m1] = parse("""
odoo.define('m2', function (require) {
var Item = require('m1');
return {
Item: Item
};
});
odoo.define('m1', function (require) {
var Class = require('Class');
var Item = Class.extend({});
return Item;
});
""")
assert type(m2.exports) == jsdoc.NSDoc
it = m2.exports.get_property('Item')
assert type(it) == jsdoc.ClassDoc
assert it['sourcemodule'] is m1
assert sorted([n for n, _ in m1.properties]) == ['<exports>', 'Class', 'Item']
def test_reassign():
[m] = parse("""
odoo.define('m', function (require) {
var Class = require('Class');
/** local class */
var Class = Class.extend({});
return Class
});
""")
assert m.exports.doc == 'local class'
# can't use equality or identity so use class comment...
assert m.exports.superclass.doc == 'Base Class'
def test_attr():
[m1, m2] = parse("""
odoo.define('m1', function (require) {
var Class = require('Class');
var Item = Class.extend({});
return {Item: Item};
});
odoo.define('m2', function (require) {
var Item = require('m1').Item;
Item.include({});
return Item.extend({});
});
""")
assert type(m2.exports) == jsdoc.ClassDoc
# that these two are resolved separately may be an issue at one point (?)
assert m2.exports.superclass.to_dict() == m1.exports.get_property('Item').to_dict()
def test_nothing_implicit():
[m] = parse("""
odoo.define('m', function () {
});
""")
assert m.exports is None
def test_nothing_explicit():
[m] = parse("""
odoo.define('m', function () {
return;
});
""")
assert m.exports is None

View File

@ -0,0 +1,177 @@
# -*- coding: utf-8 -*-
from autojsdoc.parser import jsdoc
from support import parse, params
def test_empty():
[mod] = parse("""
odoo.define('a.ns', function (r) {
return {};
});
""")
assert type(mod.exports) == jsdoc.NSDoc
assert mod.exports.properties == []
def test_inline():
[mod] = parse("""
odoo.define('a.ns', function (r) {
return {
/**
* a thing
* @type {Boolean}
*/
a: true
}
});
""")
assert isinstance(mod.exports, jsdoc.NSDoc)
[(n, p)] = mod.exports.properties
assert n == 'a'
assert params(p) == ('a', 'Boolean', 'a thing')
def test_header():
[mod] = parse("""
odoo.define('a.ns', function (r) {
/**
* @property {Boolean} a a thing
*/
return { a: true }
});
""")
assert isinstance(mod.exports, jsdoc.NSDoc)
[(_, p)] = mod.exports.properties
assert params(p) == ('a', 'Boolean', 'a thing')
def test_header_conflict():
""" should the header or the inline comment take precedence? """
[mod] = parse("""
odoo.define('a.ns', function (r) {
/**
* @property {Boolean} a a thing
*/
return {
/** @type {String} */
a: true
}
});
""")
assert isinstance(mod.exports, jsdoc.NSDoc)
[(_, p)] = mod.exports.properties
assert params(p) == ('a', 'Boolean', 'a thing')
def test_mixin():
[mod] = parse("""
odoo.define('a.mixin', function (r) {
/**
* @mixin
*/
return {
/**
* @returns {Number} a number
*/
do_thing: function other() { return 42; }
}
});
""")
assert isinstance(mod.exports, jsdoc.MixinDoc)
[(n, p)] = mod.exports.properties
assert n == 'do_thing'
assert params(p) == ('other', 'Function', '')
assert params(p.return_val) == ('', 'Number', 'a number')
def test_literal():
[mod] = parse("""
odoo.define('a.ns', function (r) {
/** whop whop */
return {
'a': 1,
/** wheee */
'b': 2,
};
});
""")
assert mod.exports.doc == 'whop whop'
[(_1, a), (_2, b)] = mod.exports.properties
assert params(a) == ('a', 'Number', '')
assert params(b) == ('b', 'Number', 'wheee')
def test_fill_ns():
[mod] = parse("""
odoo.define('a.ns', function (r) {
var Class = r('Class');
var ns = {};
/** ok */
ns.a = 1;
/** @type {String} */
ns['b'] = 2;
/** Ike */
ns.c = Class.extend({});
ns.d = function () {}
return ns;
});
""")
ns = mod.exports
assert type(ns) == jsdoc.NSDoc
[(_a, a), (_b, b), (_c, c), (_d, d)] = ns.properties
assert params(a) == ('a', 'Number', 'ok')
assert params(b) == ('b', 'String', '')
assert type(c) == jsdoc.ClassDoc
assert type(d) == jsdoc.FunctionDoc
def test_extend_other():
[o, b] = parse("""
odoo.define('a.ns', function () {
/** @name outer */
return {
/** @name inner */
a: {}
};
});
odoo.define('b', function (r) {
var o = r('a.ns');
var Class = r('Class');
/** Class 1 */
o.a.b = Class.extend({m_b: 1});
/** Class 2 */
o.a['c'] = Class.extend({m_c: 1});
});
""")
[(_, m)] = o.exports.properties
assert type(m) == jsdoc.NSDoc
b = m.get_property('b')
assert type(b) == jsdoc.ClassDoc
assert b.get_property('m_b')
assert b.doc == 'Class 1'
c = m.get_property('c')
assert type(c) == jsdoc.ClassDoc
assert c.get_property('m_c')
assert c.doc == 'Class 2'
def test_ns_variables():
[mod] = parse("""
odoo.define('A', function (r) {
var Class = r('Class');
var Thing = Class.extend({});
return {
Thing: Thing
};
});
""")
p = mod.exports.get_property('Thing')
assert type(p) == jsdoc.ClassDoc
def test_diff():
""" Have the NS key and the underlying object differ
"""
[mod] = parse("""
odoo.define('mod', function (r) {
var Class = r('Class');
var Foo = Class.extend({});
return { Class: Foo };
});
""")
c = mod.exports.get_property('Class')
assert type(c) == jsdoc.ClassDoc
assert c.name == 'Foo'

View File

@ -0,0 +1,116 @@
# -*- coding: utf-8 -*-
from autojsdoc.parser.jsdoc import ParamDoc
"Lorem ipsum dolor sit amet, consectetur adipiscing elit."
def test_basic():
d = ParamDoc("Lorem ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d == {
'name': 'Lorem',
'type': '',
'optional': False,
'default': None,
'doc': 'ipsum dolor sit amet, consectetur adipiscing elit.',
}
d = ParamDoc("{Lorem} ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d == {
'name': 'ipsum',
'type': 'Lorem',
'optional': False,
'default': None,
'doc': 'dolor sit amet, consectetur adipiscing elit.',
}
def test_optional():
d = ParamDoc("[Lorem] ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d == {
'name': 'Lorem',
'type': '',
'optional': True,
'default': None,
'doc': 'ipsum dolor sit amet, consectetur adipiscing elit.',
}
d = ParamDoc("[Lorem=42] ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d == {
'name': 'Lorem',
'type': '',
'optional': True,
'default': "42",
'doc': 'ipsum dolor sit amet, consectetur adipiscing elit.',
}
d = ParamDoc("{Lorem} [ipsum] dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d == {
'name': 'ipsum',
'type': 'Lorem',
'optional': True,
'default': None,
'doc': 'dolor sit amet, consectetur adipiscing elit.',
}
d = ParamDoc("{Lorem} [ipsum=42] dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d == {
'name': 'ipsum',
'type': 'Lorem',
'optional': True,
'default': '42',
'doc': 'dolor sit amet, consectetur adipiscing elit.',
}
def test_returns():
d = ParamDoc("{} ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d == {
'name': '',
'type': '',
'optional': False,
'default': None,
'doc': 'ipsum dolor sit amet, consectetur adipiscing elit.',
}
d = ParamDoc("{Lorem} ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d == {
'name': '',
'type': 'Lorem',
'optional': False,
'default': None,
'doc': 'ipsum dolor sit amet, consectetur adipiscing elit.',
}
def test_odd():
d = ParamDoc("{jQuery} [$target] the node where content will be prepended").to_dict()
assert d == {
'name': '$target',
'type': 'jQuery',
'optional': True,
'default': None,
'doc': 'the node where content will be prepended',
}
d = ParamDoc("""{htmlString} [content] DOM element,
array of elements, HTML string or jQuery object to prepend to $target""").to_dict()
assert d == {
'name': 'content',
'type': 'htmlString',
'optional': True,
'default': None,
'doc': "DOM element,\n array of elements, HTML string or jQuery object to prepend to $target",
}
d = ParamDoc("{Boolean} [options.in_DOM] true if $target is in the DOM").to_dict()
assert d == {
'name': 'options.in_DOM',
'type': 'Boolean',
'optional': True,
'default': None,
'doc': 'true if $target is in the DOM',
}
d = ParamDoc("Lorem\n ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d['doc'] == 'ipsum dolor sit amet, consectetur adipiscing elit.'
d = ParamDoc("Lorem - ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d['doc'] == 'ipsum dolor sit amet, consectetur adipiscing elit.'
d = ParamDoc("Lorem : ipsum dolor sit amet, consectetur adipiscing elit.").to_dict()
assert d['doc'] == 'ipsum dolor sit amet, consectetur adipiscing elit.'

View File

@ -0,0 +1,104 @@
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
import pytest
from autojsdoc.parser import types
def test_parser():
assert types.parse("MouseEvent|TouchEvent") == types.Alt([
types.Type('MouseEvent', []),
types.Type('TouchEvent', []),
])
assert types.parse('Deferred<Object>') == types.Alt([
types.Type('Deferred', [
types.Alt([types.Type('Object', [])])
])
])
assert types.parse('Object|undefined') == types.Alt([
types.Type('Object', []),
types.Literal('undefined'),
])
assert types.parse('any[]') == types.Alt([
types.Type('Array', [types.Type('any', [])])
])
assert types.parse("'xml' | 'less'") == types.Alt([
types.Literal('xml'),
types.Literal('less'),
])
assert types.parse('Foo<Bar | Bar[] | null>') == types.Alt([
types.Type('Foo', [types.Alt([
types.Type('Bar', []),
types.Type('Array', [types.Type('Bar', [])]),
types.Literal('null'),
])])
])
assert types.parse('Function<Array<Object[]>>') == types.Alt([
types.Type('Function', [types.Alt([
types.Type('Array', [types.Alt([
types.Type('Array', [
types.Type('Object', [])
])
])])
])])
])
def test_tokens():
toks = list(types.tokenize('A'))
assert toks == [(types.NAME, 'A')]
toks = list(types.tokenize('"foo" | "bar"'))
assert toks == [(types.LITERAL, 'foo'), (types.OP, '|'), (types.LITERAL, 'bar')]
toks = list(types.tokenize('1 or 2'))
assert toks == [(types.LITERAL, 1), (types.OP, '|'), (types.LITERAL, 2)]
toks = list(types.tokenize('a.b.c.d'))
assert toks == [
(types.NAME, 'a'), (types.OP, '.'),
(types.NAME, 'b'), (types.OP, '.'),
(types.NAME, 'c'), (types.OP, '.'),
(types.NAME, 'd'),
]
toks = list(types.tokenize('Function<String, Object>'))
assert toks == [
(types.NAME, 'Function'),
(types.OP, '<'),
(types.NAME, 'String'),
(types.OP, ','),
(types.NAME, 'Object'),
(types.OP, '>')
]
toks = list(types.tokenize('Function<Array<Object[]>>'))
assert toks == [
(types.NAME, 'Function'),
(types.OP, '<'),
(types.NAME, 'Array'),
(types.OP, '<'),
(types.NAME, 'Object'),
(types.OP, '['),
(types.OP, ']'),
(types.OP, '>'),
(types.OP, '>')
]
def test_peekable():
p = types.Peekable(range(5))
assert p.peek() == 0
assert next(p) == 0, "consuming should yield previously peeked value"
next(p)
next(p)
assert next(p) == 3
assert p.peek() == 4
next(p)
assert p.peek(None) is None
with pytest.raises(StopIteration):
p.peek()
with pytest.raises(StopIteration):
next(p)

View File

@ -0,0 +1,196 @@
# -*- coding: utf-8 -*-
import ast
import collections
import io
import token
from tokenize import generate_tokens
LITERAL_NAME = ('null', 'undefined', 'true', 'false')
__all__ = ['tokenize', 'parse', 'iterate', 'Alt', 'Type', 'Literal']
class _Marker(object):
__slots__ = ['name']
def __init__(self, name): self.name = name
def __repr__(self): return '<%s>' % self.name
OP = _Marker('OP')
NAME = _Marker('NAME')
LITERAL = _Marker('LITERAL')
def tokenize(typespec):
typespec = typespec.replace('$', 'jQuery')
for toktype, string, _, _, _ in generate_tokens(io.StringIO(typespec).readline):
if toktype == token.NAME:
if string == 'or': # special case "A or B"
# TODO: deprecation warning
yield (OP, '|')
else:
yield (NAME, string)
elif toktype == token.OP:
if string in '|<>[].,':
yield (OP, string)
elif string == '>>':
yield (OP, '>')
yield (OP, '>')
elif string == '*': # maybe?
yield (NAME, 'any')
elif string in '()':
# TODO: deprecation warning
# seems useless unless we add support for "..."
continue
else:
raise ValueError("Unknown typespec operator %r" % string)
elif toktype in (token.STRING, token.NUMBER): # enum-ish
yield (LITERAL, ast.literal_eval(string))
elif toktype == token.ENDMARKER:
return
elif toktype == token.NEWLINE:
pass
else:
raise ValueError("Unknown typespec token %s" % token.tok_name[toktype])
Alt = collections.namedtuple('Alt', ['params'])
Literal = collections.namedtuple('Literal', ['value'])
Type = collections.namedtuple('Type', ['name', 'params'])
def iterate(t, fortype, forliteral):
if isinstance(t, Alt):
ps = Peekable(t.params)
for param in ps:
for it in iterate(param, fortype, forliteral):
yield it
if ps.peek(None):
yield forliteral('|')
elif isinstance(t, Type):
yield fortype(t.name)
if not t.params:
return
yield forliteral('<')
ps = Peekable(t.params)
for param in ps:
for it in iterate(param, fortype, forliteral):
yield it
if ps.peek(None):
yield forliteral(',')
yield forliteral('>')
elif isinstance(t, Literal):
yield forliteral(t.value)
else:
raise ValueError('Unknown item %s' % t)
def parse(typespec):
tokens = Peekable(tokenize(typespec))
return parse_alt(tokens)
# alt = param
# | param '|' alt
def parse_alt(tokens):
alt = Alt([])
while True:
alt.params.append(parse_param(tokens))
if not next_is(tokens, (OP, '|')):
break
return alt
# param = 'null' | 'undefined' | 'true' | 'false'
# | number
# | string
# | typename '[]'
# X '[' alt ']'
# | typename '<' params '>'
# params = alt | alt ',' params
def parse_param(tokens):
t, v = tokens.peek()
if t == LITERAL or (t == NAME and v in LITERAL_NAME):
next(tokens)
return Literal(str(v))
# # [typespec] # should this be Array<T> or an n-uple allowing multiple items?
# if tok == (OP, '['):
# next(tokens)
# t = Type('Array', [parse_alt(tokens)])
# rest = next(tokens)
# expect(rest, (OP, ']'))
# return t
t = Type(parse_typename(tokens), [])
# type[]
peek = tokens.peek(None)
if peek == (OP, '['):
next(tokens) # skip peeked
expect(next(tokens), (OP, ']'))
return Type('Array', [t])
# type<typespec, ...>
if peek == (OP, '<'):
next(tokens) # skip peeked
while True:
t.params.append(parse_alt(tokens))
n = next(tokens)
if n == (OP, ','):
continue
if n == (OP, '>'):
break
raise ValueError("Expected OP ',' or OP ',', got %s '%s'" % n)
return t
# typename = name | name '.' typename
def parse_typename(tokens):
typename = []
while True:
(t, n) = next(tokens)
if t != NAME:
raise ValueError("Expected a name token, got %s '%s'" % (t, n))
if n in LITERAL_NAME:
raise ValueError("Expected a type name, got literal %s" % n)
typename.append(n)
if not next_is(tokens, (OP, '.')):
break
return '.'.join(typename)
def next_is(tokens, expected):
"""
Consumes the next token if it's `expected` otherwise does not touch the
tokens.
Returns whether it consumed a token
"""
if tokens.peek(None) == expected:
next(tokens)
return True
return False
def expect(actual, expected):
""" Raises ValueError if `actual` and `expected` are different
:type actual: (object, str)
:type expected: (object, str)
"""
if actual != expected:
raise ValueError("Expected %s '%s', got %s '%s'" % (expected + actual))
NONE = object()
class Peekable(object):
__slots__ = ['_head', '_it']
def __init__(self, iterable):
self._head = NONE
self._it = iter(iterable)
def __iter__(self):
return self
def __next__(self):
if self._head is not NONE:
r, self._head = self._head, NONE
return r
return next(self._it)
next = __next__
def peek(self, default=NONE):
if self._head is NONE:
try:
self._head = next(self._it)
except StopIteration:
if default is not NONE:
return default
raise
return self._head

View File

@ -0,0 +1,295 @@
# -*- coding: utf-8 -*-
def _name(node):
if node['type'] == 'Identifier':
return node['name']
if node['type'] == 'MemberExpression':
return "%s.%s" % (_name(node['object']), _name(node['property']))
raise ValueError("Unnamable node type %s" % node['type'])
def _value(node, strict=False):
t = node['type']
if t == 'Identifier':
return node['name']
elif t == 'Literal':
return node['value']
msg = '<%s has no value>' % _identify(node, {})
if strict:
raise ValueError(msg)
return msg
def _identify(valnode, ns):
if valnode is None:
return "None"
# already identified and re-set in the ns?
if isinstance(valnode, Printable):
return valnode
# check other non-empty returns
t = valnode['type']
if t == "Literal":
if valnode.get('regex'):
return valnode['regex']['pattern']
return valnode['value']
elif t == "Identifier":
n = valnode['name']
return ns.get(n, Global(n))
elif t == "NewExpression":
return Instance(_identify(valnode['callee'], ns))
elif t == "ObjectExpression":
return Namespace({
_value(prop['key']): _identify(prop['value'], ns)
for prop in valnode['properties']
})
elif t == "MemberExpression":
return Deref(
_identify(valnode['object'], ns),
_identify(valnode['property'], ns) if valnode['computed'] else valnode['property']['name'],
)
elif t == "CallExpression":
return Call(
_identify(valnode['callee'], ns),
[_identify(arg, ns) for arg in valnode['arguments']],
)
elif t == 'BinaryExpression':
return "%s %s %s" % (
_identify(valnode['left'], ns),
valnode['operator'],
_identify(valnode['right'], ns),
)
else:
return t
class Counter(object):
__slots__ = ['_number']
def __init__(self):
self._number = 0
def __bool__(self):
return bool(self._number)
__nonzero__ = __bool__
def __int__(self):
return self._number
def get(self):
return self._number
def increment(self):
self._number += 1
return self._number
def decrement(self):
self._number -= 1
return self._number
def __enter__(self):
self._number += 1
return self._number
def __exit__(self, *args):
self._number -= 1
class Printable(object):
__slots__ = []
depth = Counter()
def __repr__(self):
with self.depth as i:
if i > 2:
return "%s(...)" % type(self).__name__
return "%s(%s)" % (type(self).__name__, ', '.join(
"%s=%r" % (k, getattr(self, k))
for k in self.__slots__
if not k.startswith('_')
if getattr(self, k)
))
def resolve(obj, store):
if obj is None:
return None
if isinstance(obj, (type(u''), bool, int, float)):
return obj
if isinstance(obj, ModuleProxy):
return resolve(obj.get(), store)
try:
if getattr(obj, 'resolved', False):
return obj
return obj.resolve(store)
except AttributeError:
raise TypeError("Unresolvable {!r}".format(obj))
class Resolvable(Printable):
"""
For types resolved in place
"""
__slots__ = ['resolved']
def __init__(self):
super(Resolvable, self).__init__()
self.resolved = False
def resolve(self, store):
self.resolved = True
return self
class Namespace(Resolvable):
__slots__ = ['attrs']
def __init__(self, attrs):
super(Namespace, self).__init__()
self.attrs = attrs
def resolve(self, store):
r = super(Namespace, self).resolve(store)
self.attrs = {
k: resolve(v, store)
for k, v in self.attrs.items()
}
return r
def __getitem__(self, key):
assert isinstance(key, type(u'')), "%r is not a namespace key" % key
try:
return self.attrs[key]
except KeyError:
return Global(self)[key]
class Module(Resolvable):
__slots__ = ('name', 'comments', 'exports', 'augments', 'dependencies', '_store')
def __init__(self, name, store, comments=()):
super(Module, self).__init__()
self.name = name
self._store = store
self.comments = tuple(comments)
self.exports = None
self.augments = []
self.dependencies = set()
store[name] = self
def add_dependency(self, depname):
dep = ModuleProxy(depname, self._store)
self.dependencies.add(dep)
return dep
def get(self):
return self
def resolve(self, store=None):
r = super(Module, self).resolve(store)
self.exports = resolve(self.exports, self._store)
self.augments = [resolve(a, self._store) for a in self.augments]
self.dependencies = {
resolve(d, self._store)
for d in self.dependencies
}
return r
def __getitem__(self, k):
try:
return self.exports[k]
except KeyError:
return Global(self)[k]
class ModuleProxy(object):
def __init__(self, name, store):
self.name = name
self.store = store
def get(self):
# web.web_client is not an actual module
return self.store.get(self.name, '<unstored %s>' % self.name)
def __repr__(self):
return repr(self.get())
def __hash__(self):
return hash(self.name)
def __eq__(self, other):
return self.name == other.name
class Class(Resolvable):
__slots__ = ['name', 'members', 'extends', 'mixins', 'comments']
def __init__(self, name, extends):
super(Class, self).__init__()
self.name = name
self.extends = extends
self.members = {}
self.mixins = []
self.comments = []
def resolve(self, store):
r = super(Class, self).resolve(store)
self.extends = resolve(self.extends, store)
self.mixins = [resolve(m, store) for m in self.mixins]
return r
class Instance(Resolvable):
__slots__ = ['type']
def __init__(self, type):
super(Instance, self).__init__()
self.type = type
def resolve(self, store):
r = super(Instance, self).resolve(store)
self.type = resolve(self.type, store)
if isinstance(self.type, Module):
self.type = self.type.exports
return r
def __getitem__(self, key):
return Global(self)['key']
class Function(Resolvable):
__slots__ = ['comments']
def __init__(self, comments):
super(Function, self).__init__()
self.comments = comments
class Deref(Printable):
__slots__ = ['object', 'property']
def __init__(self, object, property):
self.object = object
self.property = property
def resolve(self, store):
return resolve(self.object, store)[self.property]
class Call(Resolvable):
__slots__ = ['callable']
def __init__(self, callable, arguments):
super(Call, self).__init__()
self.callable = callable
def resolve(self, store):
r = super(Call, self).resolve(store)
self.callable = resolve(self.callable, store)
return r
def __getitem__(self, item):
return Global(self)[item]
def get(node, it, *path):
if not path:
return node[it]
return get(node[it], *path)
def match(node, pattern):
"""Checks that ``pattern`` is a subset of ``node``.
If a sub-pattern is a callable it will be called with the current
sub-node and its result is the match's. Other sub-patterns are
checked by equality against the correspoding sub-node.
Can be used to quickly check the descendants of a node of suitable
type.
TODO: support checking list content? Would be convenient to
constraint e.g. CallExpression based on their parameters
"""
if node is None and pattern is not None:
return
if callable(pattern):
return pattern(node)
if isinstance(node, list):
return all(len(node) > i and match(node[i], v) for i, v in pattern.items())
if isinstance(pattern, dict):
return all(match(node.get(k), v) for k, v in pattern.items())
return node == pattern
class Global(object):
def __init__(self, name):
self.name = name
def __repr__(self):
return '<external %s>' % self.name
def __getitem__(self, name):
return Global('%s.%s' % (self.name, name))
def resolve(self, store):
return self

View File

@ -0,0 +1,126 @@
from pyjsparser.pyjsparserdata import Syntax
_binary = lambda n: [n['left'], n['right']]
_function = lambda n: n['params'] + n['defaults'] + [n['body']]
# yields children in visitation order
_children = {
Syntax.ArrayExpression: lambda n: n['elements'],
Syntax.ArrayPattern: lambda n: n['elements'],
Syntax.ArrowFunctionExpression: _function,
Syntax.AssignmentExpression: _binary,
Syntax.AssignmentPattern: _binary,
Syntax.BinaryExpression: _binary,
Syntax.BlockStatement: lambda n: n['body'],
Syntax.BreakStatement: lambda n: [],
Syntax.CallExpression: lambda n: [n['callee']] + n['arguments'],
Syntax.CatchClause: lambda n: [n['param'], n['body']],
Syntax.ClassBody: lambda n: [n['body']],
Syntax.ClassDeclaration: lambda n: [n['superClass'], n['body']],
Syntax.ClassExpression: lambda n: [n['superClass'], n['body']],
Syntax.ConditionalExpression: lambda n: [n['test'], n['consequent'], n['alternate']],
Syntax.ContinueStatement: lambda n: [],
Syntax.DebuggerStatement: lambda n: [],
Syntax.DoWhileStatement: lambda n: [n['body'], n['test']],
Syntax.EmptyStatement: lambda n: [],
Syntax.ExportAllDeclaration: lambda n: [n['source']],
Syntax.ExportDefaultDeclaration: lambda n: [n['declaration']],
Syntax.ExportNamedDeclaration: lambda n: ([n['declaration']] if n['declaration'] else n['specifiers']) + [n['source']],
Syntax.ExportSpecifier: lambda n: [n['local'], n['exported']],
Syntax.ExpressionStatement: lambda n: [n['expression']],
Syntax.ForStatement: lambda n: [n['init'], n['test'], n['update'], n['body']],
Syntax.ForInStatement: lambda n: [n['left'], n['right'], n['body']],
Syntax.FunctionDeclaration: _function,
Syntax.FunctionExpression: _function,
Syntax.Identifier: lambda n: [],
Syntax.IfStatement: lambda n: [n['test'], n['consequent'], n['alternate']],
Syntax.ImportDeclaration: lambda n: n['specifiers'] + [n['source']],
Syntax.ImportDefaultSpecifier: lambda n: [n['local']],
Syntax.ImportNamespaceSpecifier: lambda n: [n['local']],
Syntax.ImportSpecifier: lambda n: [n['local'], n['imported']],
Syntax.LabeledStatement: lambda n: [n['body']],
Syntax.Literal: lambda n: [],
Syntax.LogicalExpression: _binary,
Syntax.MemberExpression: lambda n: [n['object'], n['property']],
#Syntax.MethodDefinition: lambda n: [],
Syntax.NewExpression: lambda n: [n['callee']] + n['arguments'],
Syntax.ObjectExpression: lambda n: n['properties'],
Syntax.ObjectPattern: lambda n: n['properties'],
Syntax.Program: lambda n: n['body'],
Syntax.Property: lambda n: [n['key'], n['value']],
Syntax.RestElement: lambda n: [n['argument']],
Syntax.ReturnStatement: lambda n: [n['argument']],
Syntax.SequenceExpression: lambda n: n['expressions'],
Syntax.SpreadElement: lambda n: [n['argument']],
Syntax.Super: lambda n: [],
Syntax.SwitchCase: lambda n: [n['test'], n['consequent']],
Syntax.SwitchStatement: lambda n: [n['discriminant']] + n['cases'],
Syntax.TaggedTemplateExpression: lambda n: [n['tag'], n['quasi']],
Syntax.TemplateElement: lambda n: [],
Syntax.TemplateLiteral: lambda n: n['quasis'] + n['expressions'],
Syntax.ThisExpression: lambda n: [],
Syntax.ThrowStatement: lambda n: [n['argument']],
Syntax.TryStatement: lambda n: [n['block'], n['handler'], n['finalizer']],
Syntax.UnaryExpression: lambda n: [n['argument']],
Syntax.UpdateExpression: lambda n: [n['argument']],
Syntax.VariableDeclaration: lambda n: n['declarations'],
Syntax.VariableDeclarator: lambda n: [n['id'], n['init']],
Syntax.WhileStatement: lambda n: [n['test'], n['body']],
Syntax.WithStatement: lambda n: [n['object'], n['body']],
}
SKIP = object()
class Visitor(object):
"""
Generic visitor for the pyjsparser AST.
Visitation is driven by the ``visit`` method, which iterates the tree in
depth-first pre-order.
For each node, calls ``enter_$NODETYPE``, visits the children then calls
``exit_$NODETYPE``. If the enter or exit methods are not present on the
visitor, falls back on ``enter_generic`` and ``exit_generic``.
Any ``enter_`` method can return ``SKIP`` to suppress both the traversal
of the subtree *and* the call to the corresponding ``exit_`` method
(whether generic or specific).
For convenience, ``visit`` will return whatever is set as the visitor's
``result`` attribute, ``None`` by default.
``visit`` can be given multiple root nodes, and it can be called multiple
times. The ``result`` attribute is cleared at each call but not between
two roots of the same ``visit`` call.
"""
def __init__(self):
super(Visitor, self).__init__()
self.result = None
def enter_generic(self, node): pass
def exit_generic(self, node): pass
def visit(self, nodes):
if isinstance(nodes, dict):
nodes = [nodes]
# if multiple nodes are passed in, we need to reverse the order in
# order to traverse front-to-back rather than the other way around
nodes = list(reversed(nodes))
while nodes:
node = nodes.pop()
# should probably filter None descendants in _children...
if node is None:
continue
node_type = node['type']
if node_type == '_exit':
node = node['node']
getattr(self, 'exit_' + node['type'], self.exit_generic)(node)
continue
if getattr(self, 'enter_' + node_type, self.enter_generic)(node) is SKIP:
continue
nodes.append({'type': '_exit', 'node': node})
nodes.extend(reversed(_children[node_type](node)))
return self.result