Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crashes related to calling isinstance() on a NamedTuple #3419

Closed
JelleZijlstra opened this issue May 23, 2017 · 4 comments · Fixed by #3952
Closed

Crashes related to calling isinstance() on a NamedTuple #3419

JelleZijlstra opened this issue May 23, 2017 · 4 comments · Fixed by #3952
Labels
crash priority-0-high topic-import-cycles topic-named-tuple

Comments

@JelleZijlstra
Copy link
Member

I found two separate tracebacks playing with variations on the code from #3418:

from typing import Dict, NamedTuple

NameDict = Dict[str, 'NameInfo']

class NameInfo(NamedTuple):
    ast: bool

def parse_ast(name_dict: NameDict) -> None:
    if isinstance(name_dict[''], int):
        pass

produces

Traceback (most recent call last):
  File "/Users/jzijlstra-mpbt/py/venvs/venv36/bin/mypy", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/jzijlstra-mpbt/py/mypy/scripts/mypy", line 6, in <module>
    main(__file__)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/main.py", line 93, in type_check_only
    options=options)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 1595, in dispatch
    process_graph(graph, manager)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 1838, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 1937, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 1510, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 177, in check_first_pass
    self.accept(d)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 565, in accept
    return visitor.visit_func_def(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 510, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 569, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 718, in check_func_def
    self.accept(item.body)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 811, in accept
    return visitor.visit_block(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 1173, in visit_block
    self.accept(s)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 970, in accept
    return visitor.visit_if_stmt(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 1916, in visit_if_stmt
    if_map, else_map = self.find_isinstance_check(e)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 2477, in find_isinstance_check
    return find_isinstance_check(n, self.type_map)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 2731, in find_isinstance_check
    return conditional_type_map(expr, vartype, type)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 2531, in conditional_type_map
    and is_proper_subtype(current_type, proposed_type)):
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/subtypes.py", line 558, in is_proper_subtype
    return left.accept(ProperSubtypeVisitor(right))
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/types.py", line 411, in accept
    return visitor.visit_instance(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/subtypes.py", line 594, in visit_instance
    for base in left.type.mro:
TypeError: 'NoneType' object is not iterable
typeshed_client/parser.py:9: note: use --pdb to drop into pdb

The other stack trace I haven't been able to minimize yet, but the traceback is

/Users/jzijlstra-mpbt/py/typeshed_client/typeshed_client/resolver.py:61: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.520-dev-38453c0ddfaa6660309f782b227bd40b8c4634ba-dirty
Traceback (most recent call last):
  File "/Users/jzijlstra-mpbt/py/venvs/venv36/bin/mypy", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/jzijlstra-mpbt/py/mypy/scripts/mypy", line 6, in <module>
    main(__file__)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/main.py", line 93, in type_check_only
    options=options)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 1595, in dispatch
    process_graph(graph, manager)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 1838, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 1937, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/build.py", line 1510, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 177, in check_first_pass
    self.accept(d)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 750, in accept
    return visitor.visit_class_def(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 1080, in visit_class_def
    self.accept(defn.defs)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 811, in accept
    return visitor.visit_block(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 1173, in visit_block
    self.accept(s)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 565, in accept
    return visitor.visit_func_def(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 510, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 569, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 718, in check_func_def
    self.accept(item.body)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 811, in accept
    return visitor.visit_block(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 1173, in visit_block
    self.accept(s)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 264, in accept
    stmt.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 970, in accept
    return visitor.visit_if_stmt(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checker.py", line 1906, in visit_if_stmt
    t = self.expr_checker.accept(e)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 2058, in accept
    typ = node.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 1368, in accept
    return visitor.visit_unary_expr(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 1408, in visit_unary_expr
    operand_type = self.accept(e.expr)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 2058, in accept
    typ = node.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 1303, in accept
    return visitor.visit_call_expr(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 201, in visit_call_expr
    ret_type = self.check_call_expr_with_callee_type(callee_type, e)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 341, in check_call_expr_with_callee_type
    e.arg_names, callable_node=e.callee)[0]
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 393, in check_call
    callee, args, arg_kinds, formal_to_actual)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 538, in infer_arg_types_in_context2
    res[ai] = self.accept(args[ai], callee.arg_types[i])
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 2058, in accept
    typ = node.accept(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 1256, in accept
    return visitor.visit_member_expr(self)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 1012, in visit_member_expr
    result = self.analyze_ordinary_member_access(e, False)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkexpr.py", line 1027, in analyze_ordinary_member_access
    original_type=original_type, chk=self.chk)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkmember.py", line 100, in analyze_member_access
    original_type=original_type, chk=chk)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkmember.py", line 222, in analyze_member_var_access
    v = lookup_member_var_or_accessor(info, name, is_lvalue)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/checkmember.py", line 341, in lookup_member_var_or_accessor
    node = info.get(name)
  File "/Users/jzijlstra-mpbt/py/mypy/mypy/nodes.py", line 2033, in get
    for cls in self.mro:
TypeError: 'NoneType' object is not iterable
/Users/jzijlstra-mpbt/py/typeshed_client/typeshed_client/resolver.py:61: note: use --pdb to drop into pdb

The crashing line is if not isinstance(info.ast, parser.ImportedName):, where info is a NameInfo from the above example. I think it's likely the same underlying bug.
@ilevkivskyi
Copy link
Member

I think this is related to #3308. In particular, MRO is not recalculated in third pass for synthetic types like named tuples. I think the solution should be similar to #3322 - add corresponding recalculation in the third pass. After #3322 is merged this should be quite easy.

MeGotsThis added a commit to MeGotsThis/BotGotsThis that referenced this issue Jun 17, 2017
@MeGotsThis
Some mypy fixes
6591ae6 
There is an mypy issue with NamedTuple relates to: python/mypy#3419
@sseg
Copy link

sseg commented Jul 8, 2017 •
edited
Loading

I can reproduce the second traceback when annotating with a NamedTuple type before that type's definition:

from typing import NamedTuple

def get_state(proc: 'Process') -> int:
    return proc.state

class Process(NamedTuple):
     state: int

This kind of annotation came up for me when writing a decorator to be used on the methods of the new NamedTuple.

mypy version: 0.520-dev-7670ac9a6527708faf226039d600da6e7e596462

MeGotsThis added a commit to MeGotsThis/BotGotsThis that referenced this issue Jul 13, 2017
@MeGotsThis
Some mypy fixes
ed6c661 
There is an mypy issue with NamedTuple relates to: python/mypy#3419
@gvanrossum gvanrossum mentioned this issue Jul 22, 2017
Closed
@Daenyth Daenyth mentioned this issue Jul 28, 2017
Closed
@ilevkivskyi
Copy link
Member

Raising priority to high since this keeps coming. There is another situation where this crash appears: an import cycle originally reported by @Daenyth in #3776

@ilevkivskyi ilevkivskyi marked this as a duplicate of #3756 Jul 28, 2017
@ilevkivskyi
Copy link
Member

Note that the first crash could also appear from a call to make_simplified_union as reported in #3756

@ilevkivskyi ilevkivskyi mentioned this issue Aug 7, 2017
Closed
@ilevkivskyi ilevkivskyi mentioned this issue Aug 17, 2017
Closed
@ilevkivskyi ilevkivskyi mentioned this issue Sep 13, 2017
Merged
JukkaL pushed a commit that referenced this issue Sep 27, 2017
@ilevkivskyi @JukkaL
Fix crashes and fails in forward references (#3952)
a611b11 
Forward references didn't work with anything apart from classes, for example 
this didn't work:

```
x: A
A = NamedTuple('A', [('x', int)])
```

The same situation was with `TypedDict`, `NewType`, and type aliases. The 
root problem is that these synthetic types are neither detected in first pass, 
nor fixed in third pass. In certain cases this can lead to crashes (first six issues 
below are various crash scenarios). This fixes these crashes by applying some 
additional patches after third pass.

Here is the summary of the PR:

* New simple wrapper type `ForwardRef` with only one field `link` is introduced 
  (with updates to type visitors)
* When an unknown type is found in second pass, the corresponding 
  `UnboundType` is wrapped in `ForwardRef`, it is given a "second chance" in 
  third pass.
* After third pass I record the "suspicious" nodes, where forward references and 
  synthetic types have been encountered and append patches (callbacks) to fix 
  them after third pass. Patches use the new visitor `TypeReplacer` (which is the 
  core of this PR).

Fixes #3340
Fixes #3419
Fixes #3674
Fixes #3685
Fixes #3799
Fixes #3836
Fixes #3881
Fixes #867
Fixes #2241
Fixes #2399
Fixes #1701
Fixes #3016
Fixes #3054
Fixes #2762
Fixes #3575
Fixes #3990
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
crash priority-0-high topic-import-cycles topic-named-tuple
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix crashes and fails in forward references ilevkivskyi/mypy
3 participants
@JelleZijlstra @sseg @ilevkivskyi