Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash with "in <string>' requires string as left operand, not int" #1508

Closed
leplatrem opened this issue Feb 21, 2018 · 9 comments
Closed

Crash with "in <string>' requires string as left operand, not int" #1508

leplatrem opened this issue Feb 21, 2018 · 9 comments

Comments

@leplatrem
Copy link
Contributor

ValidationError: 'minVersion' is a required property

Failed validating 'required' in schema['properties']['versionRange']['items']['properties']['targetApplication']['items']:
    {'additionalProperties': False,
     'description': 'Target application',
     'properties': {'guid': {'description': 'The application unique '
                                            'identifier.',
                             'enum': ['{ec8030f7-c20a-464f-9b0e-13a3a9e97384}',
                                      '{3550f703-e582-4d05-9a08-453d09bdfdc6}',
                                      '{92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}',
                                      '{aa3c5121-dab2-40e2-81ca-7ea25febc110}'],
                             'enumNames': ['Firefox',
                                           'Thunderbird',
                                           'Seamonkey',
                                           'Android'],
                             'title': 'Application id',
                             'type': 'string'},
                    'maxVersion': {'$ref': '#/definitions/maxVersion'},
                    'minVersion': {'$ref': '#/definitions/minVersion'}},
     'required': ['guid', 'minVersion', 'maxVersion'],
     'title': 'Target application',
     'type': 'object'}

On instance['versionRange'][0]['targetApplication'][0]:
    {'guid': 'ec8030f7-c20a-464f-9b0e-13a3a9e97384', 'maxVersion': '57.0.*'}
  File "kinto/views/records.py", line 73, in process_record
    jsonschema.validate(data, schema)
  File "jsonschema/validators.py", line 541, in validate
    cls(schema, *args, **kwargs).validate(instance)
  File "jsonschema/validators.py", line 130, in validate
    raise error

TypeError: 'in <string>' requires string as left operand, not int
(11 additional frame(s) were not displayed)
...
  File "cornice/service.py", line 494, in wrapper
    response = view_()
  File "kinto/core/resource/__init__.py", line 463, in put
    new_record = self.process_record(post_record, old=existing)
  File "kinto/views/records.py", line 81, in process_record
    raise_invalid(self.request, name=field, description=e.message)
  File "kinto/core/errors.py", line 178, in raise_invalid
    response = json_error_handler(request)
  File "kinto/core/errors.py", line 149, in json_error_handler
    if name in description:
@leplatrem
Copy link
Contributor Author

To fix this issue, start by reproducing in a test (in /test_views_schema_collection.py for example)

  • using the schema shown above
  • using a record that does not have minVersion in a targetApplication field (in a list, itself in a list in versionRange)
  • try to validate!

Introspect the validation error attributes and see if something consistent can be done :)

@SeriousFrest
Copy link

Sorry for my english. Could you help me to configure and run tests correctly. My system info:

Arch Linux
Python 3.7.0
GNU Make 4.2.1

After all tutorials reading, I use in virtual environment(after make serve writing "Serving on http://localhost.localdomain:8888"):
$ tox -- tests/test_views_schema_collection.py
And I get large list of warnings and errors and don't find direction, which I have to go:

Results (3.51s):
       6 passed
ERROR: InvocationError for command '/home/iurez/Projects/Python-projects/kinto/.tox/py37/bin/py.test --cov-report term-missing --cov-branch --cov-fail-under 100 --cov kinto tests/test_views_schema_collection.py' (exited with code 1)
flake8 inst-nodeps: /home/iurez/Projects/Python-projects/kinto/.tox/dist/kinto-10.1.2.dev0.zip
ERROR: invocation failed (exit code 1), logfile: /home/iurez/Projects/Python-projects/kinto/.tox/flake8/log/flake8-6.log
ERROR: actionid: flake8
msg: installpkg
cmdargs: '/home/iurez/Projects/Python-projects/kinto/.tox/flake8/bin/pip install --no-deps -U /home/iurez/Projects/Python-projects/kinto/.tox/dist/kinto-10.1.2.dev0.zip'

Processing ./.tox/dist/kinto-10.1.2.dev0.zip
    Complete output from command python setup.py egg_info:
    warning: no previously-included files found matching '*.nix'
    warning: no previously-included files found matching 'appveyor.yml'
    warning: no previously-included files found matching '.travis.yaml'
    warning: no previously-included files found matching '.pre-commit-config.yaml'
    
    Installed /tmp/easy_install-5dugmz6o/pytest-runner-4.2/.eggs/setuptools_scm-3.1.0-py3.7.egg
    zip_safe flag not set; analyzing archive contents...
    Traceback (most recent call last):
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 157, in save_modules
        yield saved
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 198, in setup_context
        yield
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 255, in run_setup
        DirectorySandbox(setup_dir).run(runner)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 285, in run
        return func()
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 253, in runner
        _execfile(setup_script, ns)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 47, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-5dugmz6o/pytest-runner-4.2/setup.py", line 76, in <module>
        'paste.app_factory': [
      File "/usr/lib64/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib64/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
        os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
        return analyze_egg(self.bdist_dir, self.stubs)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
        safe = scan_module(egg_dir, base, name, stubs) and safe
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
        code = marshal.load(f)
    ValueError: bad marshal data (unknown type code)
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-5sv_1rb1/setup.py", line 119, in <module>
        entry_points=ENTRY_POINTS)
      File "/usr/lib64/python3.7/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/dist.py", line 318, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/dist.py", line 375, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/pkg_resources/__init__.py", line 851, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1123, in best_match
        return self.obtain(req, installer)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1135, in obtain
        return installer(requirement)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/dist.py", line 443, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 699, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 880, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1119, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1105, in run_setup
        run_setup(setup_script, args)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 258, in run_setup
        raise
      File "/usr/lib64/python3.7/contextlib.py", line 130, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 198, in setup_context
        yield
      File "/usr/lib64/python3.7/contextlib.py", line 130, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 169, in save_modules
        saved_exc.resume()
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 144, in resume
        six.reraise(type, exc, self._tb)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/six.py", line 692, in reraise
        raise value.with_traceback(tb)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 157, in save_modules
        yield saved
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 198, in setup_context
        yield
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 255, in run_setup
        DirectorySandbox(setup_dir).run(runner)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 285, in run
        return func()
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 253, in runner
        _execfile(setup_script, ns)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/sandbox.py", line 47, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-5dugmz6o/pytest-runner-4.2/setup.py", line 76, in <module>
        'paste.app_factory': [
      File "/usr/lib64/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib64/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 209, in run
        os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
        return analyze_egg(self.bdist_dir, self.stubs)
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
        safe = scan_module(egg_dir, base, name, stubs) and safe
      File "/home/iurez/Projects/Python-projects/kinto/.tox/flake8/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
        code = marshal.load(f)
    ValueError: bad marshal data (unknown type code)
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-5sv_1rb1/

flake8 installed: appdirs==1.4.3,flake8==3.5.0,kinto==10.1.2.dev0,mccabe==0.6.1,packaging==18.0,pycodestyle==2.3.1,pyflakes==1.6.0,pyparsing==2.2.2,six==1.11.0
___________________________________________________________________________________________ summary ____________________________________________________________________________________________
SKIPPED:  py35-raw: InterpreterNotFound: python3.5
SKIPPED:  py35: InterpreterNotFound: python3.5
SKIPPED:  py36: InterpreterNotFound: python3.6
ERROR:   py37: commands failed
ERROR:   flake8: InvocationError for command /home/iurez/Projects/Python-projects/kinto/.tox/flake8/bin/pip install --no-deps -U /home/iurez/Projects/Python-projects/kinto/.tox/dist/kinto-10.1.2.dev0.zip (see /home/iurez/Projects/Python-projects/kinto/.tox/flake8/log/flake8-6.log) (exited with code 1)

Also, when I run make test, some of them crashes too with similar errors. I was watching through issues, but didn't find anything. What is wrong?

@leplatrem
Copy link
Contributor Author

Did you try what folks suggest on the InternetZ for the flake8 ValueError: bad marshal data (unknown type code) error? (delete pyc files etc)

@SeriousFrest
Copy link

I have tried to delete all .pyc files and no changes. But on Xubuntu(from VirtualBox) this error doesn't exist. and if I run test with tox <test>, then I get IvocationError. But if I run pytest directly , then all's ok. i will goggle and I'll try solve that

@eugene-kulak
Copy link
Contributor

@leplatrem I'm trying to reproduce this issue and it looks like the schema is incomplete (minVersion and maxVersion use $ref). I'm not an expert in jsonschema, so I might be wrong.

@leplatrem
Copy link
Contributor Author

it looks like the schema is incomplete

You're right! Sorry!

The full JSON schema would be this one. You can use to try to reproduce the issue and we can simplify it afterwards :)

{
  "definitions": {
    "minVersion": {
      "type": "string",
      "title": "Min version",
      "description": "The mininum version."
    },
    "maxVersion": {
      "type": "string",
      "title": "Max version",
      "description": "The maximum version."
    }
  },
  "title": "Add-on",
  "description": "Add-on",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "guid"
  ],
  "default": {
    "guid": "",
    "prefs": [

    ],
    "versionRange": [
      {
        "minVersion": "0",
        "maxVersion": "*",
        "severity": 1
      }
    ]
  },
  "properties": {
    "enabled": {
      "type": "boolean",
      "title": "Enabled",
      "description": "blocking rule is enabled.",
      "default": true
    },
    "guid": {
      "type": "string",
      "title": "Add-on id",
      "description": "The add-on unique identifier or a regular expression.",
      "minLength": 1,
      "default": ""
    },
    "os": {
      "type": "string",
      "title": "OS",
      "description": "The comma-separated operating system identifiers, eg. Darwin,Linux",
      "pattern": "^[^,]*(,[^,]+)*$"
    },
    "prefs": {
      "type": "array",
      "title": "Preferences",
      "description": "The list of impacted preferences.",
      "uniqueItems": true,
      "default": [

      ],
      "items": {
        "title": "Preference name",
        "description": "The browser preference name, eg. browser.startup.homepage",
        "type": "string",
        "minLength": 1,
        "default": ""
      }
    },
    "versionRange": {
      "type": "array",
      "title": "Versions",
      "description": "The list of impacted versions.",
      "items": {
        "type": "object",
        "title": "Version range",
        "description": "Version range",
        "additionalProperties": false,
        "required": [
          "minVersion",
          "maxVersion",
          "severity"
        ],
        "default": {
          "minVersion": "",
          "maxVersion": "",
          "severity": 1
        },
        "properties": {
          "minVersion": {
            "$ref": "#/definitions/minVersion"
          },
          "maxVersion": {
            "$ref": "#/definitions/maxVersion"
          },
          "severity": {
            "type": "integer",
            "title": "Severity",
            "description": "The severity code number.",
            "enum": [
              1,
              3
            ],
            "enumNames": [
              "1 - Soft block",
              "3 - Hard block"
            ],
            "default": 1
          },
          "targetApplication": {
            "type": "array",
            "title": "Target applications",
            "description": "The list of target application information.",
            "default": [

            ],
            "items": {
              "type": "object",
              "title": "Target application",
              "description": "Target application",
              "additionalProperties": false,
              "required": [
                "guid",
                "minVersion",
                "maxVersion"
              ],
              "properties": {
                "guid": {
                  "type": "string",
                  "title": "Application id",
                  "description": "The application unique identifier.",
                  "enum": [
                    "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
                    "{3550f703-e582-4d05-9a08-453d09bdfdc6}",
                    "{92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}",
                    "{aa3c5121-dab2-40e2-81ca-7ea25febc110}"
                  ],
                  "enumNames": [
                    "Firefox",
                    "Thunderbird",
                    "Seamonkey",
                    "Android"
                  ]
                },
                "minVersion": {
                  "$ref": "#/definitions/minVersion"
                },
                "maxVersion": {
                  "$ref": "#/definitions/maxVersion"
                }
              }
            }
          }
        }
      }
    },
    "details": {
      "type": "object",
      "title": "Details",
      "required": [
        "name",
        "why"
      ],
      "properties": {
        "name": {
          "type": "string",
          "title": "Name"
        },
        "why": {
          "type": "string",
          "title": "Why"
        },
        "who": {
          "type": "string",
          "title": "Who"
        },
        "bug": {
          "type": "string",
          "title": "Bug"
        },
        "created": {
          "type": "string",
          "format": "date-time"
        }
      }
    },
    "blockID": {
      "type": "string",
      "title": "Internal blocklist id",
      "description": "Original block id, eg. i152",
      "pattern": "^i[0-9]+$"
    }
  }
}

@eugene-kulak
Copy link
Contributor

eugene-kulak commented Feb 5, 2019

Thank you @leplatrem, I manage to reproduce the issue now.
This schema should be enough to reproduce:

{
  "title": "Add-on",
  "description": "Add-on",
  "type": "object",
  "additionalProperties": False,
  "required": [
    "guid"
  ],
  "properties": {
    "guid": {
      "type": "string",
      "title": "Add-on id",
      "description": "The add-on unique identifier or a regular expression.",
      "minLength": 1,
      "default": ""
    },
    "versionRange": {
      "type": "array",
      "title": "Versions",
      "description": "The list of impacted versions.",
      "items": {
        "type": "object",
        "title": "Version range",
        "description": "Version range",
        "additionalProperties": False,
        "required": [
          "minVersion",
          "maxVersion"
        ],
        "properties": {
          "minVersion": {
            "type": "number"
          },
          "maxVersion": {
            "type": "number"
          }
        }
      }
    }
  }
}

With this record:
{ "guid": "TestGUID", "versionRange": [ { "minVersion": 1 } ] }

This issue related to error_handler, it doesn't expect array indexes as element names here

if name in description:

It seems quite easy to fix, but the question I have is more about the format of message we want to show.

Now it will be something like this:
"0 in body: \'maxVersion\' is a required property"

@leplatrem
Copy link
Contributor Author

Cool! Thanks!

Now it will be something like this: "0 in body: \'maxVersion\' is a required property"

I'm ok with this! ;)

@leplatrem
Copy link
Contributor Author

Validation messages are tricky, because we'd have introspect the internal of jsonschema validation... Which is not recommended :/

eugene-kulak added a commit to eugene-kulak/kinto that referenced this issue Feb 5, 2019
eugene-kulak added a commit to eugene-kulak/kinto that referenced this issue Feb 5, 2019
eugene-kulak added a commit to eugene-kulak/kinto that referenced this issue Feb 6, 2019
leplatrem added a commit that referenced this issue Feb 6, 2019
#1508 - fix json_error_handler behaviour in case field name is index
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants