Add a general lms user id column to the student table #1036

sigurdurb · 2018-10-29T20:18:04Z

Adds a lms_user_id to the ngrader db:

See issue: #1035

How to use:

nbgrader db student add <some student.id> --lms-user-id="'123'"
Notice the extra "" if your lms user id contains only numbers.

Additions that can be made:

Not yet added to nbgrader/plugins/export.py : CSV export plugin, just putting it here for reference on things this could be added to:

nbgrader/nbgrader/plugins/export.py

Line 72 in 7a81a40

score['max_score'] = assignment.max_score

Possible issues:

I had a very unhelpful error with the nbgrader db upgrade oldgradebook.db but it still put in the column. Maybe someone else could also test it.

sigurdurb · 2018-10-30T20:20:57Z

So here is the error msg when I upgrade the gradebook.db, yet it still adds the column and I can run nbgrader db student add <some student.id> --lms-user-id="'123'" successfully

$ nbgrader db upgrade gradebook.db
[DbUpgradeApp | WARNING] No nbgrader_config.py file found (rerun with --debug to see where nbgrader is looking)
[DbUpgradeApp | INFO] Backing up /home/reir20173/reir20173/gradebook.db => /home/reir20173/reir20173/gradebook.db.2018-10-30-200724.921960
[DbUpgradeApp | INFO] Upgrading sqlite:////home/reir20173/reir20173/gradebook.db
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 724cde206c17 -> 282d30a0218e, adds lms_user_id column to student table
Traceback (most recent call last):
  File "/opt/conda/bin/alembic", line 11, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.5/site-packages/alembic/config.py", line 486, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/opt/conda/lib/python3.5/site-packages/alembic/config.py", line 480, in main
    self.run_cmd(cfg, options)
  File "/opt/conda/lib/python3.5/site-packages/alembic/config.py", line 463, in run_cmd
    **dict((k, getattr(options, k, None)) for k in kwarg)
  File "/opt/conda/lib/python3.5/site-packages/alembic/command.py", line 254, in upgrade
    script.run_env()
  File "/opt/conda/lib/python3.5/site-packages/alembic/script/base.py", line 427, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/opt/conda/lib/python3.5/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/conda/lib/python3.5/site-packages/alembic/util/compat.py", line 83, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/alembic/env.py", line 69, in <module>
    run_migrations_online()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/alembic/env.py", line 64, in run_migrations_online
    context.run_migrations()
  File "/opt/conda/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/conda/lib/python3.5/site-packages/alembic/runtime/migration.py", line 186, in do_nothing
    yield
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/alembic/env.py", line 64, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/conda/lib/python3.5/site-packages/alembic/runtime/environment.py", line 836, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/conda/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/conda/lib/python3.5/site-packages/alembic/operations/base.py", line 161, in context
    yield op
  File "/opt/conda/lib/python3.5/site-packages/alembic/runtime/environment.py", line 836, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/conda/lib/python3.5/site-packages/alembic/runtime/migration.py", line 342, in run_migrations
    run_args=kw)
  File "/opt/conda/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/conda/lib/python3.5/site-packages/alembic/runtime/migration.py", line 180, in do_nothing
    yield
  File "/opt/conda/lib/python3.5/site-packages/alembic/runtime/migration.py", line 330, in run_migrations
    step.migration_fn(**kw)
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/alembic/versions/282d30a0218e_adds_lms_user_id_column_to_student_table.py", line 20, in upgrade
    op.add_column('student', sa.Column('lms_user_id', sa.String(128), unique=True, nullable=True))
  File "<string>", line 8, in add_column
  File "<string>", line 3, in add_column
  File "/opt/conda/lib/python3.5/site-packages/alembic/operations/ops.py", line 1565, in add_column
    return operations.invoke(op)
  File "/opt/conda/lib/python3.5/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/opt/conda/lib/python3.5/site-packages/alembic/operations/toimpl.py", line 127, in add_column
    operations.impl.add_constraint(constraint)
  File "/opt/conda/lib/python3.5/site-packages/alembic/ddl/sqlite.py", line 34, in add_constraint
    "No support for ALTER of constraints in SQLite dialect")
NotImplementedError: No support for ALTER of constraints in SQLite dialect
Traceback (most recent call last):
  File "/opt/conda/bin/nbgrader", line 11, in <module>
    load_entry_point('nbgrader', 'console_scripts', 'nbgrader')()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/apps/nbgraderapp.py", line 301, in main
    NbGraderApp.launch_instance()
  File "/opt/conda/lib/python3.5/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/apps/nbgraderapp.py", line 294, in start
    super(NbGraderApp, self).start()
  File "/opt/conda/lib/python3.5/site-packages/jupyter_core/application.py", line 255, in start
    self.subapp.start()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/apps/dbapp.py", line 558, in start
    super(DbApp, self).start()
  File "/opt/conda/lib/python3.5/site-packages/jupyter_core/application.py", line 255, in start
    self.subapp.start()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/apps/dbapp.py", line 505, in start
    dbutil.upgrade(self.coursedir.db_url)
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/dbutil.py", line 72, in upgrade
    ['alembic', '-c', alembic_ini, 'upgrade', revision]
  File "/opt/conda/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['alembic', '-c', '/tmp/tmpmjnnblz2/alembic.ini', 'upgrade', 'head']' returned non-zero exit status 1

Running it again I get a

sqlite3.OperationalError: duplicate column name: lms_user_id

# nbgrader db upgrade gradebook.db
[DbUpgradeApp | WARNING] No nbgrader_config.py file found (rerun with --debug to see where nbgrader is looking)
[DbUpgradeApp | INFO] Backing up /home/reir20173/reir20173/gradebook.db => /home/reir20173/reir20173/gradebook.db.2018-10-30-200748.769119
[DbUpgradeApp | INFO] Upgrading sqlite:////home/reir20173/reir20173/gradebook.db
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 724cde206c17 -> 282d30a0218e, adds lms_user_id column to student table
Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: duplicate column name: lms_user_id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/bin/alembic", line 11, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.5/site-packages/alembic/config.py", line 486, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/opt/conda/lib/python3.5/site-packages/alembic/config.py", line 480, in main
    self.run_cmd(cfg, options)
  File "/opt/conda/lib/python3.5/site-packages/alembic/config.py", line 463, in run_cmd
    **dict((k, getattr(options, k, None)) for k in kwarg)
  File "/opt/conda/lib/python3.5/site-packages/alembic/command.py", line 254, in upgrade
    script.run_env()
  File "/opt/conda/lib/python3.5/site-packages/alembic/script/base.py", line 427, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/opt/conda/lib/python3.5/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/conda/lib/python3.5/site-packages/alembic/util/compat.py", line 83, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/alembic/env.py", line 69, in <module>
    run_migrations_online()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/alembic/env.py", line 64, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/conda/lib/python3.5/site-packages/alembic/runtime/environment.py", line 836, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/conda/lib/python3.5/site-packages/alembic/runtime/migration.py", line 330, in run_migrations
    step.migration_fn(**kw)
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/alembic/versions/282d30a0218e_adds_lms_user_id_column_to_student_table.py", line 20, in upgrade
    op.add_column('student', sa.Column('lms_user_id', sa.String(128), unique=True, nullable=True))
  File "<string>", line 8, in add_column
  File "<string>", line 3, in add_column
  File "/opt/conda/lib/python3.5/site-packages/alembic/operations/ops.py", line 1565, in add_column
    return operations.invoke(op)
  File "/opt/conda/lib/python3.5/site-packages/alembic/operations/base.py", line 319, in invoke
    return fn(self, operation)
  File "/opt/conda/lib/python3.5/site-packages/alembic/operations/toimpl.py", line 123, in add_column
    schema=schema
  File "/opt/conda/lib/python3.5/site-packages/alembic/ddl/impl.py", line 172, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/opt/conda/lib/python3.5/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
    compiled
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/opt/conda/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: lms_user_id [SQL: 'ALTER TABLE student ADD COLUMN lms_user_id VARCHAR(128)'] (Background on this error at: http://sqlalche.me/e/e3q8)
Traceback (most recent call last):
  File "/opt/conda/bin/nbgrader", line 11, in <module>
    load_entry_point('nbgrader', 'console_scripts', 'nbgrader')()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/apps/nbgraderapp.py", line 301, in main
    NbGraderApp.launch_instance()
  File "/opt/conda/lib/python3.5/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.5/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/apps/nbgraderapp.py", line 294, in start
    super(NbGraderApp, self).start()
  File "/opt/conda/lib/python3.5/site-packages/jupyter_core/application.py", line 255, in start
    self.subapp.start()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/apps/dbapp.py", line 558, in start
    super(DbApp, self).start()
  File "/opt/conda/lib/python3.5/site-packages/jupyter_core/application.py", line 255, in start
    self.subapp.start()
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/apps/dbapp.py", line 505, in start
    dbutil.upgrade(self.coursedir.db_url)
  File "/etc/jupyterhub/nbgrader_stuff/lms_octave_nbgrader/nbgrader/nbgrader/dbutil.py", line 72, in upgrade
    ['alembic', '-c', alembic_ini, 'upgrade', revision]
  File "/opt/conda/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['alembic', '-c', '/tmp/tmpn286gwz5/alembic.ini', 'upgrade', 'head']' returned non-zero exit status 1

jhamrick · 2018-12-15T15:49:17Z

Looks good, thanks!

I'll try to have a look and see why the alembic update fails.

jhamrick

So the error appears to be because you enforce the column to be unique (a form of constraint), but SQLite doesn't allow you to add new constraints after the table has already been created. It seems like it would be possible to work around this using alembic batch operations (https://alembic.sqlalchemy.org/en/latest/batch.html), but it would be messy and hard to implement. Thus, I would suggest just dropping the unique requirement, and then everything should work as expected.

Additionally, can you update the API docs to reflect the addition of the new column name?

sigurdurb · 2019-01-02T23:54:33Z

I removed the unique constraint. I'm not sure why the documentation is not being generated. I ran the python tasks.py docs command but nothing seems to have changed. https://nbgrader.readthedocs.io/en/latest/command_line_tools/nbgrader-db-student-add.html This page and some other pages also give a 404 when I click edit on github.

jhamrick · 2019-01-03T10:20:37Z

You have to manually edit the rst file here to add the column name: https://github.com/jupyter/nbgrader/blob/master/nbgrader/docs/source/api/models.rst

… is called

jhamrick

Looks good! I guess the last thing is just to update the tests so that they check for the lms_user_id as well.

nbgrader/api.py

Co-Authored-By: sigurdurb <[email protected]>

jhamrick · 2019-05-30T23:05:31Z

I am not sure why the tests are failing as they seem to pass for me locally. Can you rebase (or merge master into your branch) to bring this up-to-date with the master branch of the repo? I think this time you shouldn't require too many changes, probably the main thing is you'll need to change the down_revision of your alembic file to 167914646830 (which is the task cells one).

…ask cells one)

sigurdurb added 3 commits October 25, 2018 19:16

add lms_user_id to the Student object and dict representation

f939416

adds lms_user_id to the nbgrader student add command

0d58149

adds alembic update method for the db to update+downgrade

81e2da9

sigurdurb changed the title ~~Adds a general lms user id column to the student table~~ Add a general lms user id column to the student table Oct 29, 2018

jhamrick added the enhancement label Dec 15, 2018

jhamrick added this to the 0.6.0 milestone Dec 15, 2018

jhamrick requested changes Dec 15, 2018

View reviewed changes

jhamrick added the needs work label Dec 15, 2018

removing unique constraint on lms_user_id in student table

1372369

sigurdurb added 2 commits January 4, 2019 21:02

adds documentation 1

c0ab5e8

adds lms_user_id to the print statement when nbgrader db student list…

f102a49

… is called

jhamrick requested changes Jan 6, 2019

View reviewed changes

nbgrader/api.py Outdated Show resolved Hide resolved

jhamrick and others added 3 commits January 7, 2019 11:05

Update nbgrader/api.py

0fad444

Co-Authored-By: sigurdurb <[email protected]>

adds lms_user_id to student_dicts method and to test models

a0e3dee

adds lms_user_id to get_student dict in apps/api and fixes to tests

96f16ba

sigurdurb added 2 commits May 31, 2019 09:44

Merge branch 'master' into lms_user_id

8897fcf

changed down_revision of alembic file to 167914646830 (which is the t…

8d0901e

…ask cells one)

jhamrick added Edinburgh-Hackathon and removed needs work labels May 31, 2019

jhamrick merged commit 39ea9d4 into jupyter:master May 31, 2019

jhamrick mentioned this pull request May 31, 2019

Add a lms_user_id #1035

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a general lms user id column to the student table #1036

Add a general lms user id column to the student table #1036

sigurdurb commented Oct 29, 2018 •

edited

Loading

sigurdurb commented Oct 30, 2018

jhamrick commented Dec 15, 2018

jhamrick left a comment

sigurdurb commented Jan 2, 2019 •

edited

Loading

jhamrick commented Jan 3, 2019

jhamrick left a comment

jhamrick commented May 30, 2019

Add a general lms user id column to the student table #1036

Add a general lms user id column to the student table #1036

Conversation

sigurdurb commented Oct 29, 2018 • edited Loading

Adds a lms_user_id to the ngrader db:

How to use:

Additions that can be made:

Possible issues:

sigurdurb commented Oct 30, 2018

Running it again I get a

jhamrick commented Dec 15, 2018

jhamrick left a comment

Choose a reason for hiding this comment

sigurdurb commented Jan 2, 2019 • edited Loading

jhamrick commented Jan 3, 2019

jhamrick left a comment

Choose a reason for hiding this comment

jhamrick commented May 30, 2019

sigurdurb commented Oct 29, 2018 •

edited

Loading

sigurdurb commented Jan 2, 2019 •

edited

Loading