OperationalError: database is locked

[apexxxx]

Hi,
Recently discover a bug with threading and ponyorm.
Here a litle poc : http://pastebin.com/VDgmJpBu

from pony.orm import *
import threading
from datetime import datetime
import time
import random

my_db = Database("sqlite", "my_db.sqlite", create_db=True)
sql_debug(False)


class Log(my_db.Entity):
    id = PrimaryKey(int, auto=True)
    timestamp = Required(datetime)
    message = Required(unicode, sql_type='collate nocase')

my_db.generate_mapping(create_tables=True)

@db_session
def callback_function():
    print 'fu'
    wait = random.randint(0,10)
    print "wait for {}".format(wait)
    Log(timestamp=datetime.now(), message="test")
    time.sleep(wait)
    print 'srodha'
    for i in range(0, 100000):
        Log(timestamp=datetime.now(), message="testFin {}".format(i))

threads = []
for i in range(5):
    t = threading.Thread(target=callback_function)
    threads.append(t)
    t.start()

and here the traceback :
Exception in thread Thread-5:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 763, in run
self._target(_self._args, *_self._kwargs)
File "<auto generated wrapper of callback_function() function>", line 2, in callback_function
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 356, in new_func
finally: self.exit(exc_type, exc, tb)
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 388, in exit
commit()
File "<auto generated wrapper of commit() function>", line 2, in commit
File "C:\Python27\lib\site-packages\pony\utils.py", line 88, in cut_traceback
return func(_args, *_kwargs)
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 284, in commit
transact_reraise(CommitException, exceptions)
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 268, in transact_reraise
reraise(exc_class, new_exc, tb)
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 278, in commit
try: primary_cache.commit()
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 965, in commit
if cache.modified: cache.flush()
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 1030, in flush
if obj is not None: obj.save()
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 4144, in save
if status == 'created': obj.save_created()
File "C:\Python27\lib\site-packages\pony\orm\core.py", line 4048, in save_created
% (obj, e.classname, msg), e)
File "C:\Python27\lib\site-packages\pony\utils.py", line 126, in throw
raise exc
CommitException: Object Log[new:5] cannot be stored in the database. OperationalError: database is locked

(have to remove some double underscore in the traceback to avoid automatic bold here the full traceback http://pastebin.com/XKud32q6)

The bug seems to be thread and time.sleep related.
Sometime the POC doesn't crash. Let me know if you want more informations.
Run with python 2.7.10
Hope you guys could fixe this. Your work is really good and i enjoy to use your product.

[sm0k]

Saw that too. time.sleep() called in threading context seams to be responsible.

[ApexHub]

i've not the poc but even when i've tried to do a 'real time' sleep function (a sleep function which doesn't sleep (just lot of time check)) the bug appears too. It seems to appear just with two thread and heavy entry in DB. hope this can help more. The doc speak about transaction and explicitly describe there is noway to obtain an operational error http://doc.ponyorm.com/transactions.html#sqlite. maybe the solution to our bug ? have to try with a retry in the DB decorator (@db_session(retry=3)) and tell you soon the result.

[amalashkevich]

we are working on the solution, stay tuned ;)

[ApexHub]

great. thank you to take care of this.

[ApexHub]

already ! what a good news. Thank you so much.

[kozlovsky]

We fixed this bug, your example should work correctly now. Please check and confirm.

As a side notice I think that inserting 100000 objects in a single transaction is a bit too much, because when your transaction do first INSERT or UPDATE the SQLite database becomes locked and all concurrent transactions will wait pretty long time until all this 100000 objects are inserted. If atomicity of inserting of 100000 objects at once is not important, it may be better to do commit every 1000 or 10000 objects in order to give concurrent transactions a chance to get write access to the database. It may be done by adding manual commit in your loop:

    for i in range(0, 100000):
        Log(timestamp=datetime.now(), message="testFin {}".format(i))
        if i % 10000 == 0:
            commit()

[ApexHub]

Of course those 100000 input was for the poc purpose. In my case, it was just with few heavy sql statement that the bug append. Will tell you soon if it work find. Thank you for the commit advice.