This repository has been archived by the owner on Jun 10, 2024. It is now read-only.
Fixed db inconsistency #779
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brian6peng
pushed a commit
to brian6peng/pyspider
that referenced
this pull request
Jun 20, 2018
* Fixed db creation inconsistency in taskdb, projectdb and resultdb * Fixed typo
StarUI
added a commit
to StarUI/pyspider
that referenced
this pull request
Mar 16, 2021
* fix test 1.1.1.1 took by cloudflare * Fixed db inconsistency (binux#779) * Fixed db creation inconsistency in taskdb, projectdb and resultdb * Fixed typo * using reserved ip address for testing rolling out version 0.3.10 * Fix mysql return bytes as field name type (binux#787) * use pip version of mysql-connector-python for testing * fix mysql return bytes as field names type * fix raise Unread result found error This error raise on mysql-connector-python with C extension * fix test Pure version raise InterfaceError, but C extension version raise DatabaseError * fix binux#799 * optimise scheluler dynamic select limit and improve task queue (binux#796) * optimise scheduler select-limit and task queue * fix test case in python2.6 * fix: time priority queue only compare exetime * update:add test case for time priority queue * optimise: add globally auto increasing value for task to keep priority queue in order * change async to async_ (binux#803) * change async to async_ * change async to async_ in tests * change async_ to async_mode * modify async to async_mode to support python3.7 * add python3.7 CI test * add python3.7 CI test * add python3.7 CI test * add python3.7 CI test * remove python3.7 CI test * add py3.7-dev CI test * add support py3.7-dev CI test * removed 2.6 due to lack of support, changed pip install for 3.5 due to pip versioning * feature puppeteer js engine * features: add opened pages maximum limit, default 5 * fix: python3.5 install lxml error * add puppeteer fetcher * update * fix bugs 1. some args "async" haven't been replaced completely yet 2. delete Python 3.3 in .travis.yml because the current version of lxml is not supported by Python3.3 * use suggested python3.7 build * fix build for 3.3 * 1. python2.7 image is different when using metrix 2. pip install just works now days * sudo not required any more? * try not to specify a version for apt-get * fix setup.py test for py3.3 * try manually install * try again * fix for 3.7 * try install librt * try again * allow fail * updated requirements.txt to fixed package versions * port to python 3.6 * upgrade python-six * updated travis.yml * fixed "connect to scheduler rpc error: error(111, Connection refused)" error * fixed phantomjs libssl_conf.so error * travis test * another Travis test * trying to trace "cannot find module express" error in Travis * using NODE_PATH env var * moved NODE_PATH assignment after install * making symlink to node_modules * travis test * node modules are currently missing from travis * added npm install to travis.yml * fixed travis node dependancy issues * using run_in_thread for scheduler and fetcher dispatch again * accommodate changes made in run.py to tests * changed test_90_docker_scheduler * added extra asserts to tests * test * upgraded sqlAlchemy * sqlalchemy upgrade * sqlalchemy upgrade * sqlalchemy upgrade * sqlalchemy upgrade * sqlalchemy upgrade * sqlalchemy upgrade * sqlalchemy upgrade fix * sqlalchemy upgrade * added extra assertions * sqlalchemy upgrade * sqlalchemy upgrade * sqlalchemy upgrade * undo previous * tracing errors * fix sqlalchemy data encoding * sqlalchemy changed dict encoding to pure json string * test_10_save mongodb fix * undo previous * tracing test_10_save mongodb bug * tracing test_10_save mongodb bug * upgraded pymongo * mongo tests now passing * fixed test_a110_one failing by "fetcher() got an unexpected keyword argument xmlrpc" * upgraded pika * tracing RabbitMQ ConnectionRefusedError: [Errno 111] Connection refused * fixed typo * tracing RabbitMQ ConnectionRefusedError: [Errno 111] Connection refused * tracing RabbitMQ ConnectionRefusedError: [Errno 111] Connection refused * tracing RabbitMQ ConnectionRefusedError: [Errno 111] Connection refused * switching to Pika for Rabbitmq * skip TestAmqpRabbitMQ * travis test * travis build failing with 0 errors and 0 failures, 40 "unexpected successes" * added updated docker-compose.yaml * cleanup * initial couchdb projectdb implementation * test url parser * fix couchdb connect url * fix couchdb connect url * fix couchdb json encoding * fix couchdb json encoding * fix couchdb url encoding * fix couchdb urls * fixed couchdb request headers * travis upgrade couchdb * travis upgrade couchdb * travis upgrade couchdb * travis upgrade couchdb * travis upgrade couchdb * fixed "Fields must be an array of strings, not: null" eroor * fixed responses * fixed drop database * tracing insertion issue * fixed default values * tracing update bug * fixed update bug * fixed drop bug * changed default fields * fixed drop bug * fixed _default_fields usage * fixed update bug * fixed update bug * fixed drop bug * tracing update bug * fixed drop bug * tracing drop bug * fixed drop bug * fixed db naming issue * fixed drop bug * initial resultdb implementation * added resultdb tests * fix resultdb tests * fix resultdb init * fix resultdb init * fix missing class var * fixed get_docs * fixed db naming * fixed db naming * fixed db naming * fixed get_docs * minor fixes * fixed update_doc * fixed update_doc * fixed get_doc * fixed get_docs * fixed get_docs * fixed parse * fixed get_all_docs * fixed get_doc * fixed update_doc * minor fixes * fixed select * initial taskdb implementation * added debug prints * added collection_prefix * minor fixes * minor fixes * fixed update * fixed test_25_get_task * fixed status_count selector * fixed update * tracing test_create_project bug * fixed collection naming * Revert "fixed collection naming" This reverts commit 0d89a0d. * fixed collection naming * minor fixes * minor fixes * fixed test_z10_drop * fixed test_50_load_tasks * fixed get_docs * fixed get methods * cleanup * removed python 3.3 and added 3.7 and 3.8 * added index * tracing index create bug * fixed index create bug * fixed index create bug * fixed index create bug * minor test fixes * added couchdb test run * added couchdb test run * full working example * fixed test setup * fixed test setup * updated travis file for couchdb auth * updated travis file for couchdb auth * added credentials exception * fixed credentials * fixed test auth * fixed test auth * tracing auth issue * tracing auth issue * fixed test auth issue * fixed test test_60a_docker_couchdb * fixed test test_60a_docker_couchdb * cleanup * attempting to remove "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * Revert "tracing "unexpected successes"" This reverts commit 829da8c. * tracing "unexpected successes" * tracing "unexpected successes" in crawl * tracing "unexpected successes" in crawl * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * tracing "unexpected successes" * fixed "unexpected successes" * fixed TestFetcherProcessor * fixed TestFetcherProcessor * fixed TestFetcherProcessor * fix BaseHandler * fix BaseHandler * fix BaseHandler * fix BaseHandler * fix BaseHandler * fix BaseHandler * fix BaseHandler * removed beanstalkc * cleanup * removed 3.8 from travis * removed python 3.8 from setup.py * fixed test_60_relist_projects change * fixed .travis * added https to couchdb + cleanup + added couchdb to docs * added extra comment on top of docker-compose example * fixed docker-compose issue * improve docker-compose sample * remove demo link * fix test break because couchdb failing to start * try to use non-auth for CouchDB test * more couchdb_password * improve couchdb allow empty username password * drop support for couchdb Co-authored-by: Roy Binux <[email protected]> Co-authored-by: jxltom <[email protected]> Co-authored-by: binux <[email protected]> Co-authored-by: sdvcrx <[email protected]> Co-authored-by: Lucas <[email protected]> Co-authored-by: vibiu <[email protected]> Co-authored-by: farmercode <[email protected]> Co-authored-by: Phillip <[email protected]> Co-authored-by: feiyang <[email protected]> Co-authored-by: clchen <[email protected]> Co-authored-by: v1nc3nt <[email protected]> Co-authored-by: Keith Tunstead <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It looks like creation of resultdb is not same as creation of projectdb and taskdb. This may cause error for postgres in Python3