Skip to content

A Tool to help import the content of the Learning Registry into a data store of your choice

License

Notifications You must be signed in to change notification settings

johnpaulbalagolan/LR-Data

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

LR-Data

This is a small utility to help pull the data from the Learning Registry into a datastore of your choice.

Dependencies

LR-Data requires:

Platform-Specific Requirements:

On OS X, you will also need libevent, which can be installed with homebrew: brew install libevent

Install dependencies - Ubuntu shown as example

sudo apt-get update
sudo apt-get install -y virtualenv python-dev libevent-dev libxml2-dev libxmlsec1-dev

Setup

# Create virtual environment
virtualenv env
. env/bin/activate
# Install python library requirements
pip install -U -r requirements.txt

Configuration

All configuration is done in the src/celeryconfig.py file. For information of configuring Celery please see their document. For lr-data configuration modify

config = {

	"lrUrl": "http://lrdev02.learningregistry.org/harvest/listrecords",

	"couchdb":{

		"dbUrl":"http://localhost:5984/lr-data"

	},
	"tasks": {
		"insert": "tasks.save.insertDocumentMongo",

		"validate": "tasks.validate.emptyValidate",
	}

	"redis":{

		"host":"localhost",

		"port":6379,

		"db":0
	}

	"mongodb":{

		"database":"lr",

		"collection":"envelope",

		"host": "localhost",

		"port": 27017,

	},
}

Customizable tasks are defined in the tasks hash. validate is the task name for validating incoming docs. insert is the task you wish to use to save the data

Startup

There are scripts inside of src to get you started.

  • start_celery_workers.sh - will start your default worker threads responsible for harvesting, validating, and saving LR data
  • stop_celery_workers.sh - stops workers
  • start_harvesting.py - sends harvest request into queue for workers to start working

To start harvesting, activate your virtualenv, start the celery workers, then start harvesting request

cd src
. ../env/bin/activate
./start_celery_workers.sh
./start_harvesting.py

Should you want to stop (or pause) the processing you can stop the celery workers:

./stop_celery_workers.sh

When you want to resume processing, you need only start the celery workers again:

./start_celery_workers.sh

About

A Tool to help import the content of the Learning Registry into a data store of your choice

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 90.3%
  • JavaScript 5.8%
  • Shell 3.9%