Skip to content

Latest commit

 

History

History
223 lines (136 loc) · 10.5 KB

README.md

File metadata and controls

223 lines (136 loc) · 10.5 KB
Raw

The project

Beta Production

An open source online comparator that operates ecological scoring, following a common good mindset.

Technical metrics and maven site is deployed on Github Pages:

Open4goods (o4g) is an open-source and open-data product aggregator, search engine and comparator. More over, it is a stack aiming at handling large product datasets identified by GTIN's. It is build upon Maven, Java, SpringBoot, Elasticsearch, Redis, and some other cool libraries. It is mainly designed to :

  • crawl and ingest product based datas (merchant offers, reviews, brands ratings). This is the job of the crawler sub-project. Designed to ingest any kind of data, if having an UPC/GTIN code.

  • aggregate data fragments into well structured product-data, by verticals. Features are -among others- scoring, attributes merging and conflict detection, comments NLP processing... The business intelligence of product construction is weared in the aggregation sub-project, and the orchestration and product data construction is operated through the scalable API component.

  • Renders the data through API's, UI's, and through open data sets, exposed through the API component.

  • Make the truth available for everyone on officially supported websites, (only french nudger.fr for now), through the UI component. The affiliation revenue business model is used to deliver ecological compensation, maintain the user service and the open-data delivery

How to contribute

There are several ways to contribute to the project.

Use it

By buying your products on official sites, you "create" money through the affiliation system, that allows to reverse the "by law" 20% environmental compensation. By using it you will also able to provide us some feedbacks

Speak about it

Communication will be the lack of this Odyssey, any kind of help is welcome. Talking about this the project to your granny or even to your dog could help us a lot.

Feedback us

Bugs and ideas are greatly welcome ! We have a specific feedback system that allows you to report feedback to Github Issues directly from your navigation on our frontends.

Eco-score and vertical definitions

Verticals (eg : tv's, washing machines, ...) are defined through yaml configuration files. The ecoscore is part of the verticals configurations.

Our verticals definitions are totaly open, meaning that you are very welcome to :

Geek it !

Contribute to the websites, on the UI, on the content, or on the data aspects. Quiet simple to set-up, you can run open4goods website localy with the below instructions.

Run in dev mode

We will see here how to run open4goods frontends and API's on tour computer.

Software Requirements

You will need :

running through docker-compose

Elasticsearch and Redis are used by the open4goods project. A Kibana instance is commented in the docker-compose.yml file, if you want to browse data's by yourself. Go to the project root, then start the compose file

docker compose up

Docker compose is managed by spring-boot-docker-compose. That means that the manual launching of docker-compose is not mandatory, containers will be started by the application if not present.

Elastic, kibana and Redis should be available on :

Before running the docker compose check that the value of 'vm.max_map_count' is higher or equal to 262144. If not, you will have to raise your max map args to be able to rune the Elastic image, see the Hint's section

Building the open4goods project from code base

Jars are not published to any central repo (nor planned to, it seems not to make any particular sense). To build, please go into the project folder. Then :

mvn install

This will build and run tests, hope in your terminal you'll get a

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for parent 0.0.1-SNAPSHOT:
[INFO]
[INFO] parent ............................................. SUCCESS [ 2.509 s]
[INFO] commons ............................................ SUCCESS [ 27.046 s]
[INFO] crawler ............................................ SUCCESS [ 8.990 s]
[INFO] verticals .......................................... SUCCESS [ 7.143 s]
[INFO] api ................................................ SUCCESS [ 11.289 s]
[INFO] ui ................................................. SUCCESS [ 7.895 s]
[INFO] test ............................................... SUCCESS [ 1.496 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:07 min
[INFO] Finished at: 2021-01-31T15:20:30+01:00
[INFO] ------------------------------------------------------------------------

Launching

The open4goods project is packaged under the form of several SpringBoot web applications. You will probably want to launch :

  • API component : Play with the data aspect and the business logic
  • UI component : Play with the user interface part of the question

Command line

From the project root folder, you can now launch the aspects of the platform you are interested in :

UI

java -Dspring.profiles.active=dev -jar ui/target/ui-[VERSION].jar

You should be able to access the open4goods user interface at http://localhost:8082

Api

java -Dspring.profiles.active=dev -jar api/target/api-[VERSION].jar

You should be able to access the open4goods API at http://localhost:8081

Crawler

You should not need to run a separate crawler, since an embedded one is instanciated in the API. However, if you want to play, or register as a open4goods web scrapper and help us in crawling the world, you could setup an individual crawler node.

java -Dspring.profiles.active=dev -jar target/bin/open4goods-crawler.jar

You should be able to access the open4goods crawler interface at http://localhost:8080

Using an IDE

If using any kind of IDE (tested with Eclipse and Intellij), please import as maven or SpringBoot project, then run or debug from the following Application classes :

Don't forget to specify the profile to use (probably you'll want "dev" to run in development mode)

Loading some datas

For now, you have an up and running open4good platform. You will like to play with some datasets, and like it's not so easy to get them, we provide some live datas directly from our websites.

Products auto-loading

A special service will automaticaly load sample datas on applications startup, allowing you to easily play with the project.

A note on the work directory

TODO : Document

A note on Xwiki

TODO : Document

How to's

TODO : Document

Play with the UI

TODO : Document

Play with the data

TODO : Document

Ingest new data

TODO : Document

Transform data : the aggregation mechanism

Hints

Report issues from code

TODO That means that simple PR's to TODO can directly make new products of enhance the user experience of all open4goods users.

Elastic max map count

You will probably need to raise your max map args in order to be able to start elastic

sudo  sysctl -w vm.max_map_count=262144

To permanently set the max virtual memory :

  1. edit /etc/sysctl.conf
  2. add line vm.max_map_count=262144
  3. sudo service sysctl restart

Elastic space crash

After a space crash, elastic indexes are locked. You can get hand back on them by using :

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'