 |
 |
|---|
An open source online comparator that operates ecological scoring, following a common good mindset.
- financially contributing to the environmental cause, through a repayment of 20% of the company incomes.
- provide products datasets in open data
- This project is for now deployed on the frenchy nudger.fr
Technical metrics and maven site is deployed on Github Pages:
Open4goods (o4g) is an open-source and open-data product aggregator, search engine and comparator. More over, it is a stack aiming at handling large product datasets identified by GTIN's. It is build upon Maven, Java, SpringBoot, Elasticsearch, Redis, and some other cool libraries. It is mainly designed to :
-
crawl and ingest product based datas (merchant offers, reviews, brands ratings). This is the job of the crawler sub-project. Designed to ingest any kind of data, if having an UPC/GTIN code.
-
aggregate data fragments into well structured product-data, by verticals. Features are -among others- scoring, attributes merging and conflict detection, comments NLP processing... The business intelligence of product construction is weared in the aggregation sub-project, and the orchestration and product data construction is operated through the scalable API component.
-
Renders the data through API's, UI's, and through open data sets, exposed through the API component.
-
Make the truth available for everyone on officially supported websites, (only french nudger.fr for now), through the UI component. The affiliation revenue business model is used to deliver ecological compensation, maintain the user service and the open-data delivery
There are several ways to contribute to the project.
By buying your products on official sites, you "create" money through the affiliation system, that allows to reverse the "by law" 20% environmental compensation. By using it you will also able to provide us some feedbacks
Communication will be the lack of this Odyssey, any kind of help is welcome. Talking about this the project to your granny or even to your dog could help us a lot.
Bugs and ideas are greatly welcome ! We have a specific feedback system that allows you to report feedback to Github Issues directly from your navigation on our frontends.
Verticals (eg : tv's, washing machines, ...) are defined through yaml configuration files. The ecoscore is part of the verticals configurations.
Our verticals definitions are totaly open, meaning that you are very welcome to :
- inspect them, by reviewing the verticals configuration
- comment / question, by posting issues with your concerns
- contribute, by creating new verticals or by updating existing ones with Pull Requests
Contribute to the websites, on the UI, on the content, or on the data aspects. Quiet simple to set-up, you can run open4goods website localy with the below instructions.
We will see here how to run open4goods frontends and API's on tour computer.
You will need :
- Java 21+
- Maven
- Elasticsearch and Redis, that can be transparently provided through Docker and Docker Compose
Elasticsearch and Redis are used by the open4goods project. A Kibana instance is commented in the docker-compose.yml file, if you want to browse data's by yourself. Go to the project root, then start the compose file
docker compose up
Docker compose is managed by spring-boot-docker-compose. That means that the manual launching of docker-compose is not mandatory, containers will be started by the application if not present.
Elastic, kibana and Redis should be available on :
- Elastic Search : http://localhost:9200
- Kibana : http://localhost:5601
- Redis : http://localhost:6379
Before running the docker compose check that the value of 'vm.max_map_count' is higher or equal to 262144. If not, you will have to raise your max map args to be able to rune the Elastic image, see the Hint's section
Jars are not published to any central repo (nor planned to, it seems not to make any particular sense). To build, please go into the project folder. Then :
mvn install
This will build and run tests, hope in your terminal you'll get a
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for parent 0.0.1-SNAPSHOT:
[INFO]
[INFO] parent ............................................. SUCCESS [ 2.509 s]
[INFO] commons ............................................ SUCCESS [ 27.046 s]
[INFO] crawler ............................................ SUCCESS [ 8.990 s]
[INFO] verticals .......................................... SUCCESS [ 7.143 s]
[INFO] api ................................................ SUCCESS [ 11.289 s]
[INFO] ui ................................................. SUCCESS [ 7.895 s]
[INFO] test ............................................... SUCCESS [ 1.496 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:07 min
[INFO] Finished at: 2021-01-31T15:20:30+01:00
[INFO] ------------------------------------------------------------------------
The open4goods project is packaged under the form of several SpringBoot web applications. You will probably want to launch :
- API component : Play with the data aspect and the business logic
- UI component : Play with the user interface part of the question
From the project root folder, you can now launch the aspects of the platform you are interested in :
UI
java -Dspring.profiles.active=dev -jar ui/target/ui-[VERSION].jar
You should be able to access the open4goods user interface at http://localhost:8082
Api
java -Dspring.profiles.active=dev -jar api/target/api-[VERSION].jar
You should be able to access the open4goods API at http://localhost:8081
Crawler
You should not need to run a separate crawler, since an embedded one is instanciated in the API. However, if you want to play, or register as a open4goods web scrapper and help us in crawling the world, you could setup an individual crawler node.
java -Dspring.profiles.active=dev -jar target/bin/open4goods-crawler.jar
You should be able to access the open4goods crawler interface at http://localhost:8080
If using any kind of IDE (tested with Eclipse and Intellij), please import as maven or SpringBoot project, then run or debug from the following Application classes :
- UI : ui/src/main/java/com/open4goods/ui/Ui.java
- API : api/src/main/java/com/open4goods/api/Api.java
- Crawler : crawler/src/main/java/com/open4goods/crawler/Crawler.java
Don't forget to specify the profile to use (probably you'll want "dev" to run in development mode)
For now, you have an up and running open4good platform. You will like to play with some datasets, and like it's not so easy to get them, we provide some live datas directly from our websites.
A special service will automaticaly load sample datas on applications startup, allowing you to easily play with the project.
TODO : Document
TODO : Document
TODO : Document
TODO : Document
TODO : Document
TODO : Document
TODO That means that simple PR's to TODO can directly make new products of enhance the user experience of all open4goods users.
You will probably need to raise your max map args in order to be able to start elastic
sudo sysctl -w vm.max_map_count=262144
To permanently set the max virtual memory :
- edit /etc/sysctl.conf
- add line vm.max_map_count=262144
- sudo service sysctl restart
After a space crash, elastic indexes are locked. You can get hand back on them by using :
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'