The first open source tool that operates ecological scoring, doing its best for the common good.
- Financially contributing to the environmental cause, through a repayment of 20% of the site incomes.
open4goods project is an open-source and open-data product aggregator, search engine and comparator. It is build upon Maven, Java, SpringBoot, Elasticsearch and some other cool libraries.It is designed to :
-
crawl and ingest product based datas (merchant offers, reviews, brands ratings). This is the job of the crawler sub-project. Designed to ingest any kind of data, if having an UPC code, a BRAND or a MODEL_NAME
-
aggregate data fragments into well structured product-data, by verticals. Features are -among others- scoring, attributes merging and conflict detection, comments NLP processing... The business intelligence of product construction is weared in the aggregation sub-project, and the orchestration and product data construction is operated through the scalable API component.
-
Renders the data through an open API, and through open data sets, exposed through the API component.
-
Make the truth available for everyone on officially supported websites, (only french nudger.fr for now), through the UI component. The affiliation revenue business model is used to deliver ecological compensation, maintain the user service and the open-data delivery
The project will do its best to maximize community and user contributions, with effective and fastest delivery on official websites. We would be happy when we will :
- provide the hugest set of barcodes in open data
- provides and deliver the best price comparison and product information platform, in a collaborative and open-sourced manner.
Mostly, it is a try and an attempt for an ethical and ecological business model. Made on blue time and after having lost in too much complexity, the actual platform is still missing a lot of cool features :
- green scoring
- developper documentation
- product filtering
- SEO optimisations
- A community
- And so more....
I will try to improve it over time, please feel free to get your feedback on the contact page, to contribute through code or with your ideas or issues on this github.
There are several ways to contribute.
-
Use it ! By buying your products on official sites, you "create" money through the affiliation system, that allows to reverse the "by law" 20% environmental compensation. By using it you will also able to provide us some feedback
-
Speak about it ! Communication will be the lack of this Odyssey, any kind of help is welcome. Talking about this the project to your granny or your dog could help us a lot.
-
Feedback us ! Ideas (philosophicals or pragmaticals ) are greatly welcome ! Allow us to get the truth (the user's one) with your enhancements, bug report.
-
Development (Docker, Elastic, Java, SpringBoot, Bootstrap, Jquery, ...)
You will need :
-
Elasticsearch, that can be transparently provided through Docker and Docker Compose
-
Unfortunately a Linux / Mac os, cause Java Path Separators are hardcoded. Not (really) a tooth against Windows User ;)
Elasticsearch is the only store used by the open4goods project. It is packaged with a https://www.elastic.co/fr/kibana instance in the docker-compose.yml file. Go to the project root, then start the compose file
$ docker-compose up
Elastic and kibana should be available on :
- Elastic Search : http://localhost:9200
- Kibana : http://localhost:5601
Note that you could have to raise your max vm args to be able to rune the Elastic image
$ sudo sysctl -w vm.max_map_count=262144
To permanently set the max virtual memory :
- edit /etc/sysctl.conf
- add line vm.max_map_count=262144
- sudo service sysctl restart
Jars are not published to any central repo (nor planned to, it seems not to make any particular sense). To build, please go into the project folder. Then :
$ mvn install
This will build and run tests, hope in your terminal you'll get a
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for parent 0.0.1-SNAPSHOT:
[INFO]
[INFO] parent ............................................. SUCCESS [ 2.509 s]
[INFO] commons ............................................ SUCCESS [ 27.046 s]
[INFO] crawler ............................................ SUCCESS [ 8.990 s]
[INFO] aggregation ........................................ SUCCESS [ 7.143 s]
[INFO] api ................................................ SUCCESS [ 11.289 s]
[INFO] ui ................................................. SUCCESS [ 7.895 s]
[INFO] test ............................................... SUCCESS [ 1.496 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:07 min
[INFO] Finished at: 2021-01-31T15:20:30+01:00
[INFO] ------------------------------------------------------------------------
The open4goods project is packaged under the form of several SpringBoot web applications. You will probably want to launch :
- API component : Play with the data aspect and the business logic
- UI component : Play with the user interface part of the question
From the project root folder, you can now launch the aspects of the platform you are interested in :
UI
$ java -Dspring.profiles.active=dev -jar ui/target/ui-[VERSION].jar
You should be able to access the open4goods user interface at http://localhost:8082
Api
$ java -Dspring.profiles.active=dev -jar api/target/api-[VERSION].jar
You should be able to access the open4goods API at http://localhost:8081
Crawler
You should not need to run a separate crawler, since an embedded one is instanciated in the API. However, if you want to play, or register as a open4goods web scrapper and help us in crawling the world, you could setup an individual crawler node.
$ java -Dspring.profiles.active=dev -jar target/bin/open4goods-crawler.jar
You should be able to access the open4goods crawler interface at http://localhost:8080
If using any kind of IDE (tested with Eclipse and Intellij), please import as maven or SpringBoot project, then run or debug from the following Application classes :
- UI : ui/src/main/java/com/open4goods/ui/Ui.java
- API : api/src/main/java/com/open4goods/api/Api.java
- Crawler : crawler/src/main/java/com/open4goods/crawler/Crawler.java
Don't forget to specify the profile to use (probably you'll want "dev" to run in development mode)
TODO : provide a playground dataset
You can use maven to get spring-boot and other dependencies update.
$ mvn versions:display-dependency-updates -DprocessDependencyManagement=false -DallowIncrementalUpdates=false
After a space crash, elastic indexes are locked. You can get hand back on them by using :
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'