Client engaged the eGrove developer for mining product information from different supplier sites and to import into their website. The product import script was developed to import 100K products owning 2800 categories from Dropshipping website of client with the help of Yahoo Japan API and other APIs. An algorithm was implemented within the import script to follow the rules of data mining from dropshipper/ supplier website. The imported products are translated in three (3) languages & stored in a temproary PGSQL database. The architecture used Celery Queuing system, celery flower, Rabbit MQ for asynchronous product data mining, queuing, importing to eCommerce website product catalog database. Import and product update are scheduled to execute once in a week automatically to Vivima.com from dropshipping supplier website.
- Product import script implementation to fetch products from Dropshipping supplier website
- Developed Celery Queuing system for importing 100k products from 2800 categories
- Schedule import process using Celery and RabbitMQ flower interface
- Auto synchronization of product import and update on weekly basis
- Imported Products are translated and uploaded in Vivima website
Python, Yahoo Japan API, Celery & Rabbit MQ, Celery flower, PostgreSQL, CRON, REST API, Daemon.