I have a custom code that is used to scrape data from a different website and save images and data in MySQL database. It uses simplehtmldom and Nokogiri to scrape and parse data, angrycurl for proxy connection, etc.. MVC framework is used through the code.
There are bugs and issues that need to be fixed with the code and I need to developer who understand scraping to look into it.
1. fix all the scrapers for the different website to ensure there is no errors and the correct data is being scraped with the correct logic applied before saving into database.
2. There is a refresh mechanism so if the information of a product is changed, products are listed in the refreshed products page. There is a bug prevent this from working.
3. The script also greater XML and csv with product information. Some changes are required to the data and layout of these files.
4. There are currently 2 x [login to view URL] files, so configuration information is saved in two files. Need to merge them info one fire and ensure its secure/safe.
5. Optimize code/script for security and speed of scraping.
Please refer to the attached screnshots.