Hello,
I am looking for someone to build a scraper that I will be able to run anytime, to get public data from brazilian websites.
SPECIFICATIONS:
The original url doesn't need to be scraped. Instead, it has a FORM.
I need to fill that form repeatedly using data in a csv file on the same server (or another idea).
Then export SPECIFIC data from the resulting page to a created csv file (probably).
4x this. It's 4 different original URLs in 2 different domains.
Go to url -> fill form based on csv file data -> export specific data from result after a few clicks to another csv.
Repeat several times.
There IS recaptcha at 2 of the 4 form submissions. And you will need to make a few "clicks" after some submissions to scrap all data thereafter.
I am looking for someone who can do this. My best guess would be a scrapy folder that can be run from the server itself (CentOS), as I need this to be run anytime by me in the future, editing only the original csv files with the form data to be submitted.
I also would also need antiblocking tools to be setup somehow, like connection to a VPN which I will have to buy, tell me what I would need to make this work.
I would like to hear what programs and solutions you would be using to solve this problem, including any additional problems or costs for subsequent scrapings.
Thank you very much, and I am available to provide more info quickly, for us to get this going soon.
I have 6 Years of experience in Automation Testing, I worked on a saas product startup's, I have well experience in Appium mobile automation(Android and iOS) and selenium framework design from scratch.
Skills:
1. Appium mobile automation - iOS and Android
2. Selenium- TestNg/Cucumber
3. Mobile Automation on AWS Device Farm using Appium
4. Jmeter Setup's In google cloud or docker
5. Framework Integration into CI/CD
6. Performance Testing
7. Git and Jenkins
8. Single Automation Framework design (iOS & Android)
9. Test Reporting Extent Reports and Allure Report generation