Find Jobs
Hire Freelancers

Build a Web crawler for a crawler-protected site

€30-250 EUR

Cancelled
Posted over 5 years ago

€30-250 EUR

Paid on delivery
I need a way to crawl a site which is hosted at Cloudflare. It is protected against automated access, but open to access from a real web browser. I suppose they have velocity checks, etc. But I am not sure. I need to receive the data in a PHP application. So the crawler part can be either a PHP component, which I can call from my program, or a web browser-based crawler, which then sends the data to my app via http. Both solutions are fine for me. So, in short, what I need is a component, either PHP or browser-based, which can send the contents of the URL. I can extract the data I need, it is not important - just to retrieve the content of the URL. When bidding, please explain your approach. If you need it, ask me and I will PM you the site address for you to see.
Project ID: 18108199

About the project

12 proposals
Remote project
Active 5 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
12 freelancers are bidding on average €193 EUR for this job
User Avatar
Hi there,I am Miljan,Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it within the agreed deadline. Check out my profile, portfolio and former clients feedback - that'll let you know everything about me. Please feel free to contact me so that we can discuss further details. Thank you for taking the time to read my proposal.I am looking forward to hearing from you. Best regards, Miljan
€155 EUR in 3 days
4.9 (82 reviews)
7.0
7.0
User Avatar
Hello, Hope you are doing well. I can help with you in your project Build a Web crawler for a crawler-protected site. I can assure you the quality job. I have good experience in Web Scraping. We have worked on several similar projects before! We have worked on 400+ Projects. Please check the profile reviews. I can deliver your job with in your deadline. Please ping me for more discussion. I can assure the 100% job satisfaction. Thanks,
€250 EUR in 4 days
5.0 (14 reviews)
5.8
5.8
User Avatar
Dear Client,   I have read your description carefully, and finally I think I will be good candidate for your project. Throughout these years, I have gained abundant experience with scraping using scrapy, selenium web driver, puppeteer, beautifulsoup and so on.  And also have skilled in Multi-threaded scraping, bypassing captchas, using PROXY/VPN to rotate IP. If everything is impossible, I will create real bot which uses real web browser(not test web browser). Anyway, please share url, at that moment, we can determine difficulty of this work. Thanks
€250 EUR in 3 days
5.0 (4 reviews)
4.9
4.9
User Avatar
- i will do your work . with browser based crawling approach Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON (Scrapy, Selenium) based web scraper as well as WINDOWS BASED web scraping software through which I have crawled many sites such as Craigslist, Amazon, Yelp and many others. I have also worked on complex site to bypass CAPTCHA with the use of PROXY IP bouncing techniques.. Let's work together :) Have a great day! I am glad to see your WORK HISTORY and positive reviews of other freelancers. I am really excited to work with you and would love to have a long-term business association for any of your data related needs less
€55 EUR in 3 days
4.9 (37 reviews)
4.7
4.7
User Avatar
Hello, could you please send me the link so I can test a phantomjs that is a scripted, headless browser used for automating web.
€200 EUR in 3 days
5.0 (1 review)
2.6
2.6
User Avatar
Hello, I've read your project details, According to my experience in the Creating Web Scrappers I can Help you in the project. I can bypass cloudflare protection, please give me the host URL in chat to check. Portfolio: Recently I created 2 scrapers for 2 project owners and both left excellent feedback on my profile. Scraper for checking "Google Shop" for product name price and sellers by sending Part Number and then saving in excel file: https://www.freelancer.com/projects/php/google-shopping-scraping/ Scraper for "Youtube" for Gathering Video file Titles, URLs and Publish date and saving in excel file: https://www.freelancer.com/projects/software-architecture/youtube-channel-video-listing-program/ Scraper for "Youtube Playlits" + Web Panel: https://www.freelancer.com/projects/website-design/Video-Playlist-Program-Service/ I understand the range and importance of your task and I can provide you perfect result . Please lets talk more about the project in chat. Thank you
€130 EUR in 5 days
5.0 (1 review)
1.8
1.8
User Avatar
i am expert in web scraping i did many websites scrap just like amazon your satisfaction is my priority let see more in chat
€111 EUR in 3 days
5.0 (2 reviews)
1.4
1.4
User Avatar
I just did this for a client. It was pretty tricky to get around. You have to solve cloudflare's js puzzle or you'll be stuck with: <h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1> I don't know PHP, but what I am able to do is write you a python program that can get by cloudflare's javascript wall. Let's talk and I can even demo that I have this working already
€250 EUR in 1 day
5.0 (1 review)
1.1
1.1
User Avatar
Hi, I intend to use python with packages like Mechanize or selenium to simulate a browser and also add a proxy component to help circumvent anti web scraper protection. I will provide a clean and understandable script that will send the needed data to your PHP application. Please send the website so I can see what I am up against. Thanks
€250 EUR in 3 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Buenas, somos un grupo de desarrolladores que estamos disponibles para crear e implementar su proyecto en el menor tiempo posible. Nos gustaría poder apoyarle en desarrollar el proyecto , teniendo en consideración sus requerimientos en el momento que sean planteados. Podríamos coordinar para mostrar los sistemas o proyectos similares a los que usted esta necesitando. Nos gustaría saber mas a fondo los detalles de su proyecto y en que tiempo aproximado desea tenerlo quedamos a la espera de su contacto.
€155 EUR in 5 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of SPAIN
Barcelona, Spain
5.0
29
Payment method verified
Member since Apr 30, 2004

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.