Find Jobs
Hire Freelancers

Web crawler / scrapers for bloomberg

$80-240 HKD

Closed
Posted over 5 years ago

$80-240 HKD

Paid on delivery
Detected by the web site as a robot Need to fix below to crawl the web successfully $ch = curl_init(); $headers = array(); $headers[] = "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0"; $headers[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"; $headers[] = "Accept-Language: zh-TW,zh-HK;q=0.8,en-US;q=0.5,en;q=0.3"; $headers[] = "Dnt: 1"; $headers[] = "Connection: keep-alive"; $headers[] = "Upgrade-Insecure-Requests: 1"; $headers[] = "Te: Trailers"; curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET"); curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate'); curl_setopt($ch, CURLOPT_URL, "[login to view URL]:IND"); curl_setopt($ch, CURLOPT_MAXREDIRS, 10); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 100); curl_setopt($ch, CURLOPT_REFERER, "[login to view URL]:IND"); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,2); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,false); curl_setopt($ch, CURLOPT_FAILONERROR, false); curl_setopt($ch, CURLOPT_HEADER, 1); curl_setopt($ch, CURLOPT_COOKIE, $cookie); curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE); curl_setopt($ch, CURLOPT_COOKIEFILE, COOKIE); $result = curl_exec($ch); if (curl_errno($ch)) { echo 'Error:' . curl_error($ch); } curl_close ($ch);
Project ID: 17927169

About the project

6 proposals
Remote project
Active 5 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
6 freelancers are bidding on average $277 HKD for this job
User Avatar
Hi sir, This is Lin and I am scraping expert, please check my reviews then you will know. Can we discuss more details about this? Thanks, Lin
$240 HKD in 2 days
5.0 (497 reviews)
8.1
8.1
User Avatar
I can update your PHP scraping script so it will be not detected as a robot. I can start right now and complete it in next 2 hours.
$240 HKD in 1 day
4.9 (597 reviews)
7.9
7.9
User Avatar
Hi, I have checked the requirements. Will do this script using PHP selenium web driver. I'm available to start now. Please share additional details. Regards, Mohan
$240 HKD in 1 day
4.9 (149 reviews)
6.1
6.1
User Avatar
Hi, do you know how Bloomberg website detected your web crawler as a robot when you're presenting it as Firefox 62.0. They use JavaScript based pings to monitor their website traffic. When we fetch the page using cURL only the HTML file get downloaded. There is no way to execute the JS files on that page and hence the website knows that a webbot is trying to masquerade as a browser. The solution for fetching the content out of such websites is to use a headless browser like PhantomJS. I can build a web scraper for you using PhantomJS and PHP to download content from Bloomberg website. My rate is higher than most other freelancers because programming PhantomJS is not easy. We have to use asynchronous JavaScript programming techniques to load the website and then wait for it to finish loading before retrieving content out of it. I have developed several scrapers using PhantomJS and have quite a bit of experience in programming it. So if you would like to discuss your project with me then feel free to send me a message even when I'm offline. I usually respond within a few minutes when online and within a few hours when offline.
$500 HKD in 3 days
5.0 (54 reviews)
6.2
6.2
User Avatar
I see the code is only for loading. What exactly need to be scrapped and where to store this? Want to discuss more details on the project
$240 HKD in 1 day
4.9 (77 reviews)
5.7
5.7
User Avatar
Hello! I feel your job is an interesting one, so I would like to offer you my services. I have more than 8 years of experience in web-development. I have studied your problem carefully and I am ready to start working on your project.
$200 HKD in 1 day
0.0 (0 reviews)
0.0
0.0

About the client

Flag of HONG KONG
hong kong, Hong Kong
5.0
4
Payment method verified
Member since Oct 24, 2013

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.