Closed

WEB SCRAPPING Application

We need an application hosted in windows server, that can scrap web pages and recollect data, the detail of the project are on the word document attached, please only need expert on web scrapping tasks.

Technology Stack

1. Web driver or html class

2. Language programming: C#

3. .Net Framework : 4.5.2 or netcore 2.0

4. Project type: Console Application

Milestones

1. FreeLancer create the demo scrapping application using an example scrapping feed file provided by us.

2. FreeLancer test the application using the 10 scrapping files and will be success if the result has more than 75% url found match

3. David review the Scrapping result file

4. David send 10 scrapping feed files more to test the application

5. David Approve or rejected the project according with the result on step 4

6. David pay to FreeLancer for the source code depend on the step 5

Description: See attached image on the last page

1. The VDP Scrapping console read the “scrapping feed file” text, this file contains 6 columns , with PIPE delimiter:

- Domain main url (required) : Is the main url of the dealer

- New Car Inventory Search Page (optional) : Is the page where we can find the list of new cars (can have pagination)

- Used Car Inventory Search Page (optional): Is the page where we can find the list of used cars (can have pagination)

- Vin (required) : is the vin number of the car

- StockNumber (required) : is the stock number of the car

- Condition (required): Indicate if the car is new or used ( U = used , N= New )

2. When the VDP Scrapping console has the data in memory, for each line should be found the Vdp Url in base of the domain main url Or “new/used” car inventory search page (according with the condition of the vehicle)

- The VDP Scrapping console for each line will search on the domain main url Or “new/used” car inventory search page (according with the condition of the vehicle) by VIN if doesn’t find the vehicle, it will use the StockNumber to find the car, when find the car, find the link to go to the vdp url. This is the core of the application, we need to build an algorithm to find the car in multiple different html, for example, the website1 to find the car has a htmlcontrol “lnkvehicle” , but on the website2 to find the car has a htmlcontrol “aVehicleDetailPage”, but on the website 3 to fin the car has a htmlcontrol “ilinkvehicle”, so this is the complex, we need to be able to find this link on the html ( recommendations: the photo always has a link to go to the vdp url, fin the section of each car and then test each link found on this section,etc ). The VDP Scrapping console must to have a list of htmlcontrol by default to find the links of the cars to go to the vdp url, the The VDP Scrapping console allow to the user enter the control links names of the site that will be scrapping.

- If the VDP Scrapping console doesn’t find the car on the domain main url Or “new/used” car inventory search page (according with the condition of the vehicle) on the “vdpurl” field will show “NOTFOUND”

** Vdp url: is the vehicle detail page url, where show all information of one vehicle

3. When the VDP Scrapping console finish to scrapping all vehicles from the “scrapping feed file”, write a “Scrapping result file”, that will have 7 columns, the first 6 columns will be the same of the “scrapping feed file” and the last column will be the vdp url for each car

4. Exceptions

- When the domain main url Or “new/used” car inventory search page (according with the condition of the vehicle) is down, the VDP Scrapping console, show the error message to the user.

- When the domain main url Or “new/used” car inventory search page (according with the condition of the vehicle) doesn’t allow scrapping, the VDP Scrapping console, show the error message to the user

Skills: .NET, Data Mining, Scrapy, Software Architecture, Web Scraping

See more: web scraping software, is web scraping legal, web scraping tutorial, how to use web scraper, web scraping javascript, web scraper tool free download, web scraping python, web scraper chrome, convert windows desktop application web based application, web browser application bots, web based application form design sample template, use access database web based application, web appointment application, web database application, convert windows based application web based application, web programing application programing, web cam application, convert window based application web based application, internet web based application scheduling, web based application development

About the Employer:
( 1 review ) lima, Peru

Project ID: #17617340

40 freelancers are bidding on average $590 for this job

agilepksolutions

Can easily develop what you required as already done lot of scraping projects in my experience. My main and only skill is C# Desktop and Web using MVC on .NET and developing from more than 6 years. 99% projects on p More

$1000 USD in 15 days
(76 Reviews)
7.7
mhmhz

Hi David I always implement the scraper in C#, the scraper can be multi-threading. Can work on a demo if you like Thanks

$800 USD in 5 days
(205 Reviews)
7.7
waema

I have delivered more than 70 scraping bots this year alone. I have dealt with several car websites also. Let me know once back so that we can discuss more

$250 USD in 7 days
(186 Reviews)
7.3
mingxiao2008

Hello , Dear How are you? I have checked your project description and I am very interested in your project. I have experienced in developing .NET, Data Mining, Scrapy, Software Architecture, Web Scraping I will wor More

$555 USD in 10 days
(38 Reviews)
6.8
ITPyramid85

hello,how are you. i read your bid carefully. i am webscrapping expert and have full experience for 7 years. C# is my top skill and i can complete your project by using c#. I can provide most quality and high fast. More

$500 USD in 10 days
(11 Reviews)
6.8
zekovicm

Hi there,I am Miljan,Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it within More

$555 USD in 10 days
(58 Reviews)
6.6
itpathsolutions

Hello, I am Microsoft Certified developer with more than 8 years of IT industry experience and worked on different kind of projects from different geography. So I think your search for the best Microsoft developer i More

$750 USD in 10 days
(28 Reviews)
6.3
schoudhary1553

Hello Sir, I am the expert freelancer here. I am on the 6th position through out the world to deliver the quality job. I have deliver here more than 390 + projects with 100% client satisfaction. I can help you More

$750 USD in 10 days
(39 Reviews)
6.2
shivampanchal

I have a good hands on working with Advanced Excel, R and Python and BI tools and technologies, AI, Big Data. I have quite a good knowledge of DL/ML Algorithm , have also developed Dashboards and Web Application. My ar More

$500 USD in 10 days
(51 Reviews)
6.4
donnyforu

Hello We are an IT company based in India with a team of 60 developers who are skilled in experienced. We have read your project description and we are confident to do the Scrapping work as per your instructions More

$1000 USD in 10 days
(12 Reviews)
5.6
DavidLindon

a console app that will run on your Windows environment

$750 USD in 15 days
(31 Reviews)
5.7
shinelancer

*** I have checked your description. I have good experience on scraping with C#(web driver, httpclient and htmlagility etc) I already developed many scraper with C# for e-commerce and business website . you can see More

$700 USD in 10 days
(21 Reviews)
5.4
poduzov

Hello I'm a software developer with 10+ years of experience and made many automatic web scrapers for Mac, Windows, Linux which can do - simple HTML crawling and parsing using XPath, RegExp, HTMLAgilityPack - user More

$555 USD in 10 days
(28 Reviews)
5.4
profvipabutaleb

I'm computer engineerign TA with 10+ years of experience experienced with C#/HTML/PHP/Javascripit/JSP and scarping websites after analyzing different site components using Python/Java/c# including parsing the data an More

$400 USD in 10 days
(30 Reviews)
4.8
wuyong2020

I can do it! I have done a lot of web bots and scrapers. This project is perfect with me. Hope to work with you. Best Regards

$555 USD in 10 days
(3 Reviews)
4.8
skginfosolutions

Hi, I have gone through the job description and believe we will be the good fit for this job. We are experts in ASP.NET MVC, jQuery, Angular.Js Entity Framework, WCF (Web API), WPF and MSSQL, PHP, Yii, CakePHP, Code More

$666 USD in 20 days
(4 Reviews)
4.9
nbeslie

Hi, I am interested to work in your project. I can develop your scrapper application for your needs. I read your document , there are some questions that i want to ask. Please contact me for the details. Here is my More

$600 USD in 15 days
(12 Reviews)
4.4
Shuchi144

Hi there, Yes we have alread done the scraping and crawling of the data from different website. We scraped data from many e-commerce websites and pushed it in excel and in a database. using selenium , python and m More

$750 USD in 23 days
(6 Reviews)
4.4
BestPartner4You

hello,there. I have read all your requirements for 'WEB SCRAPPING Application' and I fully understood it. I am confident and I am sure that I am able to finish this project. Please come in contact with me, so that we c More

$555 USD in 4 days
(9 Reviews)
4.3
nathandatascrape

Dear Sir/Madam, I'm very interested in and willing to do your job. I have more than 3 years of experience with web scraping and I think I can help you with your project. I'm willing to make samples or test phase f More

$400 USD in 5 days
(9 Reviews)
3.9