Find Jobs
Hire Freelancers

Expert to build a web using Scrapy + djangoCMS + AngularJS4

$130-185 NZD

Closed
Posted over 5 years ago

$130-185 NZD

Paid on delivery
IMPORTANT NOTES: 1) Please read the project details and attachment before you bid. 2) Full requirements as in attachment, which is fairly clear 3) Budget currency is NZD 4) Total price of this job is as high as you bid, no-negotiable. sources: 1)[login to view URL] 2)[login to view URL] ASUMPTIONS: A1: 1) scrap by using regex to complete the ID for the whole URL A2: 2) can only be scrap based on looping each category/subcategory to get all listings in each page If you have a better way for A2, that is great. --------------------------------- Functional requirements: --------------------------------- R1. scrap all categories (see Appendix 2: product categories in attachment) R2. scrap all listings of a website R3. scrap production information and delta information of an existing listing: price, stock^, sold qty^, see Appendix 1:schema tables below. R4. Admin backend UI to: 4.1 list all existing sites, last ran schedule and add new site(name,site url) 4.1.1 a button for pop up to set up schedule to start scrapping a site and a button to start scrap the site 4.1.2 a button to drill down the siate and redirect a new pate for displaying categories of a site 4.2 add new category in 4.1.2 and a button to scrap new category 4.3 display all listing under the category. 4.5 a page to display fields and setting for a scrapy item. Modify existing css rule for existing field of a listing 4.6 add a new css rule to scrap a new field(document/write HOWTO steps for changing the code if needed) ^: if field/data exists in page R5. webAPI to return result of listing query R6. front end pages to: 6.1 display all categories of a site on left and display all listings a site - paged 6.2 click on one listing in 6.1 and open a new page to display detail of a listing. product name, brand, image, latest price/price history(scrapped by #R3) R7. user login and add listing to watch list --------------------------------- Technical requirements: --------------------------------- Backbones: Scrapy + djangoCMS + MySQL + SQLAlchemy + AngularJS4 A. add scrapy module as an app for djangoCMS 1. support multi-threading 2. anti-scraping: [login to view URL] tool:RandomProxy + RotateUserAgentMiddleware 3. each listing has an Unique identifier(foreign key for other tables) 4. data access layer: SQLAlchemy B. admin backend 1. support login( djangoCMS support by default) 2. four *admin* pages integrated to *djangoCMS admin backend*, refer to in attachments(site page / rule page/ product page/ users) C. WebAPI as an app for django (for R5) 1. input: text(product name) or unique identifier 2. return: array of listing in JSON format [{Id, productName, brand, last price, last date, image url, [category 1, category 2, ...]}] 3. authentication token(64bit) 4. security: SSL/https D. front end to display (for R6) 1. UI(Angular4 + Bootstrap) 2. security: SSL/https 3. call webapi to retrieve data 4. support CORS: [login to view URL] 5. listing page to display products, display top 20 products by default order by last scrapped time. 6. detail page to display product details E. naming convention: CamelCase or underscore F(optional). it is great if WebAPI result can be used for both admin backend and frontend, if you want, you can use WebAPI for admin backend   Appendix Appendix 1: schema tables: 1. site (id, name, baseUrl, enable, schedule, lastScrapDateTime, lastStatus) store site as source, enable flag indicate the schedule is active for scrapping, update lastScrapDateTime each time the spider ran) 2. siteRule(id, siteId, ruleName, enable, type^, url, minId**, maxId**, linkRegex^^) ^type: depend on scrap by Id or category links (see [sources] at the beginning above) .... please find more in attachment
Project ID: 17441593

About the project

6 proposals
Remote project
Active 6 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
6 freelancers are bidding on average $572 NZD for this job
User Avatar
Hello , How are you? I just saw your project description carefully. I am very interested in your project. I have rich experience in Scrapy + djangoCMS + AngularJS4. I am ready to start your project immediately. I am sure I can satisfy you surely and complete in a short time. Please send your message so that we can discuss about your project. ************************************************************************
$175 NZD in 10 days
4.9 (46 reviews)
7.0
7.0
User Avatar
Hi, Sir. I am a senior web developer with 5 years of experience in building website. I am very experienced in web development with PHP(CodeIgniter, Laravel, WordPress, ...), Java(JSP, Spring, Hibernate, Struts, JSF...), JavaScript(Angular, ReactJS, ...) and so on. Here are my generic working style. - Communication is first, my timezone is as flexible as you want. - Best user experience and keep pixel perfect design. - Always not satisfied with my results and try harder to improve. - Quality of code is guaranteed. (Robust and bug-free, adopt design patterns) - Bug & issue track using Github, Bitbucket and so on. I am sure I can deliver high quality product and I want to have an interview with you. Please consider working with me. I will do my best. Thanks.
$500 NZD in 10 days
5.0 (6 reviews)
5.3
5.3
User Avatar
Hello, Hope you are doing well. The last 8 year's I have focused on high-quality Web, IOS, Android and Computer Vision development. Please see my range of skill sets below. After coming on chat discussions I will show previous work. My technical skills: 1. Web Development - Angular 1,2,4,5, React.js, , HTML/CSS, JavaScript/jQuery, Bootstrap... - Node.js, Express, PHP, Laravel, Codeigniter, Yii, CakePHP, Symfony, Zend, Django... - Twilio, Facebook, Instagram, Twitter, Google Map, Youtube, Paypal, Stripe... - MySQL, MongoDB, Redis... 2. Mobile/CV Development - Android, iOS(Swift, Object C), Cordova, Ionic Framework, React Native ... - Java, C++, C#, Python, Matlab, Machine Learning, OCR, Image Processing.. I take pride in my attention to detail, and technical expertise required to deliver high quality, bug-free code. I approach all projects with a positive attitude, and perspective that no project is too difficult and the timeframe is unattainable. I would be really appreciated, If you will open a line of chat to discuss further Thank you for your time and consideration
$144 NZD in 10 days
5.0 (4 reviews)
1.5
1.5

About the client

Flag of NEW ZEALAND
Wellington, New Zealand
0.0
0
Member since Apr 15, 2015

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.