Shopping comparison site

Title: Shopping comparison site CMS which delivers information/products to user by collating data from different site. A shopping/information comparison site which is user friendly and aesthetically pleasing for end users. It also has dynamic related content in product and category pages. The high level functionality is; The crawler/bot/robot to crawl a set number of sites collate data and send it back to our server. Our indexing/publisher to deploy that data (Content delivery) to the live web server to serve end users/bots. In other words there are two stages to this design, both the stages are explained below. Stage 1 (Crawl various sites): A web crawler/bot which can be controlled using a front end (preferably web based) to configure the number of sites to be crawled, the frequency with which the bot crawls the site, etc. The crawled meta data is stored in Database or in a XML file. The bot or crawler can be set to crawl sites on a weekly or daily basis. It has to be multi-threaded and crawl at a specific rate (customisable) based on the server load. The bot crawls for simple product meta data viz Title, Crumbs/Category, Image, Price, Stock, Description, Shipping, and other relevant data, these data are then stored in a database. It has to simple relational database with 2 to 3 relevant tables holding the crawled information or alternatively in a XML file. There is also possibilities for certain sites to provide us a feed/google feed/etc which needs to be directly loaded into the database using native loaders. Stage 2 (Content delivery): The crawled data from stage 1 is then published/deployed in a MVC based standard CMS (eg: Apache + MySQL). This dynamic site has different categories and product page which displays the offers available for similar product from different site comparing their data viz; price, company/manufacturer, product, shipping, etc. This site also provides extensive search facility as it would have thousands of pages with offers and vital comparison information. The search tab should also provide auto complete (Ajax) and some advanced options for users (separate html page). This CMS should be search engine optimised and linked internally and externally to target natural organic traffic. The site should also have related item/products, featured/top sellers facility to dynamically cross link between category and product pages on every category or middle level pages. In other words most html pages would have the following 1) Navigation tabs on top of every page. 2) Left side search filtering on search result and category pages. 3) Related or top seller items banner on right or bottom of the page. 4) Global footer links (common) on all pages. 5) General static pages viz: T & C's, Contact Us, Sitemap, About Us, etc. 6) Graphic design incorporating company logo, sale banners, etc. Technical Requirements: Good knowledge on internet technologies, SEO capabilities, coding crawler/bots, using Open Source softwares eg: PERL, Java, .NET, etc, proficient with Database, Web Server, Java Script (Ajax). Stage 1: Crawlers can be designed from scratch (not ideal but creative idea are welcome) or built upon existing Open Source Software (There many free Open Source Crawlers which can be build upon or customised). The crawlers have to be customisable (regular expressions for extracting meta data from different sites), scalable (add more sites to crawl as we go), and configurable (add/delete new sites, crawl rate/frequency, fields/meta data to extract). The crawlers should have web based configurable interface to accomplish the above points. Currently we are planning to crawl 100 to 150 sites, but this can released in small stages. Stage 2: The content delivery or CMS will be based on MVC design pattern as mentioned above and can use Open Source or cheap products available in the market. The site also needs extensive search facility hence suggest using Sphinx Search or similar open source search engine which can hook on the live Database. As mentioned above related, featured and top seller products on category pages. Easy addition/removal of banners in most pages. The search filters on the left side can be JavaScript (Ajax) based or do post back to the web server. Some example reference sites: MoneysupermarketDOTcom pricegrabberDOTcom twengaDOTcom etc Successful bidder/company/programmer can choose any technology or use their in-house CMS solution, but we prefer freely available tools and softwares in the market eg: Perl::Catalyst on Apache with MySQL, PHP+Apache, Java Springs, etc. If using any licensed software we need to be aware of it well in advance. There are plenty of examples for online shopping comparison sites to get more idea and similarities to gives us a quote. Please provide us the following information along with your quote; 1) Total Price of the whole above project, can bid for stage 1 alone or both stages simultaneously. 2) Estimated delivery time or simple high level schedule. 3) Technology, tools and softwares that is going to be used to accomplish both stages (simple Tech spec). Please feel free to ask any questions or shoot us an email: