Longterm Builder of Data-Mining Engine Required

We require a genius-level server-side software developer to build a data-mining engine.

This engine will:

- be a constantly growing cluster of single scripts.

- utilize a constantly evolving data-mining library that the developer will create and continually revise to make each script function efficiently.

- enable each script to be responsible for mining data from one website.

- have a complete, secure, web-based index of all scripts to be managed, reviewed and scheduled (we will provide the user interface).

- record vital statistics as to the data it collects, successes, failures, and complete history.

- be cloud powered.

This project is a minimum 5 year project. You will be paid per data-mining script, and each script will be negotiated based on its complexity. We expect the compensation per script to range from $15 to $120 each.

The following responsibilities will be yours, and will not be compensated for independently, but will be part of the agreement:

1. Your scripts must all be committed to the engine's GitHub repo directly from the server via SSH, once its output and functionality is approved.

2. You must develop an alert system that advises if something has failed with a script, i.e. the website it was mining changed structure, or went offline.

3. You will be responsible for building a script that compresses and databases the data that is mined, in a manner that allows for rapid querying.

You will require an extremely analytical mind, and should be the type of developer that enjoys algorithms, and complex mathematical scripting.

This is a minimum 5 year contract and our goal for this engine is to have it scraping tens of thousands of websites, each script on its own schedule. There is a lot of money to be made for the right person, but this person will be highly skilled, highly reliable, highly determined, and highly creative in their ways of problem solving.


There is no list of websites that you could possibly send us that will cause us to choose you over someone else. This job will be awarded to the developer who has read and understands the complexity and the potential of this engine, and explains not only why they would be the best at building it, but also why they WANT to be the one to build it.

This job is not for someone who will lose interest, someone who loses power or internet regularly, gets sick regularly, or has family problems regularly. We are quick to fire when we hear these things, as they hurt longterm projects.

If before you know whether you can do the job, you need to ask what type of technology the websites have that you'll be scraping, then you can't do the job. Because we'll be using this engine to scrape so many websites that you'll likely come across everything.

The budget means nothing. This project will likely compensate the developer 5 - 6 figures USD over time. There will be full negotiation before awarding the contract.

Good luck!

Skills: Algorithm, Data Mining, Database Administration, Machine Learning, Software Architecture

See more: why would i need data scraping, why do we need data structure, why data structure, who do you need to develop a software, who can develop a website from nothing, what's an algorithm, what's algorithm, what is time complexity of an algorithm, what is time complexity in data structure, what is time complexity in c, what is time complexity, what is server side scripting, what is record in data structure, what is data structure in c, what is data structure, what is data in data structure, what is complexity of an algorithm, what is complexity in algorithm, what is an algorithms, what is algorithm in data structure

About the Employer:
( 56 reviews ) Toronto, Canada

Project ID: #4184527

9 freelancers are bidding on average $433 for this job


I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

$149 CAD in 5 days
(27 Reviews)

I am an expert in scrapping and look forward to discuss further

$30 CAD in 30 days
(41 Reviews)

Hi sir, please check PM, thx Kimi.

$250 CAD in 5 days
(24 Reviews)

Hello I am an expert on data mining. I read the description and realized that I am the person you need.

$30 CAD in 30 days
(34 Reviews)

please check message box.

$3000 CAD in 30 days
(0 Reviews)

I have research experience in data mining and algorithm optimization. I have all ready done some good work and a research paper in data mining. I have very rich education background. I am ready to work but want detail More

$250 CAD in 5 days
(0 Reviews)

Hello, I would like to cooperate with you. Why do I think I'm the best fit for the project? because I have the academic background (BSc of computer science), good experience, and aroused interest in data-mining. Why More

$50 CAD in 300 days
(0 Reviews)

I am Data Miner , So many algorithm generate as per client requirement

$75 CAD in 10 days
(0 Reviews)

Hi, this seems very much like my PhD project, so there is your "interest shown in the topic" - I just spent five years of my life developing a proof-of-concept prototype for it... I developed a web spider which gets we More

$60 CAD in 30 days
(0 Reviews)