Local Data Management System

Closed Posted 5 years ago Paid on delivery
Closed Paid on delivery

Our company collects various modes of data in the field to conduct condition assessment of critical infrastructure. The data is typically from sensors installed on field robots and includes images, videos, laser scans, environmental data, point measurements etc. The data is also coupled with other miscellaneous attributes such as reports, documents, annotations etc.

The data is currently hosted on local servers and is accessible by the employees on their windows/linux/osx platforms using sshfs, nfs or samba.

However as the amount of data exponentially grows (currently at 30Tb and expecting to double in the next 6 months) we need to address a few challenges in data management.

Current challenges are:

- Data stored across multiple server locations and not accessible seamlessly

- Difficult to prioritise data between ssd and hdd when requiring different speeds for current projects

- No overview of all the data, where it is, when was it last saved etc.

- Data is sometime reshuffled in the back end, which means all the code pointing to hard links breaks.

- Hard to manage permissions which many users trying to do prototyping and development at the same time

- Slow network access on linux machines (could be problem with sshfs)

These problems leads to the basic requirements of the project.

A. Central repository to store all data

A1. A central repository where data can be seamlessly stored. Data is currently spread over a few local machines. This needs to be aggregated such that the end user doesn't have to worry about where the data is actually stored.

A2. Set protocols for adding new data (adding done by plugging in drives into the servers or over the network)

A3. Data access to be streamlined. Data is currently (and will be) stored on local linux servers. Data to be accessible to users positioned on compute servers (connected to monitors and running linux) or their laptops (running linux, windows or OSX). Usable file paths to data for all linux users needs to be consistent to enable code transfer between machines. The same for windows and OSX machines to enable unified file paths when working on data processing software across multiple machines.

A4. Establish correct mounting protocols/scripts needed to allow for maximum read/write speeds.

B A barebones dashboard for the data repository (1 developer, 5-10 days including validation and testing)

B1. View basic stats on all data (where stored, how to access, last edited, size of data, basic notes etc)

B2. Accessible on the network using user login details (if they are an allowed user)

B3. Perform storage medium switches - e.g. copy/move data in the backend between ssd and hdd (for speed boosts on short term projects). This should be seamless to all users, who continue to access the data using the same path.

B4. Ability to change group owner of the different mount points

B5. Ability to see if data is being uploaded to AWS and tag data to be uploaded onto AWS (currently done using cronjob and bash).

C Programming C++ Programming Linux Network Administration Python

Project ID: #16944203

About the project

14 proposals Remote project Active 5 years ago

14 freelancers are bidding on average $4748 for this job

Yknox

Dear,I am Gang Lee . I'm interested in the project you recently posted. I'm a certificated freelancer with over 1000 good reviews from clients. I have experienced knowledge in every part of Web development. I never More

$4444 AUD in 30 days
(746 Reviews)
9.0
hbxfnzwpf

I am very proficient in c and c++. I have 16 years c++ developing experience now, and have worked for more than 7 years. My work is online game developing, and mainly focus on server side, using c++ under Linux environ More

$3000 AUD in 21 days
(148 Reviews)
7.0
linuxsupport

I am having an experience of more than 16 years in this field. I have served as a VP-Technology for on of the biggest B2B company of India. I am an expert of AWS and Linux. I have worked on similar type of project befo More

$5555 AUD in 30 days
(60 Reviews)
6.0
schoudhary1553

Hello, I can help with you in your project Local Data Management System. I have more than 5 years of experience in C Programming, C++ Programming, Linux, Network Administration, Python. We have worked on several sim More

$5000 AUD in 30 days
(41 Reviews)
5.8
tekzee

Hi ****We dont do Code in Python, we can only assist you with the DB and AWS**** I just have been through your requirement of Local DB Management and AWS assignments and I found myself fit for this job As we are a More

$4705 AUD in 30 days
(15 Reviews)
4.1
donnyforu

Hello, I have more than 23 years of experience in top IT companies based in the USA. I have worked for some of the top IT clients across the globe and I have managed and worked on some of the large and complex large More

$4666 AUD in 20 days
(2 Reviews)
3.7
AhteshamAK

It is my great honor to apply for this vacant job. I am very hardworking and can work effectively as fast as I can. I have excellent skills in VB, VBA, C++, C#, OPPs, HTML5, CSS3, JS, AJAX, JQYERY, React.js, ANGULAR.JS More

$3529 AUD in 30 days
(3 Reviews)
3.3
MetaoriginLab

Our dedicated team of software and programming professionals provides our clients with state of the art programming solutions in the python language. Me and my team has 5 years of experience into Python/Django,Selenium More

$5023 AUD in 8 days
(7 Reviews)
2.9
eclipsetechno

Hi, After reading your description, following conclusions could be developed: 1) We shall introduce Mongo DB to store and distribute big data even if it crosses millions. 2) You already suggested AWS, so Clo More

$5000 AUD in 60 days
(0 Reviews)
0.0