Find Jobs
Hire Freelancers

Data Processing using Python and SQL

$30-250 USD

Completed
Posted almost 4 years ago

$30-250 USD

Paid on delivery
I am converting some hive tabular data into JSON documents in python using a UDF. So basically each row in the output text/csv files has two columns. first column has sequence number and the second column has a JSON object as a string. Each day we may have atleast 100 files (text/csv). The total row count in these files can be in millions. This project is to batch these rows by ignoring the sequence number and take atleast 1000 JSONs and create an array of JSONs and then assign a new batch number to that batch. If it can be done in hive table as post processing that is fine too.
Project ID: 24768897

About the project

19 proposals
Remote project
Active 4 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
hello complete within max 36hrs I would like to work on your project. I have a 5+ years of experience in field I can start working immediately on this project if needed. I have gone through the job description and understand your requirements.I can complete this work with high quality. I would really like to work with you on this [login to view URL] interested,Kindly contact me via chat for further details and discussion
$100 USD in 1 day
4.9 (32 reviews)
5.1
5.1
19 freelancers are bidding on average $135 USD for this job
User Avatar
Hello, I'd be glad to develop the python script that performs the required json processing . Looking forward to chat with you soon for more details. Best regards.
$140 USD in 2 days
5.0 (167 reviews)
7.4
7.4
User Avatar
Hello, I can help you with your project - Data Processing using Python and SQL I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Please visit : https://www.freelancer.com/u/schoudhary1553 I have excellent command over English. I am a hard worker, productive and worthy of your attention I hope, I would be the right candidate for this post. Awaiting an affirmative response from you. Kinds Regards, Sandeep
$220 USD in 4 days
5.0 (66 reviews)
6.8
6.8
User Avatar
so you want to take all the rows across all the days txt files, strip out the batch and import into one CSV with a new batch number?
$99 USD in 5 days
5.0 (16 reviews)
5.8
5.8
User Avatar
Hello Nice to meet you. I already read your job description and understood your requirments. I have rich experiences in this field and python is my top skill. So i can complete your project on time. Let's discuss more via chat. Thanks.
$200 USD in 7 days
5.0 (6 reviews)
4.1
4.1
User Avatar
Hi Sir, I hope you are doing well. I have round about 10 years of experience in Python and can work with you on this script. Please contact. Thanks
$99 USD in 3 days
5.0 (4 reviews)
4.1
4.1
User Avatar
Hi, I am an expert in processing data using python. I have processed many data files(csv, xlsx, jsson, xml) for SEO. I think I can complete your project successfully. Thanks
$140 USD in 7 days
5.0 (3 reviews)
4.0
4.0
User Avatar
Hello, I have read your requirements and I can build python script that fulfill task you require. You can check out my skills at my profile. lets discuss more about project in chat. thank you.
$70 USD in 3 days
5.0 (14 reviews)
4.1
4.1
User Avatar
Hi, I can build that for you. It's really about creating the right input hooks, batching the data and writing it in the right location. I'll include proper logging and design the script with flexibility in mind. Regards, Nikolay Kovachev
$99 USD in 7 days
5.0 (1 review)
3.7
3.7
User Avatar
Hello. I am an experienced Python developer. I own a wide stack of technologies. Always attentive and honest with the customer. Write me to discuss the details of the project.
$100 USD in 7 days
5.0 (2 reviews)
2.4
2.4
User Avatar
sir I am expert like this job each day I can complete 1k row if you want I can use my expert team so please contact me
$100 USD in 3 days
4.9 (2 reviews)
2.0
2.0
User Avatar
I am python expert and can do this. contact me for further deliberations............................................................. regards Ginu
$140 USD in 7 days
5.0 (1 review)
1.5
1.5
User Avatar
Hello, I am a data analyst by profession and Im expertise dealing any format of the data in Python, pandas library. I would love to take up your project. Please let me know if you are interested. -David.
$45 USD in 2 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I can deliver this project in both Hive and Python. I have 7+ years exp in SQL analytics and ETL processes in Big Data both in Cloud and on premise. I am new to freelancer so there are no past projects but I can guarantee that the code will be of good quality and quite efficient
$100 USD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Are you going to insert to SQL Server or MySQL?. if you are running this in linux machine, we can make a cron for this task. The things that is not clear is the batch sequence, are we going to put the batch no to the table?. that means in the table we have 3 column Seq, Data, and Batch ?..
$120 USD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hello There, I'm a professional back-end and I can assure of you a robust and scalable solution for a problem like this. My approach would be to concurrently process the rows in groups of X and tag them with batch_ids as we along, allowing us to get many a time faster results. Drop me a message if this interests you, I'd be happy to discuss this in detail. Regards, Mandeep
$200 USD in 7 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Greetings, We are a company working for the last 5 years, but we are new to freelancer.com, so you might find our profile new. Our reviews do not display the amount of work that we have done, so you can go through our website - hrtechservices dot com. We read your job description and surely, we can help you out. We have a team of 120+ expertise, which includes website developer, Software Developers, Mobile Developers, Testers, Project Managers, and Quality Manages. We have developed over 250+ websites, software, & application for different business sectors and industries. Skills: 1. Backend- Python, machine learning, AR/VR, PHP (Laravel, Codeigniter, CakePHP, CorePHP, etc), etc. 2. Frontend- HTML5/CSS3, Angular JS, Bootstrapping, Javascript, jQuery, Node JS, Vue JS. etc. 3. CMS: WordPress, Wix, Magento and Shopify. 4. Database- MySql, MsSql, SQLite, etc. 5. Android Coding- Java, Kotlin. 6. Hybrid framework- React Native, Flutter. 7. iOS Coding- Swift, objective-c. 8. Advance Level Skills- IoT, Alexa Skills, Data Visualization, Machine Learning. 9. Designing tools: Adobe Photoshop, Adobe Illustrator, Adobe XD, Sketch, Invision and Figma. We are ready to start work on an immediate basis. Looking forward to your reply Thanks & Regards HrTechLLC
$210 USD in 8 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I have worked extensively in python and Spark processing various kinds of files like json,csv or text files.I am also well versed in Hive.
$140 USD in 7 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
Chicago, United States
4.6
2
Payment method verified
Member since Feb 4, 2018

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.