Task - 1 Billion JSON to CSV file conversion
Structure of file - Deeply nested file (attached data structure)
Package in R --- Sparklyr package reads 1 Billion data file within 30 minutes only, whereas other packages take non-ending time. So I preferred Sparklyr package in R. (If required I can provide you that R-code which I have already tried.)
Important Note -
1) The converted file should get opened in excel properly. Many sub-columns have similar names and that should be renamed uniquely.
2) The reading process 1 billion file should not take non-ending time. Maximum half an hour time for reading will be ok. I have already used Sparklyr package. It takes half an hour for reading this big file. If you have any other package suggestion in R then that would be accepted.
Testing - I will require your R-code so that I can apply to 1 Billion data for the purpose of testing.
Attached Files -
1) Data structure - word file
2) 500 sample records (JSON format)
3) R file (whatever I have tried by using Sparklyr package)
4) One more R code (whatever I have tried by using jsonlite package)
Data size is 53.7 GB, My laptop has 8 GB RAM and 64 bit OS.
JSON file is in streaming format.
4 freelancers are bidding on average ₹6679 for this job
Hello? I am a data scientist with a very strong command in the use and scripting of R programming language. I look forward to chatting on the project details