End to End Big data project

Problem Statement:

Imagine you are part of a data team that wants to bring in daily data for COVID-19

test occurring in New York state for analysis. Your team has to design a daily

workflow that would run at 9:00 AM and ingest the data into the system.

API:

[login to view URL]

By following the ETL process, extract the data for each county in New York state from

the above API, and load them into individual tables in the database. Each county table

should contain following columns :

❖ Test Date

❖ New Positives

❖ Cumulative Number of Positives

❖ Total Number of Tests Performed

❖ Cumulative Number of Tests Performed

❖ Load date

Implementation options:

1. Python scripts to run a daily cron job

a. Utilize SQLite in memory database for data storage

b. You should have one main standalone script for a daily cron job that

orchestrates all other remaining ETL processes

c. Multi-threaded approach to fetch and load data for multiple counties

concurrently

2. Airflow to create a daily scheduled dag

a. Utilize docker to run the Airflow and Postgres database locally

b. There should be one dag containing all tasks needed to perform the end

to end ETL process

c. Dynamic concurrent task creation and execution in Airflow for each county

based on number of counties available in the response

Implement unit and/or integration tests for your application

Skills: Python, PySpark, Docker, Hadoop, PostgreSQL

See more: big data entry project, can data entry big project, give big project data entry, big data big analysis project management, i have big data entry project with high budget but i dont accept fake profiles in freelancer need original profiles with origina, explain what you understand by the term data entry and the problem, big data freelancing project, This is a simple data entry project. I have some text files. You need to work with those text files and copy them, big data hadoop project ideas, big data analytics project management pdf, big data capstone project ideas, big data analytics project report, big data hadoop project report, big data analytics project ideas, big data live project, big data ecommerce project, big data hadoop project report pdf, describe one hard technical problem you faced in your last major project., big data freelance project

Project ID: #29026380