PySpark Jobs

building python and pyspark frame work for ETL pipelines, data warehousing, azure data factory

I am looking for a male who is expert in Hadoop, Java, MapReduce, Sqoop, Hive, Hbase and pig with good experience in Design patterns, ETL and Frameworks. Need good communication skills and available to take phone interview US EST time zone during working hours. I will send the job description well in advance. There might be multiple phone interview rounds, and the funds will only be transferred...

1) Peform ETL S3 ( HDFS file) as source and Redshift as target Technical Environment : EC2,EMR, Pyspark & Hive 2) Peform ETL across S3, AWS noSQL database using Glue 3) One sample Python AWS Lambda function to run AWS Redshift SQL scripts 4) One sample on realtime streaming ( preferably Kafka( producer) + Pyspark( Consumer) 5) Executing bash commands ( jobs ) via Airflow ( or AW...

Using Spark ML Library to ReCode "Thinking Process" from 2 Similar Python Panda Scripts. Knowledge of the Codes from [login to view URL] is Required. Entails Features Transformation and Wrangling on json data. Milestone will not be released until project is delivered.

kubernetes is hosted on premise, spark need to be setup over that

Read in Protobuff file into dataframe using Databricks - Using Python

i need the Python - Scala expert for multiple jobs. Details will be shared

i need the Python - Scala expert for multiple jobs. Details will be shared