Position: Lead Data Engineer
*Work Location: Foster City, CA*
*Type: FTE *
Key Skills:
• 10+ years of experience in software development and delivery.
• 5+ of relevant work experience on large scale Data applications
• Experience working(or willing to work) with distributed teams in different time zones and offshore teams
• Experience building streaming data pipelines using event based models.
• Strong SQL skills
• Experience with programming in languages Python, and scripting tools like Bash, KSH
• Good aptitude, strong problem-solving abilities, and analytical skills, ability to take ownership as appropriate
• Should be able to do coding, debugging, performance tuning, and deploying apps to a Production environment.
• Should have good working experience in the bellow technologies and languages
o Hadoop ecosystem (HDFS, Hive, Yarn, File formats like Avro/Parquet)
o Kafka
o Spark Streaming
o PySpark
o Airflow
• Good understanding of Design Patterns in software architecture
• Experience working in Agile Methodology
• Ability to learn and help the team learn new technologies quickly
• Take up the complete planning/design/implementation/UAT/delivery of the project
• Excellent communication and coordination skills
• Knowledgeable (and preferred hands-on)
• UNIX environments, different continuous integration tools
• Must be able to integrate quickly into the team and work independently towards team goals
•
Role & Responsibilities:
• Take the complete responsibility of the project execution
• Be accountable for the delivery of the project in the defined timelines with good quality.
• Define and follow the processes for project execution and delivery.
• Drive and motivate the team to be agile and follow agile methodology
• Guide the team technically and give the team a direction on how to plan, design, implement, and deliver the projects.
• Define the architecture and discuss the pros-cons of the same with the team
• Involve the team during the brainstorming sessions as appropriate.
• Work with other team leads to get the architecture reviewed.
• Keep all the stakeholders updated about the project status/risks/issues if there are any.
• Work closely with the management wherever and whenever required, to ensure smooth execution and delivery of the project.
•
Education:
Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experience
Job Type: Full-time
Pay: $150,000.
00 - $160,000.
00 per year
Experience level:
* 10 years
Schedule:
* 8 hour shift
* Monday to Friday
Experience:
* pyspark: 1 year (Required)
* Hadoop: 1 year (Required)
* Data warehouse: 6 years (Required)
Ability to Commute:
* Foster City, CA 94404 (Required)
Ability to Relocate:
* Foster City, CA 94404: Relocate before starting work (Required)
Work Location: In person
Whatsapp
