Data Engineer
We are looking for multiple data engineers to work for our fast growing company. We are expanding our data science practice and are looking for a number of data engineers in multiple locations. You will also be partnered with a seasoned data engineer to further your training on the job. A typical project and skillset is below:
Project Objective:
The objective is to analyze consumption activity against OneLake datasets. This data will then be consolidated into a report, and provided to LOBs to know their consumers, and to better plan for the impact of remediation of OL datasets on their consumers.
Job Description:
- Collaborate with multiple agile teams to develop, test, implement and support the technical solution implemented.
- Experience in software development including design, coding and testing
- Experience working in an Agile environment
- Experience in building dashboards using EFK/ELKExperience with CI/CD pipeline tools
- Experience in developing web services, REST, GRPC, using Spring boot or Python Flask along with Hibernate/JPA/SQLAlchemy.
- Experience in unit testing, version control tools such as JUnit, Cucumber/Gherkin, Maven, Jenkins, Git.
- Experience in building data ingestion pipelines, structured, unstructured data using Apache Spark, Kafka, Hadoop, HIVE, Snowflake/RedShift, SQL, Scala, PySpark.
- Experience in cloud concepts such as Data Lakes, Lambda, Load balancers, AWS S3, EC2, RDS, Terraform, Cloudformation.
- Experience in developing, debugging in shell scripts, Java, python.
Job Skills: AWS, pyspark, python, scala, sql, Java, EFK, ELK, Jenkins, Git, CICD