Senior Data Engineer
At Voloridge Investment Management our quantitative systems are deeply dependent on vast quantities of data. The Senior Data Engineer must understand the many different and evolving use cases for data at Voloridge and design systems that supply highly performant data sets for advanced analytics. In this role the Sr. Data Engineer will provide mentorship and impart experience for the data engineering team.
Summary of Job Functions
- Collaborate effectively with Stakeholders, Project Managers, Software Engineers, Data Architects, Data Analysts, QA Analysts, DBAs and other Data Engineers
- Build and maintain data pipelines based on functional and architectural specifications
- Ensure that data pipelines incorporate best practices related to high performance, fault tolerance, instrumentation, logging and data driven functionality
- Ensure that data pipelines are scalable, maintainable and not overly engineered
- Produce and maintain engineering and operational documentation
- Analyze complex data problems and engineer elegant solutions
- Stay abreast of emerging technologies and make relevant recommendations
- Transition existing data warehouses and pipelines to newer technologies where appropriate
- Work in an Agile environment
- Participate in engineering standards evolution
- Participate in an on-call rotation with other Data Engineers
- Lead investigations to troubleshoot data issues that arise along the data pipelines
Minimum Requirements
- 10+ years’ experience building ETL/ELT pipelines using both transactional databases and data warehouses with large data volumes
- Strong initiative, collaboration, accountability, impartiality and communication
- Strong analytical skills, real passion for working with data and strong interest in solving data problems
- Strong understanding of traditional Agile SDLC best practices
- Extensive experience building ETL/ELT pipelines against data warehouse entities such as SCD’s and Facts
- Extensive experience with advanced SQL Server features such as Table and Index Partitioning, Columnstore Indexes, CLR and Service Broker
- Extensive experience in performance tuning TSQL and SSIS, execution plan analysis, blocking / deadlock analysis and index optimization
- Extensive experience with SQL Server 2016+
- Extensive experience using Visual Studio to create and maintain SSIS packages
- Extensive experience using SSMS to create and maintain SQL Server tables, views, functions, stored procedures and user defined table types
- Experience writing Python, C# / .Net, and PowerShell
- Experience Python programming using libraries such as Pandas, Numpy, csv, TraceBack, JSON, PyODBC, Math
- Experience building data pipelines using multiple external data sources and transport mechanisms
- Experience with Git source control integration with Visual Studio and SSMS
- Experience working on an Agile team
- Experience owning mission critical service(s)
- Bachelor’s degree in Computer Science, Information Systems or related disciplines
Preferred Skills and Previous Experience
- Skilled in developing automated testing, code quality, and engineering best practices for data services
- Experience working within segregated Development, QA, UAT and Production SDLC stages
- Experience with AWS
- Experience working with trading / financial / investment / accounting data
- Experience with tools such as Red Gate, Grafana, OpsGenie and AirFlow
- Experience with Big Data technology and MPP databases like Matrix/Paracel, Vertica, Netezza, PDW, Greenplum
- MS/PhD in Computer Science, Information Systems or related disciplines
Compensation and Benefits
- Relocation assistance available for the right candidate
- Highly competitive base salary
- Profit sharing bonus
- Health, dental, vision, life, and disability insurance
- 401K