Data Engineer

We are building a data platform for digital assets to facilitate the analysis of structured and unstructured data. The platform will initially be used to provide daily analysis of public and private markets, along with large fundamental and alternatives data sets. You will work in a combination of on-prem and cloud-based environments, leveraging cloud-native services as required. You will be working closely with the other core engineering teams to incorporate the data pipeline, data products, and research into our overall reference, market, and alternative data environment.

Principal Responsibilities

• Partner with COO, Head of Technology, and Engineering Leads for this asset class to evaluate data providers and implement strategic roadmap for the research platform.
• Analyze Real-time and Historical Digital Asset data sources and work to integrate into firm platforms similar to existing asset classes
• Gather and process raw data at scale (including writing systems and scripts, calling APIs, write SQL queries, etc.).
• Build solutions to process unstructured data into a form suitable for analysis and ingest into the platform. This includes a strong emphasis on data quality and resiliency.
• Work with other members of the team to ensure data is ingested, stored, and managed properly to support our analytical needs.
• Helping define automation and management of cloud infrastructure and services using native tools, open source tools, and third party products.
• Implement platform telemetry and reliability systems for performance tuning and troubleshooting, and to trigger automated actions based on events or thresholds.

Qualifications

• 4+ years of experience building scalable and reliable data pipelines based on Big Data processing technologies like Hadoop, Cassandra, MapReduce, Spark, etc.
• 4+ years of combined hands-on experience in Java and Python, with examples of reusable and efficient code to automate analyses and data processes.
• 2+ years of experience deploying and managing cloud environments for data pipelines and intelligence services.
• Experience processing large amounts of structured and unstructured data from disparate sources with strong emphasis on quality controls and resiliency.
• Strong knowledge of and experience with statistics; potentially other advanced math as well.
• Broad understanding of financial services instruments helpful