Sr. Data Engineer , San Francisco, California Phone + Skype No H1B..!! JD Position Summary We are seeking a Senior Data Engineer to join the Data Analytics department.
This team follows best-practice agile techniques to deliver impactful data products for internal partners. You will actively participate and practice in an open, highly collaborative Agile environment. This Data Engineer role with Data Analytics will have a major impact in reducing wildfire risk in California, improving the quality of life for all Californians.
Position Responsibilities Collaborate on a daily basis with the product team. This includes pairing for all aspects of software delivery. Create and maintain optimal data pipeline architecture.
Assemble large, complex data sets that meet functional and non-functional business requirements. Identify, design, and implement internal process improvements automating manual processes, optimizing data delivery, and re-designing infrastructure for greater scalability. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data' technologies.
Minimum Qualifications Bachelors Degree in Computer Science or job-related discipline or equivalent experience 7+ years-experience with software delivery Experience delivering product with Agile Scrum methodologies Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases. Experience with ETL flows using Python Experience building and optimizing 'big data' data pipelines, architectures and data sets. Strong analytic skills related to working with unstructured datasets.
Build processes supporting data transformation, data structures, metadata, dependency and workload management. Proficiency with the following tools that enable the candidate to contribute autonomously Glue, Kafka, Redshift (with a focus on infrastructure -as-code), Python. Desired Qualifications Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
Experience with Palantir Foundry platform creating datasource, creating transformations using pyspark, experience with Foundry applications include Slate, Contour and Workbooks AWS DevOps skills Humble is open to being coached, has high Emotional Quotient (EQ) and is self-aware Hungry desires to get things done while honoring people, and seeks better ways to do the job, is highly motivated by the significant impact this work will have Collaborative has strong interpersonal skills demonstrates empathy with teammates and stakeholders, cares about and works well with teammates Willingness to impact beyond defined role Experience with data analytics product development