Experienced Remote Data Engineer – Big Data Processing, Cloud Migration, and Distributed Systems Expertise
Introduction to PulsePoint and Our Mission
PulsePoint is a pioneering healthcare ad technology company that leverages real-world data in real-time to optimize campaign performance and revolutionize health decision-making. As a leading player in the industry, we utilize proprietary datasets and methodology to target healthcare professionals and patients with unprecedented accuracy, delivering unparalleled results to our clients. Our company is now a part of Internet Brands, a KKR portfolio company and owner of WebMD Health Corp. We are seeking a highly skilled and experienced Data Engineer to join our team, working remotely from anywhere in the world, to help us drive innovation and growth in the healthcare technology sector.
About Our Data Engineering Team
Our Data Engineering team plays a vital role in our technology company, which is experiencing exponential growth. We process over 80 billion impressions daily, generating more than 20 TB of data (200 TB uncompressed). This data is used to produce reports, update budgets, and drive our optimization engines, all while meeting tight SLAs and providing statistics and reports in near real-time. As a Data Engineer at PulsePoint, you will have the opportunity to work with cutting-edge technologies such as Kafka, Spark Streaming, Presto, Airflow, and Kubernetes, and contribute to the development of our data infrastructure.
Job Responsibilities
As a Data Engineer at PulsePoint, your primary responsibilities will include:
- Designing, building, and maintaining reliable and scalable enterprise-level distributed transactional data processing systems to support existing business and new initiatives
- Optimizing jobs to utilize Kafka, Hadoop, Presto, Spark, and Kubernetes resources efficiently
- Monitoring and providing transparency into data quality across systems, including accuracy, consistency, and completeness
- Increasing accessibility and effectiveness of data by working with analysts, data scientists, and developers to build and deploy tools and datasets that fit their use cases
- Collaborating with a small team of diverse technology backgrounds and providing mentorship and guidance to junior team members
Team Responsibilities
As a member of our Data Engineering team, you will be responsible for:
- Ingesting, validating, and processing internal and third-party data
- Creating, maintaining, and monitoring data flows in Python, Spark, Hive, SQL, and Presto for consistency, accuracy, and lag time
- Maintaining and enhancing frameworks for jobs, primarily aggregate jobs in Spark and Hive
- Creating different consumers for data in Kafka using Spark Streaming for near-time aggregation
- Evaluating tools and technologies to improve our data infrastructure
- Ensuring backups, retention, high availability, and capacity planning for our systems
- Reviewing and approving DDL for databases, Hive framework jobs, and Spark Streaming to ensure they meet our standards
Technologies We Use
We utilize a range of cutting-edge technologies, including:
- Python as our primary repository language
- Airflow and Luigi for job scheduling
- Docker for packaged container images with all dependencies
- Graphite for monitoring data flows
- Hive as our SQL data warehouse layer for data in HDFS
- Kafka for distributed commit log storage
- Kubernetes as our distributed cluster resource manager
- Presto and Trino for fast parallel data warehouse and data federation layer
- Spark Streaming for near-time aggregation
- SQL Server as our reliable OLTP RDBMS
- GCP, including BigQuery for performance and Looker for dashboards
Requirements
To be successful in this role, you will need:
- 5+ years of data engineering experience
- Fluency in Python and SQL
- Experience in Scala or Java is a plus, with a preference for polyglot programmers
- Hive and Presto experience
- Proficiency in Linux
- A strong understanding of RDBMS and query optimization
- A passion for engineering and computer science around data
- Availability to work during East Coast U.S. hours (9am-6pm EST) or flexibility to work until 12pm/1pm EST, with the option to work fully remotely
- Knowledge and exposure to distributed production systems, such as Hadoop
- Knowledge and exposure to cloud migration (AWS, GCP, Azure) is a plus
Selection Process
Our selection process typically involves the following steps:
- Recruiter screen (30 minutes)
- Hiring manager interview (45 minutes)
- Tech challenge take-home
- SQL and Python interview (60 minutes)
- Team interviews (60 minutes + 3 x 45 minutes) and SVP of Engineering (15 minutes)
- WebMD Sr. Director, DBA (30 minutes)
Career Growth Opportunities and Learning Benefits
At PulsePoint, we are committed to providing our employees with opportunities for growth and development. As a Data Engineer, you will have the chance to work with cutting-edge technologies, collaborate with a talented team, and contribute to the development of our data infrastructure. You will also have access to training and development programs, mentorship, and opportunities to attend industry conferences and events.
Work Environment and Company Culture
We pride ourselves on our dynamic and collaborative work environment, with a strong focus on innovation, creativity, and teamwork. Our company culture values diversity, inclusivity, and respect, and we are committed to creating a workplace where everyone feels welcome and valued. As a remote employee, you will be an integral part of our team and will have the opportunity to participate in virtual meetings, training sessions, and social events.
Compensation, Perks, and Benefits
We offer a competitive salary and a range of benefits, including:
- Comprehensive health insurance
- Retirement savings plan
- Paid time off and holidays
- Professional development opportunities
- Access to cutting-edge technologies and tools
- Flexible working hours and remote work options
Conclusion
If you are a motivated and experienced Data Engineer looking for a new challenge, we encourage you to apply for this exciting opportunity. Don't worry if you don't meet every single requirement – we value a great attitude and a willingness to learn above all. Submit your application today and join our team of talented professionals who are passionate about using data to drive innovation and growth in the healthcare technology sector.
Apply now and take the first step towards an exciting and rewarding career with PulsePoint!
Apply for this job