Overview:
We are looking for a skilled Data Engineer with strong experience across Databricks, SQL, Python, and AWS data services. The ideal candidate should be proficient in building end-to-end data pipelines, working with large datasets, and ensuring efficient data processing and transformation. Key Responsibilities Design, develop, and maintain end-to-end data pipelines* using Databricks.
Work extensively with SQL and Python to support data transformations, processing, and analysis.
Implement and optimize data workflows across AWS services such as S3, Secrets Manager, and other foundational cloud components.
Leverage AWS Athena, Glue, and Redshift for data extraction, cataloging, processing, and warehousing.
Develop and maintain shell scripts for automation and operational workflows.
Work with PostgreSQL databases for data modeling, querying, and optimization.
Collaborate with cross-functional teams to understand data requirements and ensure data quality, consistency, and availability.
Troubleshoot data issues, optimize query performance, and ensure efficient data processing.
Apply strong data engineering and data understanding skills to solve complex business challenges.
Required Skills \& Experience 4–5 years* of experience in Data Engineering.
Strong hands-on experience with:
Databricks (end-to-end development)
SQL (advanced querying, performance tuning)
Python (data processing, automation)
Solid knowledge of AWS Cloud services including S3, Secrets Manager, and other basic AWS components.
Experience with Athena, Glue, and Redshift.
Basic understanding of Shell scripting.
Strong data modeling, data pipeline design, and data troubleshooting skills.
Experience with PostgreSQL.
Nice to Have* Exposure to data governance, data validation, or CI/CD for data pipelines.
Experience with any orchestration tools (e.g., Airflow, AWS Step Functions).