Databricks Developer

We are looking for a skilled and motivated Databricks Developer to join our team and contribute to cloud-based data engineering initiatives in the healthcare domain. The ideal candidate will have hands-on experience developing and maintaining data pipelines using Azure Databricks, PySpark, Spark SQL, and Azure Data Factory.The role involves transforming raw data into reliable, high-quality datasets to support clinical operations, enterprise data warehousing, analytics, and reporting. The successful candidate will work closely with data engineers, architects, and business stakeholders to develop scalable data solutions while ensuring data quality, security, and governance.Key ResponsibilitiesDevelop, test, and maintain ETL/ELT data pipelines using Azure Databricks, PySpark, Spark SQL, and Databricks notebooks.Build and optimise data ingestion and transformation workflows using Azure Data Factory (ADF) and Databricks Workflows/Jobs.Implement and maintain Delta Lake tables and Medallion architecture (Bronze, Silver, and Gold layers).Integrate data from multiple sources, including SQL databases, REST APIs, SFTP, and file-based systems, into Azure Data Lake Storage (ADLS Gen2).Support the development and enhancement of Enterprise Data Warehouse (EDW), data lakehouse, and data mart solutions.Develop and maintain conceptual, logical, and physical data models to support analytics, reporting, and business requirements.Perform data profiling, cleansing, validation, reconciliation, and quality checks to ensure data accuracy and consistency.Apply data governance, metadata management, and data lineage practices to maintain traceability and auditability.Write and optimise SQL queries and data transformations for improved pipeline performance.Collaborate with technical, clinical, and business teams to translate requirements into effective data engineering solutions.Follow coding standards, participate in code reviews, and support unit, integration, and regression testing.Support version control, CI/CD processes, and automated deployment practices.Assist in preparing datasets for AI/ML workflows and integrating analytical outputs into data pipelines when required.Follow applicable data security, privacy, and regulatory requirements, including HIPAA and GDPR, where relevant.