Data Architect – Databricks - PamTen Inc
Bloomington, IL
About the Job
- 15+ yearsof experience in the IT/Technology and having at least 3+ years of experience in Azure/Databricks
- Design and architect Databricks-based solutions that align with business objectives, ensuring scalability, performance, and security.
- Designed, Development and implemented Datamesh using DeltaLake.
- Provide technical leadership and guidance to the Databricks development team, ensuring best practices are followed throughout the project lifecycle.
- Collaborate with cloud infrastructure teams to design and optimize the underlying infrastructure for Databricks workloads on platforms such as AWS or Azure
- Develop efficient data ingestion and ETL pipelines using Databricks, Apache Spark, and other relevant technologies.
- Integrate Databricks with data lakes and data warehouses to ensure seamless data access and analytics.
- Continuously monitor and optimize Databricks workloads for performance and cost-effectiveness.
- Implement and maintain security measures, including access controls, encryption, and compliance standards, to protect sensitive data.
- Create documentation and provide training to internal teams on Databricks best practices and usage.
- Stay up-to-date with the latest developments in Databricks and related technologies to recommend and implement improvements.
- Good knowledge on SQL Queries & Stored Procedures.
- Strong programming skills in Pyspark, Python, writing complex queries in T-SQL
- Experience in transferring data from RDBMS to Data bricks using ADF.
- Expertise in using Spark SQL with various data sources like JSON, Parquet and XML.
- Experience in creating tables, partitioning, bucketing, loading and aggregating data using Spark SQL/Python.
- Experience with Insurance Domain
Source : PamTen Inc