Blog

Hello,

Trust this email finds you well.

I am Akshay Kumar from Global Applications Solution, writing this email to share few exciting Job Opportunity with one of our premium clients.

If you are interested, please share resume on my email – [email protected]

To know more about Global Applications Solution, visit our website: https://www.globalapplications.com/

Job Description: Lead Databricks Engineer
Location: New York City, NY and/or Metropark, NJ
Sell Rate: $90/Hr
Experience Range: 12+ years (Lead Position)
Client: Persistent
Overview:
Persistent is seeking a highly skilled Databricks Engineer to lead the design and implementation of advanced data solutions. The ideal candidate will have extensive experience in Databricks architecture, distributed computing, and a strong command of SQL and PySpark, particularly within an Azure cloud environment.
Key Responsibilities:

• Design, develop, and maintain data pipelines using Databricks and Spark, along with other cloud technologies as necessary.
• Optimize data pipelines for performance, scalability, and reliability.
• Ensure data quality and integrity throughout the data lifecycle.
• Collaborate with data scientists, analysts, and other stakeholders to understand and fulfill their data needs.
• Troubleshoot and resolve data-related issues, providing root cause analysis and recommendations.
• Document data pipeline specifications, requirements, and enhancements, effectively communicating them to the team and management.
• Create new data validation methods and analysis tools, sharing best practices and learnings with the data engineering community.
• Implement ETL processes and data warehouse solutions, ensuring compliance with data governance and security policies.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent work experience.
• 5+ years of experience in data engineering with Databricks and Spark.
• Proficient in SQL and Python, with strong skills in PySpark.
• Experience with Azure Databricks Medallion Architecture, including DLT and Iceberg.
• Familiarity with financial or corporate banking contexts is a plus.
• Experience with data integration and ETL tools, such as Azure Data Factory.
• Strong experience with Azure cloud platform and services.
• Understanding of data warehouse and data lake concepts and architectures.
• Knowledge of big data technologies such as Kafka, Hadoop, and Hive is advantageous.
• Strong analytical and problem-solving skills.
• Excellent communication and teamwork abilities.

Requirements:

• Strong knowledge of Databricks architecture and tools.
• Experience in creating tasks and workflows in Databricks.
• Deep understanding of distributed computing and utilizing Spark for data processing.
• Proven command of SQL and PySpark for querying databases.
• Preferred experience with Azure for Databricks deployment.