Data Engineer

Overview

WHO WE ARE:

Beyondsoft is a leading mid-sized business IT and consulting company that combines modern technologies and proven methodologies to tailor solutions that move your business forward.Our global head office is based in Singapore, and our team is made up of a diversely talented team of experts who thrive on innovation and pushing the bounds of technology to solve our customers’ most pressing challenges.When it comes time to deliver, we set our sights on that sweet spot where brilliance, emerging technologies, best practices, and accountability converge. We have a global presence spanning four continents (North America, South America, Europe, and Asia). Our global network of talent and customer-centric engagement model enables us to provide top-quality services on an unprecedented scale.

WHAT WE’RE ABOUT:

We believe that collaboration, transparency, and accountability are the values that guide our business, our delivery, and our brand. Everyone has something to bring to the table, and we believe in working together with our peers and clients to leverage the best of one another in everything we do. When we proactively collaborate, business decisions become easier, innovation is greater, and outcomes are better.

Our ability to achieve our mission and live out our values depends upon a diverse, equitable, and inclusive culture. So, we strive to foster a workplace where people have the respect, support, and voice they deserve, where innovative ideas flourish, and where people can unleash their brilliance. For more information regarding DEI at Beyondsoft, please go to https://www.beyondsoft.com/diversity/.

ADDITIONAL REQUIREMENT FOR REMOTE POSITIONS:

For remote positions, all candidates must complete a video screen with our corporate recruiting team.

Responsibilities

POSITION SUMMARY:

We are seeking an experienced Data Engineer In India with strong expertise in Python, SQL, and Azure Databricks to design, build, and maintain scalable cloud-based data solutions.

The ideal candidate will have hands-on experience developing production-grade data pipelines, transforming and managing large datasets, and building reliable data solutions on Azure. In addition, the candidate should have a good understanding of Large Language Models (LLMs) and Generative AI, with experience or strong interest in integrating LLM capabilities into data and AI solutions.

You will work closely with engineers, analysts, and business stakeholders to build reliable data platforms and enable data-driven and AI-powered applications.

WHAT YOU WILL BE DOING:

• Design, develop, and maintain scalable data pipelines using Azure Databricks, Python, and SQL.
• Develop ETL/ELT workflows for ingesting, transforming, validating, and publishing data from multiple data sources.
• Build and maintain Azure Databricks notebooks, workflows, and jobs for production data processing.
• Develop reusable Python components for data ingestion, transformation, automation, and integration.
• Write and optimize complex SQL queries for data transformation, analytics, and reporting.
• Work with large-scale structured and semi-structured datasets.
• Design and maintain reliable data models and data processing workflows.
• Monitor data pipelines and troubleshoot data quality, reliability, and performance issues.
• Implement appropriate logging, monitoring, error handling, and data quality controls.
• Explore and integrate LLM and Generative AI capabilities into data engineering solutions and applications.
• Develop data preparation and processing workflows to support LLM-based applications.
• Work with LLM APIs and services to enable use cases such as information extraction, summarization, classification, and intelligent data processing.
• Collaborate with engineering and business teams to identify opportunities where LLM technologies can improve data processing and automation.
• Participate in technical design, code reviews, testing, deployment, and production support.
• Follow engineering best practices for source control, CI/CD, security, documentation, and maintainability.

Qualifications

MINIMUM QUALIFICATIONS:

• Strong hands-on experience with Azure Databricks.
• Strong programming skills in Python.
• Strong proficiency in SQL, including complex queries, data transformation, and query optimization.
• Hands-on experience designing and developing production-grade ETL/ELT data pipelines.
• Experience working with large-scale structured and semi-structured datasets.
• Solid understanding of data modeling, data warehousing, and modern cloud-based data architectures.
• Experience troubleshooting and optimizing data pipelines and data processing workloads.
• Good understanding of Large Language Models (LLMs) and Generative AI concepts.
• Familiarity with integrating LLM APIs or AI services into applications or data processing workflows.
• Understanding of common LLM concepts such as prompt engineering, embeddings, vector search, and Retrieval-Augmented Generation (RAG).
• Strong problem-solving and analytical skills.
• Good communication and collaboration skills.

Required Skills: Python, SQL, Azure Databricks, ETL/ELT, data pipeline development, data processing and transformation, LLM/generative AI fundamentals, LLM API integration, prompt engineering, RAG fundamentals.

PREFERRED QUALIFICATIONS:

• Experience with Azure OpenAI Service or other enterprise LLM platforms.
• Experience developing RAG-based applications or solutions.
• Familiarity with vector databases or vector search technologies.
• Experience preparing and managing enterprise data for LLM-based applications.
• Experience with Azure Data Factory for data integration and orchestration.
• Experience with Azure Data Lake Storage (ADLS).
• Experience with Azure SQL, Azure Synapse Analytics, or other cloud-based data platforms.
• Experience with REST APIs and integrating external services into data pipelines.
• Experience with CI/CD, Git, and Azure DevOps.
• Knowledge of data security, governance, access control, and data quality best practices.
• Understanding of responsible AI, data privacy, and security considerations when working with LLMs.
• Relevant Azure, Databricks, or AI certification is a plus.

Preferred Skills: Azure OpenAI, embeddings/vector search, RAG application development, Azure Data Factory, Azure Data Lake Storage, Azure SQL/Azure Synapse Analytics, REST API integration, Git/Azure DevOps/CI/CD, data governance and security.

Beyondsoft provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type with regards to race, color, religion, age, sex, national origin, disability status, genetics, veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws. This policy applies to all terms and conditions of employment, including recruiting, hiring, and the full employee lifecycle up through and including termination.