Data Engineer

Key Responsibilities :

1. Design & Develop Scalable Data Pipelines: Architect and build efficient data ingestion pipelines that empower the business with timely and actionable insights to drive strategy and operations.

2. Collaborate with Cross-Functional Teams: Work closely with technical and business teams to gather requirements and translate them into technical specifications and solutions.

3. Optimize Data Architecture: Continuously refine and improve the performance, reliability, and scalability of data pipelines to ensure seamless integration of data across systems.

4. Ensure Data Integrity: Perform rigorous data quality checks and uphold best practices to maintain the accuracy, consistency, and integrity of data.

5. Build ETL Solutions: Develop ETL frameworks to collect, transform, and integrate data from diverse sources such as Kafka, Scylla, PostgreSQL, MongoDB, APIs, and various file formats.

6. Adopt Best Practices: Implement best-in-class engineering practices around reporting and analysis, ensuring data integrity, testing, validation, and comprehensive documentation.

Basic Qualifications :

1. Bachelor’s degree in Computer Science, Information Systems, or a related technical field, or equivalent work experience.

2. 3+ years of hands-on experience with cloud-based data technologies, including message queues, event grids, relational databases, NoSQL databases, data warehouses, and big data technologies (e.g., Spark).

3. Proficiency in Spark (Java, Python, SQL): Expertise in developing and optimizing Spark-based applications for large-scale data processing.

4. Advanced SQL Skills: Ability to create complex queries, manage database structures, and ensure optimal performance.

5. Experience with Docker and Kubernetes: Familiarity with deploying applications using modern containerisation and orchestration tools.

6. DevOps and CI/CD: Solid understanding of modern DevOps practices, including Git, continuous integration, and continuous deployment pipelines.

7. Agile Development Experience: Comfortable working in an Agile environment, particularly using Scrum methodology.

Preferred Qualifications

1. API Frameworks & OOP: Experience with API frameworks and object-oriented programming to integrate services and improve data workflows.

2. Business Acumen: Ability to collaborate closely with leadership to deliver innovative solutions tailored to evolving business needs.

3. Strong Communication Skills: Excellent verbal and written communication abilities, with a knack for explaining complex technical concepts to both technical and non-technical stakeholders.

4. Ability to leverage tools like langchain, llama index ..etc to build llm based applications.