Full-Time

Big Data Engineer

Posted on 5/6/2025

Capco

Capco

5,001-10,000 employees

Management consultancy for financial and energy sectors

No salary listed

Senior

Company Historically Provides H1B Sponsorship

Pune, Maharashtra, India

Category
Data Engineering
Data & Analytics
Required Skills
Chef
Python
Airflow
Data Science
Apache Spark
SQL
Apache Kafka
Java
Go
Terraform
Ansible
Hadoop
Requirements
  • Bachelor's degree in Computer Science, Engineering, or related field, or equivalent work experience.
  • Proven 5-8 years of experience as a Senior Data Engineer or similar role.
  • Experience with big data tools: Pyspark, Hadoop, Spark, Kafka, Ansible, chef, Terraform, Airflow, and Protobuf RPC etc..
  • Expert level SQL skills for data manipulation (DML) and validation (DB2).
  • Experience with data pipeline and workflow management tools.
  • Experience with object-oriented/object function scripting languages: Python, Java, Go lang etc.
  • Strong problem solving and analytical skills.
  • Excellent verbal communication skills.
  • Good interpersonal skills.
  • Ability to provide technical leadership for the team.
Responsibilities
  • Design, construct, install, test and maintain highly scalable data management systems & Data Pipeline.
  • Ensure systems meet business requirements and industry practices.
  • Build high-performance algorithms, prototypes, predictive models, and proof of concepts.
  • Research opportunities for data acquisition and new uses for existing data.
  • Develop data set processes for data modeling, mining and production.
  • Integrate new data management technologies and software engineering tools into existing structures.
  • Create custom software components and analytics applications.
  • Install and update disaster recovery procedures.
  • Collaborate with data architects, modelers, and IT team members on project goals.
  • Provide senior level technical consulting to peer data engineers during data application design and development for highly complex and critical data projects.

Capco offers management and technology consulting services to the financial services and energy sectors, helping organizations address strategic and operational challenges. Their team of skilled professionals delivers tailored solutions, including digital consulting and technology innovation, to enhance efficiency and drive growth. Capco stands out from competitors by utilizing proprietary platforms like PlatformZero and fostering a diverse and inclusive workplace culture. The company's goal is to empower clients while promoting social responsibility and sustainable finance.

Company Size

5,001-10,000

Company Stage

Acquired

Total Funding

$2B

Headquarters

London, United Kingdom

Founded

1998

Simplify Jobs

Simplify's Take

What believers are saying

  • Capco's collaboration with OpenAI accelerates AI's transformative impact in financial services.
  • The FCA's AI testing service supports responsible innovation in the UK financial sector.
  • Capco's launch of an energy practice in the UK reflects consultancy diversification.

What critics are saying

  • AI ethics and data privacy pose risks in Capco's collaboration with OpenAI.
  • Capco's UK energy market expansion faces challenges from competitors and regulations.
  • Technical challenges in ISO 20022 implementation with NatWest could affect client satisfaction.

What makes Capco unique

  • Capco combines innovative thinking with deep industry knowledge for financial services.
  • Capco's proprietary PlatformZero offers specialized solutions enhancing client operations.
  • Capco emphasizes diversity and inclusion, fostering innovative solutions through diverse perspectives.

Help us improve and share your feedback! Did you find this helpful?

Benefits

Hybrid Work Options

Company News

IFA Magazine
May 1st, 2025
Capco – Fca’S New Ai Testing Service Incredibly Positive But Careful Design Is Key For Success

Charlotte Byrne, Head of Generative AI at financial services consultancy Capco, comments on the FCA’s latest AI announcement and its implications for the UK financial services sector: “The FCA’s live AI testing service is an incredibly positive move for the UK financial services sector, signalling that the FCA is leaning towards a collaborative regulatory environment,  with an appetite towards safe innovation rather than steps to stifle innovation. “Trust in AI is critical as organisations start to roll out scaled AI customer facing products. By providing firms with structured regulatory support, the FCA’s new service will introduce positive guardrails, helping organisations test and benchmark their AI models against key principles like fairness, transparency, and accountability before they reach the market. “It will also help level the playing field, enabling a wider range of firms to innovate responsibly and at pace. For smaller firms that may not have extensive in-house AI expertise or resources, the testing lab offers an opportunity to strengthen governance and build trust with customers and stakeholders. “This isn’t entirely selfless, the lab testing and collaboration with innovators will provide the FCA with valuable insights, from use cases to impact and common challenges to better understand how AI may impact UK financial markets and better prepare for the future. “It’s important the FCA ensures the service remains accessible and genuinely encourages responsible innovation across firms of all sizes. The success of the initiative – and whether it ends up an accelerator rather than a cumbersome and timely process – will depend on the practical support offered and the timeframes the FCA can support given the potential demand

IFA Magazine
Mar 27th, 2025
Fca Strategy Aligns With Uk Growth Agenda, But Clear Success Metrics Still To Emerge – Capco

Michael Shand, Managing Principal at financial services focused technology and management consultancy Capco, comments on the Financial Conduct Authority’s new five-year strategy announcement. “The FCA’s new five-year strategy is firmly aligned with the UK government’s growth agenda, but there could be some additional clarity around success metrics – though that task is made more challenging by factors outside of the FCA’s control, such as macroeconomic factors, competition from other jurisdictions, and the take up and future impact of technology.“While banks and other financial services will welcome the spirit of the FCA’s strategy – a growth focus, proportionate oversight, and an ambition to simplify, reduce or remove reporting obligations – its implications could vary widely in practice.“It is welcome that the FCA is placing consumer outcomes at the heart of the new strategy. It is clearly prioritising greater financial inclusion and resilience through a combination of greater participation in more complex, higher risk investing with a robust approach to protection, for example by cracking down on financial crime.“The FCA has also recognised the power of technology and innovation – from AI to Open Finance – as key enablers. There is a focus both on efficiency – for the FCA and the firms it regulates – and access, including unlocking more sources of funding.“The strategy provides a useful reference point for banks and other financial services firms to assess their priorities and opportunities in line with these themes. However, they will no doubt welcome greater clarity as the strategy translates into the FCAs specific plans and actions

FF News
Mar 14th, 2025
Capco Collaborates With Openai To Enhance Innovation, Drive Efficiencies And Deliver Lasting Benefits To Financial Services Clients

Capco, the global management and technology consultancy, today announced a collaboration with OpenAI, the maker of ChatGPT and frontier AI models such as GPT-4o, GPT-4.5, OpenAI o1 and o3 mini, to accelerate AI’s transformative impact at the world’s top financial institutions. This collaboration combines OpenAI’s cutting-edge innovations with Capco’s financial services domain and GenAI expertise to deliver powerful solutions that help clients achieve transformative results. As part of its GenAI Infused strategy, Capco is leading by example in the adoption and application of GenAI tools and solutions across the financial services industry. Through this collaboration with OpenAI, Capco is further scaling the adoption of ChatGPT across its workforce and pioneering these GenAI tools to accelerate client delivery and execution at scale in real world use cases that are making a difference for our clients

FF News
Mar 14th, 2025
Capco Collaborates with OpenAI to Enhance Innovation, Drive Efficiencies and Deliver Lasting Benefits to Financial Services Clients

Capco, the global management and technology consultancy, today announced a collaboration with OpenAI, the maker of ChatGPT and frontier AI models such as GPT-4o, GPT-4.5, OpenAI o1 and o3 mini, to accelerate AI's transformative impact at the world's top financial institutions.

Consulting.us
Mar 4th, 2025
Capco promotes Chris McNeely to global energy lead and John Janssen to Texas lead

Capco, a financial services and energy sector consulting firm, has promoted Chris McNeely to global lead of its energy practice and John Janssen to head of the Texas market.