B

Bybit

Cryptocurrency exchange offering spot and derivatives

Senior Principal Site Reliability Engineer

Site Reliability EngineeringConfirmed live in the last 24 hours
Full-timeKuala Lumpur, Federal Territory of Kuala Lumpur, MalaysiaNo salary listedIn-person

About the Job

Requirements

  • At least 8 years of backend or infrastructure engineering experience, including at least 3 years dedicated to chaos engineering or stability engineering.
  • Hands-on experience with large-scale fault injection in production environments and a deep understanding of production safety constraints.
  • Expert-level proficiency in Kubernetes fault injection using Chaos Mesh, Litmus, or custom solutions, and familiarity with custom resource definition (CRD) and Operator development.
  • Proficiency in at least one backend language, with Go preferred, and platform-level system architecture design capability.
  • Deep understanding of distributed system failure modes, including network partitions, split-brain, cascading failures, and data inconsistency.
  • Familiarity with observability technology such as Prometheus, Grafana, Thanos, and OpenTelemetry.
  • Strong technical documentation and solution design skills.
  • Ability to balance safety and validation depth while maintaining strict risk control during production fault injection.
  • Ability to collaborate across teams and influence business teams to gain buy-in for chaos engineering.
  • Ability to independently plan and execute in ambiguous situations.

Responsibilities

  • Design and build an enterprise-grade chaos engineering platform supporting multi-cluster Kubernetes and EC2 hybrid, multi-region, and testnet and mainnet deployments.
  • Develop a fault injection engine for pod-, node-, and availability-zone-level fault simulation, including network latency, packet loss, partition, dependency timeout, and error injection.
  • Implement production safety assurance capabilities including blast-radius control, a one-click kill switch, automatic rollback, and real-time impact monitoring.
  • Implement traffic tagging and isolation so fault injection does not affect real users.
  • Implement precise fault impact scoping at service, cluster, and availability-zone granularity.
  • Design experiment orchestration for complex fault scenarios, such as simultaneous network latency, downstream timeout, and cache invalidation.
  • Integrate the platform with monitoring, alerting, and service-level objective systems to automate fault injection, impact observation, and pass/fail determination.
  • Define safety standards and approval workflows for mainnet fault injection.
  • Design and drive routine chaos experiments, including automatically executed daily or weekly low-risk patrol experiments, monthly or quarterly critical-path resilience validation, and cross-availability-zone or cross-region disaster recovery failover drills.
  • Establish a resilience scoring system to quantify system health based on experiment results.
  • Deliver improvement recommendations and drive business teams to remediate identified weaknesses.
  • Evaluate and select the technology foundation, using Chaos Mesh, Litmus, and custom components as appropriate.
  • Develop chaos engineering best practices and playbooks for SRE teams and application developers.
  • Mentor and grow a team of 2–3 engineers in chaos engineering capabilities.
  • Stay current with industry developments and introduce practices such as AI-driven fault scenario discovery.

Desired Qualifications

  • Experience in financial or trading system stability, including understanding transaction consistency and fund safety constraints.
  • Experience building service-level objective or error-budget frameworks.
  • Experience building automated fault-recovery (self-healing) systems.
  • Familiarity with AWS infrastructure including EC2, EKS, multi-availability-zone, and multi-region deployments.
  • Knowledge of Netflix Chaos Engineering, AWS Fault Injection Simulator, or Gremlin.
  • Open-source community contributions to Chaos Mesh, Litmus, or similar projects.

About the Company

Bybit

Cryptocurrency exchange offering spot and derivatives

Bybit operates a cryptocurrency exchange that provides spot and derivatives trading, mining and staking products, and API access for retail and institutional clients worldwide. Its platform uses an ultra-fast matching engine to execute trades and offers multilingual customer support, aiming to deliver a fast, fair, and humane trading environment. Compared with rivals, Bybit emphasizes reliability, transparency, and strong service, backed by a team with backgrounds from investment banks and tech firms, and a global community. The company’s goal is to be the most trusted and dependable exchange in the emerging digital asset class.

Company size
1,001-5,000
Company stage
Not listed
Total funding
Not listed
Headquarters
Singapore, Singapore
Founded
2018

Recently Posted Jobs

Explore more opportunities at Bybit

See more jobs