Backend Engineer - Machine Learning Storage Infra

The mission of our AML team is to push the next-generation AI infrastructure and recommendation platform for the ads ranking, search ranking, live & e-Commerce ranking in our company. We also drive substantial impact on core businesses of the company. Responsibilities - Design and build a unified platform/middleware system that can support diverse business requirements across different scenarios, including low cost, high availability, high throughput, high performance, and large-scale storage capacity. - Design and optimize complex multi-tier storage architectures beyond GPU memory, CPU memory, and external storage, with a focus on efficient data placement and resource utilization. - Keep up with the latest advances in software and hardware architectures, and proactively evaluate and experiment with emerging technologies. - As an internal platform serving multiple teams, plan and optimize the utilization of large volumes of heterogeneous resources across multiple hardware generations, data centers, service tiers, and resource pools. Develop automated and dynamic optimization strategies based on changes in model size, service traffic, and workload characteristics. Minimum Qualification(s) - Bachelor's degree or above in Computer Science, Software Engineering, or a related field - Proficient in C++ and Python programming in Linux environments. - Strong understanding of distributed systems principles, with hands-on experience in the design, development, maintenance, and continuous optimization of large-scale distributed systems. Able to identify potential issues and bottlenecks in complex distributed systems. - Experience working on distributed systems in areas such as recommendation, search, or machine learning, with exposure to resource scheduling, task orchestration, model training, model inference, feature extraction, ML Systems (MLSys), or AIOps. - Strong logical and analytical thinking skills, with the ability to abstract and decompose complex business and technical requirements effectively. Strong teamwork and collaboration skills. Preferred Qualification(s) - Experience optimizing systems similar to Parameter Server, or optimizing indexing structures in large-scale search systems. - Experience with KV Cache systems, such as Mooncake, including system optimization and performance tuning and with mainstream machine learning frameworks such as TensorFlow, PyTorch, or MXNet. - Familiarity with open-source storage systems such as Redis, LevelDB/RocksDB, and MongoDB, or hands-on experience using or optimizing large-scale distributed storage systems such as HDFS or Ceph.