Full-Time
Staff Machine Learning Scientist
Applied AI, Remote
Posted on 10/9/2025
Testlio
501-1,000 employees
Managed app testing services platform
No salary listed
Remote in USA
Remote
EMEA region; 100% remote; excludes high-cost locations such as Benelux, DACH, France, Nordic.
 AI & Machine Learning (4) |
|---|
| , , , |
- Advanced degree (Master’s or PhD) in Computer Science, Data Science, Statistics, or a related field.
- 10+ years applying end-to-end machine learning and statistical modeling solutions to real-world problems, ideally in SaaS or data products.
- Strong Python skills and experience with deep learning frameworks such as PyTorch and TensorFlow.
- Hands-on experience with NLP, predictive modeling, recommendation systems, anomaly detection, and training ML and deep learning models from scratch.
- Expertise in data wrangling, feature engineering, and managing large, complex datasets.
- Knowledge of Large Language Models (LLMs) and Small Language Models (SLMs), including architecture, training, fine-tuning (LoRA, QLoRA, SFT), and deployment strategies.
- Proven experience designing, building, and maintaining end-to-end ML pipelines and MLOps frameworks, including model training, deployment, monitoring, and lifecycle management.
- Hands-on experience designing, deploying, and maintaining AI agents, including multi-agent systems, in production with robust APIs and error handling.
- Proficiency with SQL, data visualization tools, and cloud platforms such as AWS or Azure.
- Partner with engineering leaders to define, design, and deliver AI Data products.
- Explore and model complex datasets from Testlio’s platform using statistical, ML, and deep learning techniques.
- Prototype and validate models, then work with engineering to deploy them into production at scale.
- Research and implement techniques in areas like NLP, anomaly detection, recommendation systems, and predictive analytics.
- Translate raw outputs into clear, actionable insights that are easy for customers to understand and use.
- Measure and improve model performance continuously to ensure accuracy, fairness, and scalability.
- Contribute to building Testlio’s data science practice — influencing standards, tools, and best practices.
Testlio provides managed app testing services through a fully integrated platform that supports end-to-end software testing. It offers both fully managed and co-managed testing, using a global network of vetted testers who are paid by the hour. Testlio’s services cover functional testing, test automation, localization testing, usability testing, and more, and the platform integrates with CI/CD pipelines to manage tests and deliver insights. Expert project managers and test leads oversee projects to ensure high-quality results. The company helps product, engineering, and QA teams test applications faster, reduce risk, and improve user experiences. Unlike some competitors, Testlio combines a scalable global tester network with end-to-end test management and strong CI/CD integration, focusing on managed testing services rather than just tooling. Its goal is to help clients ship reliable apps quickly with better quality and customer satisfaction.
Company Size
501-1,000
Company Stage
Series B
Total Funding
$20.5M
Headquarters
San Francisco, California
Founded
2012
Simplify's Take
What believers are saying
- Summer Weisberg appointed CEO February 3, 2026, accelerates AI growth.
- LeoInsights launched January 21, 2026, cuts reporting time by 90%.
- Apex Partner Program unites fintechs, agencies for payments testing expansion.
What critics are saying
- BrowserStack erodes pricing power with identical crowdsourced testing in 12 months.
- Sauce Labs AI automation displaces human testers in 18 months.
- Enterprises build internal AI testing, displacing Testlio in 24 months.
What makes Testlio unique
- LeoAI Engine leverages 13 years of testing data for AI-driven QA insights.
- Global network covers 600K+ devices, 800+ payments, 150+ countries.
- Fully managed crowdsourced testing ensures expert accountability and scale.
Help us improve and share your feedback! Did you find this helpful?
Benefits
Remote Work Options
Paid Vacation
Professional Development Budget
Stock Options
Flexible Work Hours
Growth & Insights and Company News
6 month growth
↑ 0%1 year growth
↑ 1%2 year growth
↑ 0%Testlio to Showcase ai-powered ecommerce and Payment Testing capabilities at ShopTalk 2026. Posted On March 24, 2026 AUSTIN, TX - MARCH 24, 2026 - Testlio, a leading AI-powered crowdsourced testing platform, today announced its participation in ShopTalk 2026, where the company will demonstrate how its end-to-end testing solutions empower retailers and ecommerce brands to deliver flawless digital experiences at scale. With $67 billion in Cyber Week 2025 sales and one in five digital purchases now flowing through an AI agent, the stakes for seamless ecommerce and payment experiences have never been higher. Meanwhile, 67% of consumers are willing to try new products but only when brands deliver consistent omnichannel experiences. "As digital commerce continues to evolve, consumers increasingly expect error-free interactions whether they're shopping on mobile, completing a payment, or navigating a chatbot," said Dean Hickman-Smith, CRO, Testlio. "Those experiences are what Testlio delivers every day for the world's leading retailers and digital commerce brands." What Testlio Will Showcase at ShopTalk Attendees visiting Testlio at ShopTalk will get a firsthand look at testing services purposefully built for the complexity of modern commerce. Those that combine the precision of global expert testers with the speed and intelligence of AI. Key capabilities on display include: * Payment Testing across 800+ payment methods, ensuring transactions complete without friction regardless of geography or platform. * AI-Driven Testing that surfaces real-time insights, accelerates ramp time, and reduces manual overhead across the entire release cycle. * Localization & Location Testing spanning 150+ countries and 100+ languages to validate that every market receives a consistent, high-quality experience. * Mobile App Testing across 600K+ real devices to catch defects before they reach real customers. * Functional & Usability Testing that goes beyond bug detection to assess holistic product quality from the end user's perspective. The Testlio Difference Traditional crowdtesting can feel fragmented. Too many testers, not enough ownership, and results that are hard to trust. Testlio takes a fundamentally different approach by bringing structure, accountability, and scale to retail and payments testing. Testlio delivers: * Global vetted experts, intentionally matched to your domain and product. * Fully managed end-to-end test execution to minimize overhead for internal teams. * Parallel testing across time zones enables faster release cycles and tighter launch windows. * On-demand, scalable in-market testing that flexes to your roadmap and business needs. * Dedicated client teams that strengthen releases through strategic oversight and accountability. All of this is powered by Testlio's proprietary AI engine, known as LeoAI Engine(TM), that is built and trained on more than 13 years of testing data. Trusted by the World's Leading Retail and Commerce Brands Testlio's clients represent some of the most recognizable names in retail, ecommerce, and digital payments, including Away, eBay, Etsy, PayPal, Thrive Market, Wayfair, and Whatnot; all of whom rely on Testlio to ship with confidence. Meet Testlio at ShopTalk 2026 Testlio representatives will be available at booth 4358 during ShopTalk 2026 to discuss how organizations can achieve holistic quality across ecommerce, payments, and digital banking.
Summer Weisberg takes the helm as testlio's newly appointed CEO. Former Testlio COO and industry veteran takes on the role of Chief Executive Officer (CEO) for the managed crowdsourced testing company. Austin, TX - February 03, 2026 - Testlio, a leader in AI-powered managed crowdsourced testing, today announced the appointment of Summer Weisberg as CEO, marking a new chapter for the company as it moves into its next phase of growth. Since joining Testlio in 2020, Summer has played a key role in shaping how the company scales delivery, supports enterprise customers, and evolves its platform. She steps into her new role after serving as Chief Operating Officer and, most recently, Interim CEO, where she sharpened Testlio's strategic focus, aligned teams around clear priorities, and built momentum across the business. Under her leadership, Testlio strengthened its position as a trusted partner for enterprises needing fully managed, human-in-the-loop testing at scale. Reflecting on her appointment, Summer shared, "I'm grateful for the trust and confidence placed in me by the Board, our leaders, and every TestLion. It's both humbling and exciting to lead a company that cares so deeply about our customers and our community." A new era for Testlio. The appointment comes at a pivotal time for Testlio as the company continues to invest in its AI-powered platform to make quality a key decision layer within release pipelines. The recent launch of LeoAI Engine(TM) is a major step in how Testlio helps teams combine AI and human expertise to improve product and release decisions, reduce risk, and move faster without sacrificing trust. That shift is also why choosing the company's next CEO mattered so much to the Testlio Board. The Board spent six months running a comprehensive search, reviewing more than a dozen candidates, and closely observing Summer's leadership during her time as Interim CEO to ensure Testlio had the right leader to carry this momentum forward. "From the outset of the search, we were aligned on what this next phase of Testlio requires," said Kristel Kruustük, Testlio's Founder and Board Director. "A clear growth strategy, disciplined execution, and leadership that sets and upholds a high bar. As Interim CEO, Summer consistently demonstrated the strategic clarity, operational discipline, and accountability needed to lead Testlio into its next chapter. We have full confidence in her leadership." As CEO, Summer will focus on accelerating Testlio's growth strategy, expanding its AI-driven platform capabilities, and deepening its leadership in human-in-the-loop testing for modern, AI-enabled products. She added, "Our next chapter is about scaling what makes Testlio different. We will continue investing in our AI-powered platform to bring speed and efficiency, while empowering our global community to apply human judgment, context, and insight where machines cannot. That combination is how we will drive Testlio's next phase of growth." Testlio's fully managed and AI-driven crowdsourced testing platform connects global quality experts with product and engineering teams to ensure every release works for every user, everywhere. On 600K+ real devices. In 100+ languages. With 800+ payment methods. The company is 100% remote, with team members in 150+ countries. Female-founded, approximately 46% of full-time employees are women. Testlio's clients include some of the world's leading brands, such as Clari, Strava, Whatnot, Merck, and more. As an ISO 27001:2022 certified vendor and trusted Microsoft Supplier, Testlio Inc. apply rigorous security measures aligned with global privacy expectations, such as GDPR, to every client engagement. Learn more at testlio.com.
Testlio launches LeoInsights(TM) to turn software QA data into measurable business impact. New AI-powered analytics help engineering and QA leaders measure quality, reduce risk, and ship faster. Testlio January 21st, 2026 AUSTIN, TX - JAN 21, 2026 - Testlio, a leading fully managed crowdsourced testing platform, today announced LeoInsights(TM), a suite of AI-driven intelligence features that converts fragmented QA data into decision-ready insights for engineering and business leaders. LeoInsights is powered by Testlio's LeoAI Engine(TM), the company's intelligence layer trained on more than 13 years of testing data, 2.6+ million test cases, and 600,000+ devices. As organizations scale, QA leaders often struggle to prove the business value of testing. Quality data is scattered across dashboards, reporting is manual and time-consuming, and benchmarks are hard to establish. LeoInsights addresses this by unifying testing data and delivering AI-generated recommendations on quality, risk, and investment. The offering analyzes more than 100 signals across reports within seconds, reducing time spent preparing executive reports by up to 90%. Early adopters also report saving 2 - 4 hours per day on manual analysis of app reviews. "We're excited to introduce LeoInsights into our QA strategy to help us optimize for release velocity. We've already accelerated our release cycles by 30% because we can now instantly identify which test cases are delivering the most value and where risks are emerging, allowing us to make informed decisions in real-time rather than waiting weeks to piece together performance data from disconnected sources," said Risko Ruus, Principal QA Manager, Rush Street Interactive. Key capabilities. LeoInsights unifies data, signals from the Testlio platform, and anonymized cross-industry benchmarks to deliver AI-driven recommendations that guide decisions around quality, risk, and investment through four integrated capabilities: * Executive Summaries: On-demand view of key changes, emerging risks, and critical issues, converting multiple complex reports into executive-ready overviews. * Outlier Insights: Automated alerts that surface unusual trends and anomalies, enabling teams identify risks or opportunities that might otherwise go unnoticed. * App Review Analysis: Analyzes app reviews and sentiment trends to surface user-reported bugs and customer experience risks. * Value Calculator: Quantifies efficiency gains, cost savings, and quality impact to help teams demonstrate QA value to leadership. Aggregates data across workspaces, supports scenario modeling with adjustable inputs, and generates downloadable executive-ready PDFs for budgeting and investment discussions. "Quality teams have historically drowned in data while starving for business insights," said Summer Weisberg, COO and Interim CEO at Testlio. "LeoInsights changes that. With the depth of our data and the power of AI, we are transforming software quality from a technical metric into a strategic asset that drives executive decision-making." LeoInsights is available now. Additional capabilities, including expanded benchmarking and predictive analytics, are planned for future releases. For more information, visit testlio.com/platform/leoai/leoinsights/. About Testlio Testlio's fully managed and AI-driven crowdsourced testing platform connects global quality experts with product and engineering teams to ensure every release works for every user, everywhere. On 600K+ real devices. In 100+ languages. With 800+ payment methods. The company is 100% remote, with team members in 150+ countries. Female-founded, approximately 46% of full-time employees are women. Testlio's clients include some of the world's leading brands, such as Netflix, Clari, Strava, Whatnot, Merck, and more. As an ISO 27001:2022 certified vendor and trusted Microsoft Supplier, Testlio Inc. apply rigorous security measures aligned with global privacy expectations, such as GDPR, to every client engagement. Learn more at testlio.com.
Codenotary extends free SBOM.sh service to AI software supply chain. Codenotary, leaders in software supply chain protection, today announced new capabilities for its free SBOM.sh service - supporting AI applications by treating datasets as software supply chain artifacts. The update represents a necessary evolution of SBOMs that reflects how modern systems are actually built, deployed, and operated, helping to close a critical gap in security and compliance. "Traditional SBOM tools were built for an earlier era - focusing primarily on source code to improve visibility into the software supply chain," said Moshe Bar, CEO and co-founder, Codenotary. "Security teams are swimming in SBOMs, but they're not getting the actionable clarity they need - especially as AI transforms software with AI applications built on datasets which are entirely ignored by traditional SBOMs." Now, SBOM.sh delivers the following capabilities to help enforce data governance, avoid license violations, and demonstrate provenance during audits or regulatory reviews. - Model Lineage and Training Transparency - SBOM.sh captures lineage metadata including base-model origins, fine-tuning history, version identifiers, and update pathways. SBOM.sh is available as an easy-to-use service that enables developers, DevOps teams, and security organizations to upload, analyze, and share SBOMs, as well as their AI software supply chain. Check Point(R) Software Technologies Ltd.(link is external) announced Check Point Exposure Management, a new approach designed to help organizations defend against AI-era attacks by turning fragmented exposure data into prioritized, actionable, and safe remediation. Codenotary, leaders in software supply chain protection, today announced new capabilities for its free SBOM.sh service - supporting AI applications by treating datasets as software supply chain artifacts. Testlio announced LeoInsights(TM), a suite of AI-driven intelligence features that converts fragmented QA data into decision-ready insights for engineering and business leaders. Cloudflare announced that The Astro Technology Company team, the creators of the Astro web framework, will be joining Cloudflare. GitLab(link is external) announced the general availability of GitLab Duo Agent Platform. Kaggle launched Community Benchmarks, a new offering that allows developers to move beyond static academic metrics and build their own reproducible evaluations for AI models. The Cloud Native Computing Foundation(R)(CNCF(R), which builds sustainable ecosystems for cloud native software, announced the graduation of Dragonfly, a cloud native open source image and file distribution system designed to solve cloud native image distribution in Kubernetes-centered applications. Commvault announced Commvault Cloud Unified Data Vault, a cloud-native service that extends Commvault's trusted, air-gapped protection and resilience capabilities to data written using the S3 protocol, bringing S3-based application and AI data under a unified, policy-driven protection framework for enterprise-grade resilience. LambdaTest announced its rebrand to TestMu AI, marking a step in its evolution from a cloud testing platform to a full-stack Agentic AI Quality Engineering platform. Postman announced its acquisition of Fern, a developer experience company focused on helping businesses ship polished API documentation and production-ready Software Development Kits (SDKs). Keeper Security announced the launch of its JetBrains extension, offering JetBrains Integrated Development Environment (IDE) users a secure and seamless way to manage secrets within their development workflows. Red Hat announced a landmark expansion of its collaboration with NVIDIA to align enterprise open source technologies to the rapidity of enterprise AI evolution and rack-scale AI advances. The Cloud Native Computing Foundation(R)(CNCF(R), which builds sustainable ecosystems for cloud native software, announced the schedule for KubeCon + CloudNativeCon Europe 2026, taking place in Amsterdam, March 23-26. Check Point(R) Software Technologies Ltd.(link is external) announces that it has been recognized as a Leader in the 2025 Gartner(R) Magic Quadrant(TM) for Email Security. Apiiro introduced Apiiro AI SAST, a new approach to static application security testing (SAST) that automates code risk detection, validation and fixes with the precision and cognitive process of an expert application security engineer.
Testlio announced the launch of its Testlio Apex Partner Program (TAPP), a strategic initiative uniting fintechs, agencies, and technology providers to redefine how organizations approach software quality, payments assurance, usability, and competitive intelligence testing.