Senior Fullstack Engineer 2

How Khan Academy optimizes AI tutoring with experimentation. Khan Academy went from vibes-based prompt testing to running A/B experiments on their AI tutor, Khanmigo, in production. Kelli Hill shares the journey, including a fascinating iterative case study on latency vs. math accuracy. Kelli Hill gave a standout presentation at The Conference known as Experimentation island on February 24, 2026, walking the audience through Khan Academy's evolution from intuition-based testing to running A/B tests on generative AI features in production. If you missed it, the good news is Kelli will be joining GrowthBook, Inc. for a webinar on April 16, 2026. I'd highly encourage you to register here. Below are my key takeaways from her talk. A quick word on Khan Academy. Khan Academy is a nonprofit with a mission to provide a free, world-class education for anyone, anywhere. They have nearly 200 million registered users and have logged over 63 billion learning minutes on their platform. In 2023, they launched Khanmigo, a generative AI-powered tutor and teaching assistant built on top of their massive library of exercises, articles, and instructional content. Khanmigo is the focus of much of their current experimentation work, and the context for everything Kelli shared. From homegrown to a real experimentation stack. Khan Academy has been running experiments since 2011, when they built their first in-house platform on Google App Engine. At their peak, they had hundreds of A/B tests running simultaneously. But over time, the homegrown system slowed down, and when they rewrote their entire backend in 2019 (a million lines of code, migrating off Python 2), they made a deliberate decision not to port their old experimentation tooling. Instead, they evaluated what was available. Building a new platform in-house was tempting, but they recognized that experimentation infrastructure wasn't their core competency. Buying an enterprise solution would have required downsampling their data, which was a non-starter. They ultimately chose GrowthBook, self-hosting it and connecting it to their existing data warehouse and eventing pipelines. Their chief architect's top priority was that the tool not slow down a site serving a million daily active users, and GrowthBook delivered on that. The lesson here is one GrowthBook, Inc. see repeatedly: organizations that try to build their own experimentation platform almost always end up spending more than expected, moving slower than they'd like, and eventually switching to something purpose-built. Khan Academy's journey is a textbook case of making that transition well. How evals evolved from vibes to automated A/B testing. The most fascinating part of Kelli's talk was the four-phase journey Khan Academy went through to figure out how to measure AI quality. When you're building an AI tutor, you can't just measure click-through rates. The goals are harder: * Increased cognitive engagement * An increase in skills on their way to proficiency * Measurable learning gains on external assessments. And LLMs make measurement even harder because they're non-deterministic. The same prompt can produce wildly different outputs each time. Phase 1: Intuition-driven testing. In September 2022, before ChatGPT had even launched publicly, OpenAI gave Khan Academy early access to GPT-4 via Slack. The team's first experiments were literally typing prompts into Slack and reading the outputs. They quickly discovered problems (GPT-4 confidently told a user that 9 + 5 = 15, then gave the correct answer ten minutes later). Good enough for building intuition about how LLMs behave, but not for building a product. Phase 2: Structured manual testing. With a deadline to launch alongside GPT-4's public announcement in March 2023, they built an internal prompt playground for more repeatable testing. Faster than Slack, but still relied on humans to read outputs and judge quality. Phase 3: Automated post-hoc evals. This is where things got serious. They assembled a team of PhDs in education to define what good tutoring actually looks like, then had human raters apply that rubric to chat transcripts, targeting 85% inter-rater agreement. Once they had that ground-truth dataset, they used it to train an LLM-as-judge to label transcripts at scale. The key insight: many teams spin up LLM-as-judge systems with no ground truth, resulting in unreliable results. Khan Academy invested in the hard work of human annotation first. Once the machine matched human accuracy, they scaled it to process thousands of interactions nightly. Phase 4: A/B testing in production. With reliable automated evals in place, they could finally run controlled experiments on prompt changes, system instructions, and even entire model swaps, all measured against metrics like cognitive engagement, item performance, undesirable tutoring behaviors (like giving away answers), and latency as a guardrail. This is the stage they're in now, with 64 completed experiments, 29 running, and 13 queued as of February 2026. The takeaway: as AI products mature, your evaluation methods need to mature with them. You can't skip straight to production A/B testing without the foundation of knowing what "good" looks like. The math agent story: what iterative AI experimentation actually looks like. Kelli shared a concrete example that perfectly illustrates how A/B testing enables teams to "hill climb" toward better AI quality. The problem: Khanmigo had a math agent, essentially a calculator it could call to verify computations. Great for accuracy, but it added latency that was painful in classroom settings. Here's how the iterations played out: Iteration 1: Remove the math agent entirely. Latency improved, but math errors doubled. Rolled back immediately. Iteration 2: Switch to GPT-5. Latency decreased, but math accuracy still suffered. Rolled back. Iteration 3: Optimize the math agent's prompts. They tightened the system instructions to be more efficient. Latency dropped by three seconds, and math accuracy held. A real win. Iteration 4: Give the math agent a faster model. Reduced latency by another 300 milliseconds with stable accuracy. Iteration 5: Time-box the math agent's execution. Further latency reduction, accuracy still stable. Without A/B testing, the team might have shipped Iteration 1 or 2 and unknowingly degraded the learning experience. The experiments gave them the confidence to reject changes that looked good on one metric but failed on the one that mattered most. This is what "hill climbing" looks like in practice: hypothesis, test, measure, iterate. No single change was transformative. The cumulative effect was. From speed bump to safety net: the cultural shift. Perhaps the most important takeaway from Kelli's talk was about culture. Before Khanmigo, experimentation at Khan Academy was seen as a speed bump. Product teams wanted to ship based on strong founder intuition and internal conviction. Running an A/B test felt like an obstacle to velocity. Generative AI changed that completely. LLMs are unpredictable enough that even small changes to prompts or system instructions can produce dramatically different outputs. Teams quickly learned that shipping without testing was genuinely risky. The same engineers who once resisted experimentation now actively request it. Experimentation went from being perceived as something that slows you down to being the safety net that gives teams the confidence to move fast. That cultural transformation, more than any individual experiment result, may be the most valuable outcome of Khan Academy's journey. Want to hear the full story from Kelli? She'll be joining GrowthBook, Inc. for a live webinar on April 16, 2026, where she'll share this full story.

Choosing college in the age of AI. March 5, 2026 | College Planning, College Visits AI is reshaping College - and the workforce. Here's what students need to understand now. I can't scroll through my feed without seeing another headline about AI reshaping education and the workplace. At Alpha School, an AI-powered model is already delivering core academics in just two hours a day using adaptive technology - to students as young as kindergarten. The founder of Khan Academy has announced plans to use AI to build a learning model that could rival institutions like Harvard University or Stanford University. And companies are moving just as fast. Jack Dorsey recently announced that Block - the company behind Square and Cash App - is reducing its workforce effective immediately by 4,000 roles as part of restructuring driven by rapid AI advancement This isn't theoretical. It's happening now. So as students build their college lists or decide where to enroll, there's a question that matters more than ever: How is this college integrating AI across its curriculum - not just in computer science, but across business, healthcare, engineering, communications, education, and beyond? Because four years is a long time in today's world. If your education isn't evolving alongside technology, you risk graduating into a workforce that already has. But here's the nuance that often gets missed. "What careers are safe from AI?" This is the wrong question. The better question is: Which careers will be transformed by AI - and how can students prepare to work alongside it? Yes, there are fields that are less likely to be fully replaced by automation. 1. Hands-On, physical professions. Jobs that require physical presence, manual dexterity, and in-the-moment problem-solving are harder to automate. * Nurses * Physical therapists * Electricians * HVAC and appliance technicians * Construction managers * Dental hygienists * Surgeons AI can assist with diagnostics, scheduling, imaging analysis, and efficiency. But it cannot replace a nurse holding a patient's hand before surgery. It cannot physically rewire a home. It cannot adjust its touch in real time based on a patient's pain tolerance. These roles are more resilient. But they are not untouched. A nurse who understands AI-powered charting systems, predictive diagnostics, and workflow automation will outperform one who does not. An electrician who can integrate smart home systems and AI-based monitoring tools will be more competitive than one who cannot. The future isn't "AI versus humans." It's humans who know how to use AI versus humans who don't. 2. Empathy-Driven professions. AI can simulate conversation. It cannot replicate lived experience, intuition, or emotional nuance. Fields that rely heavily on empathy, trust, and human connection are more insulated: * Social workers * Therapists * Teachers * Pediatricians * Nurses * Counselors * Clergy But even here, integration matters. Teachers will use AI to differentiate instruction. Therapists may use AI tools for documentation or pattern tracking. Doctors will increasingly rely on AI-assisted diagnostics. The human connection remains central. But the tools surrounding it will evolve. 3. High-Level strategic and creative roles. Leadership and strategy. AI can generate. It can analyze. It can predict patterns. But it does not decide vision. Executives, entrepreneurs, policy makers, engineers designing new systems - these roles require judgment, ethics, creativity, and responsibility. However, those leaders will absolutely be using AI in their decision-making process. Students entering business, law, communications, or engineering need to graduate not just knowing theory - but knowing how to leverage AI as a force multiplier. What this means for choosing a college. When families tour campuses, they often ask: * What's your ranking? * What's your acceptance rate? * What's your average starting salary? Those are fine questions. But now The College Navigators also need to ask: * How is AI integrated into non-technical majors? * Are students required to take courses on emerging technologies? * Are professors using AI in research and instruction? * Does the school partner with industries that are actively implementing AI? Because four years from now, today's freshmen will graduate into a workforce that looks dramatically different. The goal isn't just to earn a degree. It's to graduate adaptable. Technologically fluent. Ethically grounded. Colleges that are integrating AI. These are just a few of my favorite: * The Ohio State University: every Ohio State student, beginning with the class of 2029, will graduate being AI fluent - fluent in their field of study, and fluent in the application of AI in that field. * Purdue University: Beginning Fall 2026, Purdue will launch an "AI working competency" graduation requirement for all undergraduate students * University of South Florida: First in Florida to bring together the disciplines of artificial intelligence, cybersecurity, and computing into a dedicated college. * Colby College: Established in 2021, the Davis Institute for Artificial Intelligence at Colby is the first cross-disciplinary institute for artificial intelligence (AI) at a liberal arts college. The real skills college students will need. The students who will thrive are not the ones trying to outrun AI. They are the ones learning to: * Think critically * Ask better questions * Adapt quickly * Communicate clearly * Integrate technology thoughtfully AI rewards clarity of thinking. It amplifies capability. But it also exposes shallow understanding. The competitive edge will belong to students who deeply understand their field - and know how to use AI to enhance, not replace, their expertise. Four years is a long time. The world will not pause while students are in college. So as you build your list, commit, and invest in your education, ask yourself: Is this school preparing me for the future I'm stepping into? Or the past I'm leaving behind? ABOUT JENNA SCHEBELL: Jenna is the founder of The College Navigators and The Navigator Network. As one of the few private counselors in the country who has led both an admissions office at a selective college and a college counseling office at a private high school, she understands the complexities of the admissions process and shares valuable insights, strategies, and advice to help students navigate their own journey with confidence. SHARING HER INSIGHTS Strategically building your college list and understanding how you'll be evaluated is a key focus in The Navigator Network, especially for the Class of 2027 as they prepare for the upcoming admissions cycle. Through live webinars and interactive worksheets, students and parents gain valuable insights into the application process - what colleges look for, how admissions decisions are made, and most importantly, how to stand out. Interested in joining? Learn more Grab The Ultimate College Admissions Guide for Students Applying to College Additional consulting services: Check them out!