Software Engineer Intern

Confluent: Stale, unreliable data could impair enterprise AI. Wed, 9th Sep 2026 (Today) Australian enterprises looking to scale artificial intelligence need to focus less on the capabilities of AI models and more on whether the data feeding those systems can be trusted, according to Confluent, an IBM Company. The data streaming company says trust is emerging as a major barrier to enterprise AI adoption as organisations move beyond experimentation and begin putting generative and agentic AI applications into production. The focus of the AI conversation has changed significantly over the past 12 months, said Andrew Foo, VP Customer Solutions APAC, Confluent, an IBM Company. "In 2026, the promise of AI and AI's agency to reason, make decisions, and take action has very much taken over a lot of the data discussions," he said. While much attention had initially centred on what AI models could do, organisations are now confronting the practical challenges of deploying those systems safely and reliably, he said. "The challenge now is increasingly shifting from what the models themselves can do, to whether the businesses can actually trust the data that's underpinning it," Foo said. Confluent's 2026 data streaming report, based on a survey of about 4600 IT leaders, found that 72 per cent cited challenges around areas including data lineage, timeliness and data quality assurance. Sixty-nine per cent said insufficient real-time data infrastructure was holding back their ability to scale AI. These findings highlight a fundamental issue for businesses seeking to move AI from pilot projects into business-critical applications. They need confidence that the information supplied to generative AI systems is accurate, current and sourced appropriately, whilst ensuring the right controls are in place over who can access that data. That challenge becomes even more significant as companies adopt agentic AI systems capable of taking actions rather than simply generating answers. As a result, organisations need to expand their approach to AI governance beyond the models themselves and towards the data continuously feeding them. Foo warned that organisations should not view governance and speed as competing priorities, particularly as they face pressure to keep pace with the rapid development of AI. Instead, security and governance should be incorporated into data architectures from the beginning rather than being added after an AI application has already been built. "What they should be thinking about is embedding this into their data architecture from the outset," he said. Confluent advocates a 'shift-left' approach to data governance, moving controls closer to the source of the data instead of attempting to introduce them further downstream. It has introduced capabilities designed to support that approach, including automated personally identifiable information detection and redaction within data streams. This allows organisations to protect sensitive information before it reaches downstream AI applications. Confluent has also introduced private connectivity capabilities intended to allow enterprises to connect AI workloads to external models without sending sensitive traffic across the public internet. Such capabilities are particularly relevant in highly regulated sectors including financial services, healthcare and insurance. Foo is adamant that governance should become an embedded part of how organisations build applications rather than being treated as a separate compliance exercise. "You don't do a data governance project. You do data governance in every project," Foo said. "The same applies for AI. You don't just do an AI governance project. You do AI governance in every project." Real-time data becomes critical Another major challenge for organisations adopting AI is the freshness of data available to those systems. Traditional approaches based on batch and micro-batch processing are becoming inadequate for AI applications that need to respond to rapidly changing business conditions. "AI is only as useful as the data and the context that's available to it," he said. "In many enterprise scenarios, yesterday's information simply isn't enough." Businesses continuously generate new events, from financial transactions and customer interactions to inventory movements, security incidents and operational updates. Connecting AI systems to those live streams allows applications to make decisions based on current conditions rather than historical snapshots. Foo cited fraud detection in the banking industry as an example. A bank assessing a new credit card application may have information indicating that a customer previously represented a relatively low risk. But in the hours since the bank's last batch update, that same customer may have made unusual transactions, logged in from another country or used an unfamiliar device. An AI system relying on stale information could therefore approve an application that should have been flagged. With real-time data, however, the system can combine information about transactions, devices, geography and other risk indicators to make a decision based on what is happening at that moment. "AI is doing the right thing. It simply didn't have access to the real-time information it needed to make the right decision," Foo said. For Confluent, the role of data streaming is to turn those continuously changing business events into governed and trusted context that AI applications and agents can use. Where technology leaders should start For companies trying to strengthen their data foundations, the first step should be understanding the information already flowing through the business. Technology leaders need to understand where data is coming from, how fresh it is, who can access it and whether it can be trusted. Foo cautioned against treating AI as a standalone project that requires an entirely separate data environment. Instead, organisations should look to securely connect AI applications to the operational data already powering the business. Leaders should also identify security and governance requirements early, particularly for sensitive and regulated information, rather than attempting to retrofit controls once an AI application reaches production. Focus then shifts to creating governed, real-time data products around the core entities of the business, including customers, products, orders and inventory. "If you have this as a strong foundation from the outset, it will ultimately make it easier to innovate without introducing unnecessary complexity or risk when it comes to advancing your AI ambitions," Foo said.

Time series foundation models change how Mgks build real-time AI. I've watched teams spend months wiring ML models into production only to watch them drift within weeks. The plumbing always takes longer than the math. IBM and Confluent just announced something that sidesteps that entire problem: time series foundation models that live natively inside streaming infrastructure, callable from SQL, swappable with a parameter change. This matters more than the typical integration announcement because it reorganizes where the work actually happens. The old way still dominates. Most forecasting today is statistical models and spreadsheets dressed up as decisions. Teams model the few hundred products or systems where the money is obvious, then cover everything else with safety margins. Extra inventory. Extra headroom. Extra tolerance. That margin gets paid every cycle, and nobody could forecast it anyway. Bespoke ML promised better. One model per series. Hand-tuned. Refit monthly. But one model per series means specialization: data scientists build pipelines, data engineers wire them, and the domain expert who actually owns the decision waits. A demand planner can't touch it. A process engineer can't tune it. A fraud analyst can't customize it for new corridors without filing a ticket. So most organizations still don't bother. The tail of their catalog, their equipment, their transactions runs on margins because modeling thousands of series is not rational when each one takes specialist time. What changes with a foundation model at the edge. A time series foundation model trained across millions of signals learns patterns that generalize. Give it a window of measurements it has never seen and it forecasts what comes next, scores how far behavior sits from normal, finds similar history, and optimizes toward a target. More important: a demand planner can point it at their data stream. A process engineer can customize it on their line. No data science ticket required. IBM ran these in their own operations first, then with design partners in cement, steel, food, telecom. The numbers are real: every point of accuracy in forecasting is worth millions in working capital. Productivity gains run 5 to 10 times. Work that waited for specialists now sits with the people who own the decision. But running a model is not the same as running it in time. A signal's value decays with time. A pump caught drifting today is a work order. The same pump next week is an outage. That's why Confluent matters here: they bring the live state of the business to the model, managed as stateful streams inside Apache Flink, keyed per series and fault-tolerant. No separate database hit. No external feature store query. The model gets the history it needs to make the next call matter. The portfolio approach beats the silver bullet. Here's what I find most pragmatic about this: IBM is not offering one model. They're offering four, all related to foundation models but built for different questions. PatchTST-FM reads time series the way language models read text, patch by patch, each variable in its own channel so noise doesn't cascade. It returns a full distribution so a planner can set reorder points off the 90th percentile. FlowState keeps a running summary with every point and handles both seconds-level sensor data and hourly market data because its dynamics are continuous in time. TTM drops attention for tiny mixing networks so a million-parameter model covers a hundred thousand series nightly on CPU. TSPulse pairs time and frequency views for anomaly detection and the question every operator asks: have Mgks seen this before? Switch models with one SQL parameter. No pipeline redesign. No separate ML stack. That's not a feature, that's a philosophy. Why this matters for how you ship. I think the deeper shift here is about where decisions live. Right now, forecasting and anomaly detection and optimization are typically batch jobs. Nightly runs. Reports. The business reacts the next morning. But in a streaming context, forecasts become triggers. A replenishment order fires automatically. A fraud alert lands before money moves. An optimizer recommends the next setpoint as conditions change. The gap between an event and knowing about it shrinks from days to seconds. And because the model understands context, the score that lands on the downstream topic is already actionable: not just anomaly detected, but here's the closest past case and here's what happened then. Streaming data was always supposed to enable this. Now the ML tooling catches up. Small model size was a deliberate choice, not a compromise: inference runs on your own CPUs or natively inside Confluent Cloud with no external API call per decision. No cloud ingress or egress. That keeps architecture simple and cost rational at scale. The real test will be adoption: whether domain experts actually use these models on their own, or whether bottlenecks just move upstream. But the path is clearer now than it was last year, and that matters.

Confluent has secured a $510 million term loan B facility to fund its acquisition of A&E Networks. The financing, finalised on 13 July 2026, forms part of a broader capital structure adjustment for the strategic deal. The senior secured debt will finance the transaction and related expenses. It will be used alongside existing equity and other debt instruments as a key component of the capital stack. The acquisition is expected to expand Confluent's market reach and diversify its revenue streams. The transaction remains subject to regulatory approvals and customary closing conditions. Investors are advised to monitor the company's upcoming earnings call for further details on the financing structure and integration plans.