Trading Automation and Operations Intern

How Optiver built real-time trading dashboards with Databricks Apps and Dash. June 16, 2026 · AI Automators Trading firms live and die by how fast they can turn market data into decisions. At the 2025 Databricks Data + AI Summit, the trading firm Optiver walked through how it rebuilt its live dashboards on Databricks Apps and Dash. The writeup of that talk is short on internal code but useful as a reference architecture for anyone building real-time analytics at scale. Here is what stands out, and where the lessons apply beyond finance. What they actually built. The headline change is consolidation. Optiver replaced siloed, on-premise systems with a single petabyte-scale Databricks platform, then hosted Dash apps directly on that platform using Databricks Apps. Dash is Plotly's Python framework for building interactive web dashboards, and running it inside Databricks means the dashboard sits next to the data instead of pulling it across infrastructure boundaries. The streaming layer is Spark Structured Streaming. The post claims end-to-end latency dropped from minutes to seconds after optimization work on Spark and the streaming pipeline. That is the specific, measurable claim worth holding onto: not real-time in the microsecond sense traders sometimes mean, but seconds instead of minutes for self-serve market insight. Three other engineering details are named: * Smart caching to keep dashboards responsive without re-querying everything. * Version control and modular dashboard generation so dashboards are reproducible and assembled from reusable parts rather than one-off scripts. * Unity Catalog for fine-grained access controls, so traders can see their data without engineers hand-managing permissions. The stated payoff is trader autonomy: people who need a dashboard can build and iterate on one without filing a ticket with engineering. That is a workflow win as much as a performance win. Why this matters for automation builders. Strip away the trading context and this is a pattern many teams will recognize. You have a fast-moving data source, a hungry set of internal users who want custom views, and an engineering team that becomes a bottleneck the moment every dashboard requires a developer. The interesting move here is not any single tool. It is co-locating the app layer, the streaming layer, and the governance layer on one platform so the friction between them mostly disappears. A few things are genuinely transferable: Modular, version-controlled dashboards. Treating dashboards as composed, reproducible artifacts instead of bespoke notebooks is the difference between a system that scales and one that rots. If your automations or reports currently live as untracked one-offs, this is the lesson to steal. Governance as a built-in, not a bolt-on. Using a catalog layer like Unity Catalog to enforce who sees what means self-serve access does not become a security liability. Any time you let non-engineers build on live data, that control plane is what keeps it safe. Latency budgets are a design choice. Cutting minutes to seconds came from deliberate Spark and Structured Streaming optimization, not from a faster box. The honest read is that low latency at petabyte scale is achievable but takes real tuning work. What the post does not give you is the how. There are no code samples, no benchmark tables, and no detail on caching strategy or how modular generation is implemented. Treat it as a directional case study, not a tutorial. If you want the specifics, the underlying conference talk is where they live. Where it fits versus alternatives. This approach makes sense when your data already lives in, or is heading toward, Databricks and you want dashboards close to it. Hosting Dash on Databricks Apps removes the separate web infrastructure you would otherwise stand up, and that is the main draw over running Dash on your own servers or a generic cloud host. If you are not on Databricks, the same shape can be built elsewhere, but you lose the tight Unity Catalog integration and the single-platform simplicity. For lighter needs, a hosted BI tool may be enough and far less work to operate. The Databricks-plus-Dash combination earns its keep specifically when you need custom interactivity, streaming freshness, and scale at the same time. If you only need two of those, simpler options usually win. It is also worth being clear about what this is not. This is a data engineering and visualization story, not an AI agent story, even though Optiver's other posts discuss agentic AI and trading models. The dashboard system described here is about moving and displaying live data reliably, not about models making decisions. For most automation work, the practical takeaways are the parts you can apply without a quant team: keep dashboards reproducible and version-controlled, enforce access at the data layer, and decide your latency budget before you build. Tools like Make, n8n, or Zapier cover lower-volume automation, but when streaming data and scale enter the picture, a platform like Databricks is the kind of foundation this case study points to. If you want help designing a streaming dashboard or real-time data pipeline like this, browse the provider directory to find people who can put it to work.

Electronic trading giant Optiver is building out a new AI Lab. 8 April 2026 Over the past few years, AI labs like OpenAI and Anthropic have been unabashed in their pursuit of top young talent from electronic trading and HFT firms. Optiver, an Amsterdam/Chicago based trading giant, has decided two can play at that game. Earlier in the year, it launched an AI Lab in Shanghai. It's not clear how large the team is right now, nor when exactly it was founded, but one of its members is Zhou Fang, who joined in February of this year. He's a PhD mathematician who had worked as a deep learning quant for rival trading firm IMC in Chicago for five months after graduating. He describes himself as a founding member of the lab but is understood not to be in charge. The Shanghai lab has job openings for both a machine learning engineer and an AI researcher; listings call the AI Lab a "newly established team [which] will operate independently from our core trading operations" to apply "advanced AI and machine learning technologies to complex problems." Optiver is looking for people with "startup mindset but Big Tech rigor." There's precedent for spinning an AI lab out of a trading firm in China. Deepseek was founded by Liang Wenfeng, founder of Chinese quant fund High-Flyer, who used the fund's profits to finance his AI research. Optiver should have plenty of money to fund the lab after its primary business saw a lot of success in 2025; its annual review published last week said that net trading revenue was up by over €1bn ($1.17bn). Seemingly happy with the results so far, Optiver is building out a team for its AI lab in the US. This week, Andrew Arnold announced he was joining the firm as its head of research for the AI lab in New York. He's an adjunct machine learning professor at NYU and was most recently a principal applied machine learning engineer for eCommerce firm Shopify. Arnold was also previously a machine learning researcher at Google. He was also previously a portfolio manager at Cubist, the systematic arm of hedge fund Point72, and was the chief technology officer of hedge fund Trexquant. Optiver did not respond to a request for comment. Have a confidential story, tip, or comment you'd like to share? Contact: WhatsApp: http://wa.me/442079977910 (+44 20 7997 7910), Telegram: @AlexMcMurray, Signal: @AlexMcMurrayEFC.88 Click here to fill in our anonymous form, or email [email protected]. Bear with us if you leave a comment at the bottom of this article: comments are moderated intermittently by human beings. Sometimes these humans might be asleep, or away from their desks, so it may take a while for your comment to appear. You must take sole responsibility for comments you post on this site. We will take reasonable steps to weed out anything that we consider to be offensive or inappropriate. The essential daily roundup of news and analysis read by everyone from senior bankers and traders to new recruits. Boost your career. Find thousands of job opportunities by signing up to eFinancialCareers today.