MLflow✓

MLflow AI Gateway has budget policies. You set a daily, weekly, or monthly spend limit. 👇 When it hits, you can send a webhook to Slack, PagerDuty, or any HTTP alerting and keep serving traffic, or block requests until the window resets. The limit applies to every app that routes through the gateway. Get started: pip install 'mlflow[genai]' 🔗

This week at ODSC AI West, Debu Sinha (@databricks) is running a workshop on Evaluating LLM Applications with MLflow. Teams already run evals. The workshop is the part after: did the model actually get better, which samples regressed, and is the delta statistically significant or just noise. You build that pipeline in open source MLflow, from mlflow.genai.evaluate() through run comparison and a CI gate that fails the build when a regression is real. 🗓️ Wed, Oct 28 🕓 11:00 AM–12:00 PM PT 📍 Hyatt Regency SFO 🔗

This week at ODSC AI West, Samraj Moorjani (@databricks) is giving a 30-min demo on Hidden Gems in Agent Traces. Traces show what the agent did. The talk is about what you do next: find repeating failure modes, auto-detect them across a large volume of traces, and check the fix with evals you can rerun. All in open source MLflow. 🗓️ Tues, Oct 27 🕓 11:05–11:35 AM PT 📍 Hyatt Regency SFO 🔗 Schedule:

Teams keep adding agent harnesses. Claude Code, Codex, Cursor, and the rest each keep their own conventions, so observing a run across them is messy. @2twitme walks through that harness layer, then how @omnigent_ai puts coding agents and custom agents behind one common API. From there, @opentelemetry traces land in MLflow so you can see how the agent actually behaved. Watch the full walkthrough on how to trace and debug AI coding agents with Omnigent and MLflow. 🎥

Part 3 of the Self-Improvement & MLflow series by MLflow Ambassador Jonathan Bown walks through automating the verification layer of a self-improving AI workflow with MLflow and CI/CD. A coding agent reads traces at scale and can propose changes to itself, its skills, tools, memory, or the verifier. CI then checks each change before it is approved automatically or by a person. Read Part 3 👇

Scattered API keys. Different SDKs. No clear view of usage or cost. MLflow AI Gateway provides one @OpenAI-compatible endpoint, encrypted provider keys, and traces for requests, latency, tokens, and cost. How it works 👇 #MLflow #LLMOps

What happens when every team has its own agents in production? You need more than LLMs. You need a control plane. For the third video in the MLflow A–Z series, MLflow Ambassador @shrinathsuresh covers C for Control Plane. He explains how MLflow AI Gateway brings access control, budgets, guardrails, traffic routing, and usage tracking into one place. 🔗 Learn more: #MLflow #GenAI #LLMOps

The Claude SDK makes it fast to build an agent. With native @opentelemetry export and MLflow, it is just as fast to know whether the agent works. 🔹 Trace every model call by pointing a standard OTel exporter at MLflow 🔹 See each run as one trace, with model calls and tool calls under one agent span 🔹 Find expensive runs from token usage on the trace 🔹 Score every run with an LLM judge that inspects the trace 🔹 Compare system prompts with the same test prompts and judge 🔗 #MLflow #Claude #OpenTelemetry

Part 2 of MLflow Ambassador Jonathan Bown’s series on self-improving AI workflows with MLflow is live. He walks a support agent through the loop: traces, feedback, expectations, and judges in; a versioned prompt and an evaluation dataset out; then a production alias move so the agent updates without code changes. @mlflow.test turns that dataset into a quality gate. When the manual pass stops scaling, a coding agent with MLflow skills proposes the themes, scorers, and tests. A human still accepts the changes. CI can run them as MLflow Evaluation runs. 🔗 #MLflow #GenAI #ODSC

Claude Platform now supports native @opentelemetry tracing across its agent ecosystem. Bring traces from the Claude family into one backend using OTLP, the OpenTelemetry Protocol: 🔹 Messages API and Claude SDKs: model requests and tool-runner activity 🔹 Claude Code and Agent SDK: agent steps, tool calls, and subagent activity 🔹 Managed Agents: agent turns, model runs, and platform-run tools We published a guide covers setup OpenTelemetry export across the Claude family, including Claude Platform UI and SDK configuration. 👉 Setup guide: #MLflow #opensource #otel

MLflow AI Gateway now supports TypeSafe System One calls. Create an endpoint for a Jev model, keep the TypeSafe API key on the server, and send structured questions using the endpoint name. You can change the configured Jev model without changing each application that calls it. Gateway tracks usage, supports budgets, and can fall back to a compatible Jev model from another provider, such as OpenRouter, after upstream errors or rate limits. Each endpoint gives you one place to manage access and monitor use for its API. 🔗 Learn more about the TypeSafe Gateway provider: #MLflow #GenAI

Yes/no evals don’t need a general-purpose LLM on every row. That gets slow and expensive at volume. In MLflow 3.17, TypeSafe’s Jev models work as judges. Keep your existing mlflow.genai.evaluate() flow and set model="typesafe:/jev-latest" on make_judge(). Jev returns a structured decision; MLflow stores it as an assessment and puts the probability in metadata, so close calls show up when you review. A versioned model ID keeps comparisons consistent across runs. 🔗 Learn more: #MLflow #GenAI #LLMOps

MLflow 3.17.0 is now available! 🎉 Major new features include: ✅ Jev Decisions: Use TypeSafe’s Jev models as judges in mlflow.genai.evaluate() for yes/no and finite-category checks. Tracing and AI Gateway support are included, so those calls can be inspected and governed in one place. 🔐 Fine-Grained Permissions: Control access to runs, traces, assessments, logged models, review queues, and model, prompt, scorer, and MCP-server versions. Grant an experiment without exposing everything inside it, including with an explicit DENY. 📈 Faster Trace Analytics: SQL-backed dashboards spend less time on trace counts, latency, token usage, costs, and assessments. Optional daily rollups reuse precomputed summaries as datasets grow. Release notes 👉 #MLflow #GenAI #LLMOps

Your AI agents can be autonomous, but your budget shouldn't be. 💰 In the second installment of the MLflow A-Z series, MLflow Ambassador Shrinath Suresh covers B for Budget Policy. Open the MLflow UI, go to AI Gateway, and create a new budget policy. Set a threshold, pick a daily, weekly, or monthly window, and choose whether to raise an alert or reject subsequent requests. 🔗 Docs: #MLflow #GenAI #LLMOps

At the next Agentic + AI Observability meetup in SF, Yuki Watanabe will show how open-source MLflow observability uses agent traces in production. 🔍 Agent traces already capture where your agent gets stuck, which mistakes repeat, and where users get frustrated. This session walks through automatic trace analysis in MLflow to surface those patterns, then turn the findings into improvements you can verify. 🗓️ Oct 13 · 5–8 PM PT · @databricks SF 🔗 View the full agenda & join the waitlist: #MLflow #OpenSource

Clinical RAG can sound confident while citing the wrong passage. MLflow Ambassador @pavan_mantha1 built a claim-level verdict gate using MLflow. One server serves the embedding model, reranker, and claim checker from the Model Registry, and the LLM sits behind MLflow AI Gateway. Each statement is checked against the passage it cites. The gate then releases the answer, drops unsupported claims, or abstains. 🔗 #MLflow #RAG #OpenSource

What does it take for an AI system to improve over time? In this blog, MLflow Ambassador Jonathan Bown breaks down the loop: → Capture signals from traces, feedback, judge verdicts, and test results → Store them in trace logs, evaluation datasets, or a prompt registry → Update a component, then measure the difference 🔗 Read Part 1: #MLflow #OpenSource #AgentOps

Jev faced a harder LLM-judge dataset. ✅ Jev: 64/72 human labels ✅ GPT-OSS-120B: 72/72 ⚡ Median latency: 0.20s vs 1.44s 💰 Estimated Jev cost: $0.020 per 1,000 judgments In retrospective analysis, routing uncertain judgments to a second model corrected the observed misses while escalating <20%. Learn more: #MLflow #LLMOps