At Computex on June 4, 2026, NVIDIA released a model that reshapes the open-weight landscape: Nemotron 3 Ultra. It carries 550 billion total parameters in a Mixture-of-Experts architecture, activates only around 10 percent of them per request, roughly 55 billion active parameters, and ships with a fully open licence covering weights, training data, and training recipes.
According to measurements by Artificial Analysis, it achieves the highest Intelligence Index score of any US-developed open-weight model (47.7). For organisations evaluating local AI strategies, the headline is simple: frontier-quality reasoning, fully deployable on-premise, with no data leaving your network.
Architecture Built for Agentic Workloads
Nemotron 3 Ultra is not simply a scaled-up transformer. It combines two architectural ideas that matter specifically for agent workloads:
Mamba2-Transformer Hybrid LatentMoE: Selective state-space layers (from the Mamba2 family) handle long sequences without the quadratic memory cost of pure attention. Transformer layers handle complex reasoning. The hybrid keeps throughput high even on very long contexts.
Multi-Token Prediction (MTP): Instead of generating one token at a time, the model predicts multiple output tokens per decoding step. For agents that need to produce long chains of reasoning or extended structured outputs, this cuts generation time significantly.
The result, as reported by practitioners and Artificial Analysis: over 300 tokens per second in standard configurations, exceeding 400 output tokens per second in optimised setups on NVIDIA hardware. These figures matter most for long-running agents that chain many steps and accumulate large outputs.
A One-Million-Token Context Window
Nemotron 3 Ultra supports a context window of up to 1 million tokens. In practice, this means an agent can ingest an entire company knowledge base, a full contract archive, or months of email history in a single call, without manual chunking or retrieval pipelines.
This is especially relevant for document-intensive industries: legal, accounting, manufacturing documentation, healthcare records. A 1M-token window shifts the architecture of RAG pipelines: instead of retrieving fragments, you can load entire domains into context.
On-Premise Deployment: Three Paths
NVIDIA published Nemotron 3 Ultra on Hugging Face and supports three primary deployment routes:
- vLLM: The most widely adopted open-source inference server for production. OpenAI-compatible API, horizontal scaling, and solid NVIDIA GPU support from H100 upwards. Existing applications using the OpenAI API can switch to a local Nemotron 3 Ultra endpoint without code changes.
- TensorRT-LLM: NVIDIA's optimised inference library, squeezing maximum throughput from H200 and B200 hardware. Recommended where throughput is the primary constraint.
- Hugging Face Transformers: Lower-barrier path for development, prototyping, and environments without dedicated inference infrastructure.
The model was pre-trained in NVFP4 format, NVIDIA's 4-bit floating-point format, which reduces model weight size while maintaining quality on NVIDIA-native hardware.
Data Sovereignty: The Core Business Case
The case for on-premise deployment of any large model is consistent: requests never leave your infrastructure. With Nemotron 3 Ultra running locally, no employee query, no document content, no business logic reaches an external API provider.
This matters across every regulated industry in Europe. Under our reading of GDPR Article 32, processing sensitive personal data, employee information, client files, financial records, through a cloud API creates a data transfer obligation and a risk profile that local deployment eliminates. The same logic applies to trade secrets and proprietary processes that companies reasonably want to keep off third-party infrastructure.
For a deeper look at how we frame data sovereignty in local AI deployments, see our dedicated resource.
What Hardware Does This Actually Need?
Honest assessment: Nemotron 3 Ultra is not a consumer-GPU or Apple Silicon model. The 550B MoE architecture, even in NVFP4, requires multiple NVIDIA GPUs with high VRAM. A practical minimum for fluid inference is two to four NVIDIA H100 GPUs or a dedicated NVIDIA DGX system.
That said, there are realistic entry points for European SMBs:
- Rented GPU servers in EU data centres: Hourly-billed access to H100 or B200 hardware, physically located in Germany, France, Spain, or the Netherlands. Keeps data inside EU jurisdiction without capital expenditure.
- Existing NVIDIA workstation capacity: Organisations already running NVIDIA high-performance hardware for rendering, simulation, or CAD can repurpose spare GPU time.
- NVIDIA DGX Spark: The compact GB10 system launched in 2026 provides 128 GB of unified memory. Whether smaller Nemotron 3 variants fit within that envelope is something we are actively testing in pilot configurations.
For Mac Studio-based setups, which remain excellent for models up to roughly 70B, Llama 4, Qwen3, and Gemma 4 continue to be the right choices. Nemotron 3 Ultra is the natural next step when reasoning quality and agent throughput requirements grow beyond what Apple Silicon handles today.
Where Nemotron 3 Ultra Fits in the 2026 Open-Weight Field
By June 2026, the strongest open-weight models were predominantly from Chinese AI labs: DeepSeek V4 Pro (released April 2026, 1.6T parameters, MIT licence), MiniMax M3 (1M-token context, native vision), Kimi K2.7 (coding focus). Nemotron 3 Ultra is the first US-developed model to reach their level on the Artificial Analysis Intelligence Index, according to reported measurements.
For European organisations subject to supply chain risk considerations, the distinction matters. A fully open model from a US vendor operates under different export controls and licence conditions than models from Chinese labs. Neither is automatically preferable, but the choice should be explicit and documented, especially for organisations preparing for EU AI Act compliance.
The GPAI Code of Practice, finalised in mid-2026 and signed by major providers, does not directly govern open-weight deployers. However, it establishes a reference baseline for transparency and documentation practices that internally deployed models benefit from matching.
Practical Starting Points
If your organisation is planning agentic AI deployments, document analysis automation, intelligent workflow routing, natural-language ERP queries, Nemotron 3 Ultra deserves a place in your infrastructure evaluation. Key questions to answer in a pilot:
- Throughput requirement: How many concurrent users or automated tasks will hit the model simultaneously?
- Context depth: Do your use cases require processing documents longer than 128K tokens?
- Hardware fit: Do you have existing NVIDIA infrastructure, or are you evaluating NVIDIA DGX or rented GPU servers?
Freshlab assists SMBs across Europe in scoping and running local AI infrastructure, from initial hardware evaluation to production deployment and EU AI Act documentation. Our kAIra Tools covers Mac-based setups for smaller model workloads; for NVIDIA-based agent infrastructure, we structure a pilot project to validate the setup against your specific workflows.
Contact us to start the conversation.