Two DGX Spark Units as a Local LLM Cluster: Full Setup Guide

28. Jun 2026 English 6 min read Also in: Deutsch, Español

local-llm nvidia dgx-spark vllm hardware

A single desktop box that runs a 400-billion-parameter model entirely on your own network, with no cloud API in the loop, was a server-room fantasy until recently. In late June 2026, a practitioner in the local AI community shared a complete recipe on X for turning two NVIDIA DGX Spark units into exactly that: a frontier-scale local LLM cluster you can put under a desk.

We took that recipe, cross-checked every step against NVIDIA's official DGX Spark playbooks, and turned it into the installation guide below. For European SMBs evaluating local AI, this is the most compact path to genuinely large-model inference that never leaves the building.

What Is the DGX Spark, and Why Two of Them?

The NVIDIA DGX Spark is a desktop AI machine built on the GB10 Grace Blackwell Superchip. The key numbers:

Spec	Single DGX Spark	Two-Unit Cluster
Unified memory	128 GB LPDDR5x	256 GB
CPU	20-core Arm (10x Cortex-X925 + 10x Cortex-A725)	40 cores combined
AI performance	up to 1 PFLOP FP4 (with sparsity)	up to 2 PFLOP FP4
Storage	up to 4 TB NVMe	up to 8 TB
Networking	ConnectX-7, 2x 200GbE (QSFP)	direct 200GbE peer link
Max practical model	~70B class	~405B class

A single unit is already a capable local inference machine. The reason to pair two is memory: 256 GB of pooled unified memory is the threshold where genuinely large, frontier-class open-weight models become serviceable on hardware that draws less power than a gaming PC. The two ConnectX-7 NICs make the pairing trivial, no switch required.

Bill of Materials

2x NVIDIA DGX Spark (128 GB each)
1x 200G QSFP56 passive DAC cable, NVIDIA part Q56-200G-CU0-5 or a DGX-Spark-validated equivalent
(Optional, for 4 to 8 units later) a MikroTik CRS504 200GbE switch

That is the entire shopping list for a two-unit cluster.

Step 1: Physical Connection

Plug one end of the QSFP cable into a ConnectX-7 port on the first unit, and the other end into the same port on the second unit (both left ports, or both right ports, viewed from the rear). One cable gives you full bandwidth. A second cable is only useful if you assign IPs to all four interfaces.

Step 2: Identify the Interface

On each unit, find the ConnectX-7 interface that is up:

ibdev2netdev

On the DGX Spark the two high-speed interfaces appear as enp1s0f0np0 and enp1s0f1np1. Note which one shows state Up on your cabled port, you will reference it in the network and NCCL steps.

Step 3: Network Configuration (netplan)

You have two options. Link-local is the fastest for a single-cable setup; static IPs are required if you cable all four interfaces for maximum bandwidth.

Option A, automatic link-local. Create /etc/netplan/40-cx7.yaml on both units:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0f0np0:
      dhcp4: no
      link-local: [ ipv4 ]
    enp1s0f1np1:
      dhcp4: no
      link-local: [ ipv4 ]

This assigns IPv4 link-local addresses in the 169.254.x.x range automatically.

Option B, manual static IPs. On node 1, /etc/netplan/40-cx7.yaml:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0f0np0:
      dhcp4: no
      addresses: [192.168.100.10/24]

On node 2, the same file with 192.168.100.11/24. Apply on both:

sudo netplan apply

Confirm the link with a ping across the new interface before moving on.

Step 4: Passwordless SSH

Distributed inference needs key-based SSH between the units. Use the same username on both. Generate a dedicated key and copy it to the other node:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_shared -N ""
ssh-copy-id -i ~/.ssh/id_ed25519_shared.pub <user>@192.168.100.11

NVIDIA's discover-sparks.sh helper automates this: it locates other Sparks on the link with avahi-browse -r -t ssh.tcp, distributes the shared key, and writes the ~/.ssh/config entries for you. Verify that ssh 192.168.100.11 connects with no password prompt.

Step 5: Verify NCCL Across the Link

Before running a real model, confirm that NCCL can move data across the 200GbE link. Set the interface variables (replace with your Up interface) and run an all-gather benchmark from the nccl-tests suite:

export NCCL_SOCKET_IFNAME=enp1s0f1np1
export UCX_NET_DEVICES=enp1s0f1np1
export OMPI_MCA_btl_tcp_if_include=enp1s0f1np1

mpirun -np 2 -H 192.168.100.10,192.168.100.11 \
  all_gather_perf -b 8 -e 256M -f 2 -g 1

A healthy run reports bus bandwidth climbing toward the link rate as message size grows. If bandwidth is low, you almost certainly have the wrong NCCLSOCKETIFNAME.

Step 6: Distributed Inference with vLLM and Ray

The common serving stack is vLLM with a Ray backend. Start Ray on the head node, then join the worker:

# On node 1 (head):
ray start --head --port=6379

# On node 2 (worker):
ray start --address=192.168.100.10:6379

Then launch vLLM on the head node with tensor parallelism across both units:

vllm serve Qwen/Qwen3.6-35B-A3B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray \
  --max-model-len 8192 \
  --reasoning-parser qwen3 \
  --gpu-memory-utilization 0.7

For a larger model that needs the full 256 GB, raise --max-model-len and tune --gpu-memory-utilization upward (0.85 is a common ceiling). The source guide also uses --max-num-seqs 4 to bound concurrency on memory-heavy models. The OpenAI-compatible endpoint is then live on port 8000.

Recommended Models

The following picks come from the original community guide on X and are community-reported. Treat throughput figures as indicative; they vary by quantisation and prompt length.

Single DGX Spark (128 GB):

Qwen 3.6 35B (NVFP4), up to 256K context, roughly 110 tokens/sec
DeepSeek V4 Flash (REAP)
Nemotron Super (NVIDIA-optimised)

Two-unit cluster (256 GB):

DeepSeek V4 Flash, up to 1M context, roughly 40 to 45 tokens/sec in a single session
Step-3.7-Flash with image input support

Scaling Beyond Two Units

The single-cable, no-switch approach is specific to the two-unit case. For four to eight units, a MikroTik CRS504 200GbE switch is the documented option, with minimal bandwidth penalty. NVIDIA publishes separate playbooks for three-Spark and switched multi-Spark topologies (linked below).

Why This Matters for EU SMBs

A local DGX Spark cluster removes an entire category of compliance risk. Inference happens on your hardware, so:

No GDPR Article 44 third-country transfer obligations, no data leaves the building
Full auditability under EU AI Act deployer obligations (Article 26), every query can be logged and retained internally
No vendor training on your inputs, no cloud retention you cannot control

Under our reading of the EU AI Act, on-premise deployment of general-purpose models materially reduces the documentation burden compared to cloud API integrations. This is not legal advice; verify your specific classification under Annex III.

Freshlab approaches this as a partner, not a supplier, with more than 25 years rooted in pharmaceutical contract manufacturing and ERP projects, and German-speaking roots in Bavaria. We know what a regulated, document-heavy environment demands from local AI.

The Bottom Line

Two DGX Spark units, one QSFP cable, and the six steps above give you a 256 GB local inference cluster capable of serving frontier-scale open-weight models, with nothing leaving your network. For a regulated European SMB, that combination of capability and data sovereignty is hard to match with any cloud offering.

Start a pilot project →, Freshlab installs and configures local AI infrastructure, including DGX Spark clustering and model deployment, for EU-based SMBs.

Sources

Exact interface names, IP ranges, and model availability may differ on your firmware revision. For installation, follow the linked official NVIDIA playbooks as the authoritative reference and use this guide as the orientation map.

Frequently asked questions

How many DGX Spark units do I need to run a frontier-scale model locally?

Two. A single DGX Spark has 128 GB of unified memory, which fits mid-size models comfortably. Two units connected over their ConnectX-7 200GbE ports pool 256 GB, which is enough to serve models up to roughly 405 billion parameters, depending on quantisation.

Do I need a network switch to connect two DGX Spark units?

No. For a two-unit cluster a single 200G QSFP56 cable directly between the two ConnectX-7 ports is sufficient and delivers full bandwidth. A switch (for example a MikroTik CRS504) is only needed when you scale to four or more units.

Which inference stack runs across both units?

vLLM with a Ray backend is the common choice. NCCL handles the GPU-accelerated collective operations across the 200GbE link, and vLLM is started with a tensor-parallel size of 2 and the Ray distributed executor.

Is a local DGX Spark cluster compliant with the EU AI Act and GDPR?

Local inference keeps all data on your own hardware, so there is no third-country transfer under GDPR Article 44 and full auditability under EU AI Act deployer obligations. This is a major reason European SMBs choose on-premise over cloud APIs. It is not legal advice; verify your specific classification.

Can Freshlab set this up for us?

Yes. Freshlab installs and configures local AI infrastructure for EU-based SMBs, including DGX Spark clustering, model deployment, and integration with your existing systems. A pilot project covers the initial scoping at no cost.