A petaFLOP on your desk. Now make it earn its keep.
Sparkathians is a field guide to the NVIDIA DGX Spark — how to actually set the thing up, which models fit in 128GB of unified memory without choking, and reviews from people who've run it past the keynote demo.
From box to first inference
The Spark boots into DGX OS, a locked-down Ubuntu with the CUDA stack pre-installed. It's built to run headless on your LAN — the steps that trip people up are the port and boot-order details in the first hour, not the software.
Ethernet before power — always, in that order
If this box is going to live on your LAN as a headless server, plug the Ethernet cable in before you plug in power. Wi-Fi isn't enabled by default and DGX OS starts pulling driver and NIM registry updates the moment it boots — you want it on the wire from the first second, not scrambling to find a network after the fact.
Keep a monitor and keyboard attached until SSH is up
Don't go full headless on day one. Keep a monitor and keyboard connected until you've confirmed you can SSH in — first boot and the initial firmware/OS update take a while, there's no power LED or any other physical indicator of what state it's in, and you'll want eyes on the screen to catch the actual IP address and confirm it's alive before you disconnect.
Check your ports: it's USB-C and Bluetooth only
There's no USB-A on this thing. If your keyboard or mouse isn't USB-C or Bluetooth, bring an adapter or you won't have input for that first setup screen. Same goes for your monitor cable — use the HDMI port specifically, not one of the USB-C ports, and make sure the monitor itself is switched to the matching HDMI input. It's easy to plug everything in correctly and still stare at a blank screen because the monitor's on the wrong source.
Let it finish booting and updating before you touch anything
First boot is slow — DGX OS is doing firmware and system updates in the background and it takes noticeably longer than a normal Ubuntu install. Resist the urge to power-cycle it because nothing seems to be happening. Again: no LED, no beep, nothing — just give it the time.
Once SSH works, this is a headless Ubuntu box — treat it like one
Under the desktop shell it's just Ubuntu/Linux, and honestly this hardware is built to be a headless server, not a desk companion with a GUI open. Once you've confirmed SSH access, disable the desktop UI and free up the resources it was using — you didn't buy a petaFLOP of inference to spend it rendering a taskbar.
Update the base image before installing anything
Run sudo nvidia-dgx-updater once you're in. DGX OS ships with whatever CUDA/driver pair was current at manufacture — on early units that's several point releases behind what NIM containers expect.
Pick your runtime: NIM, Ollama, or bare CUDA
NVIDIA's NIM microservices are the path of least resistance and are tuned for the GB10's unified memory architecture. Ollama works and is far more familiar if you're coming from a laptop setup, but as of now it doesn't fully exploit unified memory bandwidth — expect 10-20% lower throughput than a NIM container running the same weights.
Watch memory, not VRAM
There's no separate GPU VRAM to budget against — CPU and GPU share the same 128GB pool. jetson-stats (or nvidia-smi in newer DGX OS builds) reports combined usage. Leave 12-16GB of headroom for the OS and any concurrent NIM services or the whole session stalls instead of OOM-killing cleanly.
Cluster two units if you need real headroom
Two Sparks linked over the onboard ConnectX-7 (200Gb/s) present as 256GB of combined memory to a properly sharded model — this is the only way to run 405B-class weights locally without dropping to aggressive quantization.
What actually fits in 128GB
Unified memory means the ceiling is generous but shared — the OS, KV cache, and any other running service all draw from the same pool. These are the shapes that behave well on a single unit.
| Model | Precision | Footprint | Notes |
|---|---|---|---|
| Llama 3.1 70B Native fit |
FP8 | ~75 GB | Comfortable single-unit run with room for a long context window. The default "does this thing work" test. |
| Qwen2.5 72B Native fit |
FP8 | ~78 GB | Strong coding performance; noticeably faster prompt processing than Llama 70B at similar quality. |
| DeepSeek-R1-Distill-Llama 70B Native fit |
FP8 | ~76 GB | Reasoning traces eat context fast — budget KV cache accordingly, not just weight size. |
| Mixtral 8x22B Needs quantization |
INT4 | ~85 GB | FP8 doesn't fit alongside a usable context window. INT4 via NIM keeps quality loss modest. |
| Llama 3.1 405B Needs a second unit |
FP4 | ~230 GB | Only realistic path is clustering two Sparks over ConnectX-7. Single-unit INT2 runs technically boot but aren't worth it. |
| Stable Diffusion 3.5 / Flux.1 Native fit |
FP16 | < 20 GB | Image generation barely dents the memory pool — leaves headroom to run an LLM alongside it for agentic pipelines. |
What a 70B-and-up actually gets used for
Don't stop at 70B — it's the floor of what the Spark can comfortably run, not the ceiling. Everything from the 70B tier up through the quantized 8x22B and clustered 405B runs above ends up doing the same kind of work, just with more headroom for context and reasoning depth as you go:
Prompt generation & expansion — turn a rough concept into rich, nuanced positive and negative prompts tuned for diffusion models.
Workflow automation — parse user input inside visual pipelines to dynamically pick LoRAs, sampling steps, CFG scale, and aspect ratio.
Image tagging & captioning — paired with a vision model, generate detailed semantic descriptions, Danbooru-style tags, and dataset annotations for fine-tuning.
Strict JSON extraction & tool-calling — pull schema-compliant JSON out of messy text and reliably invoke functions or local APIs.
Agentic workflows & MCP execution — drive autonomous agents to query local databases, manage filesystems, and run CLI tools without syntax drift.
Private offline RAG — query, synthesize, and cross-reference sensitive local documents, proprietary codebases, or personal archives with zero network exposure.
Deep multi-file refactoring — analyze interconnected modules, diagnose race conditions and subtle logic bugs, and generate full unit test suites.
Local coding assistant backend — power IDE extensions (Continue, local Copilot alternatives) for unbounded inline chat and refactoring.
Multi-step logic & translation — faithful document translation, technical drafting, contract analysis, and synthesis across 32k+ token windows.
Synthetic dataset generation — produce high-quality reasoning traces and paired instruction data to distill smaller 1B–8B models.
Past the keynote demo
Honest takes from people who've had one on their desk for more than a weekend.
The best local-inference box that isn't a rack
It's not going to out-train a cloud cluster and nobody should expect that. What it does is put a 70B model at usable speed on a desk that draws under 240W. That's the whole pitch, and it delivers on it.
Memory bandwidth is the honest bottleneck
128GB of capacity is real, but the bandwidth feeding it isn't discrete-GPU fast. Prompt processing on dense 70B models is noticeably slower than the same weights on an A100, even though both technically "fit." Buy it for capacity, not for raw speed.
Clustering two units changed the calculus entirely
One Spark is a nice dev box. Two, bridged over ConnectX-7, is the first time I've run 405B-class weights outside a datacenter without renting anything. That's the actual headline, and it's buried in the spec sheet.