Hardware for a Self-Hosted AI Security Stack: Guard Models, Scanners and Red-Team Tools

By LLM Hardware team · Methodology ·

Key facts

Checked
2026-10-30
Author
LLM Hardware team

Checked 2026-10-30. Author: LLM Hardware team. Amazon prices are not stored or shown on this site.

Citeable facts

Key fact: An 8B guard model and a 14B chat model, both at Q4 in one Ollama server, need about 15.2 GB.

How we got it: 8 x 0.5 x 1.2 + 14 x 0.5 x 1.2 + 2 = 4.8 + 8.4 + 2, with the 2 GB reserve counted once (methodology). Parameter counts: Llama Guard 3 8B and Granite Guardian 4.1 8B model cards.

Checked:

Key fact: gpt-oss-safeguard-20b is about 14.6 GB by the site formula; OpenAI says it fits into GPUs with 16GB of VRAM.

How we got it: 21B total parameters (model card) x 0.5 x 1.2 + 2. The weights ship in MXFP4, a 4-bit format, so the Q4 row is the closest. Hugging Face's file listing says 22B, which gives 15.2 GB.

Checked:

Key fact: garak's TAP probe defaults to a local 8B attacker model (mistralai/Ministral-8B-Instruct-2410), about 21.2 GB at 16-bit by the formula.

How we got it: tap.py class defaults read 2026-10-30 (attack_model_type huggingface.Pipeline). Ministral card: 8,019,808,256 parameters, BF16. 8.02 x 2.0 x 1.2 + 2.

Checked:

Methodology explains the VRAM formula, the bytes per weight at each quantization, and the full source list.

A self-hosted AI security stack is mostly extra models. A prompt classifier reads each request before your chat model does, a guard model reads the answer before your user does, and a red-team tool drives an attacker model against the whole pipeline. Every one of them takes memory on the same GPUs and Macs this site sizes. This guide takes parameter counts from the official model cards, runs them through the site VRAM formula, and matches the combinations to hardware in our catalog. It does not rank which guard catches more attacks. For that side, the AI Defense Stack comparison of open-source and managed LLM guardrails lists what each tool screens, with sources.

Three jobs, three memory profiles

Screening classifiers are small encoder models that flag a prompt as an injection or a jailbreak. Meta's Llama Prompt Guard 2 comes in 86M and 22M backbone sizes, and the prompt injection scanner in LLM Guard loads ProtectAI's deberta-v3-base-prompt-injection-v2. Next to a chat model their weights are close to nothing. The decision is whether they need the GPU at all.

Guard models are generative LLMs trained to judge a conversation. Llama Guard 3 answers safe or unsafe and names the violated category, and Llama Guard 4 adds images to the prompts it can classify. IBM's Granite Guardian returns yes or no for a criterion you choose. Qwen3Guard sorts text into safe, controversial and unsafe. OpenAI's gpt-oss-safeguard classifies text against a written policy you supply. Size these like any chat model, because that is what they are underneath. AI Defense Stack's page on Meta's Llama protections covers the rest of Meta's suite, including LlamaFirewall and CodeShield.

Red-team tools such as garak and PyRIT are Python programs, and garak installs with pip. Their own footprint is small. The memory goes to the target model if you run it locally, to any attacker model that writes the attack prompts, and to any model that scores the replies.

Parameter counts and memory, model by model

The formula is parameters in billions × bytes per parameter × 1.2, plus 2 GB. Bytes per parameter are 2.0 at 16-bit, 1.0 at Q8 and 0.5 at Q4_K_M. Parameter counts come from each model card or its Hugging Face file listing, read on 30 October 2026. The Q4 column is the formula at 4-bit; check that a 4-bit build of the model exists for your runtime before you plan around it.

Model Parameters (source) Weights as published Formula, as published Formula, Q4
Llama Prompt Guard 2 22M 22M backbone; Hugging Face counts 70.8M F32 2.3 GB n/a
Llama Prompt Guard 2 86M 86M backbone; Hugging Face counts 0.3B F32 3.4 GB n/a
deberta-v3-base-prompt-injection-v2 (LLM Guard) Hugging Face counts 0.2B F32 3.0 GB n/a
Llama Guard 3 1B 1,123 million (pruned) BF16 4.7 GB 2.7 GB (Q8: 3.3 GB)
Qwen3Guard-Gen 0.6B / 4B / 8B 0.6B, 4B, 8B BF16 3.4 / 11.6 / 21.2 GB 2.4 / 4.4 / 6.8 GB
Llama Guard 3 8B 8B (fine-tuned Llama 3.1 8B) BF16 21.2 GB 6.8 GB
Granite Guardian 4.1 8B 8B BF16 21.2 GB 6.8 GB
Llama Guard 4 12B 12B, dense BF16 30.8 GB 9.2 GB
gpt-oss-safeguard-20b 21B total, 3.6B active MXFP4 (4-bit) 14.6 GB same row
gpt-oss-safeguard-120b 117B total, 5.1B active MXFP4 (4-bit) 72.2 GB same row

Three notes on the table. First, the Prompt Guard 2 and DeBERTa files ship in F32, which is 4 bytes per parameter. The methodology table stops at FP16, so those three rows use 4 bytes, and the 2 GB reserve is most of each result. The Prompt Guard cards quote backbone sizes while Hugging Face counts every tensor, and the table sizes from the larger count. Second, the gpt-oss-safeguard weights ship in MXFP4, a 4-bit format, so the Q4 row is the closest fit. The 14.6 GB result for the 20b sits under OpenAI's statement that it "fits into GPUs with 16GB of VRAM", and Hugging Face's listing of 22B parameters would give 15.2 GB, still under. OpenAI says the 120b "fits into a single H100 GPU". Third, both gpt-oss-safeguard models are mixture-of-experts, and the formula uses total parameters, not the active 3.6B and 5.1B, because every expert still has to be loaded.

Judging long conversations adds cache

A guard model reads the whole exchange it judges, so its context grows with your chat. The 1.2 factor covers about 4K to 8K tokens. Llama Guard 3 8B is fine-tuned from Llama 3.1 8B, which Meta's Llama 3 paper lists with 32 layers and 8 key-value heads, the same layout as the methodology page's worked example. That is about 131 KB of f16 cache per token, so judging a 32K-token transcript adds about 4.3 GB to the 6.8 GB weights estimate. Ollama lists a 128K window on its llama-guard3 tags. If you only send the latest turn, the cache stays small. If you send whole transcripts, set the context in the VRAM calculator or read the KV cache quantization guide.

Streaming checks stay loaded

Qwen3Guard also ships a Stream variant with a token-level classification head for real-time monitoring while text is generated. Its 4B card lists 4B parameters in BF16, which is 11.6 GB at 16-bit by the formula, and it has to stay resident for as long as the chat model is generating. The card says it suits chat models that share Qwen3's tokenizer, and it loads through Transformers 4.55.0 or later.

Five setups and the memory they need

When the guard and the chat model share one Ollama or llama.cpp server, this guide adds each model's weight term and counts the 2 GB reserve once. A classifier in its own Python process is a second engine, so it gets its own reserve. GPUs count at full card size. Macs lose about 4 GB to macOS, as in our hardware catalog.

Setup Arithmetic Total What fits it
1. Prompt Guard 2 86M (F32, own process) beside an 8B chat model at Q4 3.4 + 6.8 10.2 GB 12 GB GPU, 1.8 GB spare
2. Llama Guard 3 1B at Q8 beside a 14B chat model at Q4, one server 1.3 + 8.4 + 2 11.7 GB 16 GB GPU, 4.3 GB spare. A 12 GB card leaves 0.3 GB.
3. An 8B guard at Q4 (Llama Guard 3 8B or Granite Guardian 4.1 8B) beside a 14B chat model at Q4, one server 4.8 + 8.4 + 2 15.2 GB 24 GB GPU, 8.8 GB spare, or a 24 GB Mac, 4.8 GB spare. A 16 GB card leaves 0.8 GB.
4. gpt-oss-safeguard-20b beside an 8B chat model at Q4, one server 12.6 + 4.8 + 2 19.4 GB 24 GB GPU, 4.6 GB spare. A 24 GB Mac leaves 0.6 GB. Alone, the 20b is 14.6 GB and fits a 16 GB card.
5. gpt-oss-safeguard-120b beside a 27B chat model at Q4, one server 70.2 + 16.2 + 2 88.4 GB 128 GB of unified memory

Llama Guard 4 is the awkward size. Its card loads the weights in bfloat16, which comes to 30.8 GB by the formula and leaves 1.2 GB on a 32 GB card such as the RTX 5090, too little for a long conversation. A 128 GB machine holds it with room for a chat model. A 4-bit build would be about 9.2 GB, if your runtime supports one for this model.

Red-team hardware: count the attacker, not the scanner

garak's README lists a huggingface target type "for transformers models to run locally" next to hosted APIs and a REST generator. PyRIT's documentation lists Hugging Face targets, custom HTTP endpoints, and scorers "powered by LLMs". So the same tool can need no GPU, when every model it calls is remote, or room for two local models at once.

A probe can also load a model of its own. As of 30 October 2026, the source of garak's TAP probe sets the attacker to mistralai/Ministral-8B-Instruct-2410, loaded through garak's Hugging Face pipeline generator, and the evaluator to an OpenAI API model. The Ministral card lists 8,019,808,256 parameters in BF16. Loaded at that precision the formula gives 21.2 GB. Add an 8B target at Q4 in its own Ollama server and the box needs about 28.0 GB, which is past a 24 GB card and leaves 4.0 GB on a 32 GB one. Serving the target from Ollama or llama.cpp and pointing the tool at its HTTP endpoint keeps the target at Q4 sizes. The garak listing on AI Defense Stack covers what its probes look for.

LLM Guard: small hardware, upkeep is yours

The LLM Guard repository was archived by its owner on 9 July 2026, and the README now describes both the code and its Hugging Face models as unmaintained. The LLM Guard page on AI Defense Stack tracks that status. The memory math does not change: its prompt injection scanner is the 0.2B DeBERTa row above, and its documentation recommends ONNX Runtime for CPU inference. What changes is who ships fixes and model updates, which is now you.

Which machine for which setup

NVIDIA GeForce RTX 5070 (12 GB)

The 12 GB card for setup 1: an 8B chat model plus a Prompt Guard 2 classifier at 10.2 GB, with 1.8 GB spare. It is too tight for an 8B guard beside an 8B chat model (11.6 GB in one server leaves 0.4 GB), and gpt-oss-safeguard-20b at 14.6 GB does not fit.

ASUS TUF RTX 5070 12 GB on Amazon

ASIN B0DS6S98ZF. Amazon Associates link. The live price is on that page, not locked in here.

NVIDIA RTX 5070 Ti (16 GB)

Covers setup 2 (Llama Guard 3 1B beside a 14B chat model, 11.7 GB) and gpt-oss-safeguard-20b on its own at 14.6 GB, the class of card OpenAI names for the 20b. It is not enough for an 8B guard beside a 14B chat model, where 15.2 GB leaves 0.8 GB.

NVIDIA GeForce RTX 3090 (24 GB)

The 24 GB tier holds setups 3 and 4 with room: an 8B guard and a 14B chat model at 15.2 GB, or gpt-oss-safeguard-20b beside an 8B chat model at 19.4 GB. This listing is the Founders Edition of an Ampere-generation card, so check the seller and condition on the listing.

RTX 3090 Founders Edition on Amazon

ASIN B08HR6ZBYJ. Amazon Associates link. The live price is on that page, not locked in here.

Apple Mac mini (M5 Pro, 24 GB)

About 20 GB is usable after the macOS reserve. An 8B guard plus a 14B chat model (15.2 GB) fits with 4.8 GB to spare. gpt-oss-safeguard-20b plus an 8B chat model (19.4 GB) leaves 0.6 GB, which is too tight, so on this Mac run the 20b alone.

Mac mini M5 Pro 24 GB on Amazon

ASIN B0HGGHNQY6. Amazon Associates link. The live price is on that page, not locked in here.

GMKtec EVO-X2 (Ryzen AI Max+ 395, 128 GB)

A 128 GB unified-memory mini PC. By the formula it holds setup 5, gpt-oss-safeguard-120b beside a 27B chat model at 88.4 GB, and Llama Guard 4 at bfloat16 with room left over. It is an AMD machine with no CUDA, so confirm that the serving stack you plan to use runs these models on it before you buy. The other 128 GB options in our catalog are the NVIDIA DGX Spark, listed as unavailable at our last check, and the Mac Studio M5 Max with 128 GB, which is configured at Apple.

GMKtec EVO-X2 128 GB on Amazon

ASIN B0F53MLYQ6. Amazon Associates link. The live price is on that page, not locked in here.

What this guide does not tell you

No speeds and no detection rates. This site does not run hardware or guard models, and every number above comes from a model card, a vendor page, Meta's Llama 3 paper or the site formula. Before you rely on a guard, run it over a labeled sample of your own traffic and time it on the machine you plan to use.

Sources

Questions

Can a prompt classifier run without a GPU?

LLM Guard's documentation recommends ONNX Runtime for CPU inference and publishes timings for cloud instances with and without a GPU. Meta's Prompt Guard 2 cards give latency on an A100 and say nothing about CPU. At 0.2B to 0.3B parameters the weights are about 1 GB even in F32, so memory is not the limit; whether a CPU is fast enough for your traffic is something to time on your own machine.

Do I need a second GPU for a guard model?

Not for the pairings in this guide up to 24 GB. An 8B guard and a 14B chat model at Q4 come to about 15.2 GB in one server, which leaves 8.8 GB on a 24 GB card. A second machine starts to make sense for the large judges: Llama Guard 4 at 16-bit is about 30.8 GB and gpt-oss-safeguard-120b about 72.2 GB.

Why is the 2 GB reserve counted once for two models?

The methodology's 2 GB covers the operating system and the inference engine. Two models in one Ollama or llama.cpp server share one engine, so this guide adds each model's weight term and then the reserve once. A classifier in its own Python process beside Ollama is a second engine, so those two are added as full formula results.

A correction or a hardware question goes to the contact page.