GitHub - ModelCloud/GPTQModel: LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang. · GitHub

main

Go to file

Code

NameName

Last commit message

Last commit date

Latest commit History 3,261 Commits
.github		.github
docs		docs
format		format
gptqmodel		gptqmodel
gptqmodel_ext		gptqmodel_ext
hw		hw
licenses		licenses
scripts		scripts
tests		tests
.gitignore		.gitignore
CREDITS.md		CREDITS.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
setup.py		setup.py
View all files

Repository files navigation

GPT-QModel

LLM model quantization (compression) toolkit with hw acceleration support for NVIDIA CUDA, AMD ROCm, Huawei Ascend NPU, Intel XPU, and Intel/AMD/Apple CPUs via HF, vLLM, and SGLang.

Latest News

05/21/2026 7.1.0-dev main: ✨ Added nemotron_labs_diffusion model support
05/20/2026 7.1.0-dev main: ✨ Added interns1, ovis2_5, ovis2_6_moe and ovis2_6_next model support
05/15/2026 7.1.0-dev main: ✨ Added mimo_v2 model support
05/13/2026 7.1.0-dev main: ✨ Added minicpmv_4_6 and DeepSeek V4 model support
05/07/2026 7.1.0-dev main: ✨ Added GLM-4.5V, GLM-4.6V, Zamba and Zamba2 model support
04/29/2026 7.1.0-dev main: ✨ Added PoolSideAI Laguna model support for fused Laguna MoE checkpoints. Added ERNIE 4.5 VL MoE, Ling-2.6-flash and NVIDIA Nemotron 3 Nano Omni model support.
04/28/2026 7.0.0: 🚀 Added Huawei Ascend NPU support through native torch kernels for GPTQ, AWQ, ParoQuant, GGUF, QQQ, and EXL3. Added internvl_chat, gemma3n, GLM-OCR, GLM-ASR, and falcon_mamba model support.
04/16/2026 6.1.0: 🚀🔥⚡ CUDA kernels are now fully JIT-compiled, shrinking the wheel by about 300x and building only what you use; Marlin now supports NVIDIA Turing+ GPUs, Machete kernel validation now covers supported GPUs, GLM 5/5.1 joins the lineup, and LazyTurtle plus AWQ / multi-GPU MoE fixes make large-model quantization easier, lighter, and smoother.
04/03/2026 6.0.3: 🎉 New quantization methods: ParoQuant, GGUF, FP8, EXL3, and FOEM: First-Order Error Matters. Added PrismML/Bonsai 1bit model quantization (inference only), faster ParoQuant/AWQ kernels, ParoQuant optimization scope control: module (Paro Lite) or layer (Paro reference), plus Gemma4, MiniCPM-O, MiniCPM-V, and GLM4 MoE Lite model support.
03/19/2026 5.8.0: ✨HF Transformers 5.3.0 support with auto-defusing of fused models via pypi pkg: Defuser. Qwen 3.5 family support added. New fast HF cpu kernels for GPTQ/AWQ added. Experimental INT8 cpu kernel added for GPTQ.

Archived News * 02/09/2026 [5.7.0](https://github.com/ModelCloud/GPTQModel/releases/tag/v5.7.0): ✨New `MoE.Routing` config with `Bypass` and `Override` options to allow multiple brute-force MoE routing controls for higher quality quantization of MoE experts. Combined with `FailSafeStrategy`, GPT-QModel now has three separate control settings for efficient MoE expert quantization. `AWQ` `qcfg.zero_point` property has been merged with a unified `sym` symmetry property; `zero_point=True` is now `sym=False`. Fixed `AWQ` `sym=True` packing/inference and quantization compatibility with some Qwen3 models. Exaone 4.0 support.

Special Notes:

PrismAI/Bonsai inference sample script. GPT-QModel loads Prism/Bonsai GGUF checkpoints through its native GGUF loading path and internal GGUF runtime shim. No external gguf PyPI package is required.

• from gptqmodel import GPTQModel

  model = GPTQModel.load("prism-ml/Bonsai-1.7B-gguf")
  # or: model = GPTQModel.load("prism-ml/Bonsai-1.7B-gguf", profile="low_memory")

  tokens = model.generate(
      "Who wrote Romeo and Juliet?",
      max_new_tokens=128,
  )[0]

  print(model.tokenizer.decode(tokens, skip_special_tokens=True))

What is GPT-QModel?

GPT-QModel is a production-ready LLM model compression/quantization toolkit with hw-accelerated inference support for both CPU/GPU via HF Transformers, vLLM, and SGLang.

GPT-QModel currently supports GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, GPTAQ, EoRa, GAR and FOEM, with more quantization methods and enhancements planned.

Quantization Support

GPT-QModel is a modular design supporting multiple quantization methods and feature extensions.

Feature GPT-QModel Transformers vLLM SGLang Lora Training

GPTQ	✅	✅	✅	✅	✅
AWQ	✅	✅	✅	✅	✅
ParoQuant	✅	x	x	x	✅
GGUF	✅	x	x	x	x
FP8	✅	x	x	x	x
Exllama V3 / EXL3	✅	x	x	x	x
EoRA	✅	✅	✅	✅	x
Group Aware Act Reordering	✅	✅	✅	✅	✅
QQQ	✅	x	x	x	x
Rotation	✅	x	x	x	x
GPTAQ	✅	✅	✅	✅	✅
FOEM	✅	✅	✅	✅	✅

GGUF, FP8, EXL3, and ParoQuant are currently native GPT-QModel quantization/runtime paths. vLLM and SGLang integration currently targets GPTQ and AWQ.

Quant Method / Format / Backend Matrix

Canonical backend names are shown below. Method-specific aliases are only accepted where explicitly implemented by that quant method.

Quant Method Formats Backends / Kernels

METHOD.GPTQ	FORMAT.GPTQ, FORMAT.GPTQ_V2, FORMAT.MARLIN, FORMAT.BITBLAS	FORMAT.GPTQ: BACKEND.GPTQ_TORCH_ATEN, BACKEND.GPTQ_MACHETE, BACKEND.GPTQ_MARLIN, BACKEND.GPTQ_EXLLAMA_V2, BACKEND.GPTQ_TORCH_FUSED, BACKEND.GPTQ_TRITON, BACKEND.GPTQ_BITBLAS, BACKEND.GPTQ_TORCH, BACKEND.GPTQ_TORCH_INT8 FORMAT.GPTQ_V2: BACKEND.GPTQ_TORCH_ATEN, BACKEND.GPTQ_EXLLAMA_V2, BACKEND.GPTQ_TORCH_FUSED, BACKEND.GPTQ_TRITON, BACKEND.GPTQ_BITBLAS, BACKEND.GPTQ_TORCH, BACKEND.GPTQ_TORCH_INT8 FORMAT.MARLIN: BACKEND.GPTQ_MARLIN FORMAT.BITBLAS: BACKEND.GPTQ_BITBLAS
METHOD.AWQ	FORMAT.GEMM, FORMAT.GEMV, FORMAT.GEMV_FAST, FORMAT.LLM_AWQ, FORMAT.MARLIN, FORMAT.BITBLAS	FORMAT.GEMM: BACKEND.AWQ_TORCH_ATEN, BACKEND.AWQ_MACHETE, BACKEND.AWQ_MARLIN, BACKEND.AWQ_EXLLAMA_V2, BACKEND.AWQ_GEMM, BACKEND.AWQ_GEMM_TRITON, BACKEND.AWQ_TORCH_FUSED, BACKEND.AWQ_TORCH, BACKEND.AWQ_TORCH_INT8, BACKEND.AWQ_BITBLAS FORMAT.GEMV: BACKEND.AWQ_GEMV FORMAT.GEMV_FAST: BACKEND.AWQ_GEMV_FAST FORMAT.LLM_AWQ: BACKEND.AWQ_GEMV_FAST FORMAT.MARLIN: BACKEND.AWQ_MACHETE, BACKEND.AWQ_MARLIN FORMAT.BITBLAS: BACKEND.AWQ_BITBLAS
METHOD.PARO	FORMAT.PAROQUANT	BACKEND.PAROQUANT_CUDA, BACKEND.PAROQUANT_TRITON
METHOD.QQQ	FORMAT.QQQ	BACKEND.QQQ, BACKEND.QQQ_TORCH
METHOD.GGUF	FORMAT.GGUF	BACKEND.GGUF_TRITON, BACKEND.GGUF_CPP_CUDA, BACKEND.GGUF_CPP_CPU, BACKEND.GGUF_TORCH
METHOD.FP8	FORMAT.FP8	BACKEND.FP8_TORCH
METHOD.BITSANDBYTES	FORMAT.BITSANDBYTES	BACKEND.BITSANDBYTES
METHOD.EXL3	FORMAT.EXL3	BACKEND.EXL3_EXLLAMA_V3, BACKEND.EXL3_TORCH

BACKEND.VLLM, BACKEND.SGLANG, and BACKEND.MLX are external runtime backends and are not part of the native kernel matrix above.

Marlin uses GPTQMODEL_MARLIN_USE_FP32 (default: enabled) to control fp32 accumulation.

Features

✨ Native integration with HF Transformers, Optimum, and Peft
🚀 vLLM and SGLang inference integration for quantized models with format = FORMAT.[GPTQ/AWQ]
✨ GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, GPTAQ, and FOEM quantization support.
✨ Prism Bonsai Q1_0_g128 GGUF checkpoints can be loaded for post-quantized inference through the normal model_id_or_path argument. GPT-QModel normalizes the GGUF artifact internally for HF Transformers via its native GGUF runtime, and does not support Prism Bonsai quantization or export.
🚀 Quantize MoE models with ease even with extreme routing activation bias via Moe.Routing and/or FailSafe.
🚀 Data Parallelism for 80%+ quantization speed reduction with Multi-GPU.
🚀 Optimized for Python >= 3.13t (free threading) with lock-free threading.
✨ Linux, macOS, Windows platform support for CUDA (NVIDIA), NPU (Huawei Ascend), XPU (Intel), ROCm (AMD), MPS (Apple Silicon), CPU (Intel/AMD/Apple Silicon).
✨ Dynamic per-module mixed quantization control: each layer/module can have a unique quantization config or be excluded from quantization.
🚀 Intel Torch 2.8 fused kernel support for XPU [Arc + Datacenter Max] and CPU [avx, amx].
🚀 Python 3.13.3t (free-threading, GIL disabled) support for multi-GPU accelerated quantization for MoE models and multi-core CPU boost for packing.
✨ Asymmetric Sym=False support.
✨ lm_head module quant inference support for further VRAM reduction.
🚀 Microsoft/BITBLAS optimized tile based inference.
💯 CI unit-test coverage for all supported models and kernels including post-quantization quality regression.

Who's Using GPT-QModel?

Selected public references where teams or companies explicitly mention GPT-QModel in documentation, integration notes, or quantized model usage. This is not an exhaustive customer list.

Hugging Face
Intel
NVIDIA
Alibaba Cloud

Quality: GPTQ 4bit can match native BF16:

🤗 ModelCloud quantized Vortex models on HF

Model Support

Model

Apertus	✅	EXAONE 3/4	✅	Dots1	✅	Mistral3	✅	Qwen 2/3/3.5 (Next/MoE)	✅
Baichuan	✅	Falcon (H1 / Mamba)	✅	InternLM 1/2/2.5	✅	Mixtral	✅	Qwen 2/2.5/3 VL	✅
Bloom	✅	FastVLM	✅	Kimi K2	✅	MobileLLM	✅	Qwen 2.5/3 Omni	✅
ChatGLM	✅	Gemma 1-4 / 3n	✅	Klear	✅	MOSS	✅	RefinedWeb	✅
CodeGen	✅	GPTBigCode	✅	LING/RING	✅	MPT	✅	StableLM	✅
Cohere 1-2	✅	GPT-Neo / NeoX	✅	Llama 1-3.3	✅	Nemotron H / Omni	✅	StarCoder2	✅
DBRX Converted	✅	GPT-2	✅	Llama 3.2 VL	✅	Nemotron Ultra / Labs-Diffusion	✅	TeleChat2	✅
Deci	✅	GPT-J	✅	Llama 4	✅	OPT	✅	Trinity	✅
DeepSeek-V2/V3/V4/R1	✅	GPT-OSS	✅	LongCat Flash	✅	OLMo2 / LLaDA2	✅	Yi	✅
DeepSeek-V2-Lite	✅	Granite / Granite MoE	✅	LongLLaMA	✅	Ovis 1.6/2/2.5/2.6 MoE/2.6 Next	✅	Seed-OSS	✅
Dream	✅	GRIN-MoE	✅	Instella	✅	Phi 1-4	✅	Voxtral	✅
ERNIE 4.5 / MoE / VL MoE	✅	GLM 4/4V/4.5V/4.6V/5/5.1/OCR/ASR	✅	GLM4 MoE / Lite / 4.5V MoE	✅	MiniCPM 3/O/V/V 4_6	✅	PanGu-α	✅
XVERSE	✅	Brumby	✅	Hymba	✅	Mistral	✅	Qwen 1/2/3/3.5	✅
MiniMax M2	✅	AfMoE	✅	Bailing-MoE	✅	LFM2-MoE	✅	Marin	✅
InternVL Chat	✅	Laguna	✅	Mimo / Mimo V2	✅	Zamba / Zamba2	✅	Intern S1	✅

Prism Bonsai GGUF checkpoints are supported for inference only through GPT-QModel's native GGUF path and internal GGUF runtime. Bonsai checkpoints load through the normal model path or repo argument and do not require the external gguf package. Prism model quantization is not included.

Platform and HW Support

GPT-QModel is validated on Linux, macOS, and Windows 11:

Platform Device Optimized Arch Kernels

🐧 Linux	NVIDIA GPU	✅	Turing+ (sm_75+)	Machete, Marlin, Exllama V3 / EXL3, Exllama V2, AWQ GEMM/GEMV, ParoQuant CUDA/Triton, GGUF CUDA/Triton, QQQ, BitBLAS, Triton, BitsAndBytes, Torch
🐧 Linux	AMD GPU	✅	7900XT+, ROCm 6.2+	Exllama V2, AWQ GEMM/GEMV, QQQ, FP8 Torch, Torch
🐧 Linux	Huawei Ascend NPU	✅	Ascend 910B, torch-npu / CANN	Native Torch kernels for GPTQ, AWQ, ParoQuant, GGUF, QQQ, and EXL3
🐧 Linux	Intel XPU	✅	Arc, Datacenter Max	TorchFused, TorchFusedAWQ, FP8 Torch, Torch
🐧 Linux	Intel/AMD CPU	✅	avx, amx	TorchFused, TorchFusedAWQ, TorchAten int4, TorchInt8, GGUF C++, BitsAndBytes, Torch
🍎 macOS	GPU (Metal) / CPU	✅	Apple Silicon, M1+	Torch, FP8 Torch, MLX via conversion
🪟 Windows	GPU (NVIDIA) / CPU	✅	NVIDIA	Torch

Marlin and JIT CUDA kernels now support NVIDIA Turing+ (sm_75+) GPUs. Huawei Ascend NPU support uses native Torch kernels through torch-npu / CANN.

Install

PIP/UV

# You can install optional modules like autoround, ipex, vllm, sglang, bitblas.
# Example: pip install -v gptqmodel[vllm,sglang,bitblas]
pip install -v gptqmodel
uv pip install -v gptqmodel

The package depends on ninja for first-use JIT kernel compilation.

Install from source

# clone repo
git clone https://github.com/ModelCloud/GPTQModel.git && cd GPTQModel

# python3-dev is required for some source installs
apt install python3-dev

# pip: install from source
# You can install optional modules like  vllm, sglang, bitblas.
# Example: pip install -v .[vllm,sglang,bitblas]
pip install -v .

Inference

Three-line API to use GPT-QModel for GPTQ model inference:

from gptqmodel import GPTQModel

model = GPTQModel.load("ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v2.5")
result = model.generate("Uncovering deep insights begins with")[0] # tokens
print(model.tokenizer.decode(result)) # string output

To use models from ModelScope instead of HuggingFace Hub, set an environment variable:

export GPTQMODEL_USE_MODELSCOPE=True

FP32 accumulation toggle

Some AWQ and ParoQuant CUDA/Triton kernels support an fp32 accumulation mode to reduce numerical drift during fused quantized matmul. This setting defaults to True because accuracy is prioritized over speed.

# default behavior: higher accuracy, slightly lower speed on some kernels
export GPTQMODEL_FP32_ACCUM=1

# optional speed-first mode for some kernels
export GPTQMODEL_FP32_ACCUM=0

Notes:

This is a runtime toggle. It does not change model weights or saved checkpoints.
It mainly affects some fused AWQ and ParoQuant CUDA/Triton kernels. Dense/dequantize fallback paths are mostly unaffected.
1 is recommended for regression testing and quality-sensitive evaluation. 0 may be useful when chasing a small latency win and the quality tradeoff is acceptable.

OpenAI API compatible endpoint

# load model using above inference guide first
model.serve(host="0.0.0.0",port="12345")

Quantization

Basic example of using GPT-QModel to quantize an LLM model:

from datasets import load_dataset
from gptqmodel import GPTQConfig, GPTQModel

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
  ).select(range(1024))["text"]

quant_config = GPTQConfig(bits=4, group_size=128)

model = GPTQModel.load(model_id, quant_config)

# increase `batch_size` to match GPU/VRAM specs to speed up quantization
model.quantize(calibration_dataset, batch_size=1)

model.save(quant_path)

Other Quantization Formats

QuantizeConfig remains the broad factory. The concrete config classes are now GPTQConfig, AWQConfig, ParoConfig, QQQConfig, RTNConfig, GGUFConfig, FP8Config, BitsAndBytesConfig, and EXL3Config.

GPTQ, AWQ, ParoQuant, and EXL3 are calibration-based. GGUF and FP8 are weight-only and should be quantized with calibration=None.

Preprocessors

preprocessors=[...] adds optional module-weight preparation steps before quantization or repacking. They are available on GPTQConfig, AWQConfig, ParoConfig, RTNConfig, GGUFConfig, FP8Config, and BitsAndBytesConfig.

SmootherConfig: apply weight smoothing before quantization.
AutoModuleDecoderConfig: decode FP8/FP4 source modules to a dense target_dtype before downstream quantization or repacking.
TensorParallelPadderConfig: opt-in tensor-parallel padding metadata for TP-aligned packing.

import torch
from gptqmodel import GGUFConfig, GPTQConfig
from gptqmodel.quantization import (
    AutoModuleDecoderConfig,
    SmoothMAD,
    SmootherConfig,
    TensorParallelPadderConfig,
)

gptq_cfg = GPTQConfig(
    bits=4,
    group_size=128,
    preprocessors=[
        SmootherConfig(smooth=SmoothMAD(k=2.0)),
        AutoModuleDecoderConfig(target_dtype=torch.bfloat16),
        TensorParallelPadderConfig(),
    ],
)

gguf_cfg = GGUFConfig(
    bits=4,
    format="q_k_m",
    preprocessors=[
        AutoModuleDecoderConfig(target_dtype=torch.bfloat16),
        TensorParallelPadderConfig(),
    ],
)

GGUF Example: Llama 3.2 1B Instruct

from gptqmodel import BACKEND, GGUFConfig, GPTQModel

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-GGUF-Q4_K_M"

qcfg = GGUFConfig(
    bits=4,
    format="q_k_m",
)

model = GPTQModel.load(model_id, qcfg)
model.quantize(calibration=None, backend=BACKEND.GGUF_TORCH)
model.save(quant_path)

FP8 Example: Llama 3.2 1B Instruct

from gptqmodel import BACKEND, FP8Config, GPTQModel

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-FP8-E4M3"

qcfg = FP8Config(
    format="float8_e4m3fn",  # or "float8_e5m2"
    bits=8,
    weight_scale_method="row",
)

model = GPTQModel.load(model_id, qcfg)
model.quantize(calibration=None, backend=BACKEND.GPTQ_TORCH)
model.save(quant_path)

Exllama V3 / EXL3 Example: Llama 3.2 1B Instruct

from datasets import load_dataset
from gptqmodel import BACKEND, EXL3Config, GPTQModel

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-EXL3"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train",
).select(range(1024))["text"]

qcfg = EXL3Config(
    bits=4.0,        # target average bits-per-weight
    head_bits=6.0,   # optional higher bitrate for attention heads / sensitive tensors
    codebook="mcg",  # one of: mcg, mul1, 3inst
)

model = GPTQModel.load(model_id, qcfg)
model.quantize(calibration_dataset, batch_size=1, backend=BACKEND.EXL3_EXLLAMA_V3)
model.save(quant_path)

MoE Quantization

Some MoE (mixture of experts) models have extremely uneven/biased routing (distribution of tokens) to the experts causing some expert modules to receive close-to-zero activated tokens, thus failing to complete calibration-based quantization (GPTQ/AWQ). To better quantize these heavily biased MoE routed modules, GPT-QModel exposes 3 controls:

Moe.Routing = ExpertsRoutingOverride: Manually override the num_experts_per_tok used for model routing math, i.e., if a model only routes 4 experts per token out of 48 total experts, you can set this equal to 24 for 50% routing or 48 for 100% routing. ExpertsRoutingOverride requires the model exposes num_experts_per_tok or equivalent configuration control.
Moe.Routing = ExpertsRoutingBypass: Brute-force and bypass all routing math so all experts receive all activated tokens. This is akin to ExpertsRoutingOverride.num_experts_per_tok set to total number of experts. ExpertsRoutingBypass is enabled/tested for some models and, due to the lifecycle complexity, it needs to be validated for every model.
FailSafe: This is enabled by default and is a naive weight-only quantization technique using simple (naive) quantization methods such as nearest with optional smoothing. There are various FailSafeStrategy options, along with SmoothMethod options, to complement this feature. FailSafe does not require activations but has higher quantization error loss than normally activated GPTQ/AWQ. It is fast and applicable for all MoE models.

FailSafe can be combined with ExpertsRoutingOverride. There is no single best way to quantize MoE, and we recommend users to test all three methods.

Quantized Inference

# test post-quant inference
model = GPTQModel.load(quant_path)
result = model.generate("Uncovering deep insights begins with")[0] # tokens
print(model.tokenizer.decode(result)) # string output

EoRA Accuracy Recovery: Enhanced Post-Quant Error Recovery via Lora

GPT-QModel supports EoRA, a LoRA method developed by Nvidia that can further improve the accuracy of the quantized model.

# EoRa is currently only validated for GPTQ
# higher rank improves accuracy at the cost of VRAM usage
# suggestion: test rank 64 and 32 before 128 or 256 as latter may overfit while increasing memory usage
eora = Lora(
  # for eora generation, path is adapter save path; for load, it is loading path
  path=f"{quant_path}/eora_rank32", 
  rank=32,
)

# provide a previously GPTQ-quantized model path
GPTQModel.adapter.generate(
  adapter=eora,
  model_id_or_path=model_id,
  quantized_model_id_or_path=quant_path,
  calibration_dataset=calibration_dataset,
  calibration_dataset_concat_size=0,
)

# post-eora inference
model = GPTQModel.load(
  model_id_or_path=quant_path,
  adapter=eora
)

tokens = model.generate("Capital of France is")[0]
result = model.tokenizer.decode(tokens)

print(f"Result: {result}")
# For more details on EoRA, please see docs/eora/
# Please use the benchmark tools in later part of this README to evaluate EoRA effectiveness

How to Add Support for a New Model

Read the gptqmodel/models/llama.py code which explains in detail via comments how the model support is defined. Use it as a guide for PRs to add new models. Most models follow the same pattern.

Pair with Evaluation for post-quantization LLM Benchmarks

GPT-QModel evaluation is integrated into Evalution, a modern benchmarking toolkit with 150+ of the world's most widely used benchmark suites. We highly recommend using Evalution to measure post-quant accuracy recovery after quantization instead of relying on narrow regression-only language-model metrics.

# install Evalution pip install Evalution

Below is a short example running gsm8k_platinum through Evalution's native GPT-QModel engine.

import evalution as eval

run = (
    eval.GPTQModel(
        backend="marlin",
        device="cuda:0",
    )
    .model(eval.Model(path="ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v1"))
    .run(eval.benchmarks.gsm8k_platinum(apply_chat_template=True, batch_size=16))
)

print(run.to_dict()["tests"][0]["metrics"])

Dynamic Quantization (Per Module QuantizeConfig Override)

QuantizeConfig.dynamic is a dynamic control that allows specific matching modules to be skipped for quantization (negative matching) or have a unique [bits, group_size, sym, desc_act, mse, pack_dtype] property override per matching module vs base QuantizeConfig (positive match with override).

Sample QuantizeConfig.dynamic usage:

dynamic = { 
    # `.*\.` matches the layers_node prefix 
    # layer index starts at 0 
    
    # positive match: layer 19, gate module 
    r"+:.*\.18\..*gate.*": {"bits": 4, "group_size": 32},  
    
    # positive match: layer 20, gate module (prefix defaults to positive if missing)
    r".*\.19\..*gate.*": {"bits": 8, "group_size": 64},  
    
    # negative match: skip layer 21, gate module
    r"-:.*\.20\..*gate.*": {}, 
    
    # negative match: skip all down modules for all layers
    r"-:.*down.*": {},  
 } 

Group Aware Reordering (GAR)

Group Aware Reordering (GAR) is an enhanced activation reordering scheme developed by Intel to improve the accuracy of quantized models without incurring additional inference overhead. Unlike traditional activation reordering, GAR restricts permutations to within individual groups or rearrangements of entire groups. This ensures each group's associated scales and zero-points remain efficiently accessible during inference, thereby avoiding any inference-time overhead.

How to enable GAR:

Set the act_group_aware parameter to True and disable the default activation reordering by setting desc_act to False in your QuantizeConfig. For example:

quant_config = QuantizeConfig(bits=4, group_size=128, act_group_aware=True)

Experimental Features

Using GPTAQ (Experimental, not MoE compatible, and results may not be better than original)

Enable GPTAQ quantization by setting gptaq = GPTAQConfig(...).

# Note GPTAQ is currently experimental, not MoE compatible, and requires 2-4x more VRAM to execute
# We have many reports of GPTAQ not working better or exceeding GPTQ so please use for testing only
# If OOM on 1 GPU, please set CUDA_VISIBLE_DEVICES=0,1 to 2 GPUs and gptqmodel will auto use second GPU
quant_config = QuantizeConfig(bits=4, group_size=128, gptaq=GPTAQConfig(alpha=0.25, device="auto"))

Using FOEM

FOEM (First-order error matters) adds first-order error compensation for GPTQ-style quantization. Enable FOEM by setting foem = FOEMConfig(...).

# FOEM default hyperparameters are alpha=0.0 and beta=0.2
quant_config = QuantizeConfig(bits=4, group_size=128, foem=FOEMConfig(alpha=0.0, beta=0.2, device="auto"))

Migrating from AutoGPTQ and AutoAWQ:

GPT-QModel has fully supplanted AutoGPTQ and AutoAWQ for HF Transformers/Optimum/Peft integration. Model inference has drop-in support with zero changes.

For model quantization, there are some config changes for AutoAWQ:

AutoAWQ: version property is now format. zero_point is now sym (Symmetric Quantization): sym = True is equivalent to zero_point = False

Models quantized by GPT-QModel are inference compatible with HF Transformers (minus dynamic), vLLM, and SGLang.

Attributions:

GPTQ: IST-DASLab, main-author: Elias Frantar, arXiv:2210.17323
AWQ: main-authors: Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Dang, Xingyu and Han, Song
ParoQuant: Z-Lab, main-authors: Yesheng Liang, Haisheng Chen, Song Han, and Zhijian Liu. Official implementation, Paper
EoRA: Nvidia, main-author: Shih-Yang Liu, arXiv preprint arXiv:2410.21271.
GAR: Intel, main-author: T Gafni, A Karnieli, Y Hanani, Paper
GPTAQ: Yale Intelligent Computing Lab, main-author: Yuhang Li, arXiv:2504.02692.
QQQ: Meituan, main-author Ying Zhang, arXiv:2406.09904
FOEM: Zheng, Xingyu and Qin, Haotong and Li, Yuye and Chu, Haoran and Wang, Jiakai and Guo, Jinyang and Magno, Michele and Liu, Xianglong Paper

Citations:

# GPT-QModel
@misc{qubitium2024gptqmodel,
  author = {ModelCloud.ai and qubitium@modelcloud.ai},
  title = {GPT-QModel},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/modelcloud/gptqmodel}},
  note = {Contact: qubitium@modelcloud.ai},
  year = {2024},
}

# GPTQ
@article{frantar-gptq,
  title={{GPTQ}: Accurate Post-training Compression for Generative Pretrained Transformers}, 
  author={Elias Frantar and Saleh Ashkboos and Torsten Hoefler and Dan Alistarh},
  journal={arXiv preprint arXiv:2210.17323},
  year={2022}
  
}

# AWQ
@article{lin2023awq,
  title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
  author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Dang, Xingyu and Han, Song},
  journal={arXiv},
  year={2023}
}

# ParoQuant
@inproceedings{liang2026paroquant,
  title     = {{ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference}},
  author    = {Liang, Yesheng and Chen, Haisheng and Han, Song and Liu, Zhijian},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026}
}

# GGUF / llama.cpp
@misc{ggerganov2023gguf,
  author = {Georgi Gerganov and ggml-org contributors},
  title = {llama.cpp and the GGUF model format},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/ggml-org/llama.cpp}},
  note = {Canonical GGUF implementation and format reference; see also \url{https://github.com/ggml-org/llama.cpp/wiki/dev-notes}},
  year = {2023}
}

# EoRA
@article{liu2024eora,
  title={EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation},
  author={Liu, Shih-Yang and Yang, Huck and Wang, Chien-Yi and Fung, Nai Chit and Yin, Hongxu and Sakr, Charbel and Muralidharan, Saurav and Cheng, Kwang-Ting and Kautz, Jan and Wang, Yu-Chiang Frank and others},
  journal={arXiv preprint arXiv:2410.21271},
  year={2024}
}

# GPTAQ
@article{li2025gptaq,
  title={GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration}, 
  author={Yuhang Li and Ruokai Yin and Donghyun Lee and Shiting Xiao and Priyadarshini Panda},
  journal={arXiv preprint arXiv:2504.02692},
  year={2025}
}

# FOEM
@inproceedings{zheng2026first,
  title={First-order error matters: Accurate compensation for quantized large language models},
  author={Zheng, Xingyu and Qin, Haotong and Li, Yuye and Chu, Haoran and Wang, Jiakai and Guo, Jinyang and Magno, Michele and Liu, Xianglong},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={40},
  number={34},
  pages={28883--28891},
  year={2026}
}

# QQQ 
@article{zhang2024qqq,
      title={QQQ: Quality Quattuor-Bit Quantization for Large Language Models}, 
      author={Ying Zhang and Peng Zhang and Mincong Huang and Jingyang Xiang and Yujie Wang and Chao Wang and Yineng Zhang and Lei Yu and Chuan Liu and Wei Lin},
      journal={arXiv preprint arXiv:2406.09904},
      year={2024}
}

# ExLlama V3 / EXL3
@misc{turboderp2026exllamav3,
  author = {turboderp and exllamav3 contributors},
  title = {ExLlamaV3 and the EXL3 quantization format},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/turboderp-org/exllamav3}},
  note = {Project repository and EXL3 format documentation: \url{https://github.com/turboderp-org/exllamav3/blob/master/doc/exl3.md}},
  year = {2026}
}

# Group Aware Reordering (GAR)
@article{gar,
  title={Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference, CVPRW 2025.},
  author={T. Gafni, A. Karnieli, Y. Hanani},
  journal={arXiv preprint arXiv:2505.14638},
  year={2025}
}

# Marlin Kernel
@article{frantar2024marlin,
  title={MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models},
  author={Frantar, Elias and Castro, Roberto L and Chen, Jiale and Hoefler, Torsten and Alistarh, Dan},
  journal={arXiv preprint arXiv:2408.11743},
  year={2024}
}

Quick Notes

Limit log level

GPT-QModel uses a shared LogBar logger. Set the level once near process startup:

from logbar import LogBar

LogBar.shared().setLevel("WARNING")  # DEBUG, INFO, WARNING, ERROR, CRITICAL

Apply Triton nogil patch early in multi-package scripts

If your script imports multiple Triton users (for example gptqmodel, vllm, and sglang), apply the patch at the very top before other Triton-related imports:

from gptqmodel import TritonPatch

# Fix Triton crashing under nogil/free-threading Python 3.13+ where the kernel cache storage in Triton is not thread-safe
TritonPatch.apply()

About

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

x.com/Qubitium

transformers quantization optimum peft vllm gptq sglang

Readme

View license

Activity

Custom properties

1.2k stars

5 watching

184 forks

Report repository