HaoKun Lin@HaokunLin
🚀 Introducing STEPQuant for recurrent state quantization in linear attention!
6-bit states → near-FP32 accuracy + >5× compression.
📄 Paper:
💻 Code:
#LLM #Quantization
2061.8K
Posts with #quantization
#quantization
Public posts on X that use #quantization.
Related hashtags
Visual-Intelligence@VI_Journal_CSIG🔥Researchers from Beihang University and ETH Zurich conducted a systematic evaluation of Qwen3's robustness under various quantization settings. Check out the paper at: @qin_haotong #Quantization #LLM #Modelcompression00289
MT Group at FBK@fbk_mtOur pick of the week by @dhairya_su47605 : "Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression" by @hanqi_xiao, @yilin_sung, @EliasEskin and @mohitban47 #Quantization #InterpretibilityDhairya Suman @dhairya_su47605#PickoftheWeek @fbk_mt Super cool paper on leavaraging Interpretability for Compression!027922
Dhairya Suman@dhairya_su47605#PickoftheWeek @fbk_mt Super cool paper on leavaraging Interpretability for Compression!0151K
Silvio Martinico@SilvioMartinicoRaBitQ is now in Vectorium! Up to 30x compression, no training, 1/2/4/8 bits per component, 1.4-2.2x faster than the original implementation. Against PQ, on neural embeddings it wins on every axis, and the gap grows with dimensionality. #VectorSearch #Quantization #NeuralIR
2617784
(بشير) bachir@bachiirc👀Interested in learning more about vector quantization techniques? Check this talk that goes over different types of vector compression algorithms #LLMs #Quantization #Weaviate01392
low_precision@majipa007I always thought quantizing LLMs was something that needed crazy hardware or training. Finally gave it a shot today and it was way simpler than I expected. Thanks to llama.cpp for making the whole process ridiculously easy and @Alibaba_Qwen for good SMLs. #LLM #Quantization #GGUF30238
🧵 The next era: build your own local inference engine. Why now? 1/ Open models are good enough 2/ Quantization makes them fit on any hardware 3/ Cost and cost are pushing teams off the cloud #LocalAI #LLM #Quantization #GPU #OpenSourceAI #AIEngineering #BuildInPublic204142
Zack@Z_ackeOne card: GPTQ vs AWQ for 4-bit LLM weights GPTQ: layerwise 2nd-order error fix AWQ: protect ~0.1%-1% salient channels (activation-aware) Both usually W4A16. Speeds Decode; not KV. #AIInfra #Quantization #LLM
60021
Haoli BAI@haolibai#COLM2025 #LLM #Quantization #ReasoningModels #EfficientAI 🚀 Thrilled to introduce our recent work at COLM 2025: “Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models”, presented today at Poster Session 3, #74.
004154
Sensors MDPI@Sensors_MDPIEfficient Deep Learning Model Compression for Sensor-Based Vision Systems via Outlier-Aware Quantization #quantization #outlier_awareness #model_compression #image #feature_extraction #sensor #computer_vision0313380
Creepybits@creepybitsAutoregressive LLMs burn massive TDP fetching FP16 weights for low-entropy tokens. SDSIE pairs 1.58-bit speculative drafting with Schmitt-trigger entropy routing H(Y|X) to cut inference energy up to 50%. Thread 👇 @elonmusk @ibabuschkin #LLMInference #CUDA
12343- Creepybits@creepybits@creepybitsFull Zenodo blueprint: GitHub Spec: Tagging @karpathy @Tim_Dettmers @ggerganov for thoughts on the entropy hysteresis bounds. #vLLM #Quantization01238
ZuoCen Liu@liu_zc42321Launch: vllm-position-parity🚀 This position-resolved measurement tool for vLLM has been used in: 5-arm determinism isolation (#54521) – confirmed 4 co-occurring defects & full zeroing with combined fixes #vLLM #quantization #determinism202108- MT Group at FBK@fbk_mtOur pick of the week by @dhairya_su47605: "How Does #Quantization Affect #Multilingual #LLMs?" by @cheeesio, @TheyCallMeMr_, Hongyu Chen, @d_aumiller, @ahmetustun89, @sarahookr, @seb_ruder (Findings EMNLP, 2024)Dhairya Suman @dhairya_su47605Pick of the week @fbk_mt: How Does Quantization Affect Multilingual LLMs? Quantization has become a widely adopted technique for model compression. This work investigates the impact of quantization on different languages in multilingual LLMs.0481.1K
- Dhairya Suman@dhairya_su47605Pick of the week @fbk_mt: How Does Quantization Affect Multilingual LLMs? Quantization has become a widely adopted technique for model compression. This work investigates the impact of quantization on different languages in multilingual LLMs.1151.3K
Diana Marculescu@dianamarculescuHung-Yueh Chiang @yueh_chWe’re excited to pre-release our latest work: Quamba2! 🔧 W4A8 / W4A16 / W4AX / W8A8 Mamba models 🚀 Achieves 4× memory reduction and 3× generation speedup ️⚡ Enables 8B model inference on Orin Nano 8G at 13 tokens/sec 🔥 Outperforms W4A8KV4 Llama3-8B in both speed and quality005170
Hung-Yueh Chiang@yueh_chWe’re excited to pre-release our latest work: Quamba2! 🔧 W4A8 / W4A16 / W4AX / W8A8 Mamba models 🚀 Achieves 4× memory reduction and 3× generation speedup ️⚡ Enables 8B model inference on Orin Nano 8G at 13 tokens/sec 🔥 Outperforms W4A8KV4 Llama3-8B in both speed and quality
113304
'Approximations and Learning for Continuous State and Action MDPs under Average Cost Criteria', by Ali D. Kara, Serdar Yüksel. #quantization #quantized #discretization037978
🚀 The 4-bit era has arrived! Meet #SVDQuant, our new W4A4 quantization paradigm for diffusion models. Now, 12B FLUX can run on a 16GB 4090 laptop without offloading—with 3x speedups over W4A16 models (like NF4) while maintaining top-tier image quality. #AI #Quantization. 1/794517250.3K
Aaron Rose | Tech Reader@techreader22587The Azure AI Model Quantizer Fitting a massive brain into a tiny footprint #AzureAI #Quantization #VRAM #ModelCompression00234
Philippe Ensarguet@P_EnsarguetHoney, I shrunk the LLM! A beginner's guide to quantization – and testing it via @theregister If you're wondering how #quantization works and how it can make #LLMs less resource-intensive to run, this post is for you, including getting your hands dirty👍024614- Show more posts
