# of bits per weight on deepseek.r1-v1:0 running on Amazon Bedrock?

The DeepSeek-R1 model available on Amazon Bedrock is a large language model with 671 billion parameters. Regarding the quantization level of the weights for the model running on Bedrock, the documentation doesn't explicitly specify whether the Bedrock version uses 4-bit, 8-bit, or 16-bit weights.

What we do know is that the default weights provided by the DeepSeek team in their official R1 repository are of type FP8 (8-bit Floating Point). FP8 is a low-precision format that balances memory efficiency and model accuracy by representing values with fewer bits than standard FP16 floating point format.

When fine-tuning the DeepSeek-R1 model using Amazon SageMaker HyperPod recipes, it's recommended to convert the model from FP8 to BF16 as this has been empirically found to enhance generalization across diverse datasets. However, this specifically relates to fine-tuning scenarios rather than the default deployment on Bedrock.

The DeepSeek-R1 model on Bedrock supports a context length of 8192 tokens, which includes both output and reasoning tokens. Model reasoning (chain of thought reasoning) is always enabled for this model and cannot be toggled off.

If the specific quantization level is critical for your use case, I would recommend contacting AWS Support for definitive information about the exact quantization level used for the DeepSeek-R1 model on Bedrock, as this could impact your decision between using Bedrock or building an in-house solution.
Sources
Customize DeepSeek-R1 671b model using Amazon SageMaker HyperPod recipes – Part 2 | AWS Machine Learning Blog
Inference optimization for Amazon SageMaker AI models - Amazon SageMaker AI
Enhance model responses with model reasoning - Amazon Bedrock