Running Llama 3.1 405b

a11ce.com/llama-3.1-405b.html

Llama 3.1 405b (base) is no longer hosted by any public inference provider. This page has instructions for running it yourself on an on-demand GPU instance for ~$20/hr and 10 minutes of setup.

  1. Sign up on RunPod and add $20. (That's my referral link, we both get some credits if you use it.) ¹¹RunPod has consistent availability and reasonable prices, but these steps should work on other on-demand GPU hosts. The instance must have >500GB of tmpfs, which you can check with df -h /dev/shm.
  2. Deploy a 4x H200 pod using the default PyTorch template.
  3. ssh in and copy-paste the following script:
API_KEY=${API_KEY:-$(openssl rand -hex 16)}
echo "💜 api key is $API_KEY"

# NVLS causes problems on some pods so just disable it
export NCCL_NVLS_ENABLE=0

LLAMA=meta-llama/Llama-3.1-405B-FP8
WEIGHTSDIR=/dev/shm/llama

# This is a read-only token for an R2 bucket containing the weights
R2_KEY_ID=41f96e313f23edb542aa2d9f11f27d1f
R2_ACCESS_KEY=1864e92a4c9ac79581489dec9bf2e50b037fc47e35eb581ce46389cacfbdecb4
R2_ENDPOINT=https://47e39f6f60165d6392620a903e13d8b6.r2.cloudflarestorage.com

echo "💜 downloading vllm"
python3 -m pip install vllm fastsafetensors

echo "💜 connecting to r2"
curl -fsSL https://rclone.org/install.sh -o rclone-install.sh
bash rclone-install.sh
rclone config create r2 s3 provider=Cloudflare \
  access_key_id=$R2_KEY_ID \
  secret_access_key=$R2_ACCESS_KEY \
  endpoint=$R2_ENDPOINT \
  acl=private no_check_bucket=true

echo "💜 downloading weights"
mkdir -p $WEIGHTSDIR
rclone copy --transfers 16 --multi-thread-streams 16 --multi-thread-cutoff 64M \
  --progress r2:llama/Llama-3.1-405B-FP8/ "$WEIGHTSDIR"/

echo "💜 starting vllm"
vllm serve "$WEIGHTSDIR" --tensor-parallel-size 4 --load-format fastsafetensors \
  --max-model-len 65536 --allow-deprecated-quantization \
  --served-model-name "$LLAMA" --api-key "$API_KEY"
  1. When you see "Application startup complete" (after about 5 minutes), in a different terminal, run ssh -N -L 8000:localhost:8000 root@<IP> -p <PORT> using the IP and port under "SSH over exposed TCP" to forward localhost:8000 to the server.
  2. You can now send inference requests to http://localhost:8000/v1/completions with the model name meta-llama/Llama-3.1-405B-FP8 and the API key that was printed at the start of the script. ²²If you lose your API key, ctrl-Z to pause vllm, run echo $API_KEY, then fg to resume vllm. For example (on your local machine):
curl http://localhost:8000/v1/completions \
  -H "Authorization: Bearer <KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-405B-FP8", "prompt": "The capital of Poland is", "max_tokens": 32}'
  1. If you want to give other people access, use a Cloudflare tunnel. This creates a tunnel and prints only the url, then anyone with the API key can send inference requests to https://<WORDS>.trycloudflare.com/v1/completions:
curl -sL -o /usr/local/bin/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
chmod +x /usr/local/bin/cloudflared
cloudflared tunnel --url http://localhost:8000 2>&1 | grep --line-buffered -o 'https://[a-z-]*\.trycloudflare\.com'
  1. Explore. I recommend Tapestry Loom as an interface.
  2. Don't forget to stop the pod when you're done.

See the notes page for more details and other methods.

You can respond to this page on twitter, mastodon, or bluesky.