Pre-Installed Llama

Overview

Llama is a state-of-the-art, open-weight AI model ecosystem developed by Meta. It includes a comprehensive range of large language and multimodal models—from lightweight 1B and 3B text models for edge devices, to powerful 8B, 70B, and 405B dense models, to advanced Llama 4 models with up to 400B total parameters using sparse Mixture-of-Experts (MoE) architecture for efficient inference. Llama delivers leading performance in natural language understanding, coding, reasoning, image comprehension, tool use, and agentic workflows, with context windows up to 10 million tokens and support for over 8 languages.

Select a pre-installed Llama Marketplace image and get a fully configured server that's ready to use immediately—no manual installation or complex setup required.

Supported Operating Systems

The Llama Marketplace image is available for Ubuntu Server 24 LTS 64-bit.

Supported Products / Plans

1. Llama3.1-70B

The Llama3.1-70B Marketplace image is currently available across 3 GPU-dedicated server plans:

• Enterprise Dedicated GPU Server - RTX A6000
• Enterprise Dedicated GPU Server - A100(80GB)
• Enterprise Dedicated GPU Server - H100

View compatible plans →

2. Llama3.2-Vison-90B

The Llama3.2-Vison-90B Marketplace image is currently available across the following 3 GPU-dedicated server plans:

• Enterprise Multi-GPU Dedicated Server - 2xRTX 5090
• Enterprise Dedicated GPU Server - A100(80GB)
• Enterprise Dedicated GPU Server - H100

View compatible plans →

3. Llama4-16x17B

The Llama4-16x17B Marketplace image is currently available across the following 2 GPU-dedicated server plans:

• Enterprise Dedicated GPU Server - A100(80GB)
• Enterprise Dedicated GPU Server - H100

View compatible plans →

Llama Version

Database Mart offers the latest version of Llama, pre-installed and ready to use.

What's Installed

When you select the Llama Marketplace image, Database Mart automatically installs and configures:

Core Operating Environment

• Operating System: Ubuntu Server 24 LTS
• GPU Drivers & Acceleration: NVIDIA drivers, CUDA 12.x, and cuDNN pre-installed for GPU acceleration
• Python Environment: Python 3.10+ with a dedicated virtual environment (Conda or venv) for dependency management

Llama Core Components & Inference Tools

• Deep Learning Frameworks: PyTorch 2.0+, Transformers library, accelerate, and other essential libraries
• Model Weights & Configurations: Pre-trained model weights and configuration files (config.json, tokenizer.json) downloaded automatically based on the selected model
• Inference Engines: High-performance frameworks like vLLM with PagedAttention, continuous batching, and optimized CUDA kernels for high-throughput inference
• Quantization Support: Model quantization (FP8, INT8, GPTQ, AWQ) included to reduce VRAM usage while maintaining performance

Hardware & GPU Support

• NVIDIA Container Toolkit: Pre-installed for GPU-accelerated containerized deployment
• FlashAttention: Included for optimized attention computation

Installation Process

During server provisioning, Database Mart automatically performs the following steps:

1. Deploys the selected Linux operating system (Ubuntu Server 24 LTS).

2. Installs the selected Llama model version with all pre-trained weights and configuration files.

3. Installs and configures all required components, including:

   • Python 3.10+ with a dedicated virtual environment (Conda or venv)
   • PyTorch 2.0+ and the Transformers library
   • vLLM for high-performance inference with PagedAttention and continuous batching
   • CUDA and cuDNN for GPU acceleration
   • FlashAttention for optimized attention computation
   • Model weights downloaded from Hugging Face or NVIDIA NGC
   • Quantization tools (FP8, INT8, GPTQ, AWQ) for reduced memory footprint

Everything is fully automated—your Llama server is ready to use as soon as deployment is complete.

Getting Started After Deployment

After your server has been provisioned, follow these steps to begin using Llama.

Step 1: Access the Server

Once deployment is complete, connect to your server via SSH using the credentials provided by Database Mart:

ssh administrator@your-server-ip

The Llama model files are pre-installed in the designated directory (typically /opt/llama or a similar location).

Step 2: Verify the Installation

Check that all components are properly installed and the model is ready:

# Verify CUDA and GPU availability
nvidia-smi

# Check Python environment
python3 --version

# Navigate to the Llama directory
cd /opt/llama

# List model files
ls -la

The model weights should be present in the model directory.

Step 3: Start the Inference Service

The server comes with vLLM pre-configured for high-performance inference. Start the service using:

# For Llama 3.1 8B
vllm serve meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000

# For Llama 3.3 70B (FP8 optimized)
vllm serve nvidia/Llama-3.3-70B-Instruct-FP8 \
  --host 0.0.0.0 \
  --port 8000 \
  --kv-cache-dtype fp8 \
  --async-scheduling \
  --max-num-batched-tokens 8192

Important flags explained :

• --host 0.0.0.0: Binds the server to all network interfaces
• --port 8000: Specifies the API port
• --tensor-parallel-size <N>: Enables distributed inference across N GPUs
• --kv-cache-dtype fp8: Uses FP8 for KV cache, reducing memory usage and improving performance
• --async-scheduling: Reduces host overhead between decoding steps for better performance

The API server will start and be accessible at http://your-server-ip:8000/v1.

Step 4: Test the API with a Simple Request

Once the service is running, you can test it using the OpenAI-compatible API endpoint:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Llama-3.3-70B-Instruct-FP8",
    "prompt": "San Francisco is a",
    "max_tokens": 20,
    "temperature": 0
  }'

Using the OpenAI Python client:

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",  # Not required for local deployment
    base_url="http://localhost:8000/v1"
)

response = client.completions.create(
    model="nvidia/Llama-3.3-70B-Instruct-FP8",
    prompt="The capital of Canada is",
    max_tokens=50,
    temperature=0.7
)

print(response.choices[0].text)

Step 5: Use Chat Completions

For chat-optimized models like Llama 3.1 Instruct:

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    max_tokens=500,
    temperature=0.7
)

print(response.choices[0].message.content)

Notes

Prompting Guidelines

Llama models are sensitive to prompts and perform best when prompted with natural text completions rather than direct questions. For example:

• Instead of "What is the meaning of life?", try "I believe the meaning of life is"
• Instead of "Explain the theory of relativity.", try "Simply put, the theory of relativity states that"

vLLM Performance Optimization

• Use FP8 quantization for Hopper and Blackwell GPUs to improve throughput and reduce memory usage
• Enable async scheduling for better performance at high concurrency
• Adjust tensor-parallel-size: Set to 1 for best throughput per GPU, or 2-8 for better per-user latencies
• Set max-num-batched-tokens to 8192 for optimal performance with most workloads
• Limit max-model-len to match your actual input+output lengths for performance gains

Balancing Throughput and Latency

• For maximum throughput: Increase batch size and use tensor parallelism strategically
• For low latency: Reduce batch size and use higher tensor parallelism

Multilingual Support

Llama models were trained primarily on English, but also support Latin and Cyrillic alphabets. The tokenizer splits unseen characters into UTF-8 bytes, enabling limited processing of other languages like Chinese and Japanese.

Frequently Asked Questions

Q: What Llama models are available from Database Mart?

Database Mart offers three pre-installed Llama variants:

Q: What hardware do I need for each model?

• Llama 3.1-70B: Requires 80GB+ VRAM (A6000, A100 80GB, or H100 recommended)
• Llama 3.2-Vision-90B: Requires 80GB+ VRAM; 2x RTX 5090 recommended for optimal performance
• Llama 4-16x17B: Requires 80GB+ VRAM (A100 80GB or H100 recommended)

Q: What's the difference between Llama 3.1-70B and Llama 3.2-Vision-90B?

Llama 3.1-70B is a pure text model optimized for language understanding, coding, and reasoning tasks. Llama 3.2-Vision-90B adds multimodal capabilities — it can process and understand images in addition to text, making it ideal for visual reasoning, document analysis, and image-based tasks.

Q: What is Llama 4-16x17B and why is it special?

Llama 4-16x17B is a Mixture-of-Experts (MoE) model with 400B total parameters but only 17B active per forward pass. This architecture delivers flagship-level performance while being more computationally efficient than dense models of similar size — it can run on a single H100 or A100 80GB node.

Q: What inference engines are supported?

The pre-installed image comes with vLLM pre-configured and ready to use. Llama also supports:

• TorchServe + vLLM: Production deployment with custom metrics and versioning
• NVIDIA NeMo: Enterprise-grade deployment on Triton Inference Server
• Ollama: Local deployment on consumer hardware
• Transformers (Hugging Face): Flexible, code-level control

Q: Does Llama 3.2-Vision-90B support image inputs?

Yes. Llama 3.2-Vision-90B accepts image inputs alongside text prompts for visual reasoning, document understanding, and image analysis tasks.