Local RAG Stack: Running Llama-3 and Qdrant on a Budget

The OpenAI API is convenient, but it gets expensive at scale, and for many enterprises, sending data externally is a non-starter.

The solution? A fully local stack.

The Stack

• Model: Ollama running Llama-3 (8B quantized).
• Middleware: Python.
• Vector DB: Qdrant (runs in Docker container).

Quick Setup

First, ensure you have Ollama installed and run: ollama run llama3.

Then, spin up Qdrant:

docker run -p 6333:6333 qdrant/qdrant

The Code

Here is a Python snippet to ingest documents and query them locally.

import qdrant_client
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.ollama import OllamaEmbedding

# 1. Connect to Qdrant
client = qdrant_client.QdrantClient(url="http://localhost:6333")
vector_store = QdrantVectorStore(client=client, collection_name="my_docs")

# 2. Use local embeddings (nomic-embed-text is great and fast)
embed_model = OllamaEmbedding(model_name="nomic-embed-text")

# 3. Ingest Data
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(
    documents,
    vector_store=vector_store,
    embed_model=embed_model
)

# 4. Query
query_engine = index.as_query_engine()
response = query_engine.query("What is the primary advantage of local RAG?")
print(response)

This setup gives you a private, zero-cost (hardware permitting) RAG pipeline.