Build an Offline AI Assistant That Reads Your Documents (No Coding, Mac)
Ask questions about your own files. Keep them on your own Mac.
You can set up an AI assistant that reads your documents and answers from them, with the AI and your files both staying on your computer. This guide shows the no-coding way, using two free tools, and the settings that decide whether it’s truly private.
The short answer
- You need two things: Ollama, which runs the AI model, and AnythingLLM Desktop, which lets you chat with your documents.
- Both are free, and AnythingLLM needs no account. It says your models, documents and chat history stay on your machine.
- The privacy setting that matters: keep the model and the embedder local. Choosing a cloud provider sends text off your Mac.
- Check the answers. A small model can misread a document. Always confirm important figures in the original.
How it works, in plain words
Your documents are cut into small pieces and indexed. When you ask a question, the app finds the most relevant pieces and hands them to the AI model, which writes an answer from them. This is called RAG. AnythingLLM does the indexing and the chat window. Ollama runs the model.
Step 1: install Ollama and one model
Download Ollama from ollama.com/download, open it, and run a model once so it downloads. Pick a size that suits your Mac’s memory. A 16GB Mac can run a mid-sized model comfortably; an 8GB Mac needs a small one.
Step 2: install AnythingLLM Desktop
Download the Apple Silicon version from anythingllm.com and drag it into Applications, or use Homebrew if you have it:
brew install --cask anythingllm
Open it from Applications or with Command and Space. The download page says there is one-click install, no account and local-by-default behaviour.
Step 3: connect it to your local model
In AnythingLLM’s settings, choose Ollama as the LLM provider. When Ollama is running with its default settings, the address is:
http://127.0.0.1:11434
Then pick the model you downloaded. You can change the model later in Settings.
Step 4: add your documents and ask
- Create a workspace for one subject, such as one client or one project.
- Drag documents into the chat window or click the plus icon in the prompt box. Documents added this way belong to that chat thread only.
- Embed them if you want them everywhere.Embedding makes a document available to every thread in the workspace. If a document is too long for the model, AnythingLLM asks whether to embed it.
- Ask a specific question, such as the total on a particular page, not “summarise everything”.
- Check the answer against the original.Open the document and confirm the figure or clause.
The settings that keep it private
| Setting | Keep it local | What goes wrong otherwise |
|---|---|---|
| Language model | Ollama on your Mac | A cloud model receives your questions and document text |
| Embedder | The built-in default, or Ollama | Cloud embedders such as OpenAI or Cohere receive your text to index it |
| Changing the embedder | Choose once and don’t change it | You’ll have to delete and re-embed every document |
Limits to know before you rely on it
- Small models can be wrong.They may misread tables, mix up numbers or invent details. Treat the answer as a pointer to the source.
- Context is limited.A model can only read so much at once. For long documents, embedding (RAG) lets it search instead of reading everything.
- Memory sets your ceiling.A bigger model understands documents better, but needs more memory. Our coding and AI guide explains how much.
- Back up your originals.Don’t use your only copy of an important file as your test document.
Questions we get asked
Is it really free?
AnythingLLM Desktop is free to download, and Ollama’s local models are free to run. AnythingLLM says no account or credit card is needed.
Can it work with no internet?
After you have downloaded Ollama, a model and AnythingLLM, it can run on your Mac without a connection, as long as you keep everything local.
Is it safe for client documents?
With a local model and a local embedder, your document text isn’t sent to a cloud provider. That is stronger than a typical cloud chatbot, but it is not a guarantee. Keep your Mac updated and encrypted, and follow your own client rules.
Does it work on an older Intel MacBook?
AnythingLLM has an Intel build, but it says Apple M-series chips run local models considerably faster. On an Intel Mac, expect it to be slow.
Want a Mac that runs this well?
Tell Aparna how many documents you work with and your budget. She’ll suggest the right memory and chip from today’s stock.
Tool names, settings and claims checked on 28 October 2026 and can change.























