What was the situation before?
The firm keeps its project documents in a cloud file store, about 700 GB across the company. It wanted staff to be able to ask those documents questions without sending them to an outside AI service, and it already had a Proxmox server of its own.
The rules were set before anything was built:
- The file store stays the system of record, and the assistant only reads it. Nothing is written, moved or deleted.
- Billing and third-party quote folders are never indexed.
- Answers meant for the customer must never contain internal or financial data.
- In production, a person's permissions are checked before any document text reaches the model.
What did we build, step by step?
-
Gave the model a home on the firm's server
A Debian virtual machine on the firm's Proxmox server, raised to 16 CPU cores and 80 GB of memory, with a 150 GB disk just for models. Passing the host's AVX2 and AVX-512 instructions through to the virtual machine was the biggest single speed-up.
-
Chose the model by testing it
Three Qwen models ran the same six tasks on 22 September 2026. Qwen3 30B-A3B (instruct) passed 6 of 6 in 18 seconds and became the default. Its reasoning version also passed 6 of 6 but took 374 seconds, because it thinks before answering. Qwen 2.5 7B passed 3 of 6; it failed the arithmetic. Separately, a 72-billion-parameter model was tried and removed because it was far too slow without a GPU. The 30B model is a mixture of experts that reads only part of itself for each piece of text it writes, so it runs faster than the smaller, dense 7B.
-
Mirrored the documents, read-only
rclone copies the project folder from the cloud file store onto the server and never writes back. For the test, that was one project folder of about 6.7 GB.
-
Read what can be read
PDF, Word, Excel and CSV (with header detection), PowerPoint, Visio, KMZ and KML, JSON and text. Photos, DWG drawings and ZIP files are skipped. Scanned PDFs are marked as needing text recognition first.
-
Extracted the text, then indexed it on the server
During the test, a script on the builder's laptop in Ontario read each file through the file store's sync app and split it into passages. A worker on the server embeds those passages with BGE-small-en-v1.5 through fastembed and stores them in PostgreSQL 17 with pgvector. The test index holds 3,279 files and 35,381 passages. The plan is to move extraction onto the server, reading from the read-only mirror.
-
Searched by meaning and by keyword
A question is matched by meaning (pgvector) and by words (PostgreSQL full-text search). The two lists are merged by reciprocal rank fusion, with at most 2 excerpts from any one document. The search on its own takes about 0.2 seconds.
-
Answered only from the excerpts, with citations
The model is told to use only the numbered excerpts, cite them inline like [1] or [2][4], quote figures and dates exactly as written, and say plainly when the excerpts don't contain the answer. A source list (title, section and path) and related documents follow each answer.
This build's answers come from the client's files, so none are printed here. Example 03 runs the same search code, prompt and model on public Canadian privacy law, and prints its answers word for word.
-
Flagged financial content in the database
Each passage carries a financial flag, and the database has a view that leaves flagged passages out. Routing customer-facing answers through that view is the next step, so keeping money out won't depend on the model obeying an instruction.
-
Added a test chat page and a test connection
A test chat page on the office network, without sign-in yet, answers questions on the server. Its document menu lists every database on the server, including the second database in the data map below. For testing, an MCP server also lets a desktop AI assistant search the documents and query the local model. That connection returns excerpt text to the assistant, which sends it wherever that assistant runs: outside Canada for a US-hosted assistant. It is a test path, to be off or restricted in production.
Which tools does it use?
- Ollama
- Qwen3 30B-A3B
- Qwen 2.5 7B
- Proxmox VE
- Debian 13
- systemd
- PostgreSQL 17
- pgvector 0.8.6
- fastembed
- BGE-small-en-v1.5
- Python
- MCP Python SDK
- rclone
- a cloud file store
- Reciprocal rank fusion
Where does each piece live?
This is the data map for the build, the same kind of map your privacy officer gets with yours.
| Piece | Where it lives | Outside AI provider? |
|---|---|---|
| Original documents | The firm's cloud file store, as before. The assistant only reads it. | No |
| Mirrored copy | The firm's own server | No |
| Passages and index | PostgreSQL with pgvector, on the firm's server | No |
| Embeddings for the test index | Made on the firm's server with fastembed | No |
| The model | Qwen3 30B-A3B on Ollama, on the firm's server | No |
| Questions through the test chat page | Answered on the firm's server | No |
| A second database on the same server | A second, curated knowledge base from the same project, built and embedded on the builder's laptop, with a US-hosted AI coding tool used in building it. It can be chosen in the test chat page and is the default for the desktop-assistant test connection. Not part of this assistant's test index. | In building it |
| Text extraction during the test | The builder's laptop in Ontario, through the file store's sync app. Embedding and search: on the server. | No |
| Desktop assistant connection (MCP), test only | Sends excerpts to wherever that assistant runs, which is outside Canada for a US-hosted assistant. Off or restricted in production. | Yes, if used |
| Builder's AI coding assistant | Claude Code is installed on the server for building and upkeep, and its model runs outside Canada. It isn't part of the assistant; what it reads depends on the task it's given. | Build tool |
| The build's code repository | A private GitHub repository. It holds a list of file paths with sizes and dates from the test folder, and topic summaries the local model generated from the curated knowledge base, but no raw document text. | No (code host) |
| Staff sign-in | Not built yet; planned with Microsoft Entra ID | Not built |
What does it do, and what doesn't it do yet?
It does
- Answer questions from the firm's project files, with numbered citations and a source list
- Answer questions asked through the test chat page on the firm's server; no outside AI provider receives them
- Flag financial passages, with a database view that leaves them out
- Say when the excerpts don't contain the answer
It doesn't, yet
- Staff sign-in and per-person permissions are planned in the production brief, not yet built. Until they are, it's treated as a test system on the office network, not a production service.
- Keep every test path on the server. The desktop-assistant connection, the laptop extraction and the second database's build didn't; all three are in the data map above.
- Route customer-facing answers through the financial view. That's the next step.
- Answer fast. With no GPU an answer takes about 1 to 2 minutes, and a long set of excerpts takes about 2 minutes before the first word.
- Read photos, DWG drawings, ZIP files, or scanned PDFs that haven't had text recognition.
- Guarantee a perfect reading. A model can misread an excerpt, which is why every answer carries its citations for you to check.
Where does it stand today?
Working prototype. The test system has run on the firm's own server since late September 2026, indexing one project folder, built and tested with one person's file-store access. The production brief, written on 26 September 2026, sets out the next three jobs:
- An inventory of the file store's metadata (about 700 GB) through its API, without downloading every document, resumable after interruption and never changing anything in the store.
- Matching each person's Microsoft Entra ID sign-in to their file-store account, with the file store staying the authority on who can see what.
- A reviewed pilot set of 20 to 50 GB, with a manifest approved before any extraction starts.
The brief lists 12 tests that must pass before production, including: a revoked user can no longer retrieve the affected file; a General user cannot retrieve Restricted or Executive files; and no document content is sent to the model before authorization succeeds.
What would the same build look like at your firm?
- Your server
- An existing machine that can run a virtual machine, or a new one. We size it on the prototype. CPU-only works; a GPU is faster.
- Or a Canadian region
- The same assistant on a virtual machine you control in a Canadian cloud region.
- Your documents
- Read-only, from where they live now. A cloud file store works today; SharePoint Online or a file server would be a new read-only connector, shown working in the prototype.
- Your rules
- Folders that are never indexed, content kept out of certain answers, and who may see what.
- The prototype
- The assistant answering from one sample folder you choose, within 5 business days, free.
- The price
- Build from $995, fixed in writing. Setting up or upgrading your own server is quoted in writing after the prototype. Support is $99 a month.
In your build, nothing outside your server receives document text unless you approve that path in writing. AI coding tools, including the one in the data map above, are used to write code only: they work on code and made-up sample files, never on your documents or excerpts from them, and the desktop-assistant test connection is off. This test system didn't keep to that everywhere; the data map shows where, and that is why the rule is written down.
Your server, cited answers, and no outside path for document text unless you approve it in writing. Book a free 30-minute call and we'll tell you what it would take on your hardware.
Questions about this build
Can a local LLM run without a GPU?
Yes. This one runs on 16 CPU cores and 80 GB of memory with no GPU. Expect about 1 to 2 minutes per answer. A dense 72-billion-parameter model was far too slow here, while a mixture-of-experts model ran faster than a smaller dense one.
Does any document text go to an outside AI provider?
Questions asked through the test chat page are answered on the firm's server; no outside AI provider receives them. Two test paths are different, and both are in the data map: the desktop-assistant connection (MCP) sends excerpts to wherever that assistant runs, and text extraction ran on the builder's laptop in Ontario through the file store's sync app.
Is this a private ChatGPT, or what people call RAG?
It's retrieval-augmented generation (RAG): the assistant first searches your documents, then a model writes the answer only from the excerpts it found, with numbered citations. The difference from a general chat tool is that it answers from your files, shows where each answer came from, and runs where you choose: your own server or a Canadian cloud region.
Which files can it read?
In this build: PDF, Word, Excel and CSV, PowerPoint, Visio, KMZ and KML, JSON and text files. Photos, DWG drawings and ZIP files were skipped, and scanned PDFs were flagged because they need text recognition first.
Can each person see only the files they're allowed to?
That's how production is designed: each person signs in with their work account, and only excerpts from files they can already open are passed to the model. In this build that step is designed but not built yet, and until it is, the system stays a test system on the office network.
How much of our file store would you index first?
A pilot, not everything. This firm's plan is a reviewed 20 to 50 GB set out of about 700 GB, chosen folder by folder, with a manifest approved before any extraction. A free prototype uses one sample folder.