SaaSScout: A Grounded RAG Copilot for SaaS Evaluation.

Retrieval-Augmented Generation (RAG) —

Index real product knowledge instead of prompting around ignorance.

A vanilla LLM prompt for "compare Zendesk and Freshdesk" produces a plausible-sounding feature matrix drawn from training data that may be months or years stale. SaaSScout's answer is to not ask the LLM about products at all. Instead, 335 real product records are ingested, canonically normalized, and embedded into Chroma before any user query arrives. The LLM receives a prompt that already contains retrieved, ranked evidence — its role is to assemble readable output, not to recall facts. If the data does not support a claim, the claim does not appear. The RAG constraint is the product's entire trust argument.

Evidence lane architecture —

Partition embeddings into separate Chroma collections by source trust.

The default RAG approach drops all documents into one vector collection and lets semantic search surface the best match. The problem is that user reviews, enterprise metadata, and vendor-reported features have very different trust profiles: Capterra reviews are subjective but reveal real pain points; FactGrid metadata is verified but narrow; Wikidata vendor facts are CC0 and reliable but sparse; OpenAlternative surfaces open-source options that commercial indexes miss. Merging them into one corpus lets high-volume review text dominate cosine similarity, drowning out the structured metadata signals. SaaSScout keeps four separate Chroma collections — one per lane — so retrieval is independently tuned per source and output is labeled by origin. A TF-IDF fallback index mirrors each collection so retrieval degrades gracefully when the vector store is cold.

Multi-signal pre-generation ranking —

Score six signals before the LLM sees a single candidate.

Semantic similarity alone is a weak procurement ranking signal. A tool can embed close to the query keyword "enterprise ticketing" while failing on pricing, missing required features, or carrying uniformly negative reviews. SaaSScout scores every retrieval candidate across six dimensions before assembling the LLM prompt: feature-fit coverage (what fraction of required features are confirmed), pricing match (distance from the stated budget), review sentiment (aggregated Capterra rating), provenance trust (FactGrid weighted highest, then Wikidata, then reviews, then alternatives), query-keyword alignment (TF-IDF overlap with the raw query string), and category overlap (primary category match). The composite score governs prompt slot assignment — higher-ranked tools get more evidence real estate — so the generation step is biased toward the strongest candidates before any text is written.

Epistemic honesty in output design —

Treat missing evidence as a procurement finding, not a model failure.

When the retrieval layer returns no Wikidata record for a vendor, or no pricing data for an enterprise tier, the typical AI response is to either hallucinate a placeholder or silently omit the row. Both are wrong for procurement. SaaSScout renders missing evidence as an explicit cell in the evidence table, tagged with the lane it came from. A blank pricing cell is a procurement red flag — it usually signals "contact sales" or a non-public tier — and surfacing it as a gap gives the analyst the right signal: this is something you must verify before buying. The design rule is that absence of evidence is itself evidence, and the product makes it visible rather than papering over it.

Provider-neutral LLM layer: Groq Qwen 32B → Ollama Qwen 2.5 1.5B → deterministic template.

Tying a RAG pipeline to one LLM provider is a reliability risk. Groq's hosted Qwen 32B is fast — sub-2-second inference on the structured prompt — but rate-limited under demo traffic. Ollama's local Qwen 2.5 1.5B is available offline and avoids API costs, but slower and quantized. The pipeline checks availability in order: Groq first, Ollama second, deterministic grounded template last. The template fallback is not a degraded mode: it uses the same six-signal ranking and the same retrieved evidence, formats them into the same scorecard and recommendation memo, and delivers a procurement-ready output without any LLM call. Because the LLM is positioned as an assembler and narrator rather than a knowledge source, the fallback degrades in fluency but never in factual grounding.

Packaged artifact, same-origin proxy, and scheduled smoke monitor for zero-dollar reliability.

Three infrastructure decisions compound into production reliability on a free-tier stack. The processed Chroma index and normalized data are packaged as a versioned zip in a GitHub Release and pulled at backend startup via DATA_ARTIFACT_URL — dropping cold-start rebuild time from 5 minutes (Kaggle re-download plus Chroma rebuild) to roughly 30 seconds. Frontend API calls route through a Netlify /api/* edge proxy instead of cross-origin requests, collapsing CORS preflight latency and enabling same-origin caching. A scheduled GitHub Actions workflow hits /health, /api/status, and a low-cost template /api/analyze to keep the Render dyno warm during business hours and reports the first failing layer — Netlify, Render, Chroma, enrichment, or analyze — on failure. The combination delivers observable, warm, reproducible infrastructure at $0.