Concepts
Build a concept
A concept is the weighted set of terms that a random walk from your chosen term reaches much sooner than usual. Pick a model and a term to build one.
What a concept is and what to do with it
- 01
A walk from the chosen term
The model is a graph: each term is linked to the terms it shares chunks with, more strongly the more often they do. A random walk starts at the chosen term and spreads out for λ steps, and gives every term a hitting time: how soon the walk reaches it.
- 02
Measured against comparison terms
The model also walked from a set of comparison terms spread across the graph (50 unless it was built with another number), so it knows how soon each term is usually reached. A term reached much sooner from the chosen term than from those gets a high z-score, and the terms above the threshold form the concept: a query expansion, ranked and weighted on one scale, where the chosen term is 100 and a term at 25 counts a quarter as much.
- 03
A tool for discovery
On its own, a concept answers “what is related to this term?” in the text the model was built from, and can turn up the needle in the haystack, a rare term tied closely to the chosen term that a keyword search would never think to ask for.
- 04
A ranked, semantic search
Those weights score any raw text directly. Each chunk scores the sum of the weights of the concept terms it contains, so a chunk can rank high without the chosen term, through the terms that mean the same thing.