emily.index_classes.CooccurrenceIndex

Created on Thu May 7 08:43:03 2026

@author: Dr Peter J Bleackley

Attributes

stop

Classes

CooccurrenceIndex

Uses information theory to identify documents in which the query terms

Module Contents

emily.index_classes.CooccurrenceIndex.stop

class emily.index_classes.CooccurrenceIndex.CooccurrenceIndex(path: pathlib.Path | None = None)

Uses information theory to identify documents in which the query terms tend to occur in the same sentences. See this page for details of the algorithm.

async add_documents(corpus: collections.abc.AsyncIterable[tuple[str, list[list[str]]]])

Adds documents to the index

Parameters:

corpus (AsyncIterable[tuple[str,list[list[str]]]]) – Iterable of tuples of filename, and the document as a list of list of strings (parsed sentences)

Return type:

None.

__call__(query: list[list[str]], top_k: int = 10) → polars.LazyFrame

Finds candidate documents where the words in the query tend to cooccur. Uses the Pareto principal to automatically threhold the results. For N candidate documents Np results will be returned, such that the account for (1-p) of the total relevance of the sample

Parameters:

• query (list[list[str]]) – The query as a list of lists of strings, each list representing a sentence

• top_k (int, optional) – The number of results to return. Default is 10

Returns:

Contains a single column, “filename”

Return type:

pl.LazyFrame

save(path: pathlib.Path)

Saves the indices to parquet files

Parameters:

path (Path) – Directory to save indices in

Return type:

None.

async clear()

Clears indices

Return type:

None.