Livt packages

Eccelerators.Tokenization

Version 0.2.0

Bounded BPE and Unigram tokenization, text normalization, and decoding components for Livt hardware.

Published files (27)

src/Storage/IStorage.lvt

namespace Eccelerators.Tokenization.Storage

/**
 * Exclusive scheduled storage with contiguous valid addresses from zero.
 * Initial cells are unspecified. Consumers initialize every reachable cell
 * before reading. Logical consumer Reset invalidates metadata without clearing cells.
 * Hardware reset must cancel provider and consumer operations together.
 * Read/Write complete before returning and may take many cycles.
 */
interface IStorage<T>
{
	/**
	 * Reports whether address is valid, in element units. Valid addresses must form a contiguous range from zero.
	 * This is a query, not a reservation; the consumer owns exclusive access.
	 */
	fn IsValidAddress(address: int) bool

	/**
	 * Waits for the element at a valid address and returns its exact stored value.
	 * The consumer must write that location before reading; initial contents are unspecified.
	 * Calls are serialized in the shared clock/reset context.
	 */
	fn Read(address: int) T

	/**
	 * Waits until the value has been stored at a valid element address.
	 * A following serialized Read must return that value. No single-cycle latency is promised.
	 */
	fn Write(address: int, value: T)
}