Livt packages
Eccelerators.Tokenization
Version 0.2.0
Bounded BPE and Unigram tokenization, text normalization, and decoding components for Livt hardware.
Published files (27)
src/Unigram/IUnigramStorage.lvt
namespace Eccelerators.Tokenization.Unigram
/**
* Exclusive scheduled 64-bit word storage for Unigram working state.
* Read/Write complete before returning; valid writes must retain exact bits.
* No initial content is promised. Hardware context reset cancels outstanding transactions;
* callers reset the segmenter and provider together. Logical segmenter Reset only invalidates
* metadata and must not race a transaction. IsValidAddress must be
* true for each word from zero through the requested final address.
*/
interface IUnigramStorage
{
/**
* Reports whether address is valid, in 64-bit word units. Valid addresses must form a contiguous range from zero.
* This is a query, not a reservation; the consumer owns exclusive access.
*/
fn IsValidAddress(address: int) bool
/**
* Waits for the 64-bit word at a valid address and returns its exact stored value.
* The consumer must write that location before reading; initial contents are unspecified.
* Calls are serialized in the shared clock/reset context.
*/
fn Read(address: int) logic[64]
/**
* Waits until the value has been stored at a valid 64-bit word address.
* A following serialized Read must return that value. No single-cycle latency is promised.
*/
fn Write(address: int, value: logic[64])
}