Livt packages

Eccelerators.Tokenization

Version 0.2.0

Bounded BPE and Unigram tokenization, text normalization, and decoding components for Livt hardware.

Published files (27)

src/Unigram/IAsciiNormalization.lvt

namespace Eccelerators.Tokenization.Unigram

/**
 * A verified one-byte normalization subset. Map returns printable ASCII or -1.
 * This deliberately does not claim arbitrary Unicode normalization support.
 */
interface IAsciiNormalization
{
	/**
	 * Maps one printable ASCII input byte (0x20..0x7e) to an int in the same range.
	 * Returns -1 when unsupported or failed. The provider must establish that this one-byte
	 * subset is valid for its model; unrestricted Unicode normalization is outside this contract.
	 */
	fn Map(value: byte) int
}