| 摘要 |
<p>SYMBOLIC TOKENIZER FOR WORDS AND PHRASES A microprocessor capable of high speed generation of tokens. The tokens are binary symbols which represent language words and phrases. The tokens may be manipulated, stored, or transmitted, instead of the language words or phrases themselves, thereby realizing a greatly increased efficiency due to the statistically reduced number of bits required to represent individual language words.</p> |
