Efficient computation of character offsets for token-oriented representation of program code

摘要

An editor, software engineering tool or collection of such tools may be configured to encode (or employ an encoding of) an insertion point in both token-coordinates and character-coordinates. Efficient implementations of insert, remove and replace operations that employ and maintain such a representation are described herein. Some realizations further maintain a total buffer size encoding consistent with each such operations. Computational costs of such operations typically scale at worst with the size of fragments inserted into and/or removed from such a token-oriented representation, rather than with buffer size. Accordingly, such implementations are particularly well-suited to providing efficient support for programming tool environments in which a token stream is updated incrementally in correspondence with user edits.