The conversion of speech can be used to transform an utterance by a source speaker to match the speech characteristic of a target speaker. During a training phase, utterances corresponding to the same sentences by both the target speaker can source speaker can be force aligned according to the phonemes within the sentences. A target codebook and source codebook as well as a transformation between the two can be trained. After the completion of a training phase, a source utterance can be divided into entries in the source codebook and transformed into entries in the target codebook. During the transformation, the situation arises where a single source codebook entry can have several target codebook entries. The number of entries can be reduced with the application of confidence measures.