DIGITAL PROCESSING METHOD/SYSTEM/DEVICE AND PROGRAM STORAGE DEVICE

摘要

PROBLEM TO BE SOLVED: To reduce the error of conversion data by giving a step for forming a highly precise difference, a step for operating a clipped integer and a step for adding the operated integer to the highly precise difference and a step for generating an operated highly precise number after the completion of a processing stage. SOLUTION: Highly precise data 130 is converted into an integer and is clipped (140). Then, conversion data 150 is generated. Since conversion into the integer and a clipping stage 140 are executed before an operation for receiving integer input 150, data output 340 from an operation 310 contains at least small inaccuracy by the error of the result. The inaccuracy is removed by subtracting conversion data 150 from the highly precise value 130 and forming a highly precise difference 1120. The difference 1120 is added to output 340 after the operation and therefore an error in conversion into the integer and clipping is reduced or removed. Then, a highly precise number 1140 after the operation is formed.