The non-volatile storage system predicts which blocks (or other units of storage) will become bad based on performance data. User data in those blocks predicted to become bad can be re-programmed to other blocks, and the blocks predicted to become bad can be removed from further use.