<p>An accessory tool retention assembly (100) for a rotary tool (10) includes a collet (104) having a body defining a collet axis (P) and a plurality of flexible segments (130) that extend longitudinal along the body generally parallel to the collet axis (P). Each flexible segment (130) includes a curved end portion (138) that curves radially outwardly from the collet body. A collet chuck (108) is configured to be coupled to an output shaft (34) of a motor (24) of a rotary tool (10). The collet chuck (108) defines a collet receiving passage (162) configured to slidably receive the collet (104). A detent structure is positioned to block the collet (104) from exiting the collet receiving passage (162) while the flexible segments (130) of the collet (104) are in a relaxed state.</p>