A system and method for automatically operating a camera includes a controller for controlling the movement of a robotically controlled camera. The controller includes a processor and a computer readable medium having instructions for controlling the camera. The system determines a task to be performed and, based on the determined task, determines a camera control scheme that corresponds to the task. The camera control scheme includes rules for controlling the camera. The system receives input parameters in the form of particular tools being used and the position and movement of the tools. The input parameters can be automatically determined by the controller via a video image produced by the camera. The controller will move the camera in response to particular movement of the tools and the particular rules of the determined control scheme that correspond to the particular movements.
