Increases in storage capacity and network bandwith coupled with
advances in audio compression technology and computational performance
have made possible the creation of large digital audio arhives
containing thousands of hours of audio. The combination of audio
signal processing and machine learning techniques has the potential to
create sophisticated tools to interact with these large
collections. Manual audio annotation forms a necessary and commonly
tedious part of this process that is frequently sidelined in the
published literature. By carefully combining basic practical ideas
from machine learning and human-computer interaction it is possible to
significanty reduce annotation time and improve the annotation
experience without affecting the quality of training data. Specific
examples from the annotation of Orca vocalizations, religious chants
and popular music are provided.