When linking one object to another through the 'Select an object' window, the left panel set is used to filter the list of objects in the main panel. The middle filter sub-panel in the left panel is for filtering by object type. When selecting an object type, the main panel goes into an 'loading' mode, and when it completes its 'loading', the list isn't filtered at all.
After playing with this some more. It only filters for the 'Contacts' type. But once the list has been filtered by 'Contacts', switching to a different filter doesn't change the result list. You need to click back on 'All' to get the other types back.