To implement checking out when editing you could do something like this:
On FilesController (application/controllers/FilesController.class.php), on methods add_document and add_presentation, when the document is being edited (i.e. if (get_id() > 0)) you should check if the file has been checked out by another user ($file->getCheckedOutById() > 0 && $file->getCheckedOutById() != logged_user()->getId()) after other checks like if file exists and user can edit file. If it has been checked out, then show an error message, otherwise checkout the file for the logged user ($file->checkOut(true, logged_user())).
Then you should add an action somewhere to check in the file, for example, on the document edition toolbar ('application/views/files/add_document.php' around line 25). You could add another page action named "Check in" that sends a new parameter in the POST. Then on function save_document of FilesController you should check this parameter so that you checkin the file ($file->setCheckedOutById(0)) and then leave the document editor (ajx_current('back')).
Or you can wait for version 1.4 when there'll be an option to force the user to choose between checking out a document for edition or only viewing.