Ahhhh I don't use the timer in tasks, therefore I never see this.
It looks to me like you can't readily prevent clients (who can see this workspace) from seeing that. If you know how to code, you could probably come up with a workaround, otherwise you could post a feature request.
What you are saying makes sense to me, but permissions are not yet granular enough.