You want to be cleaning the vent before excessive accumulation occurs, otherwise you begin flirting with the fire hazard. This means that if you follow the industry recommendations for intervals between cleaning (typically 1 year) you probably won't experience a dramatic change in drying times or energy usage (unless birds get in your vent!), but you will be following 'best practice' in order to mitigate the fire hazard.
It is always satisfying when I leave a home knowing that the owner(s) will experience substantially shorter drying times. It is impressive removing large amounts of lint or multiple feet of a bird nest because I know that there will be a drastic difference in airflow before and after cleanings. However, truth be told, I am much more satisfied leaving a home that has been maintained with regular cleanings and has removed marginal accumulation because I know that the family, house, and appliance are being safeguarded.
There is a reason that cleanings are recommended annually, and not every 5 or 10 years.