Most definately there are downsides.
- Worker rotation and skill/satisfaction drops as you say SeaScourge.
- Goods in transit to the building are lost and any haulers on paths to make pick ups will stop that pick up
- Overseers who are not going to become crew will now work out new paths - to entertainment or back to the job. If they dont have affordable entertainment elsewhere they will turn tale and trudge back to the job. Could be a long walk and satisfaction, already low, gets lower.
- Forget to undemolish

Builders with time on their hands do come and demolish it..wasting their time and denying you the use of the building.
Use wisely

PS. use the trick on pirate housing to really change the game.
http://dynamic3.gamespy.com/~tropico/cafe/index.php?board=19;action=display;threadid=8158