Correct…unlike an RSP, you can withdrawal from your TFSA in any given year and re-contribute this amount back in subsequent years. All contributions & withdrawals within a TFSA are tracked by the FI and then reported to CRA to adjust your contribution room.
BUT you are absolutely correct…if you have multiple TFSAs you will really need to watch all transactions, as it isn't hard to make a mistake and over contribute and be subject to CRA penalties!