"What is the difference between an exchange (e.g. Coinbase, Kraken, Gemini, Poloniex, Bittrex) and Coin Wallet?"
Your coins are on the blockchain, regardless of what service you use to access them. When you move them, you are sending them from one address on the blockchain to another. These are simply lines of code. Your wallet file, the user interface you interact with, the private key — these do not have funds in them. The private key gives you the ability to prove ownership over coins that are on the blockchain.
If you use a client-side tool like Coin Wallet or Mist, Metamask, Exodus, or Jaxx, then you have the private key & you control your funds and your key. You do not rely on Coinbase or Gemini sending your funds from their account to yours.
The upside is that you, and only you, control your keys. An exchange getting hacked won't affect you. The downside is that you, and only you, control your keys. No one else has them, nor can recover them, should you lose them.
If you do lose your passphrase, private key or wallet file, you cannot prove ownership of an account and therefore you cannot ever send your coins again.
If you use an exchange like Coinbase, Gemini, Kraken, Polonix, Bittrex, then you have any account with that company, and they hold your coins and your keys for you. They have their own account on the blockchain with all their and their customers' funds in it. Then you have a username / password with them, on their servers, and they keep track of how much coins they "owe" you.
This allows you to have the more traditional username / password situation and do things like reset your password if you forget it, change your password if your password is compromised, and turn on 2FA. However, it also means that if the exchange loses coins, it's your coins that is lost.
Not your keys not your coins.