Install times are more than they are on the PS4, yes. But not enough so to worry about. For example, I was able to start playing Sniper 3 on my Xbox One after about 45 seconds.
But that is the point, you don't have to install the Xbox One games when going to friends homes. If I want to go to a friends house with my games, I can unplug my HDD that has my 30 some games on it, go to my friends, plug in the USB, and play them instantly, no install, right from the game library. With the PS4, I would need to bring over all the disks, and install all the games on their PS4 before I can play them. Then if they don't want the games on their PS4 anymore, they need to go in and delete them all. With the Xbox One, I just unplug my HDD, and I am off like a prom dress. Heck, I can even copy the game from my external HDD to their HDD, they can sign into the Xbox Store and buy the game if they liked it, and they instantly have access to the game without even having to download it. They had a 10 second transfer from my HDD.
The advantages of physical comes down to price and resale. I suppose, if for any reason PSN/XBL closed up shop, and you lost the ability to redownload a game that you had chosen to uninstall for whatever reason, you would lose all access to it.