This is actually how I envisioned his mind palace for about six months seeing the posts on Pinterest and Tumblr about some creepy looking dudes "mind palace" until I actually got around to watching the show. I thought he was gay. Turned out I was right.
I don't know why, but I imagine Hogwarts being his Mind Palace. And the Room of Requirement is where he puts knowledge that requires a specific case....like knowledge of stars and what dates they could be seen.