Taryn R.
Yelp
The views alone are worth the trek up to the castle! Take heed though, there is an easy way and a hard way up. We just got lucky and came up the easy way, went down the other way. The entire way down, all we could say was, "Thank goodness we didn't come up this way!"
So, the easy way starts up a side street off the main square, where the boats dock. You'll head uphill on a cobbled road past a gelato shop on your right and a souvenir shop on your left, toward the San Giorgio Church. Follow that road on up until it forks. To get to the castle, head left at the fork. Although, I'd recommend detouring right to see the church before continuing on to the castle. Once you take the fork towards the left, you will see ample signs pointing you to the castle. Follow those.
The other route leaves from the port and is a steep, rocky, winding trail uphill the entire way. Avoid, avoid, avoid! Unless you're coming down of course, then it's a nice, pleasant, downhill stroll with killer views the whole way.