Return harbour

There's talk of 'the' olive tree again, so this might be where the confusion started, because it can't be the aforementioned 'big olive tree', that one was at the head of the harbour, this one's close to the beach. Now we have multiple harbours and multiple olive trees, no wonder it got all muddled up. To complicate things further, there's also another cave down by Cape Lougi, but since Kypos beach at Cape Lougi wasn't the harbour Odysseus wanted to end up in, it couldn't logically have been part of any waypoints meant to point the way to the right harbour.

Cave Naiads

Olive tree?

Head of the harbour

As I said before, scholars always believed that the harbour of Phorkys, the head of the harbour and the cave were all in one spot. It's clear now that another harbour is entered, so Homer didn't only describe the harbour of return, he also described waypoints on the way towards it. Then Homer describes the manner in which the Phaeacians unloaded Odysseus, so to speak:

Kypos Beach

First of all they raised up Odysseus from the hollow ship, together with his linen cover and shining rug. They set him down on the sand, overcome by sleep. They lifted out the goods that the noble Phaeacians gave him for his journey home, thanks to the efforts of great-harted Athena. This they set all together beside the trunk of the olive tree, ...

Odussey Book 16 line 116-122 Barry B Powell

Presumably the whole area around Cape Lougi wasn't owned by Odysseus, since you'd expect him to recognise his own land.

Odysseus was brought to a harbour unknown to him and therefore unlikely to have been part of his personal kingdom.