There is not much known about the origins of the inhabitants of Pangkor. The local inhabitants believed that the island was protected by the spirits, so they called Pulau Pangkor the Spirit Island.
Another name for Pulau Pangkor in the old days was Dinding, which means 'screen' or 'partition'. This was in reference to the position of the island as it protects the mainland's estuary.
Over the years the island was frequently visited by pirates who were roaming the seas around the island. They robbed boats in the Straits of Malacca and hide themselves on the hills on the island. The names of Batu Lanun meaning Pirates' Cave and Batu Perompak meaning Pirates' Rock echo the memory of the pirates.
When the Dutch came in 1670, they build a Fort of which the remains still stand. Dutch records referred to it as the Dindings fort ('Dingdingh') - named after the Dindings River which it faced on the coast of the Peninsula. Read more on the history of the Dutch Fort.