Smart contracts are integral to blockchain's growth, but their vulnerabilities pose a significant threat. Traditional vulnerability detection methods rely heavily on expert-defined complex rules that are labor-intensive and dificult to adapt to the explosive expansion of smart contracts. Some recent studies of neural network-based vulnerability detection also have room for improvement. Therefore, we propose SCVHunter, an extensible framework for smart contract vulnerability detection. Specifically, SCVHunter designs a heterogeneous semantic graph construction phase based on intermediate representations and a vulnerability detection phase based on a heterogeneous graph attention network for smart contracts. In particular, SCVHunter allows users to freely point out more important nodes in the graph, leveraging expert knowledge in a simpler way to aid the automatic capture of more information related to vulnerabilities. We tested SCVHunter on reentrancy, block info dependency, nested call, and transaction state dependency vulnerabilities. Results show remarkable performance, with accuracies of 93.72%, 91.07%, 85.41%, and 87.37% for these vulnerabilities, surpassing previous methods.