Semantic communication is a novel communication paradigm focusing on the transmission of meaningful, task-oriented information. Recent results have shown that graphical structures represent the most robust and structurally faithful formalism for modeling the semantic knowledge within a wide range of source signals. However, previous solutions focus primarily on the pairwise relational graphs, which inherently lack the capacity to encapsulate complex, higher-order interactions fundamental to specific semantic contexts. In contrast, hypergraphs provide a more flexible mathematical framework that can more accurately map the multidimensional dependencies found in intricate data sources. In this paper, we investigate hypergraph-based semantic representation for the semantic communication system. We propose a novel hypergraph information bottleneck-based implicit semantic communication framework (HIB-SC), in which the semantic encoder is developed and optimized to extract the minimally sufficient representation of the hypergraph-based semantic information source, that maximizes the mutual information between the encoded representation and the implicit high-order semantic relations that are intended for the receiver. We theoretically prove that the proposed framework is able to extract the most informative subgraphs or motifs that are significantly more robust against adversarial attacks with improved generalization performance. Extensive experiments verify that the proposed HIB-SC achieves superior semantic compression efficiency, higher accuracy in implicit semantic inference, and enhanced resilience against noise, compared to the state-of-the-art solutions.