Large Language Models (LLMs) have achieved remarkable success in Natural Language Processing (NLP) tasks, but their deployment is severely constrained by intensive computation and memory costs. Block Floating-Point (BFP) extends the dynamic range beyond INT with shared exponents, while reducing memory and alignment overhead compared to floating-point formats. However, when bit-widths are further reduced, BFP becomes sensitive to outliers; existing mixed-precision BFP methods largely rely on heuristic settings of mantissa width and block size, and the induced bit-level sparsity has yet to be systematically leveraged in hardware. In this paper, we propose AO-BFP, an adaptive BFP framework for LLM inference. At the algorithm level, we propose an adaptive outlier exponent mapping mechanism combined with mixed-precision exploration driven by layer-wise sensitivity analysis. At the hardware level, we design a reconfigurable bit-serial accelerator with a unified datapath that efficiently leverages BFP-induced bit sparsity. Compared with prior LLM accelerators such as ANT, OliVe, and BitMoD, AO-BFP achieves superior performance while preserving model accuracy, delivering speedups of 1.61×, 1.39×, and 1.11×, respectively.