Architectural Decision Records (ADRs) play a central role in maintaining software architecture quality, yet many decision violations go unnoticed because projects lack both systematic documentation and automated detection mechanisms. Recent advances in Large Language Models (LLMs) open up new possibilities for automating architectural reasoning at scale. We investigated how effectively LLMs can identify decision violations in open-source systems by examining their agreement, accuracy, and inherent limitations. Our study analyzed 980 ADRs across 109 GitHub repositories using a multi-model pipeline in which one LLM primary screens potential decision violations, and three additional LLMs independently validate the reasoning. We assessed agreement, accuracy, precision, and recall, and complemented the quantitative findings with expert evaluation. The models achieved substantial agreement and strong accuracy for explicit, code-inferable decisions. Accuracy falls short for implicit or deployment-oriented decisions that depend on deployment configuration or organizational knowledge. Therefore, LLMs can meaningfully support validation of architectural decision compliance; however, they are not yet replacing human expertise for decisions not focused on code.

Background: The recent revolution in Large Language Models (LLMs) is transforming industries, enhancing communication, and reshaping research methodologies. LLMs have found significant applications across various sectors, notably in finance for stock market predictions, and in healthcare, where complex medical data is analyzed for diagnosis at an early stage, improving diagnostic procedures, and personalized treatment planning. In healthcare, where complex medical data is analyzed for diagnosis at an early stage. Despite the immense potential, challenges such as overwhelming Big Data, model hallucinations, and ethical concerns about patient privacy and bias persist. Method: We implemented novel strategies like CanPrompt to mitigate the accuracy and hallucination concerns to ensure responsible deployment. The CanPrompt strategy utilizes prompt engineering combined with few-shot and in-context learning to significantly enhance model accuracy by generating more relevant answers. The models were tested against a specialized dataset from MedQuAD, focusing on cancer, and evaluated using metrics like ROUGE and BERTScore to assess the semantic and syntactic accuracy of generated responses against validated "Gold Answers". Through this approach, the study seeks to outline the potential and limitations of LLMs in improving cancer care. Result: After applying CanPrompt with models Mistral 7x8b, Falcon 40b, and Llama 3-8b, BERTScore results showed Mistral leading with an accuracy around 84%, Falcon slightly lower, and Llama the least, with respective precision scores also reflecting a similar trend. Conclusion: The study demonstrates the promise of LLMs in cancer care through the introduction of CanPrompt.