Understanding the abilities of LLMs to reason about natural language plans, such as instructional text and recipes, is critical to reliably using them in decision-making systems.A fundamental aspect of plans is the temporal order in which their steps need to be executed, which reflects the underlying causal dependencies between them.We introduce CAT-BENCH, a benchmark of Step Order Prediction questions, which test whether a step must necessarily occur before or after another in cooking recipe plans.We use this to evaluate how well frontier LLMs understand causal and temporal dependencies.We find that SOTA LLMs are underwhelming (best zero-shot is only 0.59 in F1), and are biased towards predicting dependence more often, perhaps relying on temporal order of steps as a heuristic.While prompting for explanations and using few-shot examples improve performance, the best F1 result is only 0.73.Further, human evaluation of explanations along with answer correctness show that, on average, humans do not agree with model reasoning.Surprisingly, we also find that explaining after answering leads to better performance than normal chain-of-thought prompting, and LLM answers are not consistent across questions about the same step pairs.Overall, results show that LLMs' ability to detect dependence between steps has significant room for improvement. * Equal ContributionAlmond Flour Chocolate Cake … Step 6: Stir in ground almonds.Step 7: Add half flour and half milk.Step 8: Use wooden spoon to stir.… Step 12: Whip cream till stiff peaks … Q: Must Step 6 happen before Step 8? Questions about dependent steps Q: Must Step 7 happen after Step 6? Questions about non-dependent steps A: Yes, all ingredients have to be in bowl before stirring A: No, almonds can be added after flour and milk Parallel Steps Preconditions CAT-Bench Dependent Steps

Yash Kumar Lal, Niket Tandon, Tanvi Aggarwal, Horace Liu, Nathanael Chambers, Raymond Mooney, Niranjan Balasubramanian. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Qingqing Cao, Yash Kumar Lal, Harsh Trivedi, Aruna Balasubramanian, Niranjan Balasubramanian. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Online media outlets, in a bid to expand their reach and subsequently increase revenue through ad monetisation, have begun adopting clickbait techniques to lure readers to click on articles. The article fails to fulfill the promise made by the headline. Traditional methods for clickbait detection have relied heavily on feature engineering which, in turn, is dependent on the dataset it is built for. The application of neural networks for this task has only been explored partially. We propose a novel approach considering all information found in a social media post. We train a bidirectional LSTM with an attention mechanism to learn the extent to which a word contributes to the post's clickbait score in a differential manner. We also employ a Siamese net to capture the similarity between source and target information. Information gleaned from images has not been considered in previous approaches. We learn image embeddings from large amounts of data using Convolutional Neural Networks to add another layer of complexity to our model. Finally, we concatenate the outputs from the three separate components, serving it as input to a fully connected layer. We conduct experiments over a test corpus of 19538 social media posts, attaining an F1 score of 65.37% on the dataset bettering the previous state-of-the-art, as well as other proposed approaches, feature engineering or otherwise.