Language Model (LM) agents for cybersecurity that are capable of autonomously identifying vulnerabilities and executing exploits have potential to cause real-world impact. Policymakers, model providers, and researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such agents to help mitigate cyberrisk and investigate opportunities for penetration testing. Toward that end, we introduce Cybench, a framework for specifying cybersecurity tasks and evaluating agents on those tasks. We include 40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF competitions, chosen to be recent, meaningful, and spanning a wide range of difficulties. Each task includes its own description, starter files, and is initialized in an environment where an agent can execute commands and observe outputs. Since many tasks are beyond the capabilities of existing LM agents, we introduce subtasks for each task, which break down a task into intermediary steps for a more detailed evaluation. To evaluate agent capabilities, we construct a cybersecurity agent and evaluate 8 models: GPT-4o, OpenAI o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct, Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct. For the top performing models (GPT-4o and Claude 3.5 Sonnet), we further investigate performance across 4 agent scaffolds (structured bash, action-only, pseudoterminal, and web search). Without subtask guidance, agents leveraging Claude 3.5 Sonnet, GPT-4o, OpenAI o1-preview, and Claude 3 Opus successfully solved complete tasks that took human teams up to 11 minutes to solve. In comparison, the most difficult task took human teams 24 hours and 54 minutes to solve. Anonymized code and data are available at https://drive.google.com/file/d/1kp3H0pw1WMAH-Qyyn9WA0ZKmEa7Cr4D4 and https://drive.google.com/file/d/1BcTQ02BBR0m5LYTiK-tQmIK17_TxijIy.

BACKGROUND: Pediatric hospital readmissions increase healthcare costs and highlight gaps in care. Social determinants of health (SDOH), such as housing and transportation insecurity, significantly impact outcomes but are underexplored in pediatric populations. OBJECTIVES: This study evaluates the impact of housing and transportation-related SDOH on pediatric readmissions, comparing structured ICD-10-CM Z-codes alone to a combination of structured and unstructured data extracted via natural language processing (NLP). MATERIALS AND METHODS: We conducted a retrospective cohort study of pediatric patients (ages 2-17) discharged from UCSF Benioff Children's Hospital between January 2018 and January 2023. SDOH exposure was identified using structured Z-codes and NLP-extracted data. The primary outcome was hospital readmission within 365 days. Cox proportional hazards models assessed associations between SDOH and readmission risk. RESULTS: Among 8928 patients, only 0.8% were identified as exposed using structured data, compared to 31.7% using combined data. Patients identified through combined data had a higher readmission risk (HR: 2.64, 95% CI: 2.34-2.98) compared to those identified with structured data alone (HR: 1.99, 95% CI: 1.27-3.13). ED utilization was also higher among exposed patients. In the structured-only analysis, exposed patients had a significantly higher hazard of ED readmission (HR: 2.26, 95% CI: 1.65-3.10), whereas the association was slightly attenuated in the combined analysis (HR: 1.49, 95% CI: 1.37-1.62). CONCLUSION: Leveraging unstructured data enhances SDOH identification and reveals stronger associations with hospital and ED readmissions. A hybrid approach enables improved risk stratification and targeted interventions to address pediatric health disparities.