Human-robot interaction (HRI) relies on accurate human-object interaction (HOI) recognition to enable collaborative behaviors. While HOI recognition has been considered for enabling intuitive robot interactions, video-based implementations face challenges in handling complex object-pose variations and temporal dynamics. This work proposes a Multi-stream Multi-feature Spatial-Temporal Network (M2ST-Net) that synergistically integrates visual, geometric, and global relational features. The framework leverages Faster R-CNN-extracted ROI features for visual perception, while dedicated geometric and global encoders respectively capture human-object spatial dynamics and entity positional relationships. Evaluations on MPHOI-72, CAD-120, and Bimanual Actions demonstrate superior performance over existing methods. HRI validation on dual-arm robots confirms practical applicability of our methods. Code is available at https://github.com/Cyber-BoBo/M2ST-Net.git.

Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multimodal alignment data have inspired numerous researches on synthetic VLM data generation.The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation.In this paper, we present World to Code (W2C), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format.The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy.Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs.Further analysis also demonstrates that the new code parsing ability of VLMs presents better crossmodal equivalence than the commonly used detail caption ability.Our code is available at https://github.com/foundation-multimodal- models/World2Code.

Existing image-text modality alignment in Vision Language Models (VLMs) treats each text token equally in an autoregressive manner. Despite being simple and effective, this method results in sub-optimal cross-modal alignment by over-emphasizing the text tokens that are less correlated with or even contradictory with the input images. In this paper, we advocate for distinct contributions for each text token based on its visual correlation. Specifically, we present by contrasting image inputs, the difference in prediction logits on each text token provides strong guidance of visual correlation. We therefore introduce Contrastive Alignment (CAL), a simple yet effective re-weighting strategy that prioritizes training visually correlated tokens. Our experimental results demonstrate that CAL consistently improves different types of VLMs across different resolutions and model sizes on various benchmark datasets. Importantly, our method incurs minimal additional computational overhead, rendering it highly efficient compared to alternative data scaling strategies.

Though offering amazing contextualized tokenlevel representations, current pre-trained language models take less attention on accurately acquiring sentence-level representation during their self-supervised pre-training.However, contrastive objectives which dominate the current sentence representation learning bring little linguistic interpretability and no performance guarantee on downstream semantic tasks.We instead propose a novel generative self-supervised learning objective based on phrase reconstruction.To overcome the drawbacks of previous generative methods, we carefully model intra-sentence structure by breaking down one sentence into pieces of important phrases.Empirical studies show that our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods not only on the STS benchmarks, but also on downstream semantic retrieval and reranking tasks.