Incorrect usages of Application Programming Interfaces (APIs) may lead to unexpected problems during the software development process. Although there have been many attempts to address API-misuse issues, most of them are mainly for static languages. In contrast, API misuses in dynamic languages are rarely covered, mostly due to challenges about dynamic features. In this paper, we develop the first-ever comprehensive study of API misuses for Python programs. To accomplish this, we manually analyze 79,096 commits of six popular open-source Python projects on GitHub to collect true-positive cases. Based on the validation, we develop a classification of Python API Misuses, called PAM, and a dataset, PAMBench, containing 670 validated real-world API-misuse cases in popular Python programs. For each API-misuse case, we explore its root cause, symptom, program issue and repair method. Specifically, we pay attention to the effect of dynamic features on API usages in Python. The systematic study on PAMBench shows that, most importantly, dynamic features, especially type dynamics, have a non-negligible impact on API usages in Python, mainly related to incorrect assumptions about the type, callable state, attribute and existence of caller object, method call itself, passed argument(s) and return value during an API invocation. Our root-cause analysis reveals the importance of correct design, implementation, annotation, checking and recording about the types and states of all parts of API method calls during Python program development. Finally, we present possible solutions for more secure, reliable and maintainable API usages in Python.

APIs play a crucial role in contemporary software development, streamlining implementation and maintenance processes. However, improper API usage can result in significant issues such as unexpected outcomes, security vulnerabilities and system crashes. To detect API misuses, current methods primarily rely on comparing established API usage patterns with target points for automated detection, mainly based on pre-validated datasets. Nonetheless, there is a scarcity of publicly available datasets on API misuses and their corresponding fixes, which hinders data-driven research. Moreover, most existing techniques concentrate on statically typed languages, such as Java and C, with only a few addressing dynamic languages like Python effectively, due to difficulties in handling dynamic features. Therefore, it is essential to identify Python API misuses and their fixes automatically and promptly. In this paper, we introduce HatPAM, a Hybrid Analysis and Attention-based Python API-Misuse Miner, which (a) provides a method for automatically mining true-positive commits related to Python API-misuse fixes from GitHub and (b) presents the subsequent processing for classifying Python API misuses in true-positive cases. Particularly, HatPAM applies hybrid static analysis and introduces a structure-based attention mechanism to examine syntax, semantics and structural features in Python code context, and considers the consistency between code and developers’ natural intent to significantly reduce false-positive cases. Evaluation on six popular Python projects reveals that HatPAM outperforms various state-of-the-art baselines, achieving up to 92.2% Precision, 86.7% Recall and 89.3% F1-score, indicating its capability to identify and classify Python API-misuse commits.

When changing code, developers sometimes neglect updating the related comments, bringing inconsistent or outdated comments. These comments increase the cost of program understanding and greatly reduce software maintainability. Researchers have put forward some solutions, such as CUP and HEBCUP, which update comments efficiently for simple code changes (i.e. modifying of a single token), but not good enough for complex ones. In this paper, we propose an approach named HatCUP (Hybrid Analysis and Attention based Comment UPdater), to provide a new mechanism for comment updating task. HatCUP pays attention to hybrid analysis and information. First, HatCUP considers the code structure change information and introduces a structure-guided attention mechanism combined with code change graph analysis and optimistic data flow dependency analysis. With a generally popular RNN-based encoder-decoder architecture, HatCUP takes the action of the code edits, the syntax, semantics and structure code changes, and old comments as inputs and generates a structural representation of the changes in the current code snippet. Furthermore, instead of directly generating new comments, HatCUP proposes a new edit or non-edit mechanism to mimic human editing behavior, by generating a sequence of edit actions and constructing a modified RNN model to integrate newly developed components. Evaluation on a popular dataset demonstrates that HatCUP outperforms the state-of-the-art deep learning-based approaches (CUP) by 53.8% for accuracy, 31.3% for recall and 14.3% for METEOR of the original metrics. Compared with the heuristic-based approach (HEBCUP), HatCUP also shows better overall performance.

Python is a popular typical dynamic programming language. In Python, dynamic typing is one of the most critical dynamic features. The lack of type information is likely to hinder the maintenance of Python projects. However, existing work has seldom focused on studying the impact of Python dynamic typing on project maintenance. This paper focuses on the two most common practices of Python dynamic typing, i.e. inconsistent-type assignments (ITA) and inconsistent variable types (IVT). Two approaches are proposed to identify ITA and IVT, i.e. identifying ITA by analyzing Abstract Syntax Trees and comparing identifiers types and identifying IVT by constructing a type dependency graph. In empirical experiments, we first locate the usage of ITA and IVT in 10 open-source Python projects. Then, we investigate the relations between the occurrence of ITA and IVT and the results of maintenance tasks. The study results show that projects are more prone to change as the number of dynamic typing identifiers increases. There is a weak connection between change-proneness and variable dynamic typing. There is a high probability that maintenance time and the acceptance of commits decrease as dynamic typing identifiers increase in projects. These results implicate that dynamic and static variables should be divided while developing new programming languages. Dynamic typing identifiers may not be the direct root causes for most software bugs. The categories of these bugs are worth exploring.

API recommendation in real-time is challenging for dynamic languages like Python. Many existing API recommendation techniques are highly effective, but they mainly support static languages. A few Python IDEs provide API recommendation functionalities based on type inference and training on a large corpus of Python libraries and third-party libraries. As such, they may fail to recommend or make poor recommendations when type information is missing or target APIs are project-specific. In this paper, we propose a novel approach, PyART, to recommend APIs for Python programs in real-time. It features a light-weight analysis to derives so-called optimistic data-flow, which is neither sound nor complete, but simulates the local data-flow information humans can derive. It extracts three kinds of features: data-flow, token similarity, and token co-occurrence, in the context of the program point where a recommendation is solicited. A predictive model is trained on these features using the Random Forest algorithm. Evaluation on 8 popular Python projects demonstrates that PyART can provide effective API recommendations. When historic commits can be leveraged, which is the target scenario of a state-of-the-art tool ARIREC, our average top-1 accuracy is over 50% and average top-10 accuracy over 70%, outperforming APIREC and Intellicode (i.e., the recommendation component in Visual Studio) by 28.48%-39.05% for top-1 accuracy and 24.41%-30.49% for top-10 accuracy. In other applications such as when historic comments are not available and cross-project recommendation, PyART also shows better overall performance. The time to make a recommendation is less than a second on average, satisfying the real-time requirement.

JavaScript plays an important role in web applications and services, which is used by millions of web pages in optimizing interface design, embedding dynamic texts, reading and writing HTML elements, validating form data, responding to browser events, controlling cookies and much more. However, since JavaScript is cross-platform and can be executed dynamically, it has been a major vehicle for web-based attacks. Existing solutions work by performing static analysis or monitoring program execution dynamically. However, since the heavy use of obfuscation techniques, many methods no longer apply to malicious JavaScript code detection, and it has been a huge challenge to de-obfuscate obfuscated malicious JavaScript code accurately. In this paper, we propose a hybrid analysis method combining static and dynamic analysis for detecting malicious JavaScript code that works by first conducting syntax analysis and dynamic instrumentation to extract internal features that are related to malicious code and then performing classification-based detection to distinguish attacks. In addition, based on code instrumentation, we propose a new method which can deobfuscate part of obfuscated malicious JavaScript code accurately. Ultimately, we implement a browser plug-in called MJDetector and perform evaluation on 450 real web pages. Evaluation results show that our method can detect malicious JavaScript code and de-obfuscate obfucation effectively and efficiently. Specifically, MJDetector can detect JavaScipt attacks in current web pages with high accuracy 94.76% and de-obfuscate obfuscate code of specific types with accuracy 100% whereas the base line method can only detect with accuracy 81.16% and has no capacity of de-obfuscation.