Multimodal Continual Instruction Tuning (MCIT) aims to enable Multimodal Large Language Models (MLLMs) to incrementally learn new tasks without catastrophic forgetting, thus adapting to evolving requirements. In this paper, we explore the forgetting caused by such incremental training, categorizing it into superficial forgetting and essential forgetting. Superficial forgetting refers to cases where the model’s knowledge may not be genuinely lost, but its responses to previous tasks deviate from expected formats due to the influence of subsequent tasks’ answer styles, making the results unusable. On the other hand, essential forgetting refers to situations where the model provides correctly formatted but factually inaccurate answers, indicating a true loss of knowledge. Assessing essential forgetting necessitates addressing superficial forgetting first, as severe superficial forgetting can conceal the model’s knowledge state. Hence, we first introduce the Answer Style Diversification (ASD) paradigm, which defines a standardized process for data style transformations across different tasks, unifying their training sets into similarly diversified styles to prevent superficial forgetting caused by style shifts. Building on this, we propose RegLoRA to mitigate essential forgetting. RegLoRA stabilizes key parameters where prior knowledge is primarily stored by applying regularization to LoRA’s weight update matrices, enabling the model to retain existing competencies while remaining adaptable to new tasks. Experimental results demonstrate that our overall method, SEFE, achieves state-of-the-art performance.

The research of class-incremental semantic segmentation (CISS) seeks to enhance semantic segmentation methods by enabling the progressive learning of new classes while preserving knowledge of previously learned ones. A significant yet often neglected challenge in this domain is class imbalance. In CISS, each task focuses on different foreground classes, with the training set for each task exclusively comprising images that contain these currently focused classes. This results in an overrepresentation of these classes within the single-task training set, leading to a classification bias towards them. To address this issue, we propose a novel CISS method named STAR, whose core principle is to reintegrate the missing proportions of previous classes into current single-task training samples by replaying their prototypes. Moreover, we develop a prototype deviation technique that enables the deduction of past-class prototypes, integrating the recognition patterns of the classifiers and the extraction patterns of the feature extractor. With this technique, replay can be accomplished without using any storage to save prototypes. Complementing our method, we devise two loss functions to enforce cross-task feature constraints: the Old-Class Features Maintaining (OCFM) loss and the Similarity-Aware Discriminative (SAD) loss. The OCFM loss is designed to stabilize the feature space of old classes, thus preserving previously acquired knowledge without compromising the ability to learn new classes. The SAD loss aims to enhance feature distinctions between similar old and new class pairs, minimizing potential confusion. Our experiments on two public datasets, Pascal VOC 2012 and ADE20 K, demonstrate that our STAR achieves state-of-the-art performance.

Existing Continual Semantic Segmentation (CSS) methods effectively address the issue of background shift in regular training samples. However, this issue persists in exemplars, i.e. , replay samples, which is often overlooked. Each exemplar is annotated only with the classes from its originating task, while other past classes and the current classes during replay are labeled as background . This partial annotation can erase the network’s knowledge of previous classes and impede the learning of new classes. To resolve this, we introduce a new method named Trace Back and Go Ahead (TAGA), which utilizes a backward annotator model and a forward annotator model to generate pseudo-labels for both regular training samples and exemplars, aiming at reducing the adverse effects of incomplete annotations. This approach effectively mitigates the risk of incorrect guidance from both sample types, offering a comprehensive solution to background shift . Additionally, due to a significantly smaller number of exemplars compared to regular training samples, the class distribution in the sample pool of each incremental task exhibits a long-tailed pattern, potentially biasing classification towards incremental classes. Consequently, TAGA incorporates a class-equilibrium sampling strategy that adaptively adjusts the sampling frequencies based on the ratios of exemplars to regular samples and past to new classes, counteracting the skewed distribution. Extensive experiments on two public datasets, Pascal VOC 2012 and ADE20K, demonstrate that our method surpasses state-of-the-art methods. • Proposes a method to address background shift problems in all training samples. • Utilizes annotators to complete missing annotations for past and new classes. • Implements a class-equilibrium sampling strategy to long-tail challenges. • Demonstrates superior performance of TAGA over the state-of-the-art CSS methods.

Few-shot segmentation (FSS) methods aim to segment objects using only a few pixel-level annotated samples. Current approaches either derive a generalized class representation from support samples to guide the segmentation of query samples, which often discards crucial spatial contextual information, or rely heavily on spatial affinity between support and query samples, without adequately summarizing and utilizing the core information of the target class. Consequently, the former struggles with fine detail accuracy, while the latter tends to produce errors in overall localization. To address these issues, we propose a novel FSS framework, CCFormer, which balances the transmission of core semantic concepts with the modeling of spatial context, improving both macro and micro-level segmentation accuracy. Our approach introduces three key modules: 1) the Concept Perception Generation (CPG) module, which leverages pre-trained category perception capabilities to capture high-quality core representations of the target class; 2) the Concept-Feature Integration (CFI) module, which injects the core class information into both support and query features during feature extraction; and 3) the Contextual Distribution Mining (CDM) module, which utilizes a Brownian Distance Covariance matrix to model the spatial-channel distribution between support and query samples, preserving the fine-grained integrity of the target. Experimental results on the PASCAL-$5^{i}$and COCO-$20^{i}$datasets demonstrate that CCFormer achieves state-of-the-art performance, with visualizations further validating its effectiveness. Our code is available at github.com/lourise/ccformer.

Salient instance segmentation (SIS) is an emerging field that evolves from salient object detection (SOD), aiming at identifying individual salient instances using segmentation maps. Inspired by the success of dynamic convolutions in segmentation tasks, this article introduces a keypoints-based SIS network (KepSalinst). It employs multiple keypoints, that is, the center and several peripheral points of an instance, as effective geometrical guidance for dynamic convolutions. The features at peripheral points can help roughly delineate the spatial extent of the instance and complement the information inside the central features. To fully exploit the complementary components within these features, we design a differentiated patterns fusion (DPF) module. This ensures that the resulting dynamic convolutional filters formed by these features are sufficiently comprehensive for precise segmentation. Furthermore, we introduce a high-level semantic guided saliency (HSGS) module. This module enhances the perception of saliency by predicting a map for the input image to estimate a saliency score for each segmented instance. On four SIS datasets (ILSO, SOC, SIS10K, and COME15K), our KepSalinst outperforms all previous models qualitatively and quantitatively.

Existing continual image classification methods demonstrate that samples from all sequences of continual classification tasks contain common (task-invariant) features and class-specific (task-variant) features that can be decoupled for classification tasks. However, the existing feature decomposition strategies only focus on individual tasks while neglecting the essential cues that the relationship between different tasks can provide, thereby hindering the improvement of continual image classification results. To address this issue, we propose an Adversarial Contrastive Continual Learning (ACCL) method that decouples task-invariant and task-variant features by constructing all-round, multi-level contrasts on sample pairs within individual tasks or from different tasks. Specifically, three constraints on the distribution of task-invariant and task-variant features are included, i.e., task-invariant features across different tasks should remain consistent, task-variant features should exhibit differences, and task-invariant and task-variant features should differ from each other. At the same time, we also design an effective contrastive replay strategy to make full use of the replay samples to participate in the construction of sample pairs, further alleviating the forgetting problem, and modeling cross-task relationships. Through extensive experiments on continual image classification tasks on CIFAR100, MiniImageNet and TinyImageNet, we show the superiority of our proposed strategy, improving the accuracy and with better visualized outcomes.

Existing class-incremental semantic segmentation (CISS) methods mainly tackle catastrophic forgetting and background shift, but often overlook another crucial issue. In CISS, each step focuses on different foreground classes, and the training set for a single step only includes images containing pixels of the current foreground classes, excluding images without them. This leads to an overrepresentation of these foreground classes in the single-step training set, causing the classification biased towards these classes. To address this issue, we present STAR, which preserves the main characteristics of each past class by storing a compact prototype and necessary statistical data, and aligns the class distribution of single-step training samples with the complete dataset by replaying these prototypes and repeating background pixels with appropriate frequency. Compared to the previous works that replay raw images, our method saves over 100 times the storage while achieving better performance. Moreover, STAR incorporates an old-class features maintaining (OCFM) loss, keeping old-class features unchanged while preserving sufficient plasticity for learning new classes. Furthermore, a similarity-aware discriminative (SAD) loss is employed to specifically enhance the feature diversity between similar old-new class pairs. Experiments on two public datasets, Pascal VOC 2012 and ADE20K, reveal that our model surpasses all previous state-of-the-art methods.

Prior studies suggest that emotional empathy is one of the components of intercultural sensitivity - the affective dimension under the concept of intercultural communication competence. Based on existing theories and findings, this paper reports a randomised parallel longitudinal study investigating the use of virtual reality (VR) exposure to enhance intercultural sensitivity. A total of 80 participants (36 females and 44 males) joined the study and were included in the data analysis. The participants were randomly assigned to the VR group, the video group, and the control group. Their intercultural sensitivity was measured three times: one week before the exposure ($T_{1}$), right after the exposure ($T_{2}$), and three weeks after the exposure ($T_{3}$). The results suggested that (1) the intercultural sensitivity of the VR group was significantly enhanced in both within-subject comparisons and between-subject comparisons, (2) there were no significant differences in intercultural sensitivity between the VR group and the video group at $T_{2}$, but the VR group retained the enhancement better at $T_{3}$, and (3) the sense of presence and emotional empathy well predicted the change in intercultural sensitivity of the VR group. The results, together with the participants' feedback and comments, provide new insights into the practice of using VR for intercultural sensitivity training and encourage future research on exploring the contributing factors of the results.

A large number of industrial video applications are expected to work in real-time and power-constrained scenarios. For such applications, the coding complexity has a great impact on their performance. In this paper, we propose a complexity control method in the high-efficiency video coding intracoding to facilitate these video applications. The proposed method is performed on the coding tree unit (CTU) level, which consists of three steps, namely complexity estimation, complexity allocation, and prediction unit (PU) adaption. In the first step, a complexity estimation model is proposed to estimate the coding complexity of each CTU based on the sum of absolute transformed difference. Then, the complexity budget is allocated to each CTU proportionally to its estimated coding complexity. In PU adaption, only a subset of PU sizes are selected for the CTU according to its allocated complexity and prediction performance. A feedback-based error elimination scheme removes the complexity error during the encoding process. Experimental results show that the proposed method is able to adjust the complexity ratio from 100% to 20%. Meanwhile, the rate-distortion performance and complexity control accuracy of the proposed method are superior to those of the state-of-the-art methods.

Nonnegative matrix factorization (NMF) is a well-known paradigm for data representation. Traditional NMF-based classification methods first perform NMF or one of its variants on input data samples to obtain their low-dimensional representations, which are successively classified by means of a typical classifier [e.g., k -nearest neighbors (KNN) and support vector machine (SVM)]. Such a stepwise manner may overlook the dependency between the two processes, resulting in the compromise of the classification accuracy. In this paper, we elegantly unify the two processes by formulating a novel constrained optimization model, namely dual embedding regularized NMF (DENMF), which is semi-supervised. Our DENMF solution simultaneously finds the low-dimensional representations and assignment matrix via joint optimization for better classification. Specifically, input data samples are projected onto a couple of low-dimensional spaces (i.e., feature and label spaces), and locally linear embedding is employed to preserve the identical local geometric structure in different spaces. Moreover, we propose an alternating iteration algorithm to solve the resulting DENMF, whose convergence is theoretically proven. Experimental results over five benchmark datasets demonstrate that DENMF can achieve higher classification accuracy than state-of-the-art algorithms.

The quality of synthesized view plays an important role in the three dimensional (3D) video system. In this paper, to further improve the coding efficiency, a convolutional neural network (CNN) based synthesized view quality enhancement method for 3D High Efficiency Video Coding (HEVC) is proposed. Firstly, the distortion elimination in synthesized view is formulated as an image restoration task with the aim to reconstruct the latent distortion free synthesized image. Secondly, the learned CNN models are incorporated into 3D HEVC codec to improve the view synthesis performance for both view synthesis optimization (VSO) and the final synthesized view, where the geometric and compression distortions are considered according to the specific characteristics of synthesized view. Thirdly, a new Lagrange multiplier in the rate-distortion (RD) cost function is derived to adapt the CNN based VSO process to embrace a better 3D video coding performance. Extensive experimental results show that the proposed scheme can efficiently eliminate the artifacts in the synthesized image, and reduce 25.9% and 11.7% bit rate in terms of peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) index, which significantly outperforms the state-of-theart methods.

Pairwise constraint propagation studies the problem of propagating the scarce pairwise constraints across the entire dataset. Effective propagation algorithms have previously been designed based on the graph-based semi-supervised learning framework. Therefore, these previous constraint propagation methods rely critically on a good similarity measure over the data points. Improper or noisy similarity measurements may dramatically degrade the performance of the constraint propagation algorithms. In this paper, we make attempt to exploit the available pairwise constraints to learn a new set of similarities, which are consistent with the supervisory information in the pairwise constraints, before propagating these initial constraints. Our method is a local learning algorithm. More specifically, we compute the similarities at each data point through simultaneously minimizing the local reconstruction error and local constraint error. The proposed method has been tested in the constrained clustering tasks on eight real-life datasets and then shown to achieve significant improvements with respect to the state of the arts.

Canonical correlation analysis (CCA) has been extensively employed in various real-world applications of multi-label annotation. However, two major challenges are raised by the classical CCA. First, CCA frequently fails to remove noisy and irrelevant features. Second, CCA cannot effectively capture correlations between multiple labels, which are especially beneficial for multi-label learning. In this paper, we propose a novel framework that integrates joint sparsity and low-rank shared subspace into the least-squares formulation of CCA. Under this framework, multiple label interactions can be uncovered by the shared structure of the input features and a few highly discriminative features can be decided via structured sparsity inducing norm. Owing to the inclusion of the non-smooth row sparsity, a new efficient iterative algorithm is derived with proved convergence. The empirical studies on several popular web image and movie data collections consistently deliver the effectiveness of our new formulation in comparison with competing algorithms.

Graph regularized techniques have been extensively exploited in unsupervised learning. However, there exist no principled ways to select reasonable graphs and their associated hyper parameters, particularly in multiple heterogeneous data sources. Often, the graph selection process requires rather time-consuming cross-validation and discrete grid search that are not scalable to a large number of candidate graph sources. To address this issue, we propose a new formulation by integrating Ensemble Manifold structure into Low Rank approximation (EMLR). The central idea is to maximally approximate the intrinsic geometric structure by searching the optimal linear combination space of multiple different graphs. Specifically, efficient projection onto the probabilistic simplex is utilized to optimize the graph weights, resulting in the sparsity pattern of coefficients. This attractive property of sparsity can be properly interpreted as a criterion for selection of graphs, i.e., identifying most discriminative graphs and removing noisy or irrelevant graphs under the low rank decomposition model. Therefore, the compact output representation and linear combination coefficients of multiple different graphs can be simultaneously achieved by a unified objective. Exhaustive experimental results corroborate the effectiveness of our new model.

Microsoft Kinect sensors nowadays are widely used for real-time controller-free virtual reality (VR) interactions. However, there has been limited research on interaction-oriented analysis of motion data captured by Kinect from those who suffer from both intellectual and physical challenges. In this paper, a software framework based on Kinect is proposed to support a psycho-educational program (InSPAL) in a special school setting, which aims to enable students with severe intellectual disabilities (SID) to interact with the educational VR content and through which their pre-learning skills could be enhanced. Also, algorithms embedded in this framework are designed and modified in order to overcome limitations of the sensor and challenges originated from unique features of SID students' body movement.

User selection data accumulates as time goes by. Although the recent selections are usually assumed to have higher impact on the recommendation accuracy, empirical studies on this problem are limited. For old data, whether they can contribute to the recommendation accuracy is still to be determined. On one hand, changes in short-term user preference over time may limit their effectiveness in prediction, but on the other hand, one cannot rule out their potential in capturing long term user preferences. The result is important for the system owner to determine which data is useful to make the recommendation accurately. While there have been some related studies on the time dependency of data quality using neighbor-based CF methods (e.g., [4]), its effects remain unverified for other CF methods. In this paper, we study the effect of data generated over different time period on recommendation precision using several popular model-based CF algorithms (latent factor models). experiment results show that while more recent data expectedly have larger impacts, the usefulness of older data cannot be ignored as long as there are sufficient old samples. However, the addition of insufficient amount of old data seems to have negative impacts.

This paper presents a novel modalities consensus framework for multi-modal pairwise constraint propagation (MCP). We first combine multiple single-modal constraint propagation (SCP) problems together, and then explicitly introduce a new modalities consensus regularizer to force the propagation results on different modalities to be consistent with each other. With a separable consensus regularizer, the proposed approach can be effectively solved using an alternating optimization way. More importantly, based on our modalities consensus framework, two single-modal constraint propagation algorithms can be directly reformulated as two well-defined multi-modal solutions. Experimental results on constrained clustering tasks have shown that the proposed framework can achieve significant improvements with respect to the state of the arts.

Recommendations generated by Content Based method are highly related to the user's previous choices, which may not only unattractive to the user[1], but also restrict the user's horizon [1], [2]. At the same time, a new paradigm of making recommendations hat "surprise" the user poses new challenges in relation to the definition, formulation and performance metrics for surprising recommendation systems. Moreover, users may demand recommendations with varying degrees of surprising ness that satisfy their personal interests on the one hand, and encourage the explorations of new or unexpected areas of potential interests on another. To meet these challenges, in this paper, we proposed a framework, called Customizable GenPref, and the associated techniques for generating customizable surprising recommendations. Specifically, we contribute to the following aspects: firstly, through a review of the related works, we distinguish the difference between surprising ness and other concepts such as diversity, unexpectedness in non-traditional recommendations, secondly, we argue that the elements of surprise in a recommendation involve two conflicting goals, namely unusuality and relevance in the recommendation and proposed a framework of making recommendations such that by tuning a user-defined parameter a a user will receive recommendations which are either similar to his/her previous choices, or different and novel that surprises him/her, or combinations of both. We have evaluated our proposed framework using several relevant performance metrics, such as accuracy and diversity. Our experimental results show that Customizable GenPref is not only able to predict and recommend similar or surprising items that the user may like, but, at the same time, also serves the business objectives of e-commerce sites by recommending more distinct items to the users compared with baseline methods.

This paper presents a novel symmetric graph regularization framework for pairwise constraint propagation. We first decompose the challenging problem of pairwise constraint propagation into a series of two-class label propagation subproblems and then deal with these subproblems by quadratic optimization with symmetric graph regularization. More importantly, we clearly show that pairwise constraint propagation is actually equivalent to solving a Lyapunov matrix equation, which is widely used in Control Theory as a standard continuous-time equation. Different from most previous constraint propagation methods that suffer from severe limitations, our method can directly be applied to multi-class problem and also can effectively exploit both must-link and cannot-link constraints. The propagated constraints are further used to adjust the similarity between data points so that they can be incorporated into subsequent clustering. The proposed method has been tested in clustering tasks on six real-life data sets and then shown to achieve significant improvements with respect to the state of the arts.

This paper proposes novel spectral methods for learning latent semantics (i.e. high-level features) from a large vocabulary of abundant mid-level features (i.e. visual keywords), which can help to bridge the semantic gap in the challenging task of action recognition. To discover the manifold structure hidden among mid-level features, we develop spectral embedding approaches based on graphs and hypergraphs, without the need to tune any parameter for graph construction which is a key step of manifold learning. In particular, the traditional graphs are constructed by linear reconstruction with sparse coding. In the new embedding space, we learn high-level latent semantics automatically from abundant mid-level features through spectral clustering. The learnt latent semantics can be readily used for action recognition with SVM by defining a histogram intersection kernel. Different from the traditional latent semantic analysis based on topic models, our two spectral methods for semantic learning can discover the manifold structure hidden among mid-level features, which results in compact but discriminative high-level features. The experimental results on two standard action datasets have shown the superior performance of our spectral methods.

This paper presents a multi-modal constraint propagation approach to exploiting pairwise constraints for constrained clustering tasks on multi-modal datasets. Pairwise constraint propagation methods have previously been designed primarily for single modality data and cannot be directly applied to multi-modal data or a dataset with multiple representations. In this paper, we provide an effective solution to the multi-modal constraint propagation problem by decomposing it into a set of independent multi-graph based two-class label propagation subproblems which are then merged into a unified problem and solved by quadratic optimization. We also show that such a formulation yields a closed-form solution. Our approach allows the initial pairwise constraints to be propagated throughout the entire multi-modal dataset. The propagated constraints are further used to refine the similarities between the objects for subsequent clustering tasks. The proposed method has been tested in constrained clustering tasks on two real-life multi-modal image datasets and shown to achieve significant improvements with respect to the single modality methods.

This paper presents contextual kernel and spectral methods for learning the semantics of images that allow us to automatically annotate an image with keywords. First, to exploit the context of visual words within images for automatic image annotation, we define a novel spatial string kernel to quantify the similarity between images. Specifically, we represent each image as a 2-D sequence of visual words and measure the similarity between two 2-D sequences using the shared occurrences of s -length 1-D subsequences by decomposing each 2-D sequence into two orthogonal 1-D sequences. Based on our proposed spatial string kernel, we further formulate automatic image annotation as a contextual keyword propagation problem, which can be solved very efficiently by linear programming. Unlike the traditional relevance models that treat each keyword independently, the proposed contextual kernel method for keyword propagation takes into account the semantic context of annotation keywords and propagates multiple keywords simultaneously. Significantly, this type of semantic context can also be incorporated into spectral embedding for refining the annotations of images predicted by keyword propagation. Experiments on three standard image datasets demonstrate that our contextual kernel and spectral methods can achieve significantly better results than the state of the art.

This paper presents a novel discriminative stochastic method for image categorization and annotation. We first divide the images into blocks on a regular grid and then generate visual keywords through quantizing the features of image blocks. The traditional Markov chain model is generalized to capture 2-D spatial dependence between visual keywords by defining the notion of "past" as what we have observed in a row-wise raster scan. The proposed spatial Markov chain model can be trained via maximum-likelihood estimation and then be used directly for image categorization. Since this is completely a generative method, we can further improve it through developing new discriminative learning. Hence, spatial dependence between visual keywords is incorporated into kernels in two different ways, for use with a support vector machine in a discriminative approach to the image categorization problem. Moreover, a kernel combination is used to handle rotation and multiscale issues. Experiments on several image databases demonstrate that our spatial Markov kernel method for image categorization can achieve promising results. When applied to image annotation, which can be considered as a multilabel image categorization process, our method also outperforms state-of-the-art techniques.

The paper presents a novel human motion recognition method based on a new form of the Hidden Markov Models, called spatial-temporal hidden markov models (ST-HMM), which can be learnt from a sequence of joints positions. To cope with the high dimensionality of the pose space, in this paper, we exploit the spatial dependency between each pair of spatially connected joints in the articulated skeletal structure, as well as the temporal dependency due to the continuous movement of each of the joints. The spatial-temporal contexts of these joints are learnt from the sequences of joints movements and captured by our ST-HMM. Results of recognizing 11 different action classes on a large number of motion capture sequences as well as synthetic tracking data show that our approach outperforms traditional HMM approach in terms of robustness and recognition rates.

Abstract: A new classification algorithm based on multivariate analysis is proposed to discover and simulate the grading policy on school transcript data sets. The framework comprises three major steps. First, factor analysis is adopted to separate the scores of several different subjects into grading‐related ones and grading‐unrelated ones. Second, multidimensional scaling is employed for dimensionality reduction to facilitate subsequent data visualization and interpretation. Finally, a support vector machine is trained to classify the filtered data into different grades. This work provides an attractive framework for intelligent data analysis and decision making. It also exhibits the advantages of high classification accuracy and supports intuitive data interpretation.

This paper presents a novel graph-based framework which can combine context, consistency, and diversity cues for interactive image categorization. The image representation is first formed with visual keywords by dividing images into blocks and then performing clustering on these blocks. The context across visual keywords within an image is further captured by proposing a 2-D spatial Markov chain model. To develop a graph-based approach to image categorization, we incorporate intra-image context into a new class of kernel called spatial Markov kernel which can be used to define the affinity matrix for a graph. After graph construction with this kernel, the large unlabeled data can be exploited by graph-based semi-supervised learning through label propagation with inter-image consistency. For interactive image categorization, we further combine this semi-supervised learning with active learning by defining a new diversity-based data selection criterion using spectral embedding. Experiments then demonstrate that the proposed framework can achieve promising results.

In this paper, we propose a novel 3-D model retrieval framework, which is referred to as hybrid 3-D model associative retrieval. Unlike the conventional 3-D model similarity retrieval approach, the query model and the models obtained by 3-D model hybrid associative retrieval have the following properties: They belong to different model classes and have different shape characteristics in general but are semantically related and preassembled in a certain associative group. For instance, given a furniture associative group { desk, chair, bed}, we may probably like to use a desk as a query model to search for a list of matching models, which belong to the chair or bed class. We consider the following possibilities: 1) there can be more than two classes in an association group and 2) different association groups might have different numbers of classes. The hybrid associative retrieval is performed in two stages: 1) to establish the relationship between different 3-D model categories with semantic associations, we propose three approaches based on neural network learning and 2) to address the aforementioned two conditions, we use a cyclic-shift scheme to partition different associative groups into two-class pairwise associative groups and then adopt two different strategies to combine the final retrieval results. Experiments by using different data sets demonstrate the effectiveness and efficiency of our proposed framework on the new hybrid associative retrieval task.

This paper proposes an algorithm for segmenting deforming soft-body meshes based on statistical variation information extracted from the deforming meshes. The variation information is extracted by performing a global principal component analysis (PCA) on the set of meshes. eigen-variation similarity (EVS) and eigen-variation magnitude (EVM) are then defined for the vertices and triangle faces of the meshes based on the extracted variation information. A multiple-source region growing algorithm is presented for segmenting a mesh that favors grouping faces with similar variations into a same component. We apply the proposed mesh segmentation algorithm to the construction of piecewise active shape model (ASM) and use such piecewise ASM to reconstruct unseen meshes. Experimental results show that our algorithm outperforms several state-of-the-art methods in terms of reconstruction accuracy.

This paper presents a new class of 2D string kernels, called spatial mismatch kernels, for use with support vector machine (SVM) in a discriminative approach to the image categorization problem. We first represent images as 2D sequences of those visual keywords obtained by clustering all the blocks that we divide images into on a regular grid. Through decomposing each 2D sequence into two parallel 1D sequences (i.e. the row-wise and column-wise ones), our spatial mismatch kernels can then measure 2D sequence similarity based on shared occurrences of k-length 1D subsequences, counted with up to m mismatches. While those bag-of-words methods ignore the spatial structure of an image, our spatial mismatch kernels can capture the spatial dependencies across visual keywords within the image. Experiments on the natural and histological image databases then demonstrate that our spatial mismatch kernel methods can achieve superior results.

This paper presents a novel semi-supervised learning method which can make use of intra-image semantic context and inter-image cluster consistency for image categorization with less labeled data. The image representation is first formed with the visual keywords generated by clustering all the blocks that we divide images into. The 2D spatial Markov chain model is then proposed to capture the semantic context across these keywords within an image. To develop a graph-based semi-supervised learning approach to image categorization, we incorporate the intra-image semantic context into a kind of spatial Markov kernel which can be used as the affinity matrix of a graph. Instead of constructing a complete graph, we resort to a k-nearest neighbor graph for label propagation with cluster consistency. To the best of our knowledge, this is the first application of kernel methods and 2D Markov models simultaneously to image categorization. Experiments on the Corel and histological image databases demonstrate that the proposed method can achieve superior results.

When fitting Gaussian mixtures to multivariate data, it is crucial to select the appropriate number of Gaussians, which is generally referred to as the model selection problem. Under regularization theory, we aim to solve this model selection problem through developing an entropy regularized likelihood (ERL) learning on Gaussian mixtures. We further present a gradient algorithm for this ERL learning. Through some theoretic analysis, we have shown a mechanism of generalized competitive learning that is inherent in the ERL learning, which can lead to automatic model selection on Gaussian mixtures and also make our ERL learning algorithm less sensitive to the initialization as compared to the standard expectation-maximization algorithm. The experiments on simulated data using our algorithm verified our theoretic analysis. Moreover, our ERL learning algorithm has been shown to outperform other competitive learning algorithms in the application of unsupervised image segmentation.

Clustered microcalcification is an important signal for breast cancer in the early stages. In this paper, we propose a multiple kernel SVM with group features (GF-SVM) to tackle problems associated with heterogeneous features of clustered microcalcification and normal breast tissues in suspicious regions. Specifically, different types of features such as being gradient, geometric and textural are grouped and modeled by different kernels, respectively. The prior knowledge from different resources is then combined into the framework of the multiple kernel SVM based classification scheme. Experimental results demonstrate that our classification scheme reduces the false positive rate significantly while maintaining the true positive rate.

The spatial-hidden Markov model (SHMM) is a two dimensional generalization of the traditional hidden Markov model (HMM), with the capability of blockbased semantic annotation as well as classification of images. In this paper, we conduct a sensitivity analysis of SHMM in semantic classification with respect to different block sizes and from this analysis, we propose a novel multi-scales SHMM that combines multiple SHMMs, each classifying the image on a different scale. By regarding each SHMM as distinct classifiers, classifier combination algorithm can be applied to integrate the outputs of the respective SHMMs to improve image classification accuracy. Experiment results demonstrate that the multi-scale SHMM consistently outperforms single SHMMin image semantic classifications. The proposed approach can be extended to other block-based image classification algorithms.

In this paper, we propose several active learning strategies to train classifiers for phosphorylation site prediction. When combined with support vector machine, we show that active learning with SVM is able to produce classifiers that give comparable or better phosphorylation site prediction performance than conventional SVM techniques and, at the same time, require a significantly less number of annotated protein training samples. The result has both conceptual and practical implications in protein prediction: it exploits information inherent in the large scale database of non-annotated protein samples and reduces the amount of manual labor required for protein annotation. To the best of our knowledge, active learning has not been explored in phosphorylation sites prediction. Several active learning strategies: single-running mode, batch-running mode with sample and support vector diversity, were investigated for phosphorylation sites prediction in this work. Our experiments have shown that active learning with SVM is able to reduce the effort of protein annotation by 6.6% to 25.7% to yield similar prediction performance as compared with conventional SVM technique.

In this paper, a novel 3D head model retrieval approach is proposed, in which only a single 2D face view query is required. The proposed approach will be important for multimedia application areas such as virtual world construction and game design, in which 3D virtual characters with a given set of facial features can be rapidly constructed based on 2D view queries, instead of having to generate each model anew. To achieve this objective, we construct an adaptive mapping through which each 2D view feature vector is associated with its corresponding 3D model feature vector. Given this estimated 3D model feature vector, similarity matching can then be performed in the 3D model feature space. To avoid the explicit specification of the complex relationship between the 2D and 3D feature spaces, a neural network approach is adopted in which the required mapping is implicitly specified through a set of training examples. In addition, for efficient feature representation, principal component analysis (PCA) is adopted to achieve dimensionality reduction for facilitating both the mapping construction and the similarity matching process. Since the linear nature of the original PCA formulation may not be adequate to capture the complex characteristics of 3D models, we also consider the adoption of its nonlinear counterpart, i.e., the so-called kernel PCA approach, in this work. Experimental results show that the proposed approach is capable of successfully retrieving the set of 3D models which are similar in appearance to a given 2D face view.

We propose a multi-resolution integrated model for the segmentation of soft-tissue organs called MISTO. The model is constructed hierarchically to represent the most significant deformations from the training set as well as to generate representative deformation modes of the organ shapes. The clutter surrounding of the surface points are formulated in terms of an external functional which is also learnt automatically from the training samples. By combining a set of powerful shape models and context constraints, the segmentation process can be carried out very effectively. To avoid the local minimum during model optimization, the deformation strategies are designed such that the portions of the surface for which we have more reliable prior knowledge on their possible deformations are deformed first, followed by deformation on the less informed portions. The experimental and validation results verify that our proposed approaches can be robustly applied to highly deformable anatomies such as soft-tissue organs.

Given a region, it is a challenge to find a set of primitive shapes such as rectangles, circles or ellipses to cover it. This is in fact a set-covering problem, which is known to be NP-hard. The focus of this paper is on fitting a set of ellipses onto an image region. This problem was first formulated by identifying a number of criteria required for the ellipse fitting. A solution is then proposed for automatically determining the set of ellipses that best fits onto an image region. The proposed ellipse fitting algorithm has also been applied to strokes forming characters of Chinese calligraphic artwork. The results show that our proposed algorithm generates ellipses fitting onto stroke regions and capturing the characteristics of the strokes during turning, tilting and back-trace.

In this paper, we introduce the design of a Personalized Education (PE) search approach that employs multiple ontologies to automatically generate queries for educational resources retrieval based on a high level specification of the teaching/learning needs of a user. Central to this approach is the design of PEOnto which is an educational ontology that consists of FIVE interrelated ontologies that supports the delivery of various services in a Personalized Education System. We illustrate the feasibility of PEOnto design through a scenario walkthrough in the context of Grade 4 English Language learning.

This paper presents a new spatial-HMM(SHMM) for automatically classifying and annotating natural images. Our model is a 2D generalization of the traditional HMM in the sense that both vertical and horizontal transitions between hidden states are taken into consideration. The three basic problems with HMM-liked model are also solved in our model. Given a sequence of visual features, our model automatically derives annotations from keywords associated with the most appropriate concept class, and with no need of a pre-defined length threshold. Our experiments showed that our model outper-formed the previous 2D MHMM in recognition accuracy and also achieved a high annotation accuracy.

Chinese fonts with smooth outlines and solid colouring have been produced for computer displays and printings for a long time. However, the aesthetic properties of the characters produced by calligraphers could not be simulated with these methods. Ip and Wong proposed a parameterised brush model that enables efficient generation of Chinese calligraphic writings such that the rendering is scalable in resolution and it allows high quality publishing. While this graphical model facilitates the synthesis of calligraphy writing given a set of writing parameters, for the inverse problem, a lot of user stroke manipulation is required to regenerate the model parameters given a calligraphic image. Consequently, an intelligent method is required to automatically determine the model parameters from images of Chinese calligraphy. This paper describes a methodology for automatically estimating the set of 3D geometric and dynamic writing parameters along a stroke trajectory from images of calligraphic writings.

In this paper, a novel 3D head model retrieval framework is proposed. First, kernel PCA is adopted both to reduce the data dimension and to extract features for model characterization. Second, based on the derived features, a hierarchical indexing structure for 3D model database is constructed using the hierarchical self organizing map (HSOM). Third, an efficient search approach is presented based on the established indexing structure that requires only feature matching between the query model and a small number of SOM nodes. The main advantages of our approach include high retrieval precision due to the discrimination capacity of kernel PCA, and low computation cost due to the hierarchical indexing structure and data dimension reduction. In addition, the topology-preserving property of HSOM also facilitates the exploration of the model database with the possibility of further knowledge discovery.

In this paper, we propose a novel 3D head model retrieval approach in which the queries are 2D face views instead of less readily available 3D head models. The basic idea is to characterize the corresponding relations between 2D view feature and 3D model feature based on a machine learning approach. Thus the subsequent feature matching can be carried out in 3D feature space. As an effective solution to regression problems, relevance vector machine is used in this paper to establish an association between 2D and 3D features. Experimental results show that our proposed 2D query based method is comparable with the direct 3D query based one.

Cyber Composer is a novel and interactive cyber instrument that enables both musicians and music laypersons to dynamically control the tonality and the melody of the music that they generate/compose through hand motion and gestures. Cyber Composer generates music according to hand motions and gestures of the users in the absence of real musical instruments. Music theories are embedded in the design so that melody flow and musical expressions like the pitch, rhythm and volume of the melody can be controlled and generated in real-time by wearing a pair of motion-sensing gloves. Also central to the design is the mapping of the hand motions and gestures to musical expressions that is intuitive and requires minimal training. Cyber Composer is expected to find applications in the fields of performance, composing, entertainment, education as well as psychotherapy.

This paper proposes new modified constrained learning neural root finders (NRFs) of polynomial constructed by backpropagation network (BPN). The technique is based on the relationships between the roots and the coefficients of polynomial as well as between the root moments and the coefficients of the polynomial. We investigated different resulting constrained learning algorithms (CLAs) based on the variants of the error cost functions (ECFs) in the constrained BPN and derived a new modified CLA (MCLA), and found that the computational complexities of the CLA and the MCLA based on the root-moment method (RMM) are the order of polynomial, and that the MCLA is simpler than the CLA. Further, we also discussed the effects of the different parameters with the CLA and the MCLA on the NRFs. In particular, considering the coefficients of the polynomials involved in practice to possibly be perturbed by noisy sources, thus, we also evaluated and discussed the effects of noises on the two NRFs. Finally, to demonstrate the advantage of our neural approaches over the nonneural ones, a series of simulating experiments are conducted.

Recently, a number of approaches have been proposed which use compressed domain features for image retrieval and classification. While the main motivation of these approaches is to improve processing efficiency and reduce computational requirements, we propose a method which also aims at enhancing the content characterization capabilities of the compressed domain features in addition to efficiency improvement. We model the compressed domain feature values as random variables and approximate their associated probability mass functions as histograms. We then transform these histograms in such a way that the resulting classification rate based on these transformed histograms is improved. With a large number of possible transformations, we adopt an evolutionary strategy (ES) to search for the. optimal one. Experiments show that our proposed approach is able to obtain a better classification rate while the efficiency advantage of using compressed domain features is retained.

An efficient model-independent 3D texture synthesis algorithm based on texture growing and texture turbulence is presented to simulate vivid 3D solid textures from 2D growable texture patterns. Given a 2D texture pattern of some growable material, our algorithm is able to create a tileable anisotropic 3D texture pattern to simulate the natural property of the material. Target objects are directly dipped into the 3D texture pattern to generate creative, sculpture like models that can be presented with reasonable interactive frame rates. Additionally, our method is conceptually simple, computationally fast, and storage efficient. To the best of our knowledge, this is the first approach that transfers a given 2D texture naturally to point rendering systems

In this paper, we present a relational-tubular (ReTu) deformable model for segmenting a complex and the entire tubular network structure with branches in close proximity of each other. Specifically, we incorporate a priori knowledge of the target anatomy structure as well as the spatial relationship between branches to reduce possible segmentation errors due to the effects of a variety of imaging artifacts and noise. To get more robust description of the data properties than a simple 3D edge map, a new data energy functional is proposed based on testing the volumetric density within the model cross-sections. The deformation process is formulated as a two-stage procedure: tubular medial axis deformation and tubular surface deformation. The efficiency of this approach is demonstrated by our experiments which show that satisfactory quantifications of the entire zebrafish vasculature recorded from the fluorescence confocal microscope. The experiments also demonstrate the robustness of our deformable model in the presence of complex issue structure that adhered to the vessel branches.

This letter proposes a novel neural root finder based on the root moment method (RMM) to find the arbitrary roots (including complex ones) of arbitrary polynomials. This neural root finder (NRF) was designed based on feedforward neural networks (FNN) and trained with a constrained learning algorithm (CLA). Specifically, we have incorporated the a priori information about the root moments of polynomials into the conventional backpropagation algorithm (BPA), to construct a new CLA. The resulting NRF is shown to be able to rapidly estimate the distributions of roots of polynomials. We study and compare the advantage of the RMM-based NRF over the previous root coefficient method—based NRF and the traditional Muller and Laguerre methods as well as the mathematica roots function, and the behaviors, the accuracies of the resulting root finders, and their training speeds of two specific structures corresponding to this FNN root finder: the log σand the σ FNN. We also analyze the effects of the three controlling parameters {δP0 θp η} with the CLA on the two NRFs theoretically and experimentally. Finally, we present computer simulation results to support our claims.

We propose a robust 3D model watermarking scheme based on distributing the information corresponds to a bit of the watermark over the entire model via vertex scrambling. The scheme also adaptively varies the strength of the watermark signal with reference to the local geometry and embeds watermark information into the model by modifying the length of the vectors that link vertices to the centre of the model. Experiments show that this watermarking scheme is robust against attacks such as mesh simplification, addition of noise, and model cropping.

This paper focuses on how to find the maximum modulus root (MMR) (real or complex) of an arbitrary polynomial. Efficient solution to this problem is important for many fields including neural computation and digital signal processing etc. We present neural networks technique for solving this problem. Our neural root finder (NRF) is designed based on partitioning feedforward neural networks (FNN) trained with a constrained learning algorithm (CLA) by imposing the a priori information about the root moment from polynomial into the error cost function. Experimental results show that this neural root-finding method is able to find the maximum modulus roots of polynomials rapidly and efficiently.

This paper proposed a partitioning neural root finder (PNRF) to find the minimum modulus (real or complex) roots of an arbitrary polynomial by imposing a minimum m order root moment (RM) into the constrained learning algorithm (CLA), where the constraint "the minimum m order RM" will ensure the minimum modulus root to be obtained. If the PNRF is recursively updated, the ordered roots from minimum modulus to maximum one can be achieved. Simulations show that this partitioning neural root-finding method is indeed able to find the minimum modulus root and the ordered roots of arbitrary polynomials readily and efficiently.

Abstract Content‐based image retrieval (CBIR) is a process of retrieving images from an image database by exploiting the content of the images (typically the querying of an image). CBIR avoids many problems associated with traditional ways of retrieving images by keywords. Thus, a growing interest in the area of CBIR has been established in recent years. In this paper, a novel object‐oriented framework (CBIRFrame) is built for CBIR applications development. We discuss the motivations for CBIRFrame before discussing its design in detail. Two applications of CBIRFrame are also briefly discussed to show the effectiveness of applying CBIRFrame to real applications. Finally, we outline the possible uses of the design of CBIRFrame for other types of domains, such as content‐based retrieval of video clips. Copyright © 2003 John Wiley & Sons, Ltd.

The demand for automatically recognizing and retrieving medical images for screening, reference, and management is growing faster than ever. In this paper, we present an intelligent content-based image retrieval system called I-Browse, which integrates both iconic and semantic content for histological image analysis. The I-Browse system combines low-level image processing technology with high-level semantic analysis of medical image content through different processing modules in the proposed system architecture. Similarity measures are proposed and their performance is evaluated. Furthermore, as a byproduct of semantic analysis, I-Browse allows textual annotations to be generated for unknown images. As an image browser, apart from retrieving images by image example, it also supports query by natural language.

With the development of an innovative motion capture and analysis system using frontal infra-red illumination, and based on a systematic study of the relations between the human body movement and the visual art language, the Body-Brush turns the human body as a whole into a dynamic brush. The Body-Brush enables humans to interact intuitively with the machine to create a rich variety of visual forms and space within a virtual 3-D canvas.

The advent of the digital library and multimedia database require robust techniques for multimedia content searches. Content-based retrieval techniques have been developed to overcome some of the limitations associated with conventional keyword-based browsing or searching of visual data. We present an affine invariant shape-based retrieval technique for image retrieval which is efficient and robust. More importantly, the technique supports a query presented in the form of hand-drawn sketches and has the potential of supporting affine invariant partial shape retrieval.

The I-Browse project aimed to develop a prototype system to provide facilities for supporting intelligent retrieval of medical images through a combination of iconic and semantic content. The resulting prototype system, I-Browse, is able to extract and represent relevant iconic and semantic information from input images and to automatically generate textual annotations for images. Techniques were also developed for retrieving relevant images from the database given either an example image query or a textual query. The facilities provided by I-Browse were evaluated by medical colleagues and judged to have the potential of alleviating some of the time-consuming tasks that doctors now have to perform daily and providing a means of identifying previously unknown relationships between visual appearance and histological events.

This paper presents a new error concealment algorithm for the masking of damaged blocks due to data loss during the transmission of DCT based images over a packet network. The algorithm is called neighborhood regions partitioned matching (NRPM). Instead of repairing damaged block as a whole, each damaged block is divided into a number of sub-blocks. Each sub-block is recovered utilizing not only the surrounding pixels, but also remote pixels which share similar local image characteristics. The approach yields good a quality repair and since smaller block size is used, it can significantly reduce the computational time for the reconstruction process.

Complex maxillofacial malformations continue to present challenges in analysis and correction beyond modern technology. The purpose of this paper is to present a virtual-reality workbench for surgeons to perform virtual orthognathic surgical planning and soft-tissue prediction in three dimensions. A resulting surgical planning system, i.e., three-dimensional virtual-reality surgical-planning and soft-tissue prediction for orthognathic surgery, consists of four major stages: computed tomography (CT) data post-processing and reconstruction, three-dimensional (3-D) color facial soft-tissue model generation, virtual surgical planning and simulation, soft-tissue-change preoperative prediction. The surgical planning and simulation are based on a 3-D CT reconstructed bone model, whereas the soft-tissue prediction is based on color texture-mapped and individualized facial soft-tissue model. Our approach is able to provide a quantitative osteotomy-simulated bone model and prediction of postoperative appearance with photorealistic quality. The prediction appearance can be visualized from any arbitrary viewing point using a low-cost personal-computer-based system. This cost-effective solution can be easily adopted in any hospital for daily use.

We present a semantic content representation scheme and the associated techniques for supporting (a) query by image examples or by natural language in a histological image database and (b) automatic annotation generation for images through image semantic analysis. In this research, various types of query are analysed by either a semantic analyser or a natural language analyser to extract high level concepts and histological information, which are subsequently converted into an internal semantic-content representation structure code-named "Papillon". Papillon serves not only as an intermediate representation scheme but also stores the semantic content of the image that will be used to match against the semantic index structure within the image database during query processing. During the image database population phase, all images that are going to be put into the database will go through the same processing so that every image would have its semantic content represented by a Papillon structure. Since the Papillon structure for an image contains high level semantic information of the image, it forms the basis of the technique that automatically generates textual annotation for the input images. Papillon bridges the gap between different media in the database, allows complicated intelligent browsing to be carried out efficiently and also provides a well-defined semantic content representation scheme for different content processing engines developed for content-based retrieval.

Automatic interpretation and animation of human motion has become an important research topic among researchers in virtual reality and computer animation. One major problem encountered during hand motion analysis is the large amount of data that need to be captured and analyzed. Even for a human hand, although only a small part of the body is involved, there are about 30 motion parameters for each hand posture. The authors present an approach to hand motion animation using only static images of the set of target gestures. We achieve naturalistic hand motion animation by the use of an anatomy based hand model and a hand gesture coding system, which we called Hand Action Coding System (HACS). This allows complex sequences of hand gestures to be animated based only on the static image of the hand gestures to be animated. This approach greatly simplifies motion data acquisition and the process of motion analysis and synthesis.

Orthognathic surgery corrects a wide range of minor and major facial and jaw irregularities. This surgery will improve the patients' ability to chew, speak and breathe. In many cases, a better appearance will also result. With the recent advances in virtual reality (VR) and three-dimensional (3D) medical imaging technology, orthognathic surgery simulations typically requires costly volumetric data acquisition modalities such CT or MRI imaging for patient modeling. The authors present an approach for constructing 3D hard and soft tissue models of a patient based on colour portraits and conventional radiographs. This allows patient modeling to be done efficiently on low-cost platforms. Specifically, we extend the techniques developed by the author (H.S.H Ip and Lijin Yin, 1996) to hard tissue modeling. The extended technique employs a user-assisted approach to obtain the 3D coordinates of the feature points of the human face and jaw respectively from conventional photographs and radiographs. Then the displacement vectors of the feature points are computed by correspondence matching and interpolation against a generic head model and jaw bone model. The resulting combined hard and soft tissue models can be used for orthognathic surgical planning on a low-cost, PC based platform.

Medical images are usually composed of different kinds of texture components which are always so much varied that a conventional single window approach cannot capture enough salient information for comparison. This paper applies the widely used multi-channel Gabor filters to demonstrate how a multi-window approach can improve the classification accuracy rate of histological labels. In addition, a most confident window method is proposed to further increase the accuracy rate of the multi-window approach.

A gastro-intestinal (GI) tract histological image is usually composed of texture components with different dimensions and properties. To analyze a histological image, we divide it into an array of sub-images. A feature vector comprising a set of Gabor filters and the intensity statistics is computed in order to classify each sub-image to one of 63 histological labels. To retrieve an image from the database, we compare three similarity measures, shape, neighbour and sub-image frequency distribution. It is found that both neighbour and sub-image frequency distribution similarity measures perform similarly well but the shape similarity measure yields the worst result when retrieving images of different GI tract organs. In general, the sub-image frequency distribution measure is the best choice because it requires less time to compute than the neighbour measure.

We formulate the problem of detecting reflectional symmetries as the problem of minimising an asymmetric energy term. The local minima of the asymmetric energy correspond to the reflectional symmetric axes of an object. Typical experiments show the correctness and the robustness of the proposed technique.

A theorem linking reflectional symmetry and rotational symmetry of the 2D images has been established. This theorem shows that, for a rotationally symmetric image with K/spl ges/1 fold(s) (K-RSI), its number of reflection-symmetric axes must be either K or zero. To the authors' knowledge no previous studies have shown the constraint relationship between reflectional symmetry and rotational symmetry. Demonstrations on some typical images have shown the exactness of our theorem.

An affine-invariant feature vector, which captures local and semi-local features, has been used in the detection of skewed symmetries. Here, the problem of symmetry axes detection has been formulated as a line detection problem, with known orientations within a local similarity matrix computed for a shape. Moreover, our technique allows all the local reflection-symmetries within an object to be detected. Experiments on detecting skewed symmetries of self-symmetric objects and generalized objects, under noise and occlusions, have demonstrated the effectiveness of this method.

Reports a generic object-oriented framework for content-based image retrieval (CBIR) systems. It is designed so that the basic data structures and functionality of a typical CBIR system are provided without sacrificing speed and flexibility. The framework is based on a five-tier architecture that allows modules in different tiers to be developed independently, and thus flexibility is ensured. We show that our framework is able to adapt to a wide range of CBIR applications by applying the framework to the development of two on-going projects: a trademark image retrieval system and a medical (histological) image retrieval system. These applications are briefly discussed.

This paper presents a unified method for detecting both reflection-symmetry and rotation-symmetry of 2D images based on an identical set of features, i.e., the first three nonzero generalized complex (GC) moments. This method is theoretically guaranteed to detect all the axes of symmetries of every 2D image, if more nonzero GC moments are used in the feature set. Furthermore, we establish the relationship between reflectional symmetry and rotational symmetry in an image, which can be used to check the correctness of symmetry detection. This method has been demonstrated experimentally using more than 200 images.

We present a technique of content-based image retrieval based on the symmetric property of an image and its generalized complex moments. We propose a novel symmetry detection algorithm that is both fast and general compared with existing symmetry detection algorithms. Given the sensitivity of human perception to symmetry, the use of symmetry class helps to ensure that the retrieved images are visually similar. Experimental results show that these features are effective in retrieving binary images and our similarity function is able to rank them first place for nearly all the sample queries used in our experiment.

A recent research trend in Web applications is the integration of legacy applications on the World Wide Web. The motivations behind this research are the goals of producing a hybrid system where the Web can provide greater accessibility and distribution for legacy applications, and some standards to increase the interoperability and ease of use. For user interaction driven legacy applications, we propose a 3-tier conceptual architecture to support applications for the WWW, and present a approach to building sophisticated inactive Web systems. We benefit from this modelling approach in terms of universal accessibility, platform independency, modularity and migration efficiency. The interaction scenario between the client and server in the prototype implementation is an example to demonstrate the procedures for migrating a legacy application to the Web. Some basic technologies are reviewed and deployed for implementation of our design, including Java applet, servlet, JDBC and CORBA. Such an approach can also be extended to the newly developed Web based applications.

We present an efficient algorithm for detecting all the symmetry axes of a 2D planar shape. We proved theorems relating to certain distinguishing properties of the generalised complex moments computed for symmetrical objects. Based on these properties, the detection of symmetry axes can be carried out efficiently and accurately. The algorithm can also be applied to detecting partially (perceptually) symmetrical objects.

The fast morphological algorithm, overlapping search (OS) algorithm, recently proposed by Lam and Li (1998), can only be applied to a flat structuring element (FSE) whose 1D Euler-Poincare constants, N/sup (1)/(x) and N/sup (1)/(y), at any x or y coordinate are equal to 1. Thus, an arbitrarily shaped structuring element must be decomposed to a set of constrained components before employing the fast algorithm. In the paper, a decomposition algorithm, based on line-scanning process, and an improved algorithm, based on conditionally maximal convex polygon (CMCP), are proposed to generalize the use of the overlapping search morphological algorithm for any arbitrary flat structuring element.

Previously (H.S. Horace et al., 1997), we presented an anatomy based hierarchical model for hand motion simulation. We present a hand gesture coding system which we called Hand Action Coding System (HACS). It is an extension of our previous work. With HACS, the simulation allows specification of hand motion in a high level manner. The underlying computational complexities of motion simulation are separated from the high level motion descriptions. When humans want to perform some hand functions or hand gestures, the brain gives instructions and activates the corresponding hand muscles. HACS codifies hand gestures in terms of these hand muscle action units (HAU). When combined with our anatomy based hand model, HACS eases the specification of complex hand gestures/motion animation sequence.

The paper proposes a novel methodology which allows calligraphic writing to be synthesized realistically. The approach models the physical process of brush stroke creation and consists of three separate aspects, namely, the physical geometry of the writing brush, the dynamic movement, e.g., the position and orientation, of the brush along the stroke trajectory and the amount of ink absorbed in the brush bundle as well as the ink depositing process. By controlling these physical parameters associated with the writing process, very realistic calligraphic writing can be generated. In particular, the aesthetic features commonly associated with calligraphy, such as the varying widths of a stroke, the impression of physical rubbing between the brush and the underlying paper, the varying shades of grey caused by different degrees of ink content in the brush, and the black and white trails created by fast movement of a drying brush can be simulated. This is the first time physically-based model of a brush has been used to synthesize calligraphic writing and the model has been implemented on a PC-based platform.

We provide a generalized image normalization technique which basically solves all problems in image normalization. The orientation of any image can be uniquely defined by at most three non-zero generalized complex moments. The correctness of our method is demonstrated theoretically as well as in practice by applying them to a number of "degenerate" images which have failed other previously reported techniques for image normalization.

In this paper, we study three parallel models for curves. Based on their common properties we develop an algorithm for the detection of parallelism among curves. Furthermore, to speed up processing, we developed theorems for fast verification at critical points. False alarm is prevented by an enhancement in tangent representation, thus no postprocessing is required. The new tangent representation is called direction-dependent tangent (DDT). It incorporates concavity information into the tangent representation and prohibits false matching. Based on the theorems and the tangent representation, we show that parallelism detection can be formulated as a correspondence problem. The algorithm handles also partial parallelism, i.e., parallelism in certain ranges along the curves but not lasting throughout the whole curves spans. By "curve" we mean any planar curve with C3 continuity. The technique treats both curves and lines alike and is valid for both open and closed curves.

This paper presents a novel approach to face recognition based on an application of the theory of evidence (Dempster-Shafer (1990) theory). Our technique makes use of a set of visual evidence derived from two projected views (frontal and profile) of the unknown person. The set of visual evidence and their associate hypotheses are subsequently combined using the Dempster's rule to output a ranked list of possible candidates. Image processing techniques developed for the extraction of the set of visual evidence, the formulation of the face recognition problem within the framework of Dempster-Shafer theory and the design of suitable mass functions for belief assignment are discussed. The feasibility of the technique was demonstrated in an experiment.>

Medical documents consists of heterogeneous types of data generated from different diagnostic and monitoring modalities. Advances in multimedia technology have made possible the capturing, storage and visualisation of such multimedia documents. By incorporating hypermedia concept into the structure of the document, the physician is able to retrieve and view relevant information non-linearly via the hyperlinks embedded within the document. Central to the computerisation of such documents is the design of a suitable hypermedia document architecture which structures and organises the information inherent in the documents. This paper presents a hyperdocument architecture of a specific type of medical document, the cardiac catheterisation record, and a prototype multimedia document system which had been implemented on a PC-based platform.>

With the increasing availability of workstations and networking facilities, desktop conferencing systems have moved from experimental prototypes in research laboratories to the user community. One particularly attractive application is the use of multimedia conferencing system for real-time cooperative by physicians at different sites. This paper describes the design of such a system, with special emphasis on its flexible conferencing capabilities and tools to facilitate medical diagnosis.>

Presents a medical image computing system which has been designed to extract and analyze quantitative information from a spatial sequence of CT (computed tomography) images. In particular, it has been applied to the quantitation and 3D reconstruction of the diffuse pattern of a radio-opaque anti-cancer drug, lipiodol-doxorubicin, in hepatocellular carcinoma. The PC-based system computes quantitative measures which can be used to assess the effectiveness of the chemotherapeutic treatment and to study the histology of the tumour. The system can provide a visualization of the tumour volume and quantification of drug absorption using 3D surface reconstruction and graphics techniques.>