Existing self-evolution methods overlook the influence of fine-grained reasoning steps, which leads to the reasoner-verifier gap. The computational inefficiency of Monte Carlo (MC) process supervision further exacerbates the difficulty in mitigating the gap. Motivated by the Error-Related Negativity (ERN), which the reasoner can localize error following incorrect decisions, guiding rapid adjustments, we propose a Self-Adaptive Process Optimization (SAPO) method for self-improvement in Small Language Models (SLMs). SAPO adaptively and efficiently introduces process supervision signals by actively minimizing the reasoner-verifier gap rather than relying on inefficient MC estimations. Extensive experiments demonstrate that the proposed method outperforms most existing self-evolution methods on two challenging task types: mathematics and code. Additionally, to further investigate SAPO's impact on verifier performance, this work introduces two new benchmarks for process reward models in both mathematical and coding tasks.

Medical point cloud segmentation accuracy is often limited by feature confusion in boundary regions, which arises from point sparsity, shape complexity, and structural similarity. To address this, we introduce a boundary-aware perspective that categorizes boundaries into inner and outer types. We propose the Dynamic Boundary-Aware Network (DBA-Net), which employs a Boundary-aware Dual Stream (BDS) module to decouple semantic and boundary features via a Cross-Stream Attention Module (CSAM), alongside an Adaptive Boundary Pseudo-Label Calculation (ABP-LC) strategy for adaptive label generation. For outer boundaries, the Outer Boundary Adaptive Contextual Discrepancy-guided Graph Convolution (OACD-GC) module, incorporating a Soft Edge Connection (SEC) strategy and an Iterative Optimization Mechanism (IOM), enhances inter-class discrimination. For inner boundaries, the Inner Boundary Dy-namic Supervised Contrastive Enhancement (ID-SCE) module, utilizing a Multi-Positive Sample (MPS) strategy and a Dynamic Hard Negative Sample Update (DHNU) mechanism, improves intra-class aggregation and inter-class differentiation. Extensive experiments on the IntrA and 3DTeethSeg datasets demonstrate DBA-Net's superior performance in boundary recognition and overall segmentation accuracy.

In recent years, several robust contrastive text clustering methods have been proposed. While these methods have achieved significant performances, two issues remain. First, the false negative problem is still not fully resolved, and the false positive issue also arises because all in-neighborhood and out-of-neighborhood samples are simply treated as positive and negative pairs, respectively. Second, these methods treat text representation learning and clustering as independent processes, leading to a performance gap. We propose a novel robust method called Dual-Path Contrastive Short Text Clustering (DCTC) to address these two issues. DCTC employs instance-level contrastive learning based on random walks at the representation learning level to progressively identify data pairs in a global rather than local manner, identifying in-neighborhood negatives and out-of-neighborhood positives. At the clustering level, DCTC performs cluster-level contrastive learning, jointly optimizing representation learning and cluster assignments, thereby enhancing clustering performance. DCTC achieves state-of-the-art results on 8 datasets, demonstrating its effectiveness. The code is available at https://github.com/2251821381/DCTC.

The latest 3D point cloud processing research shows that large kernels are critical for improving performance. However, 3D CNNs are limited by their small receptive fields, while directly applying large kernels in Transformers faces challenges of high computational costs. To address this vital challenge, we propose a large kernel module for point cloud serialization and partitioning, called Large Kernel Point Mamba (LKPM). Combining octrees and space-filling curves allows us to serialize unordered point clouds and avoid the high memory overhead associated with the K-Nearest Neighbors (KNN). Based on this, we design a bidirectional and hierarchical State Space Model(SSM) module, which consists of Point-level SSM (PoM) and Patch-level SSM (PaM), to extract local and global features from point clouds efficiently. This design significantly expands the Mamba’s receptive field. Additionally, we introduce a patch shuffling mechanism to mitigate overfitting effectively. Experimental results show that large Mamba kernels are both feasible and efficient in achieving large receptive fields. Under the conditions of not using additional data and training from scratch, our proposed LKPM method achieves state-of-the-art (SOTA) performance among a series of previous SOTA methods, even surpassing some pretraining-based approaches1.

Question Generation (QG) focuses on creating relevant questions from a given context, but question-answering texts often contain substantial redundant or irrelevant information. The key to effective QG lies in identifying and selecting relevant phrases. For lengthy contexts, combining these discrete phrases into semantically coherent questions remains a significant challenge. To address the issue of information redundancy in long documents, this paper proposes a method that extracts key information from discrete documents to construct fine-grained text representations. This method incorporates inductive biases to support the Question Generation process. Additionally, the paper introduces an interrogative word type predictor that accurately identifies interrogative words and determines the domain of the answer, guiding the model in generating semantically aligned questions. This ensures that the generated questions are closely aligned with the answers and their respective context. The proposed method significantly reduces information redundancy and the reliance on annotated data. Experimental results on two benchmark data sets demonstrate that the proposed model achieves performance comparable to that of state-of-the-art QG methods.

Multi-label text classification (MLTC) presents significant challenges due to the need for accurate document representations and the effective modeling of complex label dependencies. Existing methods either underutilize label semantics for representation generation or face challenges in fully capturing inter-label dependencies using contrastive learning, often leading to suboptimal label predictions. To address these issues, we propose a novel end-to-end framework, Dual Representation Space Optimization (DRSO), for MLTC. DRSO tackles these challenges through two key components: a semantic-aware network that refines document representations by leveraging label semantics and an adaptive multi-label contrastive learning mechanism that captures inter-label dependencies to optimize label distributions in the prediction space. Extensive experiments on benchmark datasets demonstrate that DRSO outperforms state-of-the-art methods, showcasing its effectiveness in enhancing both representation quality and label prediction accuracy1.

This paper investigates the anti-synchronization control problem for a class of Memristor-based Neural Networks with time-delay and semi-Markov jump parameters. Firstly, to further effectively utilize the network resources, a novel Intermittent Discrete Dynamic Event-triggered (IDDET) scheme is introduced, where the dynamical update law of the IDDET scheme is designed to be related to the current sampling state. Secondly, by fully considering the information of jump parameters, time-delay, sampling period, and interaction of the current and past states, a general common Lyapunov functional is constructed. Then, with the virtue of inequalities analysis technique and quadratic polynomial negative definite lemma, a new less conservative criterion guaranteeing anti-synchronization for the underlying master-slave systems is derived in the form of Linear Matrix Inequalities (LMIs). In the end, the validity of our results is illustrated through a numerical example.

In remote sensing image processing for Earth and environmental applications, super-resolution (SR) is a crucial technique for enhancing the resolution of low-resolution (LR) images. In this study, we proposed a novel algorithm of frequency-domain super-resolution with reconstruction from compressed representation. The algorithm follows a multistep procedure: first, an LR image in the space domain is transformed to the frequency domain using a Fourier transform. The frequency-domain representation is then expanded to the desired size (number of pixels) of a high-resolution (HR) image. This expanded frequency-domain image is subsequently inverse Fourier transformed back to the spatial domain, yielding an initial HR image. A final HR image is then reconstructed from the initial HR image using a low-rank regularization model that incorporates a nonlocal smoothed rank function (SRF). We evaluated the performance of the new algorithm by comparing the reconstructed HR images with those generated by several commonly used SR algorithms, including: 1) bicubic interpolation; 2) sparse representation; 3) adaptive sparse domain selection and adaptive regularization; 4) fuzzy-rule-based (FRB) algorithm; 5) SR convolutional neural networks (SRCNNs); 6) fast SR convolutional neural networks (FSRCNNs); 7) practical degradation model for deep blind image SR; 8) the frequency separation for real-world SR (FSSR); and 9) the enhanced SR generative adversarial networks (ESRGANs). The algorithms were tested on Landsat-8 and Moderate Resolution Imaging Spectroradiometer (MODIS) multiresolution images over various locations, as well as on images with artificially added noise to assess the robustness of each algorithm. Results show that: 1) the proposed new algorithm outperforms the others in terms of the peak signal-to-noise ratio, structure similarity, and root-mean-square error and 2) it effectively suppresses noise during HR reconstruction from noisy low-resolution (LR) images, overcoming a key limitation of existing SR methods.

Chain of thought (CoT) has proven useful for problems requiring complex reasoning. Many of these problems are both textual and multimodal. Given the inputs in different modalities, a model generates a rationale and then uses it to answer a question. Because of the hallucination issue, the generated soft negative rationales with high textual quality but illogical semantics do not always help improve answer accuracy. This study proposes a rationale generation method using soft negative sampling (SNSE-CoT) to mitigate hallucinations in multimodal CoT. Five methods were applied to generate soft negative samples that shared highly similar text but had different semantics from the original. Bidirectional margin loss (BML) was applied to introduce them into the traditional contrastive learning framework that involves only positive and negative samples. Extensive experiments on the ScienceQA dataset demonstrated the effectiveness of the proposed method. Code and data are released at https://github.com/zgMin/SNSE-CoT.

Extractive summarization aims to generate summaries by extracting and concatenating critical information from long documents. However, existing extractive summarization methods typically rely on large-scale labeled datasets, which are expensive and time-consuming. In addition, due to the inherent class imbalance problem in extractive summarization, traditional approaches such as resampling make it difficult to address this challenge effectively. To address these issues, we treat extractive summarization as a class imbalance problem and propose a method called HDCSUM. Specifically, we first employ consistency-training and pseudo-labeling strategies to fully use limited labeled data and many unlabeled data. Additionally, HD-CSUM can amplify the idiosyncratic information of each sentence and focus more on the semantic differences by jointly utilizing differential amplifiers and contrastive learning. We introduce a weighted cross-entropy loss function to further address the class imbalance problem. Extensive experiments on the challenging and widely used CNN/Daily Mail and BBC XSum benchmark datasets show that our method outperforms state-of-the-art semi-supervised methods. Our code is available at GitHub1.

Mapping glacial lakes is a prerequisite for understanding their responses to climate changes and assessing potential danger of glacial lake outburst floods (GLOFs). Although remote sensing technology has enabled continuous monitoring and assessment of global glacial lake evolution, accurately and reliably extracting glacial lakes in high mountain areas remains challenging. This study proposed a glacial lake mapping framework based on multisource remote sensing technique and an improved deep learning (DL) model to address diverse challenges associated with glacial lake mapping in high mountain areas. Test results obtained in the Southeast Tibetan Plateau (TP) region demonstrate that the framework achieves high accuracy, with measures of Dice, precision, recall, and intersection over union (IOU) reaching 0.8986, 0.9009, 0.8963, and 0.8287, respectively. It effectively mitigates the impacts of cloud cover, shadowing, glacial debris, lake-water turbidity, and freeze-thaw lake-water conditions on glacial lake delineation. This study provided a concrete solution for glacial lake mapping in high mountain areas with complex topography, and it supported technical advancements in GLOF risk identification.

In natural language learning, pre-trained language models (PLM) can acquire rich knowledge and concepts from rich corpora, making it possible to use PLM-based models for knowledge graph completion (KGC) tasks. However, in previous research, when applying pre-trained models to knowledge graph completion tasks, two main challenges persist: (1) Existing knowledge graph completion models are typically evaluated based on the closed-world assumption(CWA), thus lacking evaluation methods suitable for the open-world assumption(OWA), which constitutes a significant challenge in the current field of knowledge graph completion. (2) Extracting useful information, reducing noise, and providing clear interpretability for extracting effective information from the extensive prior knowledge embedded in pre-trained language models is also a crucial issue. Although the loss function can reduce noise to a certain extent, from the perspective of information theory, only relying on the loss function has a limited effect on noise reduction, and the model needs more professional tools to reduce noise and reduce the impact of irrelevant information on model performance. To address the aforementioned challenges, we propose a dual confrontation learning model based on the variational information bottleneck method. This model restricts information flow and feature selection from the perspective of information theory to reduce noise and enhance model performance while providing clear interpretability for this process. Based on extensive experiments and comprehensive evaluations conducted under both closed-world and open-world assumptions, this model successfully extracts valuable knowledge from pre-trained language models to accomplish KGC tasks. Simultaneously, it minimizes noise, removes non-robust features, enhances model reliability, and optimizes model performance. More importantly, we offer a strong interpretability for the process in which our model constrains information flow to reduce noise.

BACKGROUND: The biomedical literature is growing rapidly, and it is increasingly important to extract meaningful information from the vast amount of literature. Biomedical named entity recognition (BioNER) is one of the key and fundamental tasks in biomedical text mining. It also acts as a primitive step for many downstream applications such as relation extraction and knowledge base completion. Therefore, the accurate identification of entities in biomedical literature has certain research value. However, this task is challenging due to the insufficiency of sequence labeling and the lack of large-scale labeled training data and domain knowledge. RESULTS: In this paper, we use a novel word-pair classification method, design a simple attention mechanism and propose a novel architecture to solve the research difficulties of BioNER more efficiently without leveraging any external knowledge. Specifically, we break down the limitations of sequence labeling-based approaches by predicting the relationship between word pairs. Based on this, we enhance the pre-trained model BioBERT, through the proposed prefix and attention map dscrimination fusion guided attention and propose the E-BioBERT. Our proposed attention differentiates the distribution of different heads in different layers in the BioBERT, which enriches the diversity of self-attention. Our model is superior to state-of-the-art compared models on five available datasets: BC4CHEMD, BC2GM, BC5CDR-Disease, BC5CDR-Chem, and NCBI-Disease, achieving F1-score of 92.55%, 85.45%, 87.53%, 94.16% and 90.55%, respectively. CONCLUSION: Compared with many previous various models, our method does not require additional training datasets, external knowledge, and complex training process. The experimental results on five BioNER benchmark datasets demonstrate that our model is better at mining semantic information, alleviating the problem of label inconsistency, and has higher entity recognition ability. More importantly, we analyze and demonstrate the effectiveness of our proposed attention.

Abstract Sparse representation is a powerful statistical image modelling technique and has been successfully applied to image denoising. For a given patch, a non‐convex non‐local similarity adaptive method is adopted for sparse representation of images. First, it uses the autoregressive model to perform dictionary learning from sample patch datasets. Second, the sparse representation of an image introduces non‐convex non‐local self‐similarity as the regularization term. In order to make better use of the sparse regularization method for image denoising, the parameters used in this study are estimated using adaptive methods. This model is more efficient and accurate, Compared with K‐means singular value decomposition (KSVD) algorithm, a generalized K‐means clustering method, total variation of population sparsity (GSTV) algorithm, adaptive sparse domain selection (ASDS) algorithm, forward denoising convolutional neural network (DnCNNs), a fast and flexible Convolutional Neural Network image denoising method (FFNNet) and operator‐splitting algorithm to minimize the Euler elastica functional (OSEEF). Image noise‐reduction experiments confirmed that using the adaptive regularization method, the results in peak signal to noise ratio (PSNR) and visual opinion are better than other algorithms.

To study the applicability of full polarimetric synthetic aperture radar (SAR) data to identify alpine glaciers in the central Himalayas, six polarimetric decomposition methods were used to obtain 20 polarimetric characteristic parameters based on the Advanced Land Observing Satellite 2 (ALOS-2) Phased Array type L-band SAR (PALSAR) data. Object-oriented multiscale segmentation was performed on a Landsat 8 Operational Land Imager (OLI) image prior to classification, and the vector boundaries of different types of training samples were selected from the segmented results. We performed a support vector machine (SVM)-based classification on the characteristic parameters from each polarimetric decomposition. All 20 parameters were then screened and combined according to different requirements: the degree of separability of different types of training samples and the type of scattering mechanisms. The results show that the classification accuracy of the incoherent decomposition characteristics based on the covariance matrix is the best, reaching 87%, and it can exceed 91% after adding the local incidence angle to the suite of classifiers. Eventually, more than 93% accuracy was achieved using a combination of multiple polarimetric parameters, which reduced the misclassification between bare ice and rock. We also analyzed the use of controlling factors on the accuracy of alpine glacier identification and found that the polarimetric information and aspect of the glacier surface are the most important factors. The former is the main basis for identification but the latter will confuse the feature distributions of different categories and cause misclassification.

On May 31, 2003, the scan line corrector (SLC) of the Enhanced Thematic Mapper Plus (ETM+) on-board the Landsat-7 satellite failed, resulting in strips of data lost in all ETM+ images acquired since then. In this paper, we proposed a novel inpainting algorithm for recovering the ETM+ SLC-off images. The two slopes of the boundaries of each missing stripe were extracted through the Hough transform, ignoring the slope of the edge of the strip that overlaps the edge of the image. An adaptive dictionary was then developed and trained using ETM+ SLC-on images acquired before May 31, 2003 so that the physical characteristics and geometric features of the ground coverage of the data-missing strips can be considered during recovery. To make the algorithm computationally efficient, data-missing strips were repaired along their slope directions by using the logdet $\left ({\cdot }\right)$ low-rank nonconvex model along with the dictionary. The algorithm was tested using the simulated ETM+ SLC-off images created from a multiband ETM+ SLC-on image file and compared to the high accuracy low-rank tensor completion (HaLRTC), logDet, and tensor nuclear norm (TNN) algorithms. The results show that the ETM+ images restored using the new algorithm have lower RMSE, higher PSNR and structure similarity (SSIM) values, and better visualization. These results indicate that the new algorithm performs better than the other three algorithms and can efficiently and accurately restore the data-missing stripes.

Summary This paper presents a convergence of distributed key‐value storage systems in clouds and supercomputers. It specifically presents ZHT, a zero‐hop distributed key‐value store system, which has been tuned for the requirements of high‐end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. ZHT has some important properties, such as being lightweight, dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append, compare and swap, callback in addition to the traditional insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 64 nodes, an Amazon EC2 virtual cluster up to 96 nodes, to an IBM Blue Gene/P supercomputer with 8K nodes. We compared ZHT against other key‐value stores and found it offers superior performance for the features and portability it supports. This paper also presents several real systems that have adopted ZHT, namely, FusionFS (a distributed file system), IStore (a storage system with erasure coding), MATRIX (distributed scheduling), Slurm++ (distributed HPC job launch), Fabriq (distributed message queue management); all of these real systems have been simplified because of key‐value storage systems and have been shown to outperform other leading systems by orders of magnitude in some cases. It is important to highlight that some of these systems are rooted in HPC systems from supercomputers, while others are rooted in clouds and ad hoc distributed systems; through our work, we have shown how versatile key‐value storage systems can be in such a variety of environments. Copyright © 2015 John Wiley & Sons, Ltd.

Summary Data‐driven programming models such as many‐task computing (MTC) have been prevalent for running data‐intensive scientific applications. MTC applies over‐decomposition to enable distributed scheduling. To achieve extreme scalability, MTC proposes a fully distributed task scheduling architecture that employs as many schedulers as the compute nodes to make scheduling decisions. Achieving distributed load balancing and best exploiting data locality are two important goals for the best performance of distributed scheduling of data‐intensive applications. Our previous research proposed a data‐aware work‐stealing technique to optimize both load balancing and data locality by using both dedicated and shared task ready queues in each scheduler. Tasks were organized in queues based on the input data size and location. Distributed key‐value store was applied to manage task metadata. We implemented the technique in MATRIX, a distributed MTC task execution framework. In this work, we devise an analytical suboptimal upper bound of the proposed technique, compare MATRIX with other scheduling systems, and explore the scalability of the technique at extreme scales. Results show that the technique is not only scalable but can achieve performance within 15% of the suboptimal solution. Copyright © 2015 John Wiley & Sons, Ltd.

Summary Conventional data serialization tools assume that objects to be coded are usually small in size so a single CPU core can encode it in a timely manner. In the era of Big Data, however, object gets increasingly complex and larger, which makes data serialization become a new performance bottleneck. This paper describes an approach to parallelize data serialization by leveraging multiple cores. Parallelizing data serialization introduces new questions such as how to split the (sub)objects, how to allocate the available cores, and how to minimize its overhead in practice. In this paper we design a framework for parallelly serializing large objects and analyze the design tradeoffs under different scenarios. To validate the proposed approach, we implemented parallel protocol buffers—the parallel version of Google's Protocol Buffers, a widely‐used data serialization utility. Experimental results confirm the effectiveness of Parallel Protocol Buffers: multiple cores employed in data serialization achieve highly scalable performance and incur negligible overhead. Copyright © 2015 John Wiley & Sons, Ltd.

In the era of big data and cloud, distributed key-value stores are increasingly used as building blocks of large-scale applications. Comparing to traditional relational databases, key-value stores are particularly compelling due to their low latency and excellent scalability. Many big companies, such as Facebook and Amazon, run multiple different applications and services on top of a single key-value store deployment to reduce the deployment and maintenance complexity as well as economic cost. However, every application has its performance requirement but most current key-value store systems are designed to serve every application request equally. This design works well when a single application accesses the key-value store, but it is not as good for the emerging concurrent multi-application scenario. In this paper, we present ZHT/Q, a flexible QoS (Quality of Service) fortified distributed key-value storage system for clouds and data centers. It improves the overall throughput by an order of magnitude and still satisfies different applications' latency requirements with QoS using dynamic and adaptive request batching mechanisms. The experiment results show that our new system delivers up to 28 times higher throughput than the base solution while more than 99% of requests' latency requirements are satisfied.

In this paper, we propose and implement a key-value store that supports MPI while allowing application access at any time without having to declaring in the same MPI communication world. This feature may significantly simplify the application design and allow programmers leverage the power of key-value store in an intuitive way. In our preliminary experiment results captured from a supercomputer at Los Alamos National Laboratory, our prototype shows linear scalability at up to 256 nodes.

The emerging applications in big data and social networks issue rapidly increasing demands on graph processing. Graph query operations that involve a large number of vertices and edges can be tremendously slow on traditional databases. The state-of-the-art graph processing systems and databases usually adopt master/slave architecture that potentially impairs their The contributions of this paper are as follows: scalability. This work describes the design and implementation of a new graph processing system based on Bulk Synchronous Parallel model. Our system is built on top of ZHT, a scalable distributed key-value store, which benefits the graph processing in terms of scalability, performance and persistency. The experiment results imply excellent scalability.

Data driven programming models like MapReduce have gained the popularity in large-scale data processing. Although great efforts through the Hadoop implementation and framework decoupling (e.g. YARN, Mesos) have allowed Hadoop to scale to tens of thousands of commodity cluster processors, the centralized designs of the resource manager, task scheduler and metadata management of HDFS file system adversely affect Hadoop's scalability to tomorrow's extreme-scale data centers. This paper aims to address the YARN scaling issues through a distributed task execution framework, MATRIX, which was originally designed to schedule the executions of data-intensive scientific applications of many-task computing on supercomputers. We propose to leverage the distributed design wisdoms of MATRIX to schedule arbitrary data processing applications in cloud. We compare MATRIX with YARN in processing typical Hadoop workloads, such as WordCount, TeraSort, Grep and RandomWriter, and the Ligand application in Bioinformatics on the Amazon Cloud. Experimental results show that MATRIX outperforms YARN by 1.27X for the typical workloads, and by 2.04X for the real application. We also run and simulate MATRIX with fine-grained sub-second workloads. With the simulation results giving the efficiency of 86.8% at 64K cores for the 150ms workload, we show that MATRIX has the potential to enable Hadoop to scale to extreme-scale data centers for fine-grained workloads.

One way to efficiently utilize the coming exascale machines is to support a mixture of applications in various domains, such as traditional large-scale HPC, the ensemble runs, and the fine-grained many-task computing (MTC). Delivering high performance in resource allocation, scheduling and launching for all types of jobs has driven us to develop Slurm++, a distributed workload manager directly extended from the Slurm centralized production system. Slurm++ employs multiple controllers with each one managing a partition of compute nodes and participating in resource allocation through resource balancing techniques. In this paper, we propose a monitoring-based weakly consistent resource stealing technique to achieve resource balancing in distributed HPC job launch, and implement the technique in Slurm++. We compare Slurm++ with Slurm using micro-benchmark workloads with different job sizes. Slurm++ showed 10X faster than Slurm in allocating resources and launching jobs -- we expect the performance gap to grow as the job sizes and system scales increase in future high-end computing systems.

Load balancing techniques (e.g. work stealing) are important to obtain the best performance for distributed task scheduling systems that have multiple schedulers making scheduling decisions. In work stealing, tasks are randomly migrated from heavy-loaded schedulers to idle ones. However, for data-intensive applications where tasks are dependent and task execution involves processing a large amount of data, migrating tasks blindly yields poor data-locality and incurs significant data-transferring overhead. This work improves work stealing by using both dedicated and shared queues. Tasks are organized in queues based on task data size and location. We implement our technique in MATRIX, a distributed task scheduler for many-task computing. We leverage distributed key-value store to organize and scale the task metadata, task dependency, and data-locality. We evaluate the improved work stealing technique with both applications and micro-benchmarks structured as direct acyclic graphs. Results show that the proposed data-aware work stealing technique performs well.

State-of-the-art, yet decades-old, architecture of high-performance computing systems has its compute and storage resources separated. It thus is limited for modern data-intensive scientific applications because every I/O needs to be transferred via the network between the compute and storage resources. In this paper we propose an architecture that hss a distributed storage layer local to the compute nodes. This layer is responsible for most of the I/O operations and saves extreme amounts of data movement between compute and storage resources. We have designed and implemented a system prototype of this architecture - which we call the FusionFS distributed file system - to support metadata-intensive and write-intensive operations, both of which are critical to the I/O performance of scientific applications. FusionFS has been deployed and evaluated on up to 16K compute nodes of an IBM Blue Gene/P supercomputer, showing more than an order of magnitude performance improvement over other popular file systems such as GPFS, PVFS, and HDFS.

With the exponential growth of supercomputers in parallelism, applications are growing more diverse, including traditional large-scale HPC MPI jobs, and ensemble workloads such as finer-grained many-task computing (MTC) applications. Delivering high throughput and low latency for both workloads requires developing a distributed job management system that is magnitudes more scalable than today's centralized ones. In this paper, we present a distributed job launch prototype, SLURM++, which is comprised of multiple controllers with each one managing a partition of SLURM daemons, while ZHT (a distributed key-value store) is used to store the job and resource metadata. We compared SLURM++ with SLURM using micro-benchmarks of different job sizes up to 500 nodes, with excellent results showing 10X higher throughput. We also studied the potential of distributed scheduling through simulations up to millions of nodes.

This paper presents ZHT, a zero-hop distributed hash table, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. The goals of ZHT are delivering high availability, good fault tolerance, high throughput, and low latencies, at extreme scales of millions of nodes. ZHT has some important properties, such as being light-weight, dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append (providing lock-free concurrent key/value modifications) in addition to insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 512-cores, to an IBM Blue Gene/P supercomputer with 160K-cores. Using micro-benchmarks, we scaled ZHT up to 32K-cores with latencies of only 1.1ms and 18M operations/sec throughput. This work provides three real systems that have integrated with ZHT, and evaluate them at modest scales. 1) ZHT was used in the FusionFS distributed file system to deliver distributed meta-data management at over 60K operations (e.g. file create) per second at 2K-core scales. 2) ZHT was used in the IStore, an information dispersal algorithm enabled distributed object storage system, to manage chunk locations, delivering more than 500 chunks/sec at 32-nodes scales. 3) ZHT was also used as a building block to MATRIX, a distributed job scheduling system, delivering 5000 jobs/sec throughputs at 2K-core scales. We compared ZHT against other distributed hash tables and key/value stores and found it offers superior performance for the features and portability it supports.

The integral equation model (IEM) is considered as a promising algorithm for soil moisture retrieval from active microwave data over bare soil and sparsely vegetated conditions. However, the soil dielectric constant is implicitly embedded in the complicated IEM; inversion of soil moisture is often accomplished through iteration and is thus computationally expensive, particularly when it is applied to retrieve soil moisture from active microwave data on a large scale. To simplify the inversion process of soil moisture directly from the active microwave data, basic math functions were adopted to fit the simulation results of the original IEM so that the radar backscattering coefficient becomes an explicit function of soil dielectric constant or the soil dielectric constant is an explicit function of radar backscattering coefficient. Soil moisture is then calculated directly from radar backscattering coefficient without iteration. We called this model empirically adopted IEM (EA-IEM). The accuracy of the EA-IEM to the original IEM and its applicability are analyzed through three processes: model intercomparison, sensitivity analysis, and model comparison withinsitumeasurements. The average differences of backscattering coefficients between the EA-IEM and the original IEM are 0.14 dB for HH-polarization and 0.12 dB (Gaussian correlation function) and 0.2 dB (exponential correlation function) for VV-polarization. The sensitivity of soil moisture variation is examined under the consideration of absolute and relative calibration errors. A comparison between the soil moisture estimated and the measurements is performed, and the root-mean-square (rms) error is found to be 3.4%, suggesting that the EA-IEM performs well in these real cases. All these analyses indicate that the EA-IEM is a good representative of the original IEM and can be used to retrieve soil moisture under the tested range of model parameters: incidence angles between 10degand 60deg, soil dielectric constants between 4 and 42, surface rms height from 4 to 31 mm, and correlation length from 50 to 250 mm.

Broadband albedo is an important geophysical parameter in the Earth surface-atmosphere interaction in global climate change, hydrological cycle and snowmelt runoff studies. To derive the broadband albedo accurately from satellite optical sensor observation at limited bands and at a single observation angle, the bidirectional reflectance factor (BRF) has to be quantitatively specified. When we tried to validate snow covered sea ice surface BRF model results using field-measured directional reflectance (FMDR). we recognized that the field measurements are actually the hemispherical-directional reflectance factor (HDRF) instead of BRF because of the existence of diffuse irradiance even under clear skies. Therefore, we made a comparison among FMDR and simulated BRF and HDRF. Our logic is as follow: if the HDRF patterns agree with FMDR while the simulated BRF patterns do not, we still consider that the simulated BRF patterns are validated. The comparison indicates that the simulated HDRF generally agrees with FMDR for the main part of the viewing hemisphere. This agreement suggests that the BRF model provides a good approximation of snow surface BRF for the central (viewing zenith angle <50/spl deg/) and side looking parts of the viewing hemisphere. The study also suggests that the surface roughness and surface heterogeneity may also affect the actual surface BRF pattern.

Drought is a serious climatic condition that affects nearly all climatic zones worldwide, with semi-arid regions being especially susceptible to drought conditions because of their low annual precipitation and sensitivity to climate changes. Drought indices such as the Standardized Precipitation Index (SPI) have been developed for quantifying drought conditions. Usually, calculation of drought indices requires a long record of climatic data, which may not be available because of the inaccessibility of a region and a lack of human activity. Remote sensing of semi-arid vegetation can provide vegetation indices which can be used to link drought conditions when correlated with various drought indices. Spectral reflectance measurements of creosote and black gramma grass were taken between January and November 2003 in the Sevilleta National Wildlife Refuge of New Mexico and various vegetation indices were derived. Each vegetation index was correlated with the SPI of various weekly timescales at varying time-lag intervals calculated from 1999 to seek the best vegetation index that can be used as the best indicator of SPI conditions. The results show a strong linear correlation between the vegetation indices NDVI, Greenness Index, ARVI and drought index SPI at various SPI measurements with various lag times

We recognized that snow surface direct beam spectral albedo over a wide range of solar zenith angles can be derived by reciprocity from surface directional spectral reflectance under overcast conditions if the ground-level sky diffuse light is perfectly isotropic. Whether the method can produce accurate results under typical overcast conditions is the focus of this study. Theoretical analysis indicates that the error in estimation of direct beam spectral albedo by reciprocity should be proportional to the covariance of the zenith-dependent and azimuthally averaged sky diffuse radiation and surface bi-directional reflectance. Consequently, the error would be small when the sky diffuse light varies randomly. For general overcast conditions, therefore, we focused on situations in which sky diffuse light would exhibit a non-random variation. Therefore, we used a multi-layer zenith- and azimuth-dependent radiative transfer model to simulate both direct beam spectral albedo under clear skies and surface directional spectral reflectance under various overcast conditions. We assessed the feasibility of the reciprocity method in practical use through comparing the two sets of simulation results. The comparison indicates that the derived direct beam albedo in the visible wavelength region, the direct beam spectral albedo derived under typical stratus cloud optical thicknesses (/spl tau/10-60) are accurate to within /spl plusmn/0.01. for all cases except for extremely large solar zenith angles (>88/spl deg/). At near infrared to shortwave infrared wavelengths, results are accurate to within 0.01 for all zenith angles smaller than 74/spl deg/ at 862 nm and for all solar zenith angles smaller than 63 /spl deg/ at 2250 nm. Thus, the reciprocity approach can provide accurate direct beam spectral albedo complemental to those from the traditional method.

Surface spectral bi-directional reflectance function (BRF) is a fundamental surface property in radiation interaction with the atmosphere-Earth surface system. Knowledge of surface BRFs is also crucial to estimating surface albedo from remote sensing data sets. During two austral summer cruises in 1999 and 2000, surface spectral directional reflectance patterns of first-year (FY) and multi-year (MY) sea ice were measured under clear skies for 512 spectral channels between 350-1050 nm. After smoothing raw data, calibrating spectral albedo of the reference panel, and removing the impact of sky diffuse radiation, representative patterns of BRF of FY and MY sea ice types in the Southern Ocean have been produced. The solar incidence angles (50 to 65 degrees) associated with the derived patterns are moderate. Some common features are seen in the BRF patterns of both FY and MY sea ice types. A moderately strong peak occurs in the principal plane when the sensor is looking into the azimuth from which the solar illumination comes. When the sensor is viewing towards the azimuth perpendicular to the principal plane, the BRF values are close to the surface albedo. However, the FY ice exhibits stronger surface anisotropy than the MY ice because the former has a smoother surface than the latter. The BRF pattern of the FY ice has a minimum in the backward direction. From the backward minimum towards the forward peak, BRF values of the FY sea ice show a monotonous increase along almost all great circle scans in the hemisphere. On the contrary, the MY ice looks similar to an isotropic (or Lambertian) surface when it is viewed from almost all directions except for the forward viewing azimuth with large viewing zenith angles (60-80 degrees). Besides, there is a discernible secondary peak in the backward direction of the MY ice BRF, presumably caused by reflectance from the fore-slopes of surface undulation. The resulting patterns will be especially useful in deriving surface albedo from NASA moderate resolution imaging spectrometer (MODIS).

We investigate derivation of kernel functional expression of Antarctic sea ice surface BRDF using field measurements. Two new methods we developed recently are used to expand the scope and enhance the use of field albedo and reflectance measurements. The first normalizes, among ice stations of the same ice type, the BRDF difference caused by slight differences in the physical properties of the ice cover. The second method derives direct beam spectral albedo from hemispherical directional spectral reflectance measured under overcast skies using the reciprocity between them. The derived direct beam spectral albedo estimates cover the full range of solar zenith angles. We combine the resulting data sets together to determine the coefficients of the BRDF kernel functions through stepwise linear regression. The derived BRDF kernel functional expression is representative of the ice type of interest, and is optimized in the sense that the ordinary limitation of narrow solar zenith angle range no longer exists. The resulting BRDF expression will contribute to improving derivation of sea ice surface albedo from remote sensing measurements.

In comparison between in situ and satellite-derived spectral reflectance so as to validate a newly launched satellite sensor, the difficulty is that the ground measurement is generally carried out at a higher resolution than the satellite sensor, especially for moderate or high resolution sensors such as the Moderate Resolution Imaging Spectroradiometer (MODIS) or the Advanced Very High Resolution Radiometer (AVHRR) etc. To overcome this difficulty, we adopt sub-pixel correction method in a comparison study between in situ and MODIS-derived spectral reflectances of snow and sea ice in the Amundsen Sea, Antarctica. The position of the in situ measurement is accurately located within the imagery granule using the combination of small angle approximation and direct matching algorithms. Apparent reflectance retrieved from MODIS granule for the station pixel is atmospherically corrected. The corresponding spectral albedo and directional reflectance at the same viewing geometry as MODIS are derived from ground-based spectroradiometer measurements. The averaged spectral reflectance and albedo in the vicinity of each ice station are simulated for the corresponding MODIS pixel from ground spectral measurements by weighing over different surface types (various ice types and open water) so that reflectance difference at a sub-pixel level is appropriately corrected. Values of ground-based MODIS spectral simulation are compared with satellite-derived values so that the discrepancy is estimated.

Large snow meltclusters of 1 cm-order size are ubiquitously observed in summer snow cover on sea ice in the Southern Ocean. To study their effect on remotely sensed signals, we have to understand the absorption and single scattering properties of electromagnetic waves by these meltclusters. In this study, the meltclusters are characterized as spheres and scattering phase function and asymmetry factor are calculated on the basis of geometrical optics methods. The number of internal reflections and transmissions are truncated based on the ratio of incident irradiance at the n-th interface to the initial incident irradiance. The effect of finite size in absorption efficiency calculation using GOM method is corrected using Mie calculation. We use a combination of bracketing and a hybrid algorithm of bisection and Newton-Raphson methods to find the incident angles for a given scattering angle. The absorption efficiency, scattering efficiency, and phase function of a large single snow grain are calculated for the whole solar spectrum.