Speech contains rich information on the emotions of humans, and Speech Emotion Recognition (SER) has been an important topic in the area of human-computer interaction. The robustness of SER models is crucial, particularly in privacy-sensitive and reliability-demanding domains like private healthcare. Recently, the vulnerability of deep neural networks in the audio domain to adversarial attacks has become a popular area of research. However, prior works on adversarial attacks in the audio domain primarily rely on iterative gradient-based techniques, which are time-consuming and prone to overfitting the specific threat model. Furthermore, the exploration of sparse perturbations, which have the potential for better stealthiness, remains limited in the audio domain. To address these challenges, we propose a generator-based attack method to generate sparse and transferable adversarial examples to deceive SER models in an end-to-end and efficient manner. We evaluate our method on two widely-used SER datasets, Database of Elicited Mood in Speech (DEMoS) and Interactive Emotional dyadic MOtion CAPture (IEMOCAP), and demonstrate its ability to generate successful sparse adversarial examples in an efficient manner. Moreover, our generated adversarial examples exhibit model-agnostic transferability, enabling effective adversarial attacks on advanced victim models.

This paper presents the development of an independent algorithm at NOAA/STAR for processing radio occultation (RO) bending angle and refractivity data from RO observations explicitly designed for multi-GNSS RO missions. The primary aim is to understand the uncertainties introduced in the processing, from excess phase data to bending angle and refractivity profiles. This study investigates three algorithms that convert RO excess phases to bending angles. These three algorithms are full spectrum inversion (FSI), canonical transform type 2 (CT2), and phase matching (PM). The STAR-developed FSI algorithm has been fully integrated into the Radio Occultation Processing Package (ROPP) version 10.0. This integration provides users with an alternative to wave optics and geometry optics through configurable settings. A detailed comparison of bending angle and refractivity results generated by these three algorithms, explicitly focusing on COSMIC-2 and Spire data, is presented. The analysis highlights discrepancies and uncertainties inherent in bending angle and refractivity processing, providing valuable insights into the performance of each algorithm.

Launched in November 2017, the National Oceanic and Atmospheric Administration-20 (NOAA-20) Visible Infrared Imaging Radiometer Suite (VIIRS) has successfully operated over five years and produced high-quality sensor data records (SDRs), which have significantly contributed to the Earth’s environmental and climate change studies. The VIIRS instrument collects data in the reflective solar bands (RSBs) from bands M1 to M11 and I1 to I3 with two spatial resolutions of 375 m for imaging ($I$) bands and 750 m for moderate resolution ($M$) bands covering a wavelength range from 401 to 2284 nm. For the RSBs on-orbit radiometric calibration, VIIRS primarily uses solar diffuser (SD) observations along with alternative long-term lunar calibrations and deep convective cloud (DCC) trends. The NOAA VIIRS SDR team observed upward long-term trends after three years in the lunar calibration coefficients (called lunar F-factors) compared to the initial on-orbit SD F-factors. These long-term lunar trend changes were validated with the DCC observation results and the operational radiometric calibration coefficient (called F-PREDICTED) lookup table (LUT) was updated in November of 2021, which was proportional to observed radiance. After five years of on-orbit operations, the performance of the current operational F-PREDICTED LUT was evaluated in comparison with the long-term DCC trends. After application of the LUT, the results showed excellent on-orbit radiometric calibration stability providing confidence for the user communities of NOAA-20 VIIRS SDR products.

The Joint Polar Satellite System-2 (JPSS-2) was successfully launched on November 10, 2022 and renamed as NOAA-21 after it reached its final polar orbit. Twenty-five days after launch, on December 5, 2022, the NOAA-21 Visible Infrared Imaging Radiometer Suite (VIIRS) started collecting science data. After intensive analysis, calibration and validation of VIIRS science raw data record (RDR), telemetry RDR, and sensor data record (SDR) radiometric and geolocation data products, the NOAA-21 VIIRS SDRs reached Beta maturity on Feb. 23, 2023 and provisional maturity on March 30, 2023. This paper provides an update on the NOAA-21 VIIRS SDR availability, post-launch calibration and validation activities, and SDR quality assessments.

Passive microwave radiometer sounders are the backbones for numerical weather prediction under all sky conditions. Modern 5G technology represents a significant advancement in internet and communication technology, but may overlap with some of the passive microwave sounder channels, in particular the 5G’s 24 GHz channel may interfere with the 23.8 GHz total column water vapor channel measurements over land. This paper assesses the importance of the 23.8GHz channel for atmospheric sounding by analyzing the vertical weighting functions at different pressure levels to address the issue of sensitivity of this channel over land, and related issues such as land surface emissivity. Modeling experiments are also performed to assess the impacts of this channel on water vapor retrievals using the Microwave Integrated Retrieval System (MiRS). The role of this channel in detecting planetary boundary layer (PBL) water vapor is also discussed.

In the NOAA operational processing, the Thermal Emissive Band (TEB) data from the VIIRS onboard the NOAA-20 and S- NPP satellites are not calibrated if all scans in a granule are flagged as lunar intrusion into space view (SV). As a result, more than 100 NOAA-20 and S-NPP VIIRS single-gain TEB granules are un-calibrated each year. For M13 (dual-gain fire detection band), the number of un -calibrated granules due to lunar intrusion is doubled because of an additional bug in the operational processing software. This study presents a Lowest N Algorithm for calibrating VIIRS TEB during lunar intrusions. It takes advantage of the fact that the extent of the full moon image is smaller than the field of view of VIIRS SV. Moreover, the bug that affects M13 calibration was also fixed. A VIIRS SDR algorithm code change package has been implemented in the NOAA operational processing since March 30,2021.

In this paper, we present recent progress in the development of interactive integrated Cal/Val system enhanced by 3D globe visualization. Examples of two of such enhanced systems: Small satellite (Smallsat) integrated Cal/Val system (ICVS) in support of the Smallsat program and VIIRS Global Validation System (VGVS) are presented. For the Smallsat ICVS, the global contextual visualization of Global Navigation Satellite System (GNSS) Radio Occultation data enables orbital phasing analysis and temperature profile quality monitoring from the six COSMIC2 satellites. The 3D visualization of TEMPEST-D microwave SmallSat multi-channel observations facilitates geolocation accuracy and data inventory monitoring, both are critical to the inter-sensor Cal/Val. The VGVS consists of ~20 vicarious validation sites. Mission-long time series for instrument performance trending and degradation monitoring are supported. The interactive WebGL based 3D interface provides integrated monitoring of VIIRS multi-channel radiometric performance in the context of types, locations, radiometric uncertainties and stability of validation sites.

As a panchromatic band, the Day-Night Band (DNB) on National Oceanic Atmospheric Administration (NOAA)-20 Visible Infrared Imaging Radiometer Suite (VIIRS) has a unique capability of observing very low radiances down to Nano-watt [cm-1sr-1] level, because of its three different focal planes with gain stages. The DNB radiometric calibration is based on the Solar Diffuser observations near the night to day time termination points. Besides the primary calibration source of SD, the moon provides alternative source of calibration with a specific lunar roll maneuvers. This study provides detailed methodology and results of lunar calibration for NOAA-20 VIIRS sensor and initial two year DNB radiometric calibration results from SD and scheduled lunar calibration. Over two years of NOAA-20 VIIRS operation, the Low Gain Stage (LGS) slope (inverse gain) trend showed a decreasing trend, whereas the calculated monthly lunar F-factor showed a stable response (inverse gain normalized to the lunar irradiance model) using a lunar irradiance model. These differences are monitored, compared, and applied for the best quality of NOAA-20 VIIRS DNB product.

This paper presents NOAA-20 and S-NPP Visible Infrared Imaging Radiometer Suite (VIIRS) Reflective Solar Bands (RSB) and Thermal Emissive Bands (TEB) Sensor Data Records (SDR) performance and recent improvements to support user communities. Results for NOAA-20 VIIRS and in 2019 are emphasized. NOAA-20 VIIRS geolocation errors are within ±100 m, comparable to S-NPP; RSBs have been stable after achieved validated maturity status, except that small upward trends were observed; TEBs agree with co-located Cross-track Infrared Sounder (CrIS) observations within ~0.1 K at nadir, more stable (relative to CrIS) compared to S-NPP TEBs in 2019. NOAA-20 RSBs continue bias ~2-4.5% lower than S-NPP. Three major improvements to VIIRS SDRs, including the new operational M6 saturation rollover flagging method, the operational TEB warm-up/cool-down bias correction, and the latest results for correcting NOAA-20 TEB scan angle/scene temperature dependent biases, are also presented to address users' concerns.

The NOAA-20 (N-20) VIIRS has successfully operated for more than two years since its launch in November 2017. Shortly after completing its initial instrument check-outs and post-launch testing (PLT) activities, the N-20 VIIRS sensor data records (SDR) achieved the beta, provisional, and validated maturity status in January, February, and April 2018, respectively. In this paper, we briefly describe the instrument on-orbit operation and calibration activities, provide an overall assessment of its on-orbit performance, and discuss the methodologies developed to maintain and improve sensor calibration and data quality. As illustrated in this paper, the N-20 VIIRS continues to perform with excellent stability, allowing high-quality environmental data records (EDR) to be generated from its well-calibrated SDR.

The National Oceanic and Atmospheric Administration (NOAA)-20 Visible Infrared Imaging Radiometer Suite (VIIRS) was launched on November 18, 2017, and it has been operational for more than two years and follows the first Joint Polar Satellite System (JPSS) series of the Suomi National Polar-orbiting Partnership (S-NPP) mission. VIIRS has 14 reflective solar bands (RSBs) covering a spectral range of 0.41-2.3 μm. The primary source of RSB calibration is the solar diffuser (SD), and the time-dependent SD degradation is monitored by the SD stability monitor (SDSM). The initial instability of the SD degradation (H-factor) was resolved by updating SDSM sun screen transmittance function combining yaw maneuver data and on-orbit SDSM data sets. After the H-factor improvements, the VIIRS RSB calibration coefficients (F-factors) are updated and applied to the operational Sensor Data Record (SDR) product generation. To validate the SD F-factors, the lunar F-factors are calculated by using a lunar irradiance model and comparing the trend differences between them. Over the two years of operation, decreasing trends have been calculated with the SD F-factors, whereas constant lunar F-factors were observed in bands M1-M4. With these discrepancies, the operational F-factors remained unchanged since April 2018 because the deep convective cloud (DCC) and cross-calibration comparison results did not show any further degradations in these bands. All the possible radiometric calibration sources, such as SD and lunar F-factors, DCC trends, and cross-calibration results, are monitored, compared, and applied by the NOAA VIIRS SDR science team for the best quality of the VIIRS SDR product.

The National Oceanic and Atmospheric Administration (NOAA)-20 Visible Infrared Imaging Radiometer Suite (VIIRS) was launched on November 18, 2017 and has been in operation for about one year. It has 14 Reflective Solar Bands (RSBs) covering a spectral range of 412nm to 2.25 um at two spatial resolutions at nadir 750m for Moderate resolution (M) bands and 350m for Imaging (I) bands. The spatial characterization of the VIIRS instrument was performed during the prelaunch calibration with parameters such as Band-to-Band Registration (BBR). Accurate estimation of BBR is a key spatial parameter when the inter-band calculation is required. Because there is no on-board spatial calibration source like Spectro Radiometric Calibration Assembly (SRCA) with Terra and Aqua Moderate Resolution Imaging Spectroradiometer (MODIS), VIIRS instrument does not have on-board capability of measuring on-orbit BBR. As an alternative method, scheduled moon collections can be used to estimate on-orbit BBR characterization within the RSBs. In this paper, NOAA-20 VIIRS on-orbit BBR estimation methodology and initial results are presented.

Kernel canonical correlation analysis (KCCA) is an efficient dimensionality reduction tool in the application of remote sensing image classification. However, it suffers from the problem of parametric sensitivity since a single kernel is used. In this letter, a KCCA ensemble framework is put forward to improve the robustness of KCCA. Following the philosophy that two heads are better than one, multiple KCCA models are incorporated into the framework. And more importantly, their terms are weighted to adjust their contribution to the result according to their performance. In addition, over-fitting is overcome by introducing a Laplacian regularization term in our framework, hence, the name Laplacian regularized kernel canonical correlation ensemble. Experimental results on NWPU-RESISC45 data set show that our proposed method achieves better classification performances as compared to state-of-the-art methods in both shallow and deep features.

Solar diffusers (SDs) have often been used as the onboard calibrators for the radiometric calibration of reflective solar band imaging sensors. After being spaceborne, the reflectance of SDs is observed to degrade with spectral dependence due to exposure to solar UV and energetic particle radiation. Long-term spectral reflectance data of SDs onboard multiple LEO imaging sensors, such as the Moderate Resolution Imaging Spectroradiometer (MODIS) on Terra and Aqua and the Visible Infrared Imaging Radiometer Suite (VIIRS) on SNPP, are analyzed. The reflectance of SDs on these three instruments degrades faster for the shorter wavelength (0.4-0.6 μm) bands than the longer wavelength bands. The Surface Roughness-induced Rayleigh Scattering (SRRS) model is applied to simulate the SD degradation on these instruments, and the growth of the surface roughness parameter of the SDs is derived. It is determined that the change of surface roughness scale length is ~tens of nanometers. To show the consistency of roughness growth rates among the SDs on Terra/Aqua MODIS and SNPP VIIRS instruments, the functional dependences of the growth rates are characterized according to the SD exposure time and the stage of surface roughness. It is also found that the flattening or reverse in the growth trend of the surface roughness for these three SDs occurred around the same interval between October 2013 and October 2015. The confirmation of the applicability of SRRS model with the long-term spectral reflectance data from three independent spaceborne SDs facilitates a better understanding of the origin and physical processes of the SD degradation.

The Visible Infrared Imaging Radiometer Suite (VIIRS) onboard Suomi-NPP is equipped with a Day/Night Band (DNB), a major advancement in nighttime imaging capability. DNB data quality has been largely improved through several calibration updates since early launch. The reprocessed VIIRS DNB sensor data record (SDR) data at the NOAA Center for Satellite Applications and Research (STAR) accommodate all updates, such as DNB modulated relative spectral responses (RSRs), monthly DNB offset and gain ratio, straylight correction, and geolocation terrain correction since launch. In addition, one of the major improvements is in the DNB offset computing using the pitch maneuver-based deep space view that is free of airglow. The reprocessed DNB data results in radiometrically consistent calibrated SDR that allows the user community to use high-quality DNB data in long-term applications. Until the end of 2016, DNB high gain stage (HGS) operational calibration involved estimation of dark offsets using the new moon-based nighttime measurements over the Pacific Ocean. However, the faint emission from airglow during the new moon nights led to an overestimation of the dark offset by an amount equivalent to that of the airglow. This directly impacts the calibrated data by underestimating the calibrated radiance values. The impact is prominent at low light radiance, which can also lead to negative radiances. This paper quantifies the reprocessing led improvements in the calibrated VIIRS DNB data, analyzes the impact of airglow on operational calibrated radiance product, and explains how the absolute accuracy for low light radiance has been largely improved for the entire reprocessed DNB archive.

Emerging technologies such as IoT(Internet of Things) and UAS(Unmanned Aircraft Systems) enable new observing capabilities. In particular, infrared sensing of CO2and surface temperature with satellite infrared radiometers have been operationally used since the 1970s for atmospheric soundings, and sea/land surface temperatures, such as with NOAA's HIRS/AVHRR radiometers. Recent advances also allow the unprecedented CO2retrieval accuracy with the launch of OCO-2, GOSAT, IASI, CrIS, as well as ground measurements by the TCCON network. However, the retrievals from satellites have very coarse spatial resolution (a few kilometers) which limits their usefulness especially for urban areas, in addition to latency issues. This study explores the use of portable infrared sensors inter-connected through IOT as payloads on mobile platforms such as UAS, automobile, and on distributed networks to demonstrate in-situ dynamic measurements of CO2, as well as surface temperature over water and land. While the IoT enables realtime monitoring with high temporal resolution, the UAS allows high spatial resolution measurements in areas with greater CO2and surface temperature variability. The low system cost allows its potential proliferation towards a large distributed network with IOT. Such a system could fill in a gap in existing observation systems, and complement the ground based network for the monitoring and satellite validation of a large number of geophysical variables for environmental intelligence, which is a desired observation capability but currently limited to a number of sparsely distributed stations.

Both the Advanced Himawari Imager (AHI) aboard Japanese geostationary weather satellite Himawari-8 and Advanced Baseline Imager (ABI) aboard the American GOES-16 satellite are the primary imaging instruments with multispectral channels and have very similar optical design. The Himawari-8 was launched ~two years earlier than the GOES-16 ABI. Since the AHI imagery data become available, characterization, monitoring and trending the stray light performance in AHI imageries have been routinely performed. Difference data between consecutive AHI full-disk images were processed and the stray light performance was quantitatively assessed for the regions inside and outside the restricted zone around the sun that intercept with the AHI full-disk images. It has been found that the major stray light contamination occurred in the full disk imagery of the AHI VNIR and IR channels over a few weeks around February and October-November. Because of the observed AHI stray light in 3.9 um channel, the ABI design has been modified. Analysis of ABI imagery data confirms that the stray light performance of the ABI 3.9 um channel is better than AHI.

The Visible Infrared Imaging Radiometer Suite (VIIRS) onboard Suomi-NPP is equipped with a Day/Night Band (DNB), a major advancement in nighttime imaging capability as compared to the heritage Operational Linescan System (OLS) onboard Defense Meteorological Satellite Program (DMSP). DNB High Gain Stage (HGS) operational radiometric calibration involves estimation of dark offsets using the nighttime measurements over Pacific Ocean during new moon. However, the DNB detectors were found to be highly sensitive, enough to detect faint emission from airglow even during the moonless nights. Thus, the on orbit calculated dark offset is overestimated by an amount equivalent to that of airglow. This causes direct impact in calibrated data such that the radiance values are underestimated in the absolute scale. The impact is prominent at low light radiance. This study focuses on analyzing the airglow using DNB measurements. In addition, the impact on operational calibrated radiance product and how the absolute accuracy can be improved especially at low light radiance are also explored.

The Advanced Baseline Imager (ABI) is the primary payload for NOAA's Geostationary Operational Environmental Satellite R-series (GOES-R). The first of this new generation GOES was launched on November 19, 2016, and named GOES-16 when it reached the orbit on November 29, 2016. ABI incorporated numerous advanced technologies to meet the ever-demanding users' requirements. There is great interest in the GOES-R Program, user community, and international partners whether ABI lives up to its expectations and how does it compare to Imager, the existing GOES imaging system. This paper will offer a preliminary glimpse of answers addressing those questions.

The Day/Night Band (DNB) of the Visible Infrared Imaging Radiometer Suite (VIIRS) onboard Suomi-NPP represents a major advance in night time imaging capabilities. During geomagnetic storms, auroras can be observed by the DNB on the night side over both hemispheres. The radiometrically calibrated DNB observations can enable quantitative analysis of the spatial distribution and temporal evolution of aurora during geomagnetic storms. Two coronal mass ejections (CME) occurred on June 19 and 21, 2015 and had made their way to Earth to cause a G4 (severe) geomagnetic storm on June 22 afternoon. This paper presents an analysis of the radiance data from DNB observations of the aurora during the geomagnetic storm on June 22, 2015. Regions of aurora during each orbital pass are identified and the evolution of aurora is characterized with time series of the auroral boundary, area and total light emission of the aurora region in the DNB observation. The good correlation of aurora activities with ground geomagnetic index during the geomagnetic storm suggests that DNB observations of aurora can provide new details of the magnetospheric and ionospheric responses during severe geomagnetic storms.

The mid-wavelength infrared (MWIR) imageries of Suomi-NPP Visible Infrared Imaging Radiometer Suite (VIIRS) and HIMAWARI-8 Advanced Himawari Imager (AHI) enable monitoring temperature variation of heat sources such as hot spots and urban heat islands. The 3.75 um band of VIIRS provides imagery in high spatial resolution (~375 m) twice a day and can be used to monitor temperature variation of hot spots due to industrial activities. The 3.9 um channel of AHI imagery in 10-minute time resolution enables continuous monitoring of temporal variation of urban heat island temperature. This paper compared using imageries of VIIRS and AHI MWIR channel to monitor both spatial distribution and temporal (seasonal and diurnal) variation of hot spots and urban heat islands. It was shown that the MWIR imageries of VIIRS and AHI complement each other in understanding the spatial structure and temporal variation of social economic activities-related heat sources.

Japan Meteorological Agency (JMA) Himawari-8 Advanced Himawari Imager (AHI) is the first in the series of next-generation geostationary (GEO) weather instruments. It has 16 spectral solar reflective and emissive bands located in three focal plane modules (FPM): one visible and near-infrared (VNIR) FPM, one midwave infrared (MWIR) FPM, and one longwave infrared (LWIR) FPM. All the AHI bands are geo-spatially calibrated to provide continuous environmental measurements from the Asian-Pacific area for the weather forecasting, disaster monitoring, and longterm climatic change studies. This study is to evaluate the navigation and co-registration accuracies of three AHI bands at the three FPMs using the simultaneous nadir observations (SNO) of Suomi National Polar-orbiting Partnership (SNPP) Visible Infrared Imaging Radiometer Suite (VIIRS) I-band images. The preliminary results showed that the mean navigation difference between the two instruments is within 0.5 AHI pixels at 2km spatial resolution. The AHI band-to-band co-registration (BBR) difference between the two thermal bands is generally less than 0.2 pixels, better than the BBR between the VNIR and IR bands which ranges between 0.2 to 0.7 pixels during the study period.

Power outages after a major storm affect the lives of millions of people and cause massive light outages. The launch of the Suomi National Polar-orbiting Partnership satellite with the Visible Infrared Imaging Radiometer Suite (VIIRS) significantly enhances our capability to monitor and detect light outages with the well-calibrated day/night band (DNB) and to use light loss signatures as indication of regional power outages. This study explores the use of the DNB in quantifying light outages due to the derecho storm in the Washington DC metropolitan area in June 2012 and Hurricane Sandy at the end of October 2012 on the East Coast of U.S. The results show that the DNB data are very useful in detecting power outages by quantifying light loss, but it also has some challenges due to clouds, lunar illumination, and straylight effect. Comparison of light outage and recovery trend determined from DNB data with power company survey shows reasonable agreement, demonstrating the usefulness of DNB in independently verifying and complementing the statistics from power companies.

In this study, two methods were used to evaluate radiometric calibration of the VIIRS sensor data records: (1) imaging of radiometrically stable Earth's surfaces and (2) SNO (Simultaneous Nadir Overpass) observations by VIIRS and other satellite instruments. Measurements acquired by VIIRS at the stable Dome C calibration site in Antarctica confirm that a faster-than-expected degradation of the radiometric response occurs for selected spectral bands in the visible and near-infrared region, with other bands remaining stable. SNO comparisons with MODIS show that the implemented regular updates of the radiometric calibration coefficients have stabilized the calibration and that the VIIRS and MODIS measurements are in agreement.

Many application-specific wireless sensor network (WSN) systems require small size and low power features due to their limited resources, and their use in distributed, wireless environments. In this paper, we present a light-weight distributed algorithm for line-crossing recognition, together with its analysis, implementation, and experimental evaluation within a prototype wireless sensor network platform. The algorithm is developed in conjunction with a TDMA-based communication protocol such that the proposed system provides for low duty cycle and energy efficient operation. An accurate lifetime model is proposed with consideration of detailed energy usage to analyze and estimate the system lifetime. Our experimental results demonstrate the accuracy of this lifetime model, and its utility in optimizing network implementation. The design and experimental evaluation of our prototype network demonstrates the compactness and functionality of the proposed distributed WSN system for line-crossing recognition.

A novel content-adaptive music watermarking technique is proposed in this article. To optimally balance inaudibility and robustness when embedding and extracting watermarks, the embedding scheme is highly related to the music structure and human auditory system (HAS). A note-based segmentation method is proposed and used for music vocal/instrumental boundary detection. A multiple bit hopping and hiding scheme with different embedding parameters is applied to vocal and instrumental frames of the music. The experimental results in inaudibility and robustness are provided to support all novel features in the proposed watermarking scheme.

We present a novel method for predominant vocal pitch detection in two-channel polyphonic music. The proposed method contains two stages. In the first stage, we apply the frequency domain independent component Analysis (FD-ICA) for the two-channel polyphonic music to separate the vocal content from the background music. Considering the vocal singing voice and background music are two heterogeneous signals, we employ a statistical learning based method to solve the permutation inconsistency problem in FD-ICA. In the second stage, a noise insensitive vocal pitch detection method is proposed, which is robust to noise and errors introduced by the separation process in the first stage. The proposed method has been tested on the two-channel polyphonic music signals, and experimental results show promising performance

In this article, we propose a novel approach for automatic music video summarization. The proposed summarization scheme is different from the current methods used for video summarization. The music video is separated into the music track and video track. For the music track, a music summary is created by analyzing the music content using music features, an adaptive clustering algorithm, and music domain knowledge. Then, shots in the video track are detected and clustered. Finally, the music video summary is created by aligning the music summary and clustered video shots. Subjective studies by experienced users have been conducted to evaluate the quality of music summaries and effectiveness of the proposed summarization approach. Experiments are performed on different genres of music videos and comparisons are made with the summaries generated based on music track, video track, and manually. The evaluation results indicate that summaries generated using the proposed method are effective in helping realize users' expectations.

In this paper, we present a novel approach for music summarization based on music structure analysis. From the audio signal, we first extract the note onset representing the time tempo of the song and the music structure analysis can be performed based on this tempo information. After music content has been structured into different semantic regions such as introduction (intro), verse, chorus, ending (outro), etc., the final music summary can be created with chorus and music phrases which are included anterior or posterior to the selected chorus to get the desired length of the final summary. In this way, we can guarantee that the summaries begin and end at meaningful music phrase boundaries, which is a difficult problem for existing music summarization methods. Experiments show our proposed method can capture the main theme of the music compared to the ideal summaries selected by music experts and user subjective evaluation indicates our proposed method has a good performance.

In this paper, we propose a novel approach for automatic music video summarization based on audio-visual-text analysis and alignment. The music video is separated into the music and video tracks. For the music track, the chorus is detected based on music structure analysis. For the video track, we first segment the shots and classify the shots into close-up face shots and non-face shots, then we extract the lyrics and detect the most repeated lyrics from the shots. The music video summary is generated based on the alignment of boundaries of the detected chorus, shot class and the most repeated lyrics from the music video. The experiments on chorus detection, shot classification, and lyrics detection using 20 English music videos are described. Subjective user studies have been conducted to evaluate the quality and effectiveness of summary. The comparisons with the summaries based on our previous method and the manual method indicate that the results of summarization using the proposed method are better at meeting users' expectations.

Automatic music classification and summarization are very useful to music indexing, content-based music retrieval and on-line music distribution, but it is a challenge to extract the most common and salient themes from unstructured raw music data. In this paper, we propose effective algorithms to automatically classify and summarize music content. Support vector machines are applied to classify music into pure music and vocal music by learning from training data. For pure music and vocal music, a number of features are extracted to characterize the music content, respectively. Based on calculated features, a clustering algorithm is applied to structure the music content. Finally, a music summary is created based on the clustering results and domain knowledge related to pure and vocal music. Support vector machine learning shows a better performance in music classification than traditional Euclidean distance methods and hidden Markov model methods. Listening tests are conducted to evaluate the quality of summarization. The experiments on different genres of pure and vocal music illustrate the results of summarization are significant and effective.

A novel compressed domain automatic music summarization approach is presented in this paper. The proposed method works directly in the compressed domain. Only the encoded subband samples are extracted and processed for characterizing music content and discovering the music structure. The experimental results and the evaluation by a subjective study have shown that the summarization based on MPEG-1 Layer 3 (MP3) music is comparable to the summarization based on uncompressed PCM music samples.

A new automatic summarization approach for music videos is presented. The proposed method detects and recognizes lyric captions appearing commonly in karaoke music videos and uses the captions to analyze music video structure and identify the most salient music part. The music video summary is created based on the salient part. Experimental results show our proposed method is promising.

Music genre classification can be of great utility to musical database management. Most current classification methods are supervised and tend to be based on contrived taxonomies. However, due to the ambiguities and inconsistencies in the chosen taxonomies, these methods are not applicable for a much larger database. We proposed an unsupervised clustering method, based on a given measure of similarity which can be provided by hidden Markov models. In addition, in order to better characterize music content, a novel segmentation scheme is proposed, based on music intrinsic rhythmic structure analysis and features are extracted based on these segments. The performance of this feature segmentation scheme performs better than the traditional fixed-length method, according to experimental results. Our preliminary results also suggest that the proposed method is comparable to the supervised classification method.

Automatic music summarization is very useful for music indexing, content-based music retrieval and on-line music distribution, but it is a challenge to extract automatically the most common and salient themes from unstructured raw music data. We propose an effective approach to summarize music content automatically. First, a number of features are extracted to characterize the music content. Based on the extracted features, an adaptive clustering algorithm is then applied to structure the music content. Finally, the music summary is created in terms of the clustering results and domain-related music knowledge. A user study is conducted to evaluate the quality of summarization. The experiments on different genres of music illustrate the results of summarization are significant and effective to actual expectation.

In this paper, we present a novel approach for music structure analysis. A new segmentation method, beat space segmentation, is proposed and used for music chord detection and vocal/instrumental boundary detection. The wrongly detected chords in the chord pattern sequence and the misclassified vocal/instrumental frames are corrected using heuristics derived from the domain knowledge of music composition. Melody-based similarity regions are detected by matching sub-chord patterns using dynamic programming. The vocal content of the melody-based similarity regions is further analyzed to detect the content-based similarity regions. Based on melody-based and content-based similarity regions, the music structure is identified. Experimental results are encouraging and indicate that the performance of the proposed approach is superior to that of the existing methods. We believe that music structure analysis can greatly help music semantics understanding which can aid music transcription, summarization, retrieval and streaming.

Automatic musical genre classification is very useful for music indexing and retrieval. In this paper, an efficient and effective automatic musical genre classification approach is presented. A set of features is extracted and used to characterize music content. A multi-layer classifier based on support vector machines is applied to musical genre classification. Support vector machines are used to obtain the optimal class boundaries between different genres of music by learning from training data. Experimental results of multi-layer support vector machines illustrate good performance in musical genre classification and are more advantageous than traditional Euclidean distance based method and other statistic learning methods.

In this paper, we propose a novel approach to automatically summarize musical videos. The proposed summarization scheme is different from the current methods used for video summarization. The musical video is separated into the musical and visual tracks. A music summary is created by analyzing the music content based on music features, adaptive clustering algorithm and musical domain knowledge. Then, shots are detected and clustered in the visual track. Finally, the music video summary is created by aligning the music summary and clustered video shots. Subjective studies by experienced users have been conducted to evaluate the quality of summarization. The experiments on different genres of musical video and comparisons with the summaries only based on music track and video track indicate that the results of summarization using proposed method are significant and effective to help realize user's expectation.