Nowadays, with the advancement of technology, many retail companies require in-house data scientist teams to build machine learning tasks, such as user segmentation and item price prediction. These teams typically use a trial-and-error process to obtain a good model for a given dataset and machine learning task, which is time-consuming and requires expertise. However, the team may have built models for other tasks on different datasets. This article proposes a framework to obtain a model architecture using the previous solved machine learning tasks and datasets. By analyzing real datasets with over 70,000 images from 11 online retail e-commerce websites, it is demonstrated that the performance of a model is related to the similarity among datasets, models, and machine learning tasks. A framework is hence proposed to obtain the model using the similarities among them. It was proven that the model was 26.6% better in accuracy, and using only 20% of the runtime while comparing to an auto network architecture search library, Auto-Keras, in predicting the attributes of fashion images. To the best of our knowledge, this is the first article to obtain the best model based on the similarity among machine learning tasks, models, and datasets.

Selling counterfeit online has become a serious problem, especially with the advancement of social media and mobile technology. Instead of investigating the products directly, one can only check the images, tags annotated by the sellers on the images, or the price to decide if a seller sells counterfeits. One of the ways to detect counterfeit sellers is to investigate their social graphs, in which counterfeit sellers show different behaviour in network measurements, such as those in centrality and EgoNet. However, social graphs are not easily accessible. They may be kept private by the operators, or there are no connections at all. This article proposes a framework to detect counterfeit sellers using their connection graphs discovered from their shared images. Based on 153 K shared images from Taobao, it is proven that counterfeit sellers have different network behaviours. It is observed that the network measurements follow Beta function well. Those distributions are formulated to detect counterfeit sellers by the proposed framework, which is 60% better than approaches using classification.

Social events are very common activities, where people can interact with each other. During an event, the organizer often hires photographers to take images, which provide rich information about the participants’ behaviour. In this work, we propose a method to discover the social graphs among event participants from the event images for social network analytics. By studying over 94 events with 32,330 event images, it is proven that the social graphs can be effectively extracted solely from event images. It is found that the discovered social graphs follow similar properties of online social graphs; for instance, the degree distribution obeys power law distribution. The usefulness of the proposed method for social graph discovery from event images is demonstrated through two applications: important participants detection and community detection. To the best of our knowledge, it is the first work to show the feasibility of discovering social graphs by utilizing event images only. As a result, social network analytics such as recommendations become possible, even without access to the online social graph.

In recent years, online social networks (OSNs) have become extremely popular and been one of the most common ways for storing and distributing images. Naturally, such widespread availability of OSN makes it a viable channel for transmitting additional data along with the image sharing. However, various lossy operations, e.g., resizing and compression, conducted by OSN platforms impose great challenges for designing a robust watermarking scheme over OSN shared images. In this paper, we tackle this challenge and propose a robust high-capacity watermarking technique, by using Facebook as a representative OSN. To achieve the satisfactory robustness, we first probe into Facebook and recover the image manipulation mechanism via a deep convolutional neural network (DCNN) approach. Assisted with the precise knowledge on the lossy channel offered by Facebook, we then suggest a DCT-domain image watermarking method that is highly robust against the lossy operations on Facebook, even without any error correcting codes (ECC). The proposed technique is also extended to other popular OSNs, e.g., Wechat and Twitter. Extensive experimental results are provided to show the superior performance of our method in terms of the embedding capacity, data extraction accuracy, and quality of the reconstructed images.

With the advance of mobile technology and social media, image sharing has become part of our daily lives. For many applications, such as follower/followee recommendation, shared images are an excellent source to discover connections among users who shared them. Shared images on social media are like invitations for user interactions, such as comment, like, and more. Social signals are in those images, and those signals can be objects that interest related users such that they will start to interact. Conventionally, connections among users are discovered through recognizing objects among those shared images, such as a using convolutional neural network (CNN) to extract features that are sensitive to object recognition. However, social signals are not limited to object, they can be colour, textual, or even a concept that may not be captured effectively by conventional CNN. This paper proposes a CNN-based analytic framework to detect social signals among users. The CNN is optimized using a triplet network with user-shared images, and the relationships among users who upload them. It is observed that images from 2 users with a connection have a shorter distance after encoding, than 2 users without a connection. A framework is implemented, which is verified with over 1.7 million images by over 2000 users from two image-oriented social networks, Skyrock and Flickr. It is proven that the proposed analytic framework shows an up to 89% improvement on approaches using object recognition for follower/followee recommendation. To the best of our knowledge, this paper is the first to propose an analytic framework to detect social signals from visual features for connection discovery.

With the advancement of social media and mobile technology, any smartphone user can easily become a seller on social media and e-commerce platforms, such as Instagram and Carousell in Hong Kong or Taobao in China. A seller shows images of their products and annotates their images with suitable tags that can be searched easily by others. Those images could be taken by the seller, or the seller could use images shared by other sellers. Among sellers, some sell counterfeit goods, and these sellers may use disguising tags and language, which make detecting them a difficult task. This article proposes a framework to detect counterfeit sellers by using deep learning to discover connections among sellers from their shared images. Based on 473K shared images from Taobao, Instagram, and Carousell, it is proven that the proposed framework can detect counterfeit sellers. The framework is 30% better than approaches using object recognition in detecting counterfeit sellers. To the best of our knowledge, this is the first work to detect online counterfeit sellers from their shared images.

Visual arts, especially paintings, appear everywhere in our daily lives. They are not only liked by art lovers but also by ordinary people, both of whom are curious about the stories behind these artworks and also interested in exploring related artworks. Among various methods, the mobile visual search has its merits in providing an alternative solution to text and voice searches, which are not always applicable. Mobile visual search for visual arts is far more challenging than the general image visual search. Conventionally, visual search, such as searching products and plant, focuses on locating images containing similar objects. Hence, approaches are designed to locate objects and extract scale-invariant features from distorted photos that are captured by the mobile camera. However, the objects are only part of the visual art piece; the background and the painting style are both important factors that are not considered in the conventional approaches. In this article, an empirical investigation is conducted to study issues in photos taken by mobile cameras, such as orientation variance and motion blur, and how they influence the results of the mobile visual arts search. Based on the empirical investigation results, a photo-rectification pipeline is designed to rectify the photos into perfect images for feature extraction. A new method is proposed to learn high discriminative features for visual arts, which considers both the content information and style information in visual arts. Apart from conducting solid experiments, a real-world system is built to prove the effectiveness of the proposed methods. To the best of our knowledge, this is the first article to solve problems for visual arts search on mobile devices.

Billions of user images, which are shared on social media, can be widely accessible by others due to their sharing nature. Using machine-generated labels to annotate those images is a reliable for user connections discovery on social networks. The machine-generated labels are obtained from encoded vectors using up-to-date image processing and computer vision techniques, such as convolution neural network. By analyzing 2 million user-shared images from 8 online social networks, a phenomenon is observed that the distribution of user similarity based on their shared images follows exponential functions. Users who share visually similar images are likely having follower/followee relationships, regardless of the origins and the content sharing mechanisms of a social network. This phenomenon is nicely formulated for a multimedia big data recommendation engine as an alternative to social graphs for recommendation. By utilizing the formulation of the distribution, it is proven the proposed engine can be 46 percent better than previous approaches in Fl score and achieves a comparable performance of friends-of-friends approach. To the best of our knowledge, this is the first attempt in related fields to characterize such phenomenon by massive user-shared images collected from real-world SNs, and then formulate into practical analytics engine for connection discovery.

In this paper, user log data of mobile notificationsare collected from a real mobile news application, and notificationopening rate and reaction time are identified as key parametersthat characterize user behavior. The results indicate that notificationopening patterns depend on the content of notifications, the time at which they are sent, the activeness of users, andthe reaction time. This paper also demonstrates that there existsan inverse relationship between the notification opening rate ofusers and their notification reaction time, and between theirnotification opening rate and notification opening volume. Then, a novel method of quantifying notification usefulness for users isdeveloped, where 3 models are presented to describe usefulnessof notifications against the volume of notifications received, eachrelevant in different contexts. Utilizing this usefulness model, a notification optimization framework is developed for sendingnotifications from different news categories. The optimized notificationallocation improved the average notification opening rateby by 65.24% from 4.92% to 8.13%, and the average reactiontime improved by 13.13% from 20.4 hours to 17.68 hours.

This paper aims to generate a better representation of visual arts, which plays a key role in visual arts analysis works. Museums and galleries have a large number of artworks in the database, hiring art experts to do analysis works (e.g., classification, annotation) is time consuming and expensive and the analytic results are not stable because the results highly depend on the experiences of art experts. The problem of generating better representation of visual arts is of great interests to us because of its application potentials and interesting research challenges---both content information and each unique style information within one artwork should be summarized and learned when generating the representation. For example, by studying a vast number of artworks, art experts summary and enhance the knowledge of unique characteristics of each visual arts to do visual arts analytic works, it is non-trivial for computer. In this paper, we present a unified framework, called DeepArt, to learn joint representations that can simultaneously capture contents and style of visual arts. This framework learns unique characteristics of visual arts directly from a large-scale visual arts dataset, it is more flexible and accurate than traditional handcraft approaches. We also introduce Art500k, a large-scale visual arts dataset containing over 500,000 artworks, which are annotated with detailed labels of artist, art movement, genre, etc. Extensive empirical studies and evaluations are reported based on our framework and Art500k and all those reports demonstrate the superiority of our framework and usefulness of Art500k. A practical system for visual arts retrieval and annotation is implemented based on our framework and dataset. Code, data and system are publicly available at http://deepart.ece.ust.hk.

With the advent and popularity of the social network, social graphs become essential to improve services and information relevance to users for many social media applications to predict follower/followee relationship, community membership, and so on. However, the social graphs could be hidden by users due to privacy concerns or kept by social media. Recently, connections discovered from user-shared images using machine-generated labels are proved to be more accessible alternatives to social graphs. But real-time discovery is difficult due to high complexity, and many applications are not possible. This article proposes an efficient computation framework for connection discovery using user-shared images, which is suitable for any image processing and computer vision techniques for connection discovery on the fly. The framework includes the architecture of online computation to facilitate real-time processing, offline computation for a complete processing, and online/offline communication. The proposed framework is implemented to demonstrate its effectiveness by speeding up connection discovery through user-shared images. By studying 300K+ user-shared images from two popular social networks, it is proven that the proposed computation framework reduces 90% of runtime with a comparable accurate with existing frameworks.

Many social media applications, such as recommendation, virality prediction, and marketing, make use of user gender, which may not be explicitly specified or kept privately. Meanwhile, advanced mobile devices have become part of our lives and a huge amount of content is being generated by users every day, especially user shared images shared by individuals in social networks. This particular form of user generated content is widely accessible to others due to the sharing nature. When user gender is only accessible to exclusive parties, these user shared images are proved to be an easier way to identify user gender. This work investigated 3,152,344 images by 7,450 users from Fotolog and Flickr, two image-oriented social networks. It is observed that users who share visually similar images are more likely to have the same gender. A multimedia big data system that utilizes this phenomenon is proposed for user gender identification with 79% accuracy. These findings are useful for information or services in any social network with intensive image sharing.

With the advances in mobile devices and the popularity of social networks, users can share multimedia content anytime, anywhere. One of the most important types of emerging content is video, which is commonly shared on platforms such as Instagram and Facebook. User connections, which indicate whether two users are follower/followee or have the same interests, are essential to improve services and information relevant to users for many social media applications. But they are normally hidden due to users’ privacy concerns or are kept confidential by social media sites. Using user-shared content is an alternative way to discover user connections. This article proposes to use user-shared videos for connection discovery with the Bag of Feature Tagging method and proposes a distributed streaming computation framework to facilitate the analytics. Exploiting the uniqueness of shared videos, the proposed framework is divided into Streaming processing and Online and Offline Computation. With experiments using a dataset from Twitter, it has been proved that the proposed method using user-shared videos for connection discovery is feasible. And the proposed computation framework significantly accelerates the analytics, reducing the processing time to only 32% for follower/followee recommendation. It has also been proved that comparable performance can be achieved with only partial data for each video and leads to more efficient computation.

Social graphs, representing online friendships among users, are one of the fundamental types of data for many applications, such as recommendation, virality prediction and marketing in social media. However, this data may be unavailable due to the privacy concerns of users, or kept private by social network operators, which makes such applications difficult. Inferring users' interests and discovering users' connections through their shared multimedia content has attracted more and more attention in recent years. This paper proposes a Gaussian relational topic model for connection discovery using user shared images in social media. The proposed model not only models users' interests as latent variables through their shared images, but also considers the connections between users as a result of their shared images. It explicitly relates user shared images to user connections in a hierarchical, systematic and supervisory way and provides an end-to-end solution for the problem. This paper also derives efficient variational inference and learning algorithms for the posterior of the latent variables and model parameters. It is demonstrated through experiments with over 200k images from Flickr that the proposed method significantly outperforms the methods in previous works.

Informative directories have always responded to a fundamental need of humanity: providing available information around people. However, the escalating amount of content to be visualized on directories makes relevant information search extremely time-consuming. Meanwhile, digital displays based on screen-smart device interaction become an emerging interface of smart services to deal with daily-life challenges like information seeking. Also, multimedia content, such as movies, can be understood by multimedia analytics for recommendation, but there is no effective way to visualize the content of a directory. This paper proposes a novel directory visualization framework, called analytics-driven dynamic visualization on digital directory (AVDD): understanding user preferences via smartphone-based interaction and optimizing visualization by visual analytics in terms of high content relevancy and screen utilization for advanced directories. With experiments in laboratory and real-world settings, AVDD is proven to be effective for visualizing directory with screen utilization over 98% and the score for Likert-scale surveys achieving 73% on average in a movie directory.

User-shared images are shared on social media about a user’s life and interests that are widely accessible to others due to their sharing nature. Unlike for online profiles and social graphs, most users are unaware of the privacy risks relating to shared images, as they do not directly disclose characteristics such as gender and origin. Recently, however, user-shared images have been proven to be an accessible alternative to social graphs for online friendship recommendation and gender identification. This article evaluates 1.6M user-shared images from an image-oriented social network, Fotolog, and concludes how they can create privacy risks by proposing a system for de-anonymization, as well as inferring information on online profiles with the user-shared images. It is concluded that given user-shared images, using social graphs is 2 and 2.5 times more effective in de-anonymization than using origins or genders. With two showcases, it is also proven that using user-shared images is effective in online friendship recommendation, gender identification, and origin inference. To the best of our knowledge, this is the first article to evaluate the privacy issue qualitatively with big multimedia data from a real social network.

Predicting content going viral in social networks is attractive for viral marketing, advertisement, entertainment, and other applications, but it remains a challenge in the big data era today. Previous works mainly focus on predicting the possible popularity of content rather than the timing of reaching such popularity. This work proposes a novel yet practical iterative algorithm to predict virality timing, in which the correlation between the timing and growth of content popularity is captured by using its own big data naturally generated from users’ sharing. Such data is not only able to correlate the dynamics and associated timings in social cascades of viral content but also can be useful to self-correct the predicted timing against the actual timing of the virality in each iterative prediction. The proposed prediction algorithm is verified by datasets from two popular social networks—Twitter and Digg—as well as two synthesized datasets with extreme network densities and infection rates. With about 50% of the required content virality data available (i.e., halfway before reaching its actual virality timing), the error of the predicted timing is proven to be bounded within a 40% deviation from the actual timing. To the best of our knowledge, this is the first work that predicts content virality timing iteratively by capturing social cascades dynamics.

Billions of user-shared images are generated by individuals in many social networks today, and this particular form of user data is widely accessible to others due to the nature of online social sharing. When user social graphs are only accessible to exclusive parties, these user-shared images are proved to be an easier and effective alternative to discover user connections. This work investigated over 360 000 user shared images from two social networks, Skyrock and 163 Weibo, in which 3 million follower/ followee relationships are involved. It is observed that the shared images from users with a follower / followee relationship show relatively higher similarities . A multimedia big data system that utilizes this observed phenomenon is proposed as an alternative to user- generated tags and social graphs for follower/followee recommendation and gender identification. To the best of our knowledge, this is the first attempt in this field to prove and formulate such a phenomenon for mass user-shared images along with more practical prediction methods. These findings are useful for information or services recommendations in any social network with intensive image sharing, as well as for other interesting personalization applications, particularly when there is no access to those exclusive user social graphs.

This paper conducts a summarized reality check of a P2P-based IPTV system from the operator's perspective (NextTV) through the analysis of large-scale operational data. NextTV is an IPTV system by a leading Asia media company based in Taiwan — NextMedia, in which P2P delivery mechanism is adopted to deal with system and network scalability issues for popular videos. This paper contributes by revealing 1) the relationship between the user properties and video popularity and 2) how effective the P2P mechanism is in the IPTV system for popular videos with different user properties. The results demonstrate a successful showcase of using the P2P delivery mechanism for IPTV services by analyzing the real operational data of all users. The approach is different from many previous works which has used simulations, test-beds and traffic measurements of an incomplete collection of users. The results potentially helps the design of the next generation of the P2P delivery mechanism for IPTV services.

Human Tracking in a video sequence is an important task in civilian surveillance. Successful human tracking provides data for security-purposes. However, human tracking in video sequences is always a challenging problem. Due to the rapid changes in shape with irregular motion, typical methods may not have good results, especially under occlusion and interaction. Recently, methods based on multiple cameras have been proposed. However, this requires a high computation cost. In the stereo cameras approach, 3D information is obtained and projected onto an occupancy map. In this paper, we propose an algorithm combining the occlusion and interaction model and Markov Chain Monte Carlo (MCMC) such that humans can be tracked under frequent interaction and occlusion. We have successfully reduced the number of failures by 78%. We present an efficient and effective algorithm under frequent interaction and occlusion.