Massive gatherings require hosting authorities to plan and deploy extensive arrangements to facilitate attendees while considering crowd safety the core objective. The task becomes more challenging when crowd is diverse in terms of language, culture, ethics and age-group. Hajj is an annual event where more than 2 million pilgrims from all over the world, gather in Makkah Al-Mukarramah to perform a series of spatio-temporal rituals. To plan and execute the event, the management needs to understand the behavior of both individuals and groups of pilgrims belonging to different nationalities. In this paper, we proposed a big data framework that collects mobile sensory data from all pilgrims by providing them location-based context-aware services through a mobile application. We utilize the collected data to study pedestrians' flow rate, mobility patterns for individuals and groups and their behaviour and preferences during movement. The extracted information can be analyzed to propose solutions to crowd related challenges, including mobility schedules, crowd modelling and simulation and emergency evacuation plans.

Publicly available data is increasing rapidly, and will continue to grow with the advancement of technologies in sensors, smartphones and the Internet of Things. Data from multiple sources can improve coverage and provide more relevant knowledge about surrounding events and points of Interest. The strength of one source of data can compensate for the shortcomings of another source by providing supplementary information. Maps are also getting popular day-by-day and people are using it to achieve their daily task smoothly and efficiently. Starting from paper maps hundred years ago, multiple type of maps are available with point of interest, real-time traffic update or displaying micro-blogs from social media. In this paper, we introduce Hadath, a system that displays multi-resolution live events of interest from a variety of available data sources. The system has been designed to be able to handle multiple type of inputs by encapsulating incoming unstructured data into generic data packets. System extracts local events of interest from generic data packets and identify their spatio-temporal scope to display such events on a map, so that as a user changes the zoom level, only events of appropriate scope are displayed. This allows us to show live events in correspondence to the scale of view - when viewing at a city scale, we see events of higher significance, while zooming in to a neighbourhood, events of a more local interest are highlighted. The final output creates a unique and dynamic map browsing experience. Finally, to validate our proposed system, we conducted experiments on social media data.

Turbulence in crowds due to random, unintentional and uncontrolled movement of the mass is a critical and deadly situation. Live concerts, football matches, festivals are examples of large crowd gatherings. One example of a regularly occurring highly crowded event is Hajj, where more than 3 million people gather in Makkah for five days every year to perform their rituals. Every individual performing Hajj wants to complete their rituals correctly by visiting specific places at specific dates and time. In the past, severe crowd stampedes and casualties have taken place, especially at the site of stoning the Jamarat. This paper presents an interactive tool for safe rescheduling of groups based on their preferences for performing the stoning ritual at Jamarat. The interactive visual tool allows end-users to handle the request of rescheduling on two parameters, safety and temporal preferences. The safety parameter considers the capacity of each road so that it will not cross the limit set by authorities, and the temporal preference parameter considers the time as per the Islamic school of Jurisprudence of the pilgrim. The tool can help schedule the movement of pilgrims incrementally and interactively. The tool was used successfully in Hajj'19. Furthermore, the tool provides visual analytics for stakeholders that helps in decision-making.

Movement of a large crowd from one place to another smoothly is a challenging task. It becomes more critical where the large crowd from different nationalities move together in a defined time to perform specific duties such as Hajj. Hajj is one of the large congregations in the world, where more than 2 million pilgrims coming from all over the world gather in Makkah to perform the fifth pillar of Islam. To perform the Hajj rituals, the pilgrims should stay and move between the holy sites of Mashaer area1on specific days, and time. These spatio-temporal constraints are mandatory to follow to perform the Hajj acceptably. To manage the crowd efficiently, the authorities of Saudi Arabia grouped the pilgrims from all over the world in seven establishments based on nationalities. Specific places, routes and train stations are allocated to each establishment to perform spatio-temporal rituals in Mashaer area. In this paper, we present an optimised scheduling process that schedules the movement of pilgrims to perform spatio-temporal rituals safely using trains. The scheduling process depends on a variety of factors. Firstly, it depends on the train movement type, whether its B (Mina - Arafat), C (Arafat - Muzdalifah), or D (Muzdalifah - Mina). Secondly, the model also relies on the current pilgrims' camp location and the other camp location in the destination. Thirdly, the roads allowed from/to camps-stations also affect the scheduling process. Moreover, to avoid occurring crowd disasters, the scheduling process should monitor station capability to accommodate pilgrims. Our algorithm has been used in Hajj 2019 for three movements (B, C and D) in Mashaer Area. The result shows the analytics of our scheduling algorithm along with comparison between the scheduled time and actual time for each station in Mashaer area in Hajj 2019.

Monitoring and understanding traffic behaviour plays a vital role to handle the daily needs of the users in a smart way, especially in highly populated big cities. Events pulling in big crowds like religious gatherings or sports events, clashes between vehicular traffic movements are unavoidable which results in stoppages and hazardous conditions for the pilgrims, sometimes causes disorder and jam. The problem becomes more challenging when millions of users move from one place to another in a specific duration of time during Hajj. Hajj is one of the pillars of Islam where 3+ millions of pilgrims gather in Makkah, Saudi Arabia to perform their rituals. In this paper, we present an interactive platform to understand the movement of buses during Hajj using large scale GPS traces. We collected the GPS traces of more than 20,000 buses for 60 days for the movement of pilgrims to perform their Spatio-temporal rituals. An interactive platform is developed to store big spatial data and visualize the same. We analyzed the data and identified the routes, parking areas allocated to each establishment and commuting time of routes for different establishments. The knowledge extracted will be beneficial for stakeholders and decision-makers to enhance pilgrims mobility during the hajj by developing an intelligent transportation system such as scheduling, evacuation, optimizing the number of buses or roads for each establishment.

Large gatherings such as concerts, football matches, festivals need an efficient and effective tool to deal with various emergency scenarios. Such events require a proper plan to evacuate the affected areas smoothly and peacefully. The problem of evacuation becomes more complicated when we talk about Haj where 3+ million pilgrims from all over the world gather to perform rituals on specific date and time. In this work-in-progress paper, we present a “Capacity Aware Evacuation Planning using Multimodal Transportation (CAEPM)” platform, to evacuate pilgrims from Mashaer1area in case of disaster using multimodal transportation. The mode can be pedestrian, public/private buses and trains. The pilgrims from all over the world are grouped into seven establishments. Specific routes are allocated to each establishment to perform spatio-temporal rituals in Mashaer area. To manage the crowd and avoid stampede, internal roads are closed in Mashaer. To evacuate any area in Mashaer in case of any unfortunate event, it is very necessary to understand the allowed roads and mode of movements. In order to achieve the same, we received large scale GPS data of more than 20,000 buses from Haj 2018, train movement details, and camps assignments from different authorities of Kingdom of Saudi Arabia. We extracted knowledge from these collected data to identify the bus routes for movements during Haj for different establishments, number of buses assigned to each establishment, pick up points, drop off points, average speed, capacity of train stations, trains and roads using big data platform. We will use this knowledge to develop our novel CAEPM platform. Our proposed platform will provide innovative decision-analysis capabilities to various stakeholders at city and province levels supervising Haj and Umrah activities. Our proposed platform can be handy for pre-planning exercises, thus serving potentially to the control center for managing the Haj event.

Crowdsourcing offers great opportunities to recognise user context and prescribe relevant services for both offline and real-time activities. In this work, we present a zoning model that leverages spatio-temporal dimensions and then employs different contexts to recommend necessary customised services. The context model takes into consideration three context sets: fully restricted, fully unrestricted and semi-restricted with respect to both spatial and temporal dimensions. As a proof of concept, we apply this zoning model in a scenario where a very large crowd get together to perform spatio-temporal activities. The user context of the heterogeneous crowd is captured using the carried smartphones, i.e. via crowdsourcing. Depending on the context sets and zone, the system can recommend a set of services to each user. The system has been deployed since 2014 to support the spatio-temporal activities of a very large crowd. We present our implementation details and the user feedback, which is very encouraging.

International audience

Recommending an optimized path for a large crowd poses a unique challenge to existing routing algorithms due to the interactions between users and the dynamic changes over road networks. Industries, researchers and end users show an enormous interest in crowdsourced data comprising social networks and user-generated content to remain updated with their concerns. In this paper, we present a data collection framework that helps users to find optimized routes in a dynamic environment. We have developed a data collection framework to collect dynamic road conditions via a set of location-based services to support a very large Hajj crowd by capturing their locations using smartphones. We also collect geotagged social network data that provides more details about road conditions. The system leverages geotagged crowdsourced information to identify constraints such as accidents, congestions, and roadblocks. Moreover, by continuously collecting real-time geotagged data of moving users, the system can also find the flow of traffic and road conditions. We propose a spatial grid index to compute the optimized path, and to identify the affected users within impact zones. The plan is to test the whole application and back-end server during Hajj 2016, where over three million pilgrims from all over the world gather to perform their rituals.

Crowdsourced multimedia data poses several challenges when it is collected, stored, indexed, retrieved, and visualized. Examples of crowd source multimedia data are social sensors, vehicle sensors, physical sensors, human sensors, etc. Analyzing such multimodal and diversified crowdsourced data provides very rich understanding about the need of individuals within a crowd. Such understanding makes it possible to tailor services to individuals' needs, also called context-aware services. In this paper, we propose a spatial multimedia big data framework that can collect multimedia data from 1) a very large crowd equipped with multi-sensory smartphones, 2) vehicles, and 3) social networks. A set of multimedia services are offered to users to support their spatio-temporal activities. These include but not limited to 1) simple user interfaces to utilize multimedia services for instant guidance, 2) navigation to points of interests (POI), and 3) efficient and cost effective intra-city rides to users. The big data framework is designed to handle a very large number of multimedia spatio-temporal queries in real-time. The system is a pilot project and will be deployed during the event of Hajj 2015 when over three million pilgrims from all over the world will visit Makkah, Saudi Arabia to perform their Hajj rituals.

Social networks, such as Facebook, Instagram, and Twitter, provide intuitive ways to share a variety of information including geotagged multimedia data within users' communities of interest (COI) or publicly in real-time. Real-time geotagged multimedia data can provide semantics to conventional spatial queries in order to enrich the user navigation experience. In this paper, we introduce a mechanism to process spatio-temporal queries by leveraging geotagged multimedia data such as images, audio, video, and text, in order to add semantics to the conventional queries. Our framework collects, stores, and spatially tags multimedia data shared by users through social networks or through our developed mobile application. The system then uses such data in order to enhance the conventional routing services by resolving existing usability issues and by providing semantics to the routes in terms of enriched points of interest while taking dynamic road conditions into account. A proof of concept of the system will be demonstrated with the following spatio-temporal queries on road networks: 1) multimedia-enhanced shortest path queries; 2) multimedia-enhanced k-nearest neighbor queries; and 3) multimedia-enhanced range queries. Finally, a novel technique for finding lost individuals using geotagged multimedia data is also introduced. The results are tailor-made to the users' smartphone bandwidth and resolution requirements

Traditional routing algorithms for calculating the fastest or shortest path become ineffective or difficult to use when both source and destination are dynamic or unknown. To solve the problem, we propose a novel semantic routing system that leverages geo-tagged rich crowdsourced multimedia information such as images, audio, video and text to add semantics to the conventional routing. Our proposed system includes a Semantic Multimedia Routing Algorithm (SMRA) that uses an indexed spatial big data environment to answer multimedia spatio-temporal queries in real-time. The results are customized to the users' smartphone bandwidth and resolution requirements. The system has been designed to be able to handle a very large number of multimedia spatio-temporal requests at any given moment. A proof of concept of the system will be demonstrated through two scenarios. These are 1) multimedia enhanced routing and 2) finding lost individuals in a large crowd using multimedia. We plan to test the system's performance and usability during Hajj 2015, where over four million pilgrims from all over the world gather to perform their rituals.

Logging on to a system using a conventional keyboard may not be feasible in certain environments, such as, in a surgical operation theatre or in an industrial manufacturing facility. We have developed a multi-sensory gesture based login system that allows a user to access secure information using body gestures. The system can be configured to use different types of gestures according to the type of sensors available to the user. We have proposed a simple scheme to represent all alphanumeric characters required for password entry as gestures within the multi-sensory environment. Our scheme is scalable enough to support sensors that detect a large number of gestures to those that can only accept a few. This allows the system to be used in a variety of situations such as usage by disabled persons with limited ability to perform gestures. We are in the midst of deploying our developed system in a clinical environment.

Traditional routing algorithms for calculating the fastest or shortest path become ineffective or difficult to use when both source and destination are dynamic or unknown. To solve the problem, we propose a novel semantic routing system that leverages geo-tagged rich crowdsourced multimedia information such as images, audio, video and text to add semantics to the conventional routing. Our proposed system includes a Semantic Multimedia Routing Algorithm (SMRA) that uses an indexed spatial big data environment to answer multimedia spatio-temporal queries in real-time. The results are customized to the users' smartphone bandwidth and resolution requirements. The system has been designed to be able to handle a very large number of multimedia spatio-temporal requests at any given moment. A proof of concept of the system will be demonstrated through two scenarios. These are 1) multimedia enhanced routing and 2) finding lost individuals in a large crowd using multimedia. We plan to test the system's performance and usability during Hajj 2015, where over four million pilgrims from all over the world gather to perform their rituals.

This paper presents a system that enables efficient and scalable real-time user and vehicle discovery using textual, audio and video mechanisms. The system allows users to group together for shared intra-city transportation with the aid of multimedia that helps individuals to 1) find community of common interest (CoCI), 2) locate individual users in a large crowd and 3) locate vehicles for mobility in an efficient and cost effective manner. The system is a pilot project and will be deployed during Hajj 2015 when over three million pilgrims from all over the world visit Makkah, Saudi Arabia.

We propose a context aware framework that offers a set of cloud-based services to support a very large Hajj and Umrah crowd by capturing their contexts using smartphones. The proposed framework captures the individual's context, provides a set of adapted services, and allows being in touch with a subset of one's community of interest. We leverage the spatiotemporal sensory data captured by our framework to define users' contexts for optimized services. Our proposed framework is also envisioned to assist the Hajj and Umrah authorities to (1) improve Hajj & Umrah documentation, (2) improve Hajj organization through better understanding of pilgrims' (individual and crowd) spatial and temporal behavior and needs, and (3) protect pilgrims' environment through environmental monitoring. In particular, the developed methods, techniques, and algorithms will support the pilgrimage quality of experience. We have tested our system through end-user subjects and due to apply for the upcoming Hajj events. We present our implementation details and the general impression of end users about our system.

In this paper, we present a novel idea of a map-based therapy environment for people with Hemiplegia. The therapy environment is designed according to the suggestions of therapists, which consists of a spatial map browsing serious game augmented with our novel multi-sensory natural user interface (NUI). The NUI is based on 3D motion sensors that can recognize different hand and body gestures used for browsing a 3D or 2D map. The 3D motion sensors work in a non-invasive way; hence, they do not require any wearable body attachments and can be used at home without assistance from the therapists. The map-browsing environment provides an immersive experience to the disabled users, which helps in performing therapy in an interesting and entertaining manner. We have developed analytics for measuring certain quality of health improvement metrics from each type of spatial map browsing movements. The 3D motion sensors have been tested with Nokia, Google, ESRI, and a number of other maps that allow a subject to visualize and browse the 3D and 2D maps of the world. The map browsing session data shows the nature of big data; hence, the session data is stored in a cloud environment. Our developed serious game environment is web-based; thus anyone having the appropriate low cost sensor hardware can plug it in and start experiencing a natural way of hands free map browsing. We have deployed our framework in a hospital that treats Hemiplegic patients. Based on the feedback obtained, the developed platform shows a huge potential for use in hospitals that provide physiotherapy services as well as at patients' home as an assistive therapeutic service.

Recently, the use of non-invasive ways to track joint motion in the human body has drawn significant attention in the therapy domain. One of the reasons for this popularity is due to availability of economically priced 3D motion sensors. In this paper, we present a web-based 3D interactive serious game interface that uses non-invasive methods to recognize the movements of the body. Motion data of a subject is collected through two motion sensors, a Kinect and a LEAP, in a non-invasive manner. Joint motion along with inverse kinematic joint information is displayed in 3D environment in a live manner or recorded for offline replaying and data analysis. To facilitate the complex therapy authoring process, the system incorporates an authoring tool that allows a therapist to design a complex therapy in terms of primitive therapies and assign it to a patient. The subject as well as other members of the community of interest such as therapists, parents and caregivers can view the results at any time and can follow up with patient's progress.

This paper presents a multimedia e-health framework to conduct therapy sessions by collecting live therapeutic data of patients in a non-invasive way. Using our proposed framework, a therapist can model complex gestures by mapping them to a set of primitive actions and generate high-level therapies. Two inexpensive 3D motion tracking sensors, a Kinect and a Leap, are used to collect motion data of a given subject. Data can then be displayed on screen in a live manner or recorded for offline replaying and analysis. The system incorporates an intelligent authoring tool for therapy design, and produces live plots that show the quality of improvement metrics for a given patient.