In recent years, with the advancement of artificial intelligence research, the importance of accurate emotion estimation has been increasingly recognized. In this study, we propose an emotion estimation method for processing novel multi-resolution integrated images, based on a learning-type fuzzy template matching (L-FTM) approach. The proposed method enables the transformation of 30 s EEG data from 8 channels—corresponding to a spectrum of emotional intensities ranging from unpleasant to pleasant—into images suitable for image classification models. Compared with the traditional StI method, the StI method proposed in this study combines the brain electrical features from five frequency bands across eight channels by using spatial frequency to form a more informative multi-resolution integrated image with higher density. The results demonstrate that the images generated by the proposed StI method contain more discriminative features for image-based classification compared to those generated using a single spatial frequency. These enhanced image representations are expected to improve the performance of emotion intensity estimation models in future research.

At HAI 2023, a dialogue agent using a cultured biological neuronal network as a semi-artificial brain was presented: "A semi-living broken dialogue agent depending on the internal state of a living neuronal network." This system identifies neuronal activity patterns elicited by "pleasant" and "unpleasant" inputs. To improve identification accuracy, we propose a method utilizing instantaneous spatial patterns and transfer learning in Deep Learning techniques (Figure 1). Using a multi-electrode array from cultured neural networks, the method analyzes electrical spikes within a 1 ms window to define instantaneous spatial patterns (ISP) capturing signal propagation pathways. Leveraging advanced deep learning techniques, particularly transfer learning with the VGG16 CNN model, enabled the method to identify neuronal activity from different responses with over 90% accuracy. Additionally, the study introduced two imaging techniques: TIP-NAP and SIP-NAP, effectively distinguishing neuronal activities. The results underscore the importance of temporal continuity in neuronal activity for accurate information representation.

We have been exploring a system that constructs a hybrid semi-artificial life that fuses a biological neuronal network and a hardware information processing device, and expresses the phenomena that occur there as "intelligent behavior". In this study, we implemented a broken dialogue agent that is a wetware-hardware hybrid that has a sound input/output interface that connects the biological neural network to the outside world, in order to reflect the "fluctuation" unique to living organisms in the system. The agent extracts words contained in the voice input, evaluates the impression of the dialogue based on the "emotional polarity dictionary", and applies electrical stimulation to the cultured living neuronal network according to the evaluation result. The induced neuronal electrical activity pattern was defined as the "internal state" of the agent, and this determined the emotional polarity value of the agent itself at that time. The agent selects words that are compatible with the emotional polarity of agent extracting from co-occurring words and outputs them in "broken sentences". On the human side, we aimed to establish a conversation by complementing and understanding the meaning from this broken word sequence.

In recent years, the development of AI technology has brought increased importance to the accurate assessment of emotions. To explore brainwaves strongly associated with emotions and achieve more precise assessments of emotional intensity, this study utilized the L-FTM (Learning type Fuzzy-Template-Matching) method to process the amplitude power spectra area of EEG data in the DEAP dataset for predicting the emotional intensity from unhappy to happy. The results indicate that the data from the selected eight channels in this study do not exhibit a strong correlation with emotions. This suggests the exploration space we used in the experiments did not contain brainwave features suitable for inferring the emotional intensity of the Unhappy-Happy axis.

Analyzing the dynamics of neural activity patterns using an electrophysiological approach is important for understanding the basis of information processing in a brain. In this study, we attempted to extract units of neuronal activity representing certain information and to discriminate activity pattern evoked by electrical stimulus and spontaneous activity without stimulation. We applied X-means clustering to the triplet of “2D-spacial coordinates and timestamps” of neuronal electrical spike bursts and the extracted single spatiotemporal neuronal activity pattern was converted into a standardized $8\times 84$ 2D-spatial pattern map. Most 2D-spatial patterns emerged repeatedly during recording time, corresponding to a representative motif. Then we gathered similar 2D-spatial patterns (primary clusters) into one ”pattern repertoire” and attempted to classify these pattern repertoires depending on the stimulus inducing the pattern repertoire by VGG16 deep convolutional neural networks (deep CNN). For that, we prepared two types of $224\times 224$ images by converting the 2D-spatial pattern maps, the spatial information priority neural activity pattern image (SIP-NAP image) and the time information priority neural activity pattern image (TIP-NAP image). As a result, over 80% of high discrimination accuracy was obtained for both types of VGG16 input images, especially 99.7% for TIP-NAP image. It suggested that spatiotemporal features contributing to the discrimination between spontaneous and evoked response activities were extracted by 3-D-crustering. In addition, it is shown that the spatial pattern of neuroelectric activity is able to be discriminated with high accuracy by combining transfer learning and SIP-NAP / TIP-NAP image translation method, even with a small amount of training data.

BCI based on the EEG signal of previously assumed frequency bands and measurement sites, adapted only for users who stably exhibit EEG features with high reproducibility against the preassumed cognitive tasks. However, users of BCI system is not always well-adapters. In this study, to solve the problem of the incompatibility, we propose a heuristic type BCI which automatically extracts characteristic patterns of EEG induced by certain cognitive tasks. Learning-type-fuzzy-template-matching method (L-FTM) was implemented in the heuristic BCI. L-FTM is an application of the leaning type "fuzzy singleton-type reasoning" to template matching method. The search space of the EEG pattern consists of combinations of fuzzy labels in the antecedent-clause of the fuzzy rule (if-clause), and such specific EEG feature patterns are linked to specific output values of consequent-clauses (then-clause). We also implemented "pruning" that deletes inadequate rule with high compatibility degree to EEG features appearing both of task (target) and non-task status. In the experimental results, output of BCI system during task status was larger than during non-task status in each participant. We confirmed that using the heuristic BCI system, fuzzy rules corresponding to specific EEG feature patterns appearing during motor imagery task were automatically extracted by the learning process. In this study, we confirmed that the developed BCI system recognized a certain cognitive situation from EEG patterns varied among individuals induced by the same cognitive task, adapting to the individual characteristics of EEG feature without any assumptions.

To elucidate the brain information system, it is important to understand dynamics of nonlinear neuronal activity patterns, fluctuating by the internal state of the neuronal network in the brain. In this study, we attempted to identify neuronal activity patterns including evoked responses in the autonomously reconstructed rat neuronal network. We adopted the multilayered artificial neural network (ml-ANN) as the Deep-Learning method with stacked-autoencoder as the pre-training method and classified the neuronal feature expressions. As a result of comparing the discrimination accuracy with several different hyper-parameters, the activity pattern of later time domain after the stimulation was not distinguished with high accuracy. In contrast, activity in the time domain 2 s after the electrical stimulation was discriminated into several patterns, although the discrimination ability of ml-ANN was not enough, because of the insufficient amount of learning-data, which is difficult to acquire in large amount. It is considered that a huge number of pretraining data is absolutely necessary to get the discrimination accuracy better in order to identify patterns by the Deep- Learning method for large phenomena with "fluctuation" such as brain activity.

Toward neuroprosthetic technology, it is critical that a simple model system for interaction between brain and electric devices. For this purpose, we developed neurorobot system, Vitroid, equipped with a living neuronal network and a miniature moving robot as a body of the neurorobot. Self-Organization-Map (SOM) was employed as a generator for behavior of Vitroid. SOM was designed to map a high-dimensional feature vector to a 2-dimentional vector as the winner unit in output layer of SOM. Furthermore, neighboring units were assigned to resemble input vectors. Thus, SOM also performs pattern classifying analysis for inputted feature vector of neuronal activity. Cultured neuronal networks on Multi-Electrodes-Array (MEA) dish was alternately stimulated by two different electrodes. SOM mapped patterns induced by electrical stimulation to a 30 × 30 - 2D output layer. Only in the first step of the learning, SOM is forced to select a specific winner unit previously assigned in order to associate specific behaviors. We call this process “Seeding”. After seeding process, the winner-units correspond to the response patterns induced by two different stimuli were separately mapped. We confirmed that response patterns by two different electrical stimuli could be classified and they were almost stable. Furthermore, it revealed that spontaneous activity and evoked response shared the same patterns, suggesting that the internal autonomous activity is not only a noise, but is almost equivalent to a meaningful response. We also succeeded in collision avoidance of Vitroid by SOM-based behavior generator.

The culture dish describes the small fundamental world resembling human brain function. Multi-site recording system for extracellular action potentials is used for recording the activity of living neuronal networks. The living neuronal network is able to express several patterns independently, and that's meaning that it has fundamental mechanisms for intelligent information processing. In this paper, we propose a model to analyse logicality of signals and connectivity of electrodes in a culture dish of rat hippocampal neurons. We call it "fuzzy bio-indicator". This indicator is a kind of mapping methods to show logicality and connectivity of pulse frequency from active potential of neuronal network. We try to analyze the dynamics of action potentials of neuronal networks by the fuzzy bio-indicator, and identify the logicality and connectivity of neuronal networks through the indicator. We show here the usefulness of fuzzy bio-indicator through numerical examples and action potential detected from the culture neuronal network.

To mimic biological intelligence, it is critical to elucidate the network dynamics of a neural network. The dissociated culture system possesses a simple network comparing to a whole brain, thus it is suitable for exploration of spatiotemporal dynamics of electrical activity of a neuronal circuit. Cultured neuronal network has no input-output system, so it requires an artificial peripheral system to interact with outer world. We are developing the neurorobot as the model system for biological information processing with vital components and the artificial peripheral system. The behavior of the neuro-robot is determined by the response pattern of neuronal electrical activity evoked by a current stimulation from outer world. In this study, we developed a novel type of neurorobot with Self-Organization Map (SOM) for a neuronal output pattern decoder. The robot with SOM is expected to perform non-stop learning and generation of behavior simultaneously. The spatiotemporal electrical patterns evoked by the inputs according to the value of the IR sensors on the robot body are translated to 64 dimension feature vectors and inputted to the SOM. Then the 64 dimension feature vectors are mapped to a certain winner vector in the 10 × 10 output layer of SOM. Winner nodes are linked to the purposive behaviors adequate to the inputs according to outer phenomenon. Only at the beginning of the behavior, neurorobot SOM selects two winner nodes premisely assigned to the specific inputs for the obstacles near the L and R side of the robot body. We call the process as “seeding”. After the seeding process, the distribution of winner units for the two inputs were separated each other, when the spatiotemporal pattern of electrical activity were not overlapped. In addition, the position of the centers of winner nodes gravities, updated with every input, are almost stable in the output layer of the SOM.

Recently, many attractive brain-computer interface and brain-machine interface have been proposed. The outer computer and machine are controlled by brain action potentials detected through a device such as near-infrared spectroscopy (NIRS) and electroencephalograph (EEG), and some discriminant model determines a control process. In this paper, we introduce a fuzzy bio-interface between a culture dish of rat hippocampal neurons and the khepera robot. We propose a model to analyze logic of signals and connectivity of electrodes in a culture dish, and show the bio-robot hybrid we developed. We believe that the framework of fuzzy system is essential for BCI and BMI, thus name this technology “fuzzy bio-interface”. We show the usefulness of a fuzzy bio-interface through some examples.

Brain-computer-interface has been come into the research limelight. Network dynamics of neurons strongly effects to a control of computer or machine in the outer world. Dissociated culture system with multi-electrode array is useful for elucidation of network dynamics of neurons. We have been investigating action potentials of rat hippocampal neurons cultured on the dish connecting with the outer robot. However, we don't exactly comprehend logicality of living neuronal networks. In this paper, we identify logicality of living neuronal networks with three electrodes in mult-electrode array using fuzzy connective operators consisting of t - norm and t - conorm operators, and we introduce a straight running of fuzzy bio-robot. We concluded that the logicality of living neuronal networks is dynamically changed to weak OR connection from strong AND connection. Additionally, by applying the fuzzy bio-robot to a straight running, we analyzed plasticity of living neuronal network connected to the robot, and we discussed regularity of logical potential response of the neuronal networks.

The temporal patterns of spontaneous action potentials are analyzed, using the multi-site recording system for extracellular potentials of neurons and the living neuronal networks cultured on a 2-dimensional electrode arrays. We carried out the system integration for Khepera II robot and living neuronal network. We call the system as "biomodeling system". Our goal is reconstruction of the neuronal network, which can process "thinking" in the dissociated culture system.

In dissociated neurons, we induced a long lasting modification of synaptic transmission, and found out that functional connections between neurons in the living neuronal network was changed dynamically by a transient increase of electrical activities of the whole neural network. Using electrical inputs in the network, we were able to operate the spatio-temporal patterns autonomously stored in this neural network. The high frequency stimulation (HFS), which is known to trigger long-term potentiation in vivo, was suitable for that manipulation. We tried to make an association of a particular activity pattern with another pattern by simultaneous recall of stored pattern. After HFS, One of the divided subsets of patterns frequently merged to another one suggested that particular cue stimulation linked to two-stored pattern simultaneously. Although these are still preliminary results, they have suggested that it is possible to operate the information processing in a neural network by the external electrical inputs.