We study the greedy (exploitation-only) algorithm in bandit problems with a known reward structure. We allow arbitrary finite reward structures, while prior work focused on a few specific ones. We fully characterize when the greedy algorithm asymptotically succeeds or fails, in the sense of sublinear vs. linear regret as a function of time. Our characterization identifies a partial identifiability property of the problem instance as the necessary and sufficient condition for the asymptotic success. Notably, once this property holds, the problem becomes easy—\emph{any} algorithm will succeed (in the same sense as above), provided it satisfies a mild non-degeneracy condition. Our characterization extends to contextual bandits and interactive decision-making with arbitrary feedback. Examples demonstrating broad applicability and extensions to infinite reward structures are provided.

I study adversarial attacks against stochastic bandit algorithms. At each round, the learner chooses an arm, and a stochastic reward is generated. The adversary strategically adds corruption to the reward, and the learner is only able to observe the corrupted reward at each round. Two sets of results are presented in this paper. The first set studies the optimal attack strategies for the adversary. The adversary has a target arm he wishes to promote, and his goal is to manipulate the learner into choosing this target arm $T - o(T)$ times. I design attack strategies against UCB and Thompson Sampling that only spends $\widehat{O}(\sqrt{\log T})$ cost. Matching lower bounds are presented, and the vulnerability of UCB, Thompson sampling and $\varepsilon$-greedy are exactly characterized. The second set studies how the learner can defend against the adversary. Inspired by literature on smoothed analysis and behavioral economics, I present two simple algorithms that achieve a competitive ratio arbitrarily close to 1.

I study the problem of learning a Lipschitz function with corrupted binary signals. The learner tries to learn a $L$-Lipschitz function $f: [0,1]^d \rightarrow [0, L]$ that the adversary chooses. There is a total of $T$ rounds. In each round $t$, the adversary selects a context vector $x_t$ in the input space, and the learner makes a guess to the true function value $f(x_t)$ and receives a binary signal indicating whether the guess is high or low. In a total of $C$ rounds, the signal may be corrupted, though the value of $C$ is unknown to the learner. The learner’s goal is to incur a small cumulative loss. This work introduces the new algorithmic technique agnostic checking as well as new analysis techniques. I design algorithms which: for the absolute loss, the learner achieves regret $L\cdot O(C\log T)$ when $d = 1$ and $L\cdot O_d(C\log T + T^{(d-1)/d})$ when $d > 1$; for the pricing loss, the learner achieves regret $L\cdot \widetilde{O} (T^{d/(d+1)} + C\cdot T^{1/(d+1)})$.

Recent works have shown a reduction from contextual bandits to online regression under a realizability assumption (Foster and Rakhlin, 2020; Foster and Krishnamurthy, 2021). In this work, we investigate the use of neural networks for such online regression and associated Neural Contextual Bandits (NeuCBs). Using existing results for wide networks, one can readily show a ${\mathcal{O}}(\sqrt{T})$ regret for online regression with square loss, which via the reduction implies a ${\mathcal{O}}(\sqrt{K} T^{3/4})$ regret for NeuCBs. Departing from this standard approach, we first show a $\mathcal{O}(\log T)$ regret for online regression with almost convex losses that satisfy QG (Quadratic Growth) condition, a generalization of the PL (Polyak-\L ojasiewicz) condition, and that have a unique minima. Although not directly applicable to wide networks since they do not have unique minima, we show that adding a suitable small random perturbation to the network predictions surprisingly makes the loss satisfy QG with unique minima. Based on such a perturbed prediction, we show a ${\mathcal{O}}(\log T)$ regret for online regression with both squared loss and KL loss, and subsequently convert these respectively to $\tilde{\mathcal{O}}(\sqrt{KT})$ and $\tilde{\mathcal{O}}(\sqrt{KL^*} + K)$ regret for NeuCB, where $L^*$ is the loss of the best policy. Separately, we also show that existing regret bounds for NeuCBs are $\Omega(T)$ or assume i.i.d. contexts, unlike this work. Finally, our experimental results on various datasets demonstrate that our algorithms, especially the one based on KL loss, persistently outperform existing algorithms.

Many bandit problems are characterized by the learner making decisions under constraints. The learner in Linear Contextual Bandits with Knapsacks (LinCBwK) receives a resource consumption vector in addition to a scalar reward in each time step which are both linear functions of the context corresponding to the chosen arm. For a fixed time horizon $T$, the goal of the learner is to maximize rewards while ensuring resource consumptions do not exceed a pre-specified budget. We present algorithms and characterize regret for LinCBwK in the smoothed setting where base context vectors are assumed to be perturbed by Gaussian noise. We consider both the stochastic and adversarial settings for the base contexts, and our analysis of stochastic LinCBwK can be viewed as a warm-up to the more challenging adversarial LinCBwK. For the stochastic setting, we obtain $O(\sqrt{T})$ additive regret bounds compared to the best context dependent fixed policy. The analysis combines ideas for greedy parameter estimation in \cite{kmrw18, siwb20} and the primal-dual paradigm first explored in \cite{agde17, agde14}. Our main contribution is an algorithm with $O(\log T)$ competitive ratio relative to the best context dependent fixed policy for the adversarial setting. The algorithm for the adversarial setting employs ideas from the primal-dual framework \cite{agde17, agde14} and a novel adaptation of the doubling trick \cite{isss19}.

Adversarial Transferability is an intriguing property - adversarial perturbation crafted against one model is also effective against another model, while these models are from different model families or training processes. To better protect ML systems against adversarial attacks, several questions are raised: what are the sufficient conditions for adversarial transferability, and how to bound it? Is there a way to reduce the adversarial transferability in order to improve the robustness of an ensemble ML model? To answer these questions, in this work we first theoretically analyze and outline sufficient conditions for adversarial transferability between models; then propose a practical algorithm to reduce the transferability between base models within an ensemble to improve its robustness. Our theoretical analysis shows that only promoting the orthogonality between gradients of base models is not enough to ensure low transferability; in the meantime, the model smoothness is an important factor to control the transferability. We also provide the lower and upper bounds of adversarial transferability under certain conditions. Inspired by our theoretical analysis, we propose an effective Transferability Reduced Smooth (TRS) ensemble training strategy to train a robust ensemble with low transferability by enforcing both gradient orthogonality and model smoothness between base models. We conduct extensive experiments on TRS and compare with 6 state-of-the-art ensemble baselines against 8 whitebox attacks on different datasets, demonstrating that the proposed TRS outperforms all baselines significantly.

Research has demonstrated that spatial visualization skills are crucial for success in Science, technology, engineering, and mathematics (STEM) disciplines. With an increasing number of students entering STEM disciplines, the question of how to effectively train students» spatial visualization skills has become very important. While a scalable existing solution is to implement online workshops for students, the problem of how to motivate students to participate in these online workshops remains unsolved. In this study, we studied gamification as a way to motivate first year engineering students to take part in an online workshop designed to train their spatial visualization skills. Our game contains eight modules, each designed to train a different component of spatial visualization. The game records players» in-game behavior with high granularity, which allows us to provide automated, scalable feedback on players» problem-solving strategies. Ten students with different levels of spatial ability played our game and expressed a strong interest in using the game to train their spatial visualization skills in the future. In addition, our analysis of players» in-game behaviors shows the potential benefits of implementing adaptive and personalized learning guidance.

Augmented Reality (AR) allows virtual object projection with an unblocked view of the physical world which provides reference and other people. The mixed scene provides an agile platform for communication and collaboration, especially on a product that would be difficult or expensive to present otherwise. In the boating industry, high customization leaves dealers with a high cost on inventory, financially and spatially. In this work, we present BoatAR, a multi-user AR boat configuration system designed for addressing these issues. A prototype system was implemented using HoloLens with shared experience, and demonstrated to a group of boat dealers and received positive feedback. BoatAR provided an example of how a multi-user AR system could help in the conventional industry.

Flow-based microfluidic biochips have progressed significantly in the past decade. Thanks to innovations in multilayer soft lithography (MSL) fabrication technology, the integration of thousands of microvalves along with large-scale networks of microchannels on a chip has been enabled. This progress has even been compared to the evolution of VLSI circuits following Moore's Law. In flow-based microfluidic biochips, microvalves are critical components to control the fluidic transportation for complex operations. To activate the open/close states of a microvalve, off-chip control pins are required. Due to the tremendous increase of the number of microvalves, a software-programmable microfluidic platform has been proposed to reduce the number of off-chip control pins, which integrates a microfluidic multiplexer on a separate control layer to control the array of microvalves. The multiplexer needs to be switched when the states of microvalves are changed between every two adjacent time slots. High switching frequency will make the multiplexer vulnerable and decrease the chip's reliability. We observe that different switching orders of microvalves lead to different switching frequencies of a multiplexer. Based on this observation, this paper proposes the first Hamming-distance-based switching order optimization method for microvalves to enhance the reliability of the multiplexer. Experimental results show that our method can significantly reduce the switching frequency of multiplexer, and the solution is very close to the theoretical optimal lower bound.