# R Packages worth a look

Time Series with Matrix Profile (tsmp)
A toolkit implementing the Matrix Profile concept that was created by CS-UCR <<a href="http://www.cs.ucr.edu/~eamonn/MatrixProfile.html&gt;." target …

Provides an interface to the Spotify API <https://…/>.

Simulation of Various Risk Processes (ruin)
A (not yet exhaustive) collection of common models of risk processes in actuarial science, represented as formal S4 classes. Each class (risk model) ha …

Management Strategy Evaluation Toolkit (MSEtool)
Simulation tools for management strategy evaluation are provided for the ‘DLMtool’ operating model to inform data-rich fisheries. ‘MSEtool’ provides co …

Stratified Heterogeneity Measure, Dominant Driving Force Detection, Interaction Relationship Investigation (geodetector)
Spatial stratified heterogeneity (SSH), referring to the within strata are more similar than the between strata, a model with global parameters would b …

# Document worth reading: “A Survey on Resilient Machine Learning”

Machine learning based system are increasingly being used for sensitive tasks such as security surveillance, guiding autonomous vehicle, taking investment decisions, detecting and blocking network intrusion and malware etc. However, recent research has shown that machine learning models are venerable to attacks by adversaries at all phases of machine learning (eg, training data collection, training, operation). All model classes of machine learning systems can be misled by providing carefully crafted inputs making them wrongly classify inputs. Maliciously created input samples can affect the learning process of a ML system by either slowing down the learning process, or affecting the performance of the learned mode, or causing the system make error(s) only in attacker’s planned scenario. Because of these developments, understanding security of machine learning algorithms and systems is emerging as an important research area among computer security and machine learning researchers and practitioners. We present a survey of this emerging area in machine learning. A Survey on Resilient Machine Learning

# Book Memo: “Computational Intelligence for Pattern Recognition”

 The book presents a comprehensive and up-to-date review of fuzzy pattern recognition. It carefully discusses a range of methodological and algorithmic issues, as well as implementations and case studies, and identifies the best design practices, assesses business models and practices of pattern recognition in real-world applications in industry, health care, administration, and business. Since the inception of fuzzy sets, fuzzy pattern recognition with its methodology, algorithms, and applications, has offered new insights into the principles and practice of pattern classification. Computational intelligence (CI) establishes a comprehensive framework aimed at fostering the paradigm of pattern recognition. The collection of contributions included in this book offers a representative overview of the advances in the area, with timely, in-depth and comprehensive material on the conceptually appealing and practically sound methodology and practices of CI-based pattern recognition.

# Magister Dixit

“People refer to neural networks as just ‘another tool in your machine learning toolbox’…. Unfortunately, this interpretation completely misses the forest for the trees. Neural networks are not just another classifier; they represent the beginning of a fundamental shift in how we write software. They are Software 2.0.” Andrej Karpathy ( 2017 )

# R Packages worth a look

Diffusion Map (diffusionMap)
Implements diffusion map method of data parametrization, including creation and visualization of diffusion map, clustering with diffusion K-means and r …

Uniform Sampling (uniformly)
Uniform sampling on various geometric shapes, such as spheres, ellipsoids, simplices.

Mixed Effects Regression for Linear, Non-Linear and User-Defined Models (merlin)
Fits linear, non-linear, and user-defined mixed effects regression models following the framework developed by Crowther (2017) <arXiv:1710.02223> …

A Toolkit for Year-Quarter and Year-Month Dates (dint)
S3 classes and methods to create and work with year-quarter and year-month vectors. Basic arithmetic operations (such as adding and subtracting) are su …

Landscape Metrics for Categorical Map Patterns (landscapemetrics)
Calculates landscape metrics for categorical landscape patterns in a tidy workflow. ‘landscapemetrics’ reimplements the most common metrics from ‘FRAGS …

# If you did not already know

Network Theory
In computer and network science, network theory is the study of graphs as a representation of either symmetric relations or, more generally, of asymmetric relations between discrete objects. Network theory is a part of graph theory. It has applications in many disciplines including statistical physics, particle physics, computer science, electrical engineering, biology, economics, operations research, and sociology. Applications of network theory include logistical networks, the World Wide Web, Internet, gene regulatory networks, metabolic networks, social networks, epistemological networks, etc; see List of network theory topics for more examples. Euler’s solution of the Seven Bridges of Königsberg problem is considered to be the first true proof in the theory of networks. …

TypeSQL
Interacting with relational databases through natural language helps users of any background easily query and analyze a vast amount of data. This requires a system that understands users’ questions and converts them to SQL queries automatically. In this paper we present a novel approach, TypeSQL, which views this problem as a slot filling task. Additionally, TypeSQL utilizes type information to better understand rare entities and numbers in natural language questions. We test this idea on the WikiSQL dataset and outperform the prior state-of-the-art by 5.5% in much less time. We also show that accessing the content of databases can significantly improve the performance when users’ queries are not well-formed. TypeSQL gets 82.6% accuracy, a 17.5% absolute improvement compared to the previous content-sensitive model. …

Conditional Extreme Value Models
Extreme value theory (EVT) is often used to model environmental, financial and internet traffic data. Multivariate EVT assumes a multivariate domain of attraction condition for the distribution of a random vector necessitating that each component satisfy a marginal domain of attraction condition. Heffernan and Tawn [2004] and Heffernan and Resnick [2007] developed an approximation to the joint distribution of the random vector by conditioning on one of the components being in an extreme value domain. The usual method of analysis using multivariate extreme value theory often is not helpful either because of asymptotic independence or due to one component of the observation vector not being in a domain of attraction. These defects can be addressed by using the conditional extreme value model. …

# Document worth reading: “Cogniculture: Towards a Better Human-Machine Co-evolution”

Research in Artificial Intelligence is breaking technology barriers every day. New algorithms and high performance computing are making things possible which we could only have imagined earlier. Though the enhancements in AI are making life easier for human beings day by day, there is constant fear that AI based systems will pose a threat to humanity. People in AI community have diverse set of opinions regarding the pros and cons of AI mimicking human behavior. Instead of worrying about AI advancements, we propose a novel idea of cognitive agents, including both human and machines, living together in a complex adaptive ecosystem, collaborating on human computation for producing essential social goods while promoting sustenance, survival and evolution of the agents’ life cycle. We highlight several research challenges and technology barriers in achieving this goal. We propose a governance mechanism around this ecosystem to ensure ethical behaviors of all cognitive agents. Along with a novel set of use-cases of Cogniculture, we discuss the road map ahead for this journey. Cogniculture: Towards a Better Human-Machine Co-evolution

# Whats new on arXiv

We study the ensemble Kalman filter (EnKF) algorithm for sequential data assimilation in a general situation, that is, for nonlinear forecast and measurement models with non-additive and non-Gaussian noises. Such applications traditionally force us to choose between inaccurate Gaussian assumptions that permit efficient algorithms (e.g., EnKF), or more accurate direct sampling methods which scale poorly with dimension (e.g., particle filters, or PF). We introduce a trimmed ensemble Kalman filter (TEnKF) which can interpolate between the limiting distributions of the EnKF and PF to facilitate adaptive control over both accuracy and efficiency. This is achieved by introducing a trimming function that removes non-Gaussian outliers that introduce errors in the correlation between the model and observed forecast, which otherwise prevent the EnKF from proposing accurate forecast updates. We show for specific trimming functions that the TEnKF exactly reproduces the limiting distributions of the EnKF and PF. We also develop an adaptive implementation which provides control of the effective sample size and allows the filter to overcome periods of increased model nonlinearity. This algorithm allow us to demonstrate substantial improvements over the traditional EnKF in convergence and robustness for the nonlinear Lorenz-63 and Lorenz-96 models.
Approaches to decision-making under uncertainty in the belief function framework are reviewed. Most methods are shown to blend criteria for decision under ignorance with the maximum expected utility principle of Bayesian decision theory. A distinction is made between methods that construct a complete preference relation among acts, and those that allow incomparability of some acts due to lack of information. Methods developed in the imprecise probability framework are applicable in the Dempster-Shafer context and are also reviewed. Shafer’s constructive decision theory, which substitutes the notion of goal for that of utility, is described and contrasted with other approaches. The paper ends by pointing out the need to carry out deeper investigation of fundamental issues related to decision-making with belief functions and to assess the descriptive, normative and prescriptive values of the different approaches.
Given a partial description like ‘she opened the hood of the car,’ humans can reason about the situation and anticipate what might come next (‘then, she examined the engine’). In this paper, we introduce the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning. We present SWAG, a new dataset with 113k multiple choice questions about a rich spectrum of grounded situations. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, we propose Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data. To account for the aggressive adversarial filtering, we use state-of-the-art language models to massively oversample a diverse set of potential counterfactuals. Empirical results demonstrate that while humans can solve the resulting inference problems with high accuracy (88%), various competitive models struggle on our task. We provide comprehensive analysis that indicates significant opportunities for future research.
This article proposes a unified framework, the balancing weights, for estimating causal effects with multi-valued treatments using propensity score weighting. These weights incorporate the generalized propensity score to balance the weighted covariate distribution of each treatment group, all weighted toward a common pre-specified target population. The class of balancing weights include several existing approaches such as inverse probability weights and trimming weights as special cases. Within this framework, we propose a class of target estimands based on linear contrasts and their corresponding nonparametric weighting estimators. We further propose the generalized overlap weights, constructed as the product of the inverse probability weights and the harmonic mean of the generalized propensity scores, to focus on the target population with the most overlap in covariates. These weights are bounded and thus bypass the problem of extreme propensities. We show that the generalized overlap weights minimize the total asymptotic variance of the nonparametric estimators for the pairwise contrasts within the class of balancing weights. We also develop two new balance check criteria and a sandwich variance estimator for estimating the causal effects with generalized overlap weights. We illustrate these methods by simulations and apply them to study the racial disparities in medical expenditure.
Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation. One crucial aspect for this progress are novel neural architectures. Currently employed architectures have mostly been developed manually by human experts, which is a time-consuming and error-prone process. Because of this, there is growing interest in automated neural architecture search methods. We provide an overview of existing work in this field of research and categorize them according to three dimensions: search space, search strategy, and performance estimation strategy.
While deep learning models and techniques have achieved great empirical success, our understanding of the source of success in many aspects remains very limited. In an attempt to bridge the gap, we investigate the decision boundary of a production deep learning architecture with weak assumptions on both the training data and the model. We demonstrate, both theoretically and empirically, that the last weight layer of a neural network converges to a linear SVM trained on the output of the last hidden layer, for both the binary case and the multi-class case with the commonly used cross-entropy loss. Furthermore, we show empirically that training a neural network as a whole, instead of only fine-tuning the last weight layer, may result in better bias constant for the last weight layer, which is important for generalization. In addition to facilitating the understanding of deep learning, our result can be helpful for solving a broad range of practical problems of deep learning, such as catastrophic forgetting and adversarial attacking. The experiment codes are available at https://…/NN_decision_boundary
When neural networks process images which do not resemble the distribution seen during training, so called out-of-distribution images, they often make wrong predictions, and do so too confidently. The capability to detect out-of-distribution images is therefore crucial for many real-world applications. We divide out-of-distribution detection between novelty detection —images of classes which are not in the training set but are related to those—, and anomaly detection —images with classes which are unrelated to the training set. By related we mean they contain the same type of objects, like digits in MNIST and SVHN. Most existing work has focused on anomaly detection, and has addressed this problem considering networks trained with the cross-entropy loss. Differently from them, we propose to use metric learning which does not have the drawback of the softmax layer (inherent to cross-entropy methods), which forces the network to divide its prediction power over the learned classes. We perform extensive experiments and evaluate both novelty and anomaly detection, even in a relevant application such as traffic sign recognition, obtaining comparable or better results than previous works.
Given a social network with diffusion probabilities as edge weights and an integer k, which k nodes should be chosen for initial injection of information to maximize influence in the network? This problem is known as Target Set Selection in a social network (TSS Problem) and more popularly, Social Influence Maximization Problem (SIM Problem). This is an active area of research in computational social network analysis domain since one and half decades or so. Due to its practical importance in various domains, such as viral marketing, target advertisement, personalized recommendation, the problem has been studied in different variants, and different solution methodologies have been proposed over the years. Hence, there is a need for an organized and comprehensive review on this topic. This paper presents a survey on the progress in and around TSS Problem. At last, it discusses current research trends and future research directions as well.
Accurate time-series forecasting is vital for numerous areas of application such as transportation, energy, finance, economics, etc. However, while modern techniques are able to explore large sets of temporal data to build forecasting models, they typically neglect valuable information that is often available under the form of unstructured text. Although this data is in a radically different format, it often contains contextual explanations for many of the patterns that are observed in the temporal data. In this paper, we propose two deep learning architectures that leverage word embeddings, convolutional layers and attention mechanisms for combining text information with time-series data. We apply these approaches for the problem of taxi demand forecasting in event areas. Using publicly available taxi data from New York, we empirically show that by fusing these two complementary cross-modal sources of information, the proposed models are able to significantly reduce the error in the forecasts.
Recent work on adversarial attack has shown that Projected Gradient Descent (PGD) Adversary is a universal first-order adversary, and the classifier adversarially trained by PGD is robust against a wide range of first-order attacks. However, it is worth noting that the objective of an attacking/defense model relies on a data distribution, typically in the form of risk maximization/minimization: $\max\!/\!\min \mathbb{E}_{p(\mathbf{x})} \mathcal{L}(\mathbf{x})$, with $p(\mathbf{x})$ the data distribution and $\mathcal{L}(\cdot)$ a loss function. While PGD generates attack samples independently for each data point, the procedure does not necessary lead to good generalization in terms of risk maximization. In the paper, we achieve the goal by proposing distributionally adversarial attack (DAA), a framework to solve an optimal {\em adversarial data distribution}, a perturbed distribution that is close to the original data distribution but increases the generalization risk maximally. Algorithmically, DAA performs optimization on the space of probability measures, which introduces direct dependency between all data points when generating adversarial samples. DAA is evaluated by attacking state-of-the-art defense models, including the adversarially trained models provided by MadryLab. Notably, DAA outperforms all the attack algorithms listed in MadryLab’s white-box leaderboard, reducing the accuracy of their secret MNIST model to $88.79\%$ (with $l_\infty$ perturbations of $\epsilon = 0.3$) and the accuracy of their secret CIFAR model to $44.73\%$ (with $l_\infty$ perturbations of $\epsilon = 8.0$). Code for the experiments is released on https://…/Distributionally-Adversarial-Attack
Given a response $Y$ and a vector $X = (X^1, \dots, X^d)$ of $d$ predictors, we investigate the problem of inferring direct causes of $Y$ among the vector $X$. Models for $Y$ that use its causal covariates as predictors enjoy the property of being invariant across different environments or interventional settings. Given data from such environments, this property has been exploited for causal discovery: one collects the models that show predictive stability across all environments and outputs the set of predictors that are necessary to obtain stability. If some of the direct causes are latent, however, there may not exist invariant models for $Y$ based on variables from $X$, and the above reasoning breaks down. In this paper, we extend the principle of invariant prediction by introducing a relaxed version of the invariance assumption. This property can be used for causal discovery in the presence of latent variables if the latter’s influence on $Y$ can be restricted. More specifically, we allow for latent variables with a low-range discrete influence on the target $Y$. This assumption gives rise to switching regression models, where each value of the (unknown) hidden variable corresponds to a different regression coefficient. We provide sufficient conditions for the existence, consistency and asymptotic normality of the maximum likelihood estimator in switching regression models, and construct a test for the equality of such models. Our results on switching regression models allow us to prove that asymptotic false discovery control for the causal discovery method is obtained under mild conditions. We provide an algorithm for the overall method, make available code, and illustrate the performance of our method on simulated data.
The Linear Attention Recurrent Neural Network (LARNN) is a recurrent attention module derived from the Long Short-Term Memory (LSTM) cell and ideas from the consciousness Recurrent Neural Network (RNN). Yes, it LARNNs. The LARNN uses attention on its past cell state values for a limited window size $k$. The formulas are also derived from the Batch Normalized LSTM (BN-LSTM) cell and the Transformer Network for its Multi-Head Attention Mechanism. The Multi-Head Attention Mechanism is used inside the cell such that it can query its own $k$ past values with the attention window. This has the effect of augmenting the rank of the tensor with the attention mechanism, such that the cell can perform complex queries to question its previous inner memories, which should augment the long short-term effect of the memory. With a clever trick, the LARNN cell with attention can be easily used inside a loop on the cell state, just like how any other Recurrent Neural Network (RNN) cell can be looped linearly through time series. This is due to the fact that its state, which is looped upon throughout time steps within time series, stores the inner states in a ‘first in, first out’ queue which contains the $k$ most recent states and on which it is easily possible to add static positional encoding when the queue is represented as a tensor. This neural architecture yields better results than the vanilla LSTM cells. It can obtain results of 91.92% for the test accuracy, compared to the previously attained 91.65% using vanilla LSTM cells. Note that this is not to compare to other research, where up to 93.35% is obtained, but costly using 18 LSTM cells rather than with 2 to 3 cells as analyzed here. Finally, an interesting discovery is made, such that adding activation within the multi-head attention mechanism’s linear layers can yield better results in the context researched hereto.
Convolutional neural networks have gained a remarkable success in computer vision. However, most usable network architectures are hand-crafted and usually require expertise and elaborate design. In this paper, we provide a block-wise network generation pipeline called BlockQNN which automatically builds high-performance networks using the Q-Learning paradigm with epsilon-greedy exploration strategy. The optimal network block is constructed by the learning agent which is trained to choose component layers sequentially. We stack the block to construct the whole auto-generated network. To accelerate the generation process, we also propose a distributed asynchronous framework and an early stop strategy. The block-wise generation brings unique advantages: (1) it yields state-of-the-art results in comparison to the hand-crafted networks on image classification, particularly, the best network generated by BlockQNN achieves 2.35% top-1 error rate on CIFAR-10. (2) it offers tremendous reduction of the search space in designing networks, spending only 3 days with 32 GPUs. A faster version can yield a comparable result with only 1 GPU in 20 hours. (3) it has strong generalizability in that the network built on CIFAR also performs well on the larger-scale dataset. The best network achieves very competitive accuracy of 82.0% top-1 and 96.0% top-5 on ImageNet.
We present a new approach for learning graph embeddings, that relies on structural measures of node similarities for generation of training data. The model learns node embeddings that are able to approximate a given measure, such as the shortest path distance or any other. Evaluations of the proposed model on semantic similarity and word sense disambiguation tasks (using WordNet as the source of gold similarities) show that our method yields state-of-the-art results, but also is capable in certain cases to yield even better performance than the input similarity measure. The model is computationally efficient, orders of magnitude faster than the direct computation of graph distances.

# Distilled News

With the rise of IoT devices (Internet of Things), being able to analyze and visualize live streams of data is becoming more and more important. For example, you could have sensors like thermometers in machines or portable medical devices like pacemakers, continuously streaming data to a streaming service like Kafka. PixieDust makes it easier to work with live data inside Jupyter Notebooks by providing simple integration APIs to both the PixieApp and display() framework. On the visualization level, PixieDust uses Bokeh support for efficient data source update to plot streaming data into live charts (note that at the moment, only line chart and scatter plot are supported, but more will be added in the future). The display() framework also supports geospatial visualization of streaming data using the Mapbox rendering engine.
The European Parliament, Council and Commission created a regulation which toughens and combines data protection for people inside the EU – This regulation is called GDPR. It is a single set of rules which are created in order to govern as to how personal data is used. This is done regardless of the source and across all uses. GDPR protect the personal privacy laws pertaining to the data rights of the EU citizens. GDPR is not restricted to organizations inside the EU alone, on the contrary, any organization with customers in the EU will be affected. The way companies handle personal data will change forever with the introduction of GDPR. Europe´s data protection rules have undergone a huge change with GDPR being introduced. GDPR replaced the 1995 Data Protection Directive. The internet is growing at a rapid pace. Digital content has increased at an unimaginable rate. This has led to loads of personal data being held digitally. With so much personal data out there, the need for an enhanced data protection regulation arose and hence, GDPR. What GPR does is, it empowers individuals to gain access and control over the information held on them. While empowering individuals, GDPR also holds organizations accountable for the way thy handle and store personal data. Companies will be required to have the latest documentation and communication when it comes to data protection.
There are many different cloud APIs for computer vision on the market. In addition, this field is under rapid development. In the article, we made a brief overview of the various providers. At first sight, all of them provide fairly similar capabilities, yet some put an emphasis on face recognition like Kairos, or on building custom models like IBM and Azure. However, if you need to accomplish some very specific task, you still have to build the model using Deep Learning frameworks yourself.
A list of machine learning frameworks has come into the picture for the development and deployment of the AI apps. These frameworks will ditch the entire flow of development, testing, optimization, and final production. Developers are quite bewildered which framework to pick and which to ditch. Some of the frameworks would focus on the easy usability while others may put emphasis on the production deployment and parameter optimization. Every framework will have highs and lows of their own. They would have their own areas of excellence and downfalls making the choice for the developers even more difficult. The frameworks that make the top of the list of the best ones include MXNET, Keras, PyTorch, and Tensorflow.
The reason topological analysis is useful in this type of analytical challenge is that it provides a way of compressing complicated data sets into understandable and potentially actionable form. Here, as in many other data analytic problems, it is crucial to obtain an understanding of the ‘frequently occurring motifs’ within the data. The above observations suggest that topological analysis can be used to obtain control and understanding of the learning and generalization capabilities of CNN´s. There are many further ideas along these lines, which we will discuss in future posts.
An introduction to Project Hydrogen: how it can assist machine learning and AI frameworks on Apache Spark and what distinguishes it from other open source projects.
Every wanted to make R talk to you? Now you can, with the mscstts package by John Muschelli. It provides an interface to the Microsoft Cognitive Services Text-to-Speech API (hence the name) in Azure, and you can use it to convert any short piece of text to a playable audio file, rendering it as speech using a number of different voices.
Apache Drill 1.14.0 was recently released, bringing with it many new features and a temporary incompatibility with the current rev of the MapR ODBC drivers. The Drill community expects new ODBC drivers to arrive shortly. The sergeant?? is an alternative to ODBC for R users as it provides a dplyr interface to the REST API along with a JDBC interface and functions to work directly with the REST API in a more programmatic fashion.
A new release of Bio7 is available. The new Bio7 2.9 release comes with a plethora of new R features and bugfixes.
Linear programming is a technique to solve optimization problems whose constraints and outcome are represented by linear relationships.
There are many different R packages for dealing with spatial data. The main distinctions between them involve the types of data they work with – raster or vector – and the sophistication of the analyses they can do. Raster data can be thought of as pixels, similar to an image, while vector data consists of points, lines, or polygons. Spatial data manipulation can be quite complex, but creating some basic plots can be done with just a few commands. In this post, we will show simple examples of raster and vector spatial data for plotting a watershed and gage locations, and link to some other more complex examples.
Did you know that you can execute R and Python code remotely in SQL Server from Jupyter Notebooks or any IDE? Machine Learning Services in SQL Server eliminates the need to move data around. Instead of transferring large and sensitive data over the network or losing accuracy on ML training with sample csv files, you can have your R/Python code execute within your database. You can work in Jupyter Notebooks, RStudio, PyCharm, VSCode, Visual Studio, wherever you want, and then send function execution to SQL Server bringing intelligence to where your data lives. This tutorial will show you an example of how you can send your python code from Juptyter notebooks to execute within SQL Server. The same principles apply to R and any other IDE as well.