Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

Model-free deep reinforcement learning methods have successfully learned complex behavioral strategies for a wide range of tasks, but typically require many samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that medium-sized neural network models can in fact be combined with model predictive control to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits to accomplish various complex locomotion tasks. We also propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We perform this pre-initialization by using rollouts from the trained model-based controller as supervision to pre-train a policy, and then fine-tune the policy using a model-free method. We empirically demonstrate that this resulting hybrid algorithm can drastically accelerate model-free learning and outperform purely model-free learners on several MuJoCo locomotion benchmark tasks, achieving sample efficiency gains over a purely model-free learner of 330x on swimmer, 26x on hopper, 4x on half-cheetah, and 3x on ant. Videos can be found at https://…/mbmf


Exponential Random Graph Models with Big Networks: Maximum Pseudolikelihood Estimation and the Parametric Bootstrap

With the growth of interest in network data across fields, the Exponential Random Graph Model (ERGM) has emerged as the leading approach to the statistical analysis of network data. ERGM parameter estimation requires the approximation of an intractable normalizing constant. Simulation methods represent the state-of-the-art approach to approximating the normalizing constant, leading to estimation by Monte Carlo maximum likelihood (MCMLE). MCMLE is accurate when a large sample of networks is used to approximate the normalizing constant. However, MCMLE is computationally expensive, and may be prohibitively so if the size of the network is on the order of 1,000 nodes (i.e., one million potential ties) or greater. When the network is large, one option is maximum pseudolikelihood estimation (MPLE). The standard MPLE is simple and fast, but generally underestimates standard errors. We show that a resampling method—the parametric bootstrap—results in accurate coverage probabilities for confidence intervals. We find that bootstrapped MPLE can be run in 1/5th the time of MCMLE. We study the relative performance of MCMLE and MPLE with simulation studies, and illustrate the two different approaches by applying them to a network of bills introduced in the United State Senate.


Multilayer Spectral Graph Clustering via Convex Layer Aggregation: Theory and Algorithms

Multilayer graphs are commonly used for representing different relations between entities and handling heterogeneous data processing tasks. Non-standard multilayer graph clustering methods are needed for assigning clusters to a common multilayer node set and for combining information from each layer. This paper presents a multilayer spectral graph clustering (SGC) framework that performs convex layer aggregation. Under a multilayer signal plus noise model, we provide a phase transition analysis of clustering reliability. Moreover, we use the phase transition criterion to propose a multilayer iterative model order selection algorithm (MIMOSA) for multilayer SGC, which features automated cluster assignment and layer weight adaptation, and provides statistical clustering reliability guarantees. Numerical simulations on synthetic multilayer graphs verify the phase transition analysis, and experiments on real-world multilayer graphs show that MIMOSA is competitive or better than other clustering methods.


Using JAGS for Bayesian Cognitive Diagnosis Models: A Tutorial

In this article, JAGS software was systematically introduced to fit common Bayesian cognitive diagnosis models (CDMs), such as the deterministic inputs, noisy ‘and’ gate model, the deterministic inputs, noisy ‘or’ gate model, the linear logistic model, and the log-linear CDM. The unstructured structural model and the higher-order structural model were both employed. We also showed how to extend those models to consider the testlet-effect. Finally, an empirical example was given as a tutorial to illustrate how to use our JAGS code in R.


Anomaly Detection in Multivariate Non-stationary Time Series for Automatic DBMS Diagnosis

Anomaly detection in database management systems (DBMSs) is difficult because of increasing number of statistics (stat) and event metrics in big data system. In this paper, I propose an automatic DBMS diagnosis system that detects anomaly periods with abnormal DB stat metrics and finds causal events in the periods. Reconstruction error from deep autoencoder and statistical process control approach are applied to detect time period with anomalies. Related events are found using time series similarity measures between events and abnormal stat metrics. After training deep autoencoder with DBMS metric data, efficacy of anomaly detection is investigated from other DBMSs containing anomalies. Experiment results show effectiveness of proposed model, especially, batch temporal normalization layer. Proposed model is used for publishing automatic DBMS diagnosis reports in order to determine DBMS configuration and SQL tuning.


TensorFlow Estimators: Managing Simplicity vs. Flexibility in High-Level Machine Learning Frameworks

We present a framework for specifying, training, evaluating, and deploying machine learning models. Our focus is on simplifying cutting edge machine learning for practitioners in order to bring such technologies into production. Recognizing the fast evolution of the field of deep learning, we make no attempt to capture the design space of all possible model architectures in a domain- specific language (DSL) or similar configuration language. We allow users to write code to define their models, but provide abstractions that guide develop- ers to write models in ways conducive to productionization. We also provide a unifying Estimator interface, making it possible to write downstream infrastructure (e.g. distributed training, hyperparameter tuning) independent of the model implementation. We balance the competing demands for flexibility and simplicity by offering APIs at different levels of abstraction, making common model architectures available out of the box, while providing a library of utilities designed to speed up experimentation with model architectures. To make out of the box models flexible and usable across a wide range of problems, these canned Estimators are parameterized not only over traditional hyperparameters, but also using feature columns, a declarative specification describing how to interpret input data. We discuss our experience in using this framework in re- search and production environments, and show the impact on code health, maintainability, and development speed.


A Review of Self-Exciting Spatio-Temporal Point Processes and Their Applications

Self-exciting spatio-temporal point process models predict the rate of events as a function of space, time, and the previous history of events. These models naturally capture triggering and clustering behavior, and have been widely used in fields where spatio-temporal clustering of events is observed, such as earthquake modeling, infectious disease, and crime. In the past several decades, advances have been made in estimation, inference, simulation, and diagnostic tools for self-exciting point process models. In this review, I describe the basic theory, survey related estimation and inference techniques from each field, highlight several key applications, and suggest directions for future research.


A discriminative view of MRF pre-processing algorithms

While Markov Random Fields (MRFs) are widely used in computer vision, they present a quite challenging inference problem. MRF inference can be accelerated by pre-processing techniques like Dead End Elimination (DEE) or QPBO-based approaches which compute the optimal labeling of a subset of variables. These techniques are guaranteed to never wrongly label a variable but they often leave a large number of variables unlabeled. We address this shortcoming by interpreting pre-processing as a classification problem, which allows us to trade off false positives (i.e., giving a variable an incorrect label) versus false negatives (i.e., failing to label a variable). We describe an efficient discriminative rule that finds optimal solutions for a subset of variables. Our technique provides both per-instance and worst-case guarantees concerning the quality of the solution. Empirical studies were conducted over several benchmark datasets. We obtain a speedup factor of 2 to 12 over expansion moves without preprocessing, and on difficult non-submodular energy functions produce slightly lower energy.


Generalized Entropy Agglomeration

Entropy Agglomeration (EA) is a hierarchical clustering algorithm introduced in 2013. Here, we generalize it to define Generalized Entropy Agglomeration (GEA) that can work with multiset blocks and blocks with rational occurrence numbers. We also introduce a numerical categorization procedure to apply GEA to numerical datasets. The software REBUS 2.0 is published with these capabilities: http://…/rebus2


Neural Vector Spaces for Unsupervised Information Retrieval

We propose the Neural Vector Space Model (NVSM), a method that learns representations of documents in an unsupervised manner for news article retrieval. In the NVSM paradigm, we learn low-dimensional representations of words and documents from scratch using gradient descent and rank documents according to their similarity with query representations that are composed from word representations. We show that NVSM performs better at document ranking than existing latent semantic vector space methods. The addition of NVSM to a mixture of lexical language models and a state-of-the-art baseline vector space model yields a statistically significant increase in retrieval effectiveness. Consequently, NVSM adds a complementary relevance signal. Next to semantic matching, we find that NVSM performs well in cases where lexical matching is needed. NVSM learns a notion of term specificity directly from the document collection without feature engineering. We also show that NVSM learns regularities related to Luhn significance. Finally, we give advice on how to deploy NVSM in situations where model selection (e.g., cross-validation) is infeasible. We find that an unsupervised ensemble of multiple models trained with different hyperparameter values performs better than a single cross-validated model. Therefore, NVSM can safely be used for ranking documents without supervised relevance judgments.


Recent Trends in Deep Learning Based Natural Language Processing

Deep learning methods employ multiple processing layers to learn hierarchical representations of data, and have produced state-of-the-art results in many domains. Recently, a variety of model designs and methods have blossomed in the context of natural language processing (NLP). In this paper, we review significant deep learning related models and methods that have been employed for numerous NLP tasks and provide a walk-through of their evolution. We also summarize, compare and contrast the various models and put forward a detailed understanding of the past, present and future of deep learning in NLP.


Latent Gaussian modeling and INLA: A review with focus on space-time applications

Bayesian hierarchical models with latent Gaussian layers have proven very flexible in capturing complex stochastic behavior and hierarchical structures in high-dimensional spatial and spatio-temporal data. Whereas simulation-based Bayesian inference through Markov Chain Monte Carlo may be hampered by slow convergence and numerical instabilities, the inferential framework of Integrated Nested Laplace Approximation (INLA) is capable to provide accurate and relatively fast analytical approximations to posterior quantities of interest. It heavily relies on the use of Gauss-Markov dependence structures to avoid the numerical bottleneck of high-dimensional nonsparse matrix computations. With a view towards space-time applications, we here review the principal theoretical concepts, model classes and inference tools within the INLA framework. Important elements to construct space-time models are certain spatial Mat\’ern-like Gauss-Markov random fields, obtained as approximate solutions to a stochastic partial differential equation. Efficient implementation of statistical inference tools for a large variety of models is available through the INLA package of the R software. To showcase the practical use of R-INLA and to illustrate its principal commands and syntax, a comprehensive simulation experiment is presented using simulated non Gaussian space-time count data with a first-order autoregressive dependence structure in time.


Structural Break Detection in High-Dimensional Non-Stationary VAR models

Assuming stationarity is unrealistic in many time series applications. A more realistic alternative is to allow for piecewise stationarity, where the model is allowed to change at given time points. In this article, the problem of detecting the change points in a high-dimensional piecewise vector autoregressive model (VAR) is considered. Reformulated the problem as a high-dimensional variable selection, a penalized least square estimation using total variation LASSO penalty is proposed for estimation of model parameters. It is shown that the developed method over-estimates the number of change points. A backward selection criterion is thus proposed in conjunction with the penalized least square estimator to tackle this issue. We prove that the proposed two-stage procedure consistently detects the number of change points and their locations. A block coordinate descent algorithm is developed for efficient computation of model parameters. The performance of the method is illustrated using several simulation scenarios.


Maximum Volume Inscribed Ellipsoid: A New Simplex-Structured Matrix Factorization Framework via Facet Enumeration and Convex Optimization

Consider a structured matrix factorization scenario where one factor is modeled to have columns lying in the unit simplex. Such a simplex-structured matrix factorization (SSMF) problem has spurred much interest in key topics such as hyperspectral unmixing in remote sensing and topic discovery in machine learning. In this paper we develop a new theoretical framework for SSMF. The idea is to study a maximum volume ellipsoid inscribed in the convex hull of the data points, which has not been attempted in prior literature. We show a sufficient condition under which this maximum volume inscribed ellipsoid (MVIE) framework can guarantee exact recovery of the factors. The condition derived is much better than that of separable non-negative matrix factorization (or pure-pixel search) and is comparable to that of another powerful framework called minimum volume enclosing simplex. From the MVIE framework we also develop an algorithm that uses facet enumeration and convex optimization to achieve the aforementioned recovery result. Numerical results are presented to demonstrate the potential of this new theoretical SSMF framework.


How Do People Differ? A Social Media Approach

Research from a variety of fields including psychology and linguistics have found correlations and patterns in personal attributes and behavior, but efforts to understand the broader heterogeneity in human behavior have not yet integrated these approaches and perspectives with a cohesive methodology. Here we extract patterns in behavior and relate those patterns together in a high-dimensional picture. We use dimension reduction to analyze word usage in text data from the online discussion platform Reddit. We find that pronouns can be used to characterize the space of the two most prominent dimensions that capture the greatest differences in word usage, even though pronouns were not included in the determination of those dimensions. These patterns overlap with patterns of topics of discussion to reveal relationships between pronouns and topics that can describe the user population. This analysis corroborates findings from past research that have identified word use differences across populations and synthesizes them relative to one another. We believe this is a step toward understanding how differences between people are related to each other.


An Error Detection and Correction Framework for Connectomics
Learning Feedforward and Recurrent Deterministic Spiking Neuron Network Feedback Controllers
Caterpillars Have Antimagic Orientations
A combinatorial method for connecting BHV spaces representing different numbers of taxa
Protecting Genomic Privacy by a Sequence-Similarity Based Obfuscation Method
Regenerative multi-type Galton-Watson processes
Distributed rank-1 dictionary learning: Towards fast and scalable solutions for fMRI big data analytics
Extractor-Based Time-Space Lower Bounds for Learning
Time-Space Tradeoffs for Learning from Small Test Spaces: Learning Low Degree Polynomial Functions
Embracing a new era of highly efficient and productive quantum Monte Carlo simulations
DM-PhyClus: A Bayesian phylogenetic algorithm for infectious disease transmission cluster inference
Which Encoding is the Best for Text Classification in Chinese, English, Japanese and Korean?
Gramian Tensor Decomposition via Semidefinite Programming
Learning Visual Importance for Graphic Designs and Data Visualizations
Gradient-enhanced kriging for high-dimensional problems
Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017)
Power packet transferability via symbol propagation matrix
Randomly coloring graphs of bounded treewidth
Generative Adversarial Network-based Synthesis of Visible Faces from Polarimetric Thermal Faces
Statistics of Deep Generated Images
On the number of proper paths between vertices in edge-colored hypercubes
Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
A kind of conditional connectivity of transposition networks generated by $k$-trees
Human Skin Detection Using RGB, HSV and YCbCr Color Models
What Actions are Needed for Understanding Human Actions in Videos?
Ellipsoidal Prediction Regions for Multivariate Uncertainty Characterization
Jackknife multiplier bootstrap: finite sample approximations to the $U$-process supremum with applications
Generalized Fréchet Bounds for Cell Entries in Multidimensional Contingency Tables
Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
Stochastic representation and pathwise properties of fractional Cox-Ingersoll-Ross process
Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition
Deep Face Feature for Face Alignment and Reconstruction
Universality in the fluctuation of eigenvalues of random circulant matrix
Sample-Optimal Identity Testing with High Probability
Simultaneous confidence sets for ranks using the partitioning principle – Technical report
Weakly- and Self-Supervised Learning for Content-Aware Deep Image Retargeting
Probabilistic Neural Network with Complex Exponential Activation Functions in Image Recognition using Deep Learning Framework
Joint Face Alignment and 3D Face Reconstruction with Application to Face Recognition
Gaussian Prototypical Networks for Few-Shot Learning on Omniglot
Demand-Independent Tolls
A Data Prism: Semi-Verified Learning in the Small-Alpha Regime
Minimum message length inference of the Poisson and geometric models using heavy-tailed prior distributions
An automatic water detection approach based on Dempster-Shafer theory for multi spectral images
Extreme clicking for efficient object annotation
Optimal control of a Vlasov-Poisson plasma by an external magnetic field – Analysis of a tracking type optimal control problem
Analysis of Analog Network Coding noise in Multiuser Cooperative Relaying for Spatially Correlated Environment
Isointense infant brain MRI segmentation with a dilated convolutional neural network
Fast Algorithm for Finding Maximum Distance with Space Subdivision in E2
Learning to Disambiguate by Asking Discriminative Questions
Intermittency of trawl processes
Ephemeral Context to Support Robust and Diverse Music Recommendations
Multi-dimensional Gated Recurrent Units for Automated Anatomical Landmark Localization
Space Subdivision to Speed-up Convex Hull Construction in E3
On Maximum Common Subgraph Problems in Series-Parallel Graphs
Syntactic aspects of hypergraph polytopes
Sequential testing for structural stability in approximate factor models
Non-Adaptive Randomized Algorithm for Group Testing
Time-dependent probability density functions and information geometry in stochastic logistic and Gompertz models
On the Whitney extension property for continuously differentiable horizontal curves in sub-Riemannian manifolds
Enhancing Cellular M2M Random Access with Binary Countdown Contention Resolution
BlitzNet: A Real-Time Deep Network for Scene Understanding
Finslerian Metrics in the Cone of Spectral Densities
Mutual Visibility by Robots with Persistent Memory
Anveshak – A Groundtruth Generation Tool for Foreground Regions of Document Images
A New Upper Bound for Cancellative Pairs
ExaGeoStat: A High Performance Unified Framework for Geostatistics on Manycore Systems
SPLODE: Semi-Probabilistic Point and Line Odometry with Depth Estimation from RGB-D Camera Motion
Decoupled Learning of Environment Characteristics for Safe Exploration
Speaker Diarization using Deep Recurrent Convolutional Neural Networks for Speaker Embeddings
Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanism
Measuring Inconsistency in Argument Graphs
Functional estimation and hypothesis testing in nonparametric boundary models
On Borwein’s conjectures for planar uniform random walks
Multi-Cell-Aware Opportunistic Random Access for Machine-Type Communications
WebVision Database: Visual Learning and Understanding from Web Data
CoupleNet: Coupling Global Structure with Local Parts for Object Detection
Simulated Annealing with Levy Distribution for Fast Matrix Factorization-Based Collaborative Filtering
Privacy Preserving Face Retrieval in the Cloud for Mobile Users
Multi-message Authentication over Noisy Channel with Secure Channel Codes
Area difference bounds for dissections of a square into an odd number of triangles
Interacting with Acoustic Simulation and Fabrication
An evaluation of large-scale methods for image instance and class discovery
Transitive Invariance for Self-supervised Visual Representation Learning
Implementing $\Diamond P$ with Bounded Messages on a Network of ADD Channels
Thresholding tests and confidence regions
TPC Together with Overlapped Time Domain Multiplexing System Based on Turbo Structure
KeyXtract Twitter Model – An Essential Keywords Extraction Model for Twitter Designed using NLP Tools
Spectral Learning of Restricted Boltzmann Machines
The Tensor Memory Hypothesis
SUBIC: A supervised, structured binary code for image search

Advertisements