RIVA: Efficient relational inference with variate attention by Ruizi Wu instant download

RIVA: Efficient relational inference with variate attention by Ruizi Wu instant download

A B S T R A C TDataset link: systems are omnipresent across various domains, ranging from dynamic systems in physics to/RIVAintricate societal dynamics. Relational inference aims to uncover implicit interactions between componentsKeywords:based on observed system trajectories. Existing neural relational inference methods rely on fully connectedgraphs for message passing, leading to computational inefficiency and redundant message passing. WhileRelational inferenceTransformer-based models excel in multivariate time series forecasting, their contextual attention mechanismTransformerVariate attentiondisrupts the integrity and independence of dynamics in time-invariant relational inference. Besides, the implicitinteractions should have non-trivial correlations with the attention coefficients, yet how to read out anexplicit interaction graph from Transformer also still needs to be explored. In this study, we propose RIVA,a novel relational inference model with a variate attention mechanism. Unlike contextual attention in vanillaTransformer, which encodes multiple variables at the same time point as a single token, RIVA encodes entiredynamics. Furthermore, we incorporate the inferred graph structure as a mask in the causal attention, allowingeach variable to effectively aggregate features from its neighbors and greatly enhancing the model’s abilityto capture complex interactions. Through extensive experimental evaluations, RIVA demonstrates superiorperformance in time-invariant continuous interaction inference and future state prediction, outperformingexisting methods and providing highly accurate predictions in dynamic environments.