These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.
Semi-Supervised Recognition under a Noisy and Fine-grained Dataset
Metrics concerned:Accuracy
The design choices in the Transformer attention mechanism, including weak
inductive bias and quadratic computational complexity, have limited its
application for modeling long sequences. In this paper, we introduce Mega, a
simple, theoretically grounded, single-head gated attention mechanism equipped
with (exponential) moving average to incorporate inductive bias of
position-aware local dependencies into the position-agnostic attention
mechanism. We further propose a variant of Mega that offers linear time and
space complexity yet yields only minimal quality loss, by efficiently splitting
the whole sequence into multiple chunks with fixed length. Extensive
experiments on a wide range of sequence modeling benchmarks, including the Long
Range Arena, neural machine translation, auto-regressive language modeling, and
image and speech classification, show that Mega achieves significant
improvements over other sequence models, including variants of Transformers and
recent state space models.
About the metric use case
You can click on the links to see the associated metric use cases
Objective(s):
Purpose(s):
Target sector(s):
