Any one knows how this holds up on long context retrieval (needle in haystack , ruler) vs same size full attention model? efficiency gains look great but that usually where linear attention hybrids fall apart.
Any one knows how this holds up on long context retrieval (needle in haystack , ruler) vs same size full attention model? efficiency gains look great but that usually where linear attention hybrids fall apart.