Kaiwei Che ¹ Zhaokun Zhou ¹ Jun Niu ¹ Zhengyu Ma ² Wei Fang ³ Yanqi Chen ³ Shuaijie Shen ⁴ Li Yuan ^1* Yonghong Tian ^1*

• ¹ Shenzhen Graduate School, Peking University, Shenzhen, Guangdong, China
• ² Peng Cheng Laboratory, Shenzhen, Guangdong Province, China
• ³ Peking University, Beijing, Beijing Municipality, China
• ⁴ Southern University of Science and Technology, Shenzhen, Guangdong Province, China

The integration of self-attention mechanisms into Spiking Neural Networks (SNNs) has garnered considerable interest in the realm of advanced deep learning, primarily due to their biological properties. Recent advancements in SNN architecture, such as Spikformer, have demonstrated promising outcomes. However, we observe that Spikformer may exhibit excessive energy consumption, potentially attributable to redundant channels and blocks. To mitigate this issue, we propose a one-shot Spiking Transformer Architecture Search method, namely Auto-Spikformer. \textcolor{blue}{Auto-Spikformer optimizes the transformer architecture using a weight entanglement method and the SNN inner parameters using the proposed Discrete Spiking Parameters Search (DSPS) methods.} Benefiting from these methods, the performance of subnets with weights inherited from the supernet without even retraining is comparable to the original Spikformer. Moreover, we propose a new fitness function aiming to find a Pareto optimal combination balancing energy consumption and accuracy. Our experimental results demonstrate the effectiveness of Auto-Spikformer, which outperforms the original Spikformer and most CNN or ViT models with even fewer parameters and lower energy consumption.