Leveraging tensor methods in neural architecture search for the automatic development of lightweight convolutional neural networks

By Mayur Dhanaraj, Huyen Do, Dinesh Nair, Cong Xu
2022
Download Copy BibTeX
Copy BibTeX
Most state-of-the-art Convolutional Neural Networks (CNNs) are bulky and cannot be deployed on resource-constrained edge devices. In order to leverage the exceptional generalizability of CNNs on edge-devices, they need to be made efficient in terms of memory usage, model size, and power consumption, while maintaining acceptable performance. Neural architecture search (NAS) is a recent approach for developing efficient, edge-deployable CNNs. On the other hand, CNNs used for classification, albeit developed using NAS, often contain large fully-connected (FC) layers with thousands of parameters, contributing to the bulkiness of CNNs. Recent works have shown that FC layers can be compressed, with minimal loss in performance, if any, using tensor processing methods. In this work, for the first time in literature, we leverage tensor methods in the NAS framework to discover efficient CNNs. Specifically, we employ tensor contraction layers (TCLs) to compress fully connected layers in the NAS framework and control the trade-off between compressibility and classification performance by handcrafting the ranks of TCLs. Additionally, we modify the NAS procedure to incorporate automatic TCL rank search in an end-to-end fashion, without human intervention. Our numerical studies on a wide variety of datasets including CIFAR-10, CIFAR-100, and Imagenette (a subset of ImageNet) demonstrate the superior performance of the proposed method in the automatic discovery of CNNs, whose model sizes are many-fold smaller than other cutting-edge mobile CNNs, while maintaining similar classification performance.

Latest news

IN, TS, Hyderabad
Welcome to the Worldwide Returns & ReCommerce team (WWR&R) at Amazon.com. WWR&R is an agile, innovative organization dedicated to ‘making zero happen’ to benefit our customers, our company, and the environment. Our goal is to achieve the three zeroes: zero cost of returns, zero waste, and zero defects. We do this by developing products and driving truly innovative operational excellence to help customers keep what they buy, recover returned and damaged product value, keep thousands of tons of waste from landfills, and create the best customer returns experience in the world. We have an eye to the future – we create long-term value at Amazon by focusing not just on the bottom line, but on the planet. We are building the most sustainableRead more