Tunable Fine Learning Rate Controlled by Pulse Width Modulation

Tunable Fine Learning Rate Controlled by Pulse Width Modulation in Charge Trap Flash (CTF) for Synaptic Application Shalini Shrivastava, Udayan Ganguly Department of Electrical Engineering, Indian Institute of Technology Bombay, Mumbai, India Presented by: Shalini Shrivastava NVMW 2021 [email protected]

Outline Outline Part 1: Introduction Brain Inspired Learning Electronic Synapse in the literature Evolution of Flash Memory as Synapse Charge Trap Flash as synapse Part 2: Tunable Learning rate with Flash Synapse Pulse width Modulation Tunable Learning rate by CTF & its benchmarking Part 3: Variability & Scalability Effect of Variability on learning rate Scalability of Charge trap flash Conclusion

Brain Inspired Learning Synapse Neurons in the human brain synapses in the human brain The human brain is extremely efficient when compared to current generation supercomputers for certain cognitive tasks like learning and recognition Ref: Merolla, P. et al., Custom Integrated Circuits Conference (CICC), 2011 IEEE (pp. 1-4) [email protected] Long term Potentiation (LTP) Long term depression (LTD) 3 Spike Time Dependent Plasticity

SRAM STTRAM PCM RRAM ONW ST Tunneling gate Flash SST's ESF Stressed MOSFET FB ST Human Brain 0.5 1 10 120 20 4 4 8 75 20 8 8 - 1 2 50 10 Industry Emerging Emerging Emerging Low High Emerging Industry Industry - Gradual & Symmetric Learning level LTP/ LTD cycles Timing (ns) Area ( Energy (fJ) Synapse Technology Technology Maturity Literature For Various Electronic Synapse 1012 1012 1011 104 104 2 100 100 100 20 1000 No No No No Yes Yes NR NR NR 100 200 20 yes No Yes - The electronic synapse which is Industry based, has low energy and adequate level of learnings, is still missing [email protected] 4

Evolution of Flash Memory as Electronic Synapse 4 terminal – Carver Mead Flash Synapse Split Gate Flash Synapse F.M. Bayat, et al., 2015, ISCAS NOR based CTF Gu, X., et al., 2017, EDL. SiN based CTF Our work (SiN based CTF) Mechanism CHI/ Tunneling CHI/ Tunneling CHI/ Tunneling FN Tunneling FN Tunneling Energy (fJ) 2 2.5 Learning level 1000 100 200 100 - Learning rate 45-50 100 50-500 Fixed Tunable, 0.5-100 Application LTP/LTD/STDP LTD/LTP LTP/LTD/ STDP LTP/LTD/ MINST LTP/LTD/ STDP/SNNFisher Irish/RPU 1989-2008 2015 2017 Diorio. et al., IEEE TED,1972-1980. Agarwal, et al., 2019. JESSCDC. Structure 2019 2020 Evolution of Flash Memory for Neuromorphic Computing Highly mature & commercial technology High voltage application results in extra peripheral circuits [email protected] 5

Experimental Device/Concept 𝐕𝐠 time Read Charge trap flash transistor as synapse n Poly (V) (V) 12.5 V Pulse number Pulse number

Experimental Device/Concept 𝐕𝐠 𝐕𝐠 time n Poly (V) (V) Charge trap flash transistor as synapse 12.5 12.5 12.5 12.5 VVV V 12.0 12.0 VV 11.5 V - 13.5 V - 14.0 - 14.0 VV -14.5 14.5 - -14.5 VVV Pulse number Pulse number time

Relation between and G Working as synapse LTD LTP Change in results in conductance change G LTD G LTP LTP Increase in G post-neuron spikes because of pre- neuron LTD Decrease in G post-neuron spikes is not related to pre-neuron spike

Brain Inspired Learning: Requirement for Electronic Synapse Requirement for Synapse: Gradual and linear conductance change Low energy, small area 10 fJ Low variability CMOS Compatible Learning rate 2% Learning Level 1000 Less peripherals Synapse 1. 2. 3. 4. Neurons in the human brain synapses in the human brain Rajendran, B., et al. 2012, IEEE Transactions on Electron Devices Kim, H., et.al., 2017. Nanotechnology Aditya S, et al., IJCNN 2018 V. Bhatt, et al., IJCNN 2020. Long term Potentiation (LTP) Long term depression (LTD) [email protected] 9 Spike Time Dependent Plasticity

Tunable Learning rate with Flash Synapse Learning rate: The amount that the weights are updated during training is referred to as the step size or the “learning rate.” [email protected] () () 10

Experimental Results & Benchmarking [email protected] () () The tunable learning rate (0.2%-100%) from a single synaptic device is remarkable compare to the other devices in state of art. 11

Variability & Scalability Variability C2C/D2D for 10 cycles Variability {of shift per spike} : 10 cycles C2C : for C2C constant. D2D: limited for 8000 pulse number. Scalability The comparison of of scaled devices: Flash synapse has lower sensitivity to charge fluctuation even with scaled device. Charge Trap Flash, the highly manufaturable devices has lower sensitivity for variability and scalability, and tunable learning rate is the promising device as Synapse for Neuromorphic Application. [email protected] 12

Conclusions We demonstrate designable gradual conductance change on highly manufacturable CTF (Charge Trap Flash) by Fowler Nordheim (FN) tunneling. The tunable learning rate (0.2%-100%) from a single synaptic device is remarkable compare to the other devices in state of art. The C2C, in LTP/LTD is 0.05 and fixed. C2C variation does not limit the learning rate. The D2D – increases with decreases in pulse width, hence the tunability of learning rate is limited by D2D variability for 8000 learning level. The fundamental limits of charge fluctuation in CTF synapse has lower sensitivity even with scaling compare to the state-of-the-art. [email protected] 13

Requirement for Synapse: Gradual and linear conductance change Low energy, small area 10 fJ Low variability CMOS Compatible Learning rate 2% Learning Level 1000 Less peripherals Detail Work published: Shalini, Shrivastava, Chavan Tanmay, and Ganguly Udayan. "Ultra-low Energy charge trap flash based synapse enabled by parasitic leakage mitigation." arXiv preprint arXiv:1902.09417 (2019). Bhatt, V., Shrivastava, S., Chavan, T., & Ganguly, U. (2020, July). Software-Level Accuracy Using Stochastic Computing With ChargeTrap-Flash Based Weight Matrix. In 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8). IEEE. Thank you [email protected] 14