LEAKY RELU IN TEXT GRAPH CONVOLUTIONAL NETWORKS: SHORTER CPU TRAINING TIME WITH UNCHANGED CLASSIFICATION PERFORMANCE

Abstract

Text GCNs create a graph atop the entire corpus and assign classification to each document as a collaborative node classification process. Text GCNs work well on long document benchmarks, but the corpus graph is slow to train for computers without a graphics processing unit (GPU). Training time for a process is decreased if the rectified linear unit (ReLU) in the hidden layer of a Text GCN is replaced with a Leaky ReLU, but it is not known if the classification performance is also reduced. The Graph constructions, Architectures and Training schedules of Text GCNs were kept the same, and hidden layers were changed to Leaky ReLUs, while everything else was kept constant. The experiments were run on an 8th generation Intel Core i5 CPU with 16 GB of RAM, and the hidden layer activation was Leaky ReLU for all experiments. In general, Test accuracy for all corpora shows the Leaky ReLU was within a percentage point of the ReLU baseline, and macro-averaged F1 changed by no more than 1.8 points. Total training time for the 8 corpora decreased by about 20% overall, with 20 Newsgroups showing a 20% decrease, R52 a 26% decrease, and R8 a 32% decrease. Since the two activations cost almost the same per epoch, the most likely reason for the decrease is validation-based early stopping. All results come from single training runs, therefore the time saving should be noted as a practical observation that still needs repeated runs in order to confirm early stopping.