Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3846 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4105. Min Range: -2.4105

Max Act: 2.4105. Min Act: -0.0000

Data Index: 152077 (C4 (Web Text))

Max Activating Token Index: 767

Text #1

Max Range: 2.4105. Min Range: -2.4105

Max Act: 2.1589. Min Act: -0.0000

Data Index: 929156 (C4 (Web Text))

Max Activating Token Index: 85

Text #2

Max Range: 2.4105. Min Range: -2.4105

Max Act: 2.2194. Min Act: -0.0000

Data Index: 1265114 (C4 (Web Text))

Max Activating Token Index: 978

Text #3

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.7279. Min Act: -0.0000

Data Index: 249760 (C4 (Web Text))

Max Activating Token Index: 680

Text #4

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.7572. Min Act: -0.0000

Data Index: 90682 (C4 (Web Text))

Max Activating Token Index: 654

Text #5

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.6188. Min Act: -0.0000

Data Index: 891539 (C4 (Web Text))

Max Activating Token Index: 900

Text #6

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.6907. Min Act: -0.0000

Data Index: 562032 (C4 (Web Text))

Max Activating Token Index: 218

Text #7

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.4762. Min Act: -0.0000

Data Index: 840246 (C4 (Web Text))

Max Activating Token Index: 80

Text #8

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.5861. Min Act: -0.0000

Data Index: 760932 (C4 (Web Text))

Max Activating Token Index: 401

Text #9

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.5755. Min Act: -0.0000

Data Index: 670237 (C4 (Web Text))

Max Activating Token Index: 493

Text #10

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.5588. Min Act: -0.0000

Data Index: 12609 (C4 (Web Text))

Max Activating Token Index: 522

Text #11

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.5557. Min Act: -0.0000

Data Index: 984057 (C4 (Web Text))

Max Activating Token Index: 75

Text #12

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.4489. Min Act: -0.0000

Data Index: 372073 (C4 (Web Text))

Max Activating Token Index: 276

Text #13

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.4520. Min Act: -0.0000

Data Index: 941337 (C4 (Web Text))

Max Activating Token Index: 363

Text #14

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.4430. Min Act: -0.0000

Data Index: 547946 (C4 (Web Text))

Max Activating Token Index: 838

Text #15

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.5416. Min Act: -0.0000

Data Index: 510971 (C4 (Web Text))

Max Activating Token Index: 880

Text #16

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.4390. Min Act: -0.0000

Data Index: 1003870 (C4 (Web Text))

Max Activating Token Index: 382

Text #17

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.6128. Min Act: -0.0000

Data Index: 362036 (C4 (Web Text))

Max Activating Token Index: 954

Text #18

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.4337. Min Act: -0.0000

Data Index: 94001 (C4 (Web Text))

Max Activating Token Index: 504

Text #19

Max Range: 2.4105. Min Range: -2.4105

Max Act: 1.2974. Min Act: -0.0000

Data Index: 1345122 (C4 (Web Text))

Max Activating Token Index: 290