Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2120 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.4151. Min Act: -0.0000

Data Index: 1096660 (C4 (Web Text))

Max Activating Token Index: 806

Text #1

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.3999. Min Act: -0.0000

Data Index: 989334 (C4 (Web Text))

Max Activating Token Index: 743

Text #2

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.3208. Min Act: -0.0000

Data Index: 103094 (C4 (Web Text))

Max Activating Token Index: 85

Text #3

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.9574. Min Act: -0.0000

Data Index: 116520 (C4 (Web Text))

Max Activating Token Index: 886

Text #4

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.1060. Min Act: -0.0000

Data Index: 981779 (C4 (Web Text))

Max Activating Token Index: 983

Text #5

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.2073. Min Act: -0.0000

Data Index: 790814 (C4 (Web Text))

Max Activating Token Index: 77

Text #6

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.0453. Min Act: -0.0000

Data Index: 1305809 (C4 (Web Text))

Max Activating Token Index: 111

Text #7

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.1496. Min Act: -0.0000

Data Index: 1262733 (C4 (Web Text))

Max Activating Token Index: 844

Text #8

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.9852. Min Act: -0.0000

Data Index: 543476 (C4 (Web Text))

Max Activating Token Index: 884

Text #9

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.9750. Min Act: -0.0000

Data Index: 1059664 (C4 (Web Text))

Max Activating Token Index: 510

Text #10

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.3242. Min Act: -0.0000

Data Index: 53453 (C4 (Web Text))

Max Activating Token Index: 884

Text #11

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.8963. Min Act: -0.0000

Data Index: 240779 (C4 (Web Text))

Max Activating Token Index: 131

Text #12

Max Range: 3.4151. Min Range: -3.4151

Max Act: 3.1609. Min Act: -0.0000

Data Index: 141561 (C4 (Web Text))

Max Activating Token Index: 392

Text #13

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.8451. Min Act: -0.0000

Data Index: 1244724 (C4 (Web Text))

Max Activating Token Index: 666

Text #14

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.8069. Min Act: -0.0000

Data Index: 352248 (C4 (Web Text))

Max Activating Token Index: 692

Text #15

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.9793. Min Act: -0.0000

Data Index: 825723 (C4 (Web Text))

Max Activating Token Index: 533

Text #16

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.8400. Min Act: -0.0000

Data Index: 1255753 (C4 (Web Text))

Max Activating Token Index: 183

Text #17

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.8332. Min Act: -0.0000

Data Index: 556518 (C4 (Web Text))

Max Activating Token Index: 283

Text #18

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.9756. Min Act: -0.0000

Data Index: 551624 (C4 (Web Text))

Max Activating Token Index: 438

Text #19

Max Range: 3.4151. Min Range: -3.4151

Max Act: 2.5550. Min Act: -0.0000

Data Index: 240759 (C4 (Web Text))

Max Activating Token Index: 805