Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2055 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2802. Min Range: -2.2802

Max Act: 2.2802. Min Act: -0.0000

Data Index: 376613 (C4 (Web Text))

Max Activating Token Index: 59

Text #1

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.6734. Min Act: -0.0000

Data Index: 336764 (C4 (Web Text))

Max Activating Token Index: 54

Text #2

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.5355. Min Act: -0.0000

Data Index: 315215 (C4 (Web Text))

Max Activating Token Index: 491

Text #3

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.6016. Min Act: -0.0000

Data Index: 226793 (C4 (Web Text))

Max Activating Token Index: 631

Text #4

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.5293. Min Act: -0.0000

Data Index: 1048552 (C4 (Web Text))

Max Activating Token Index: 257

Text #5

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.6002. Min Act: -0.0000

Data Index: 145235 (C4 (Web Text))

Max Activating Token Index: 935

Text #6

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.5008. Min Act: -0.0000

Data Index: 832055 (C4 (Web Text))

Max Activating Token Index: 767

Text #7

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.4354. Min Act: -0.0000

Data Index: 431224 (C4 (Web Text))

Max Activating Token Index: 640

Text #8

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.3287. Min Act: -0.0000

Data Index: 375243 (C4 (Web Text))

Max Activating Token Index: 440

Text #9

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.4693. Min Act: -0.0000

Data Index: 18811 (C4 (Web Text))

Max Activating Token Index: 915

Text #10

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.2939. Min Act: -0.0000

Data Index: 391671 (C4 (Web Text))

Max Activating Token Index: 764

Text #11

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.3218. Min Act: -0.0000

Data Index: 1320905 (C4 (Web Text))

Max Activating Token Index: 542

Text #12

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.1097. Min Act: -0.0000

Data Index: 456391 (C4 (Web Text))

Max Activating Token Index: 646

Text #13

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.1577. Min Act: -0.0000

Data Index: 687842 (C4 (Web Text))

Max Activating Token Index: 515

Text #14

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.3145. Min Act: -0.0000

Data Index: 783916 (C4 (Web Text))

Max Activating Token Index: 985

Text #15

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.2959. Min Act: -0.0000

Data Index: 9236 (C4 (Web Text))

Max Activating Token Index: 509

Text #16

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.2760. Min Act: -0.0000

Data Index: 1175074 (C4 (Web Text))

Max Activating Token Index: 82

Text #17

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.2974. Min Act: -0.0000

Data Index: 653192 (C4 (Web Text))

Max Activating Token Index: 181

Text #18

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.2242. Min Act: -0.0000

Data Index: 1257826 (C4 (Web Text))

Max Activating Token Index: 375

Text #19

Max Range: 2.2802. Min Range: -2.2802

Max Act: 1.2250. Min Act: -0.0000

Data Index: 381459 (C4 (Web Text))

Max Activating Token Index: 57