Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3846 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 10.6516. Min Range: -10.6516

Max Act: 10.6516. Min Act: -0.0000

Data Index: 305514 (C4 (Web Text))

Max Activating Token Index: 522

Text #1

Max Range: 10.6516. Min Range: -10.6516

Max Act: 9.9456. Min Act: -0.0000

Data Index: 1233561 (C4 (Web Text))

Max Activating Token Index: 595

Text #2

Max Range: 10.6516. Min Range: -10.6516

Max Act: 9.2146. Min Act: -0.0000

Data Index: 811719 (C4 (Web Text))

Max Activating Token Index: 602

Text #3

Max Range: 10.6516. Min Range: -10.6516

Max Act: 8.6824. Min Act: 0.0000

Data Index: 990723 (C4 (Web Text))

Max Activating Token Index: 448

Text #4

Max Range: 10.6516. Min Range: -10.6516

Max Act: 9.2064. Min Act: -0.0000

Data Index: 1202134 (C4 (Web Text))

Max Activating Token Index: 582

Text #5

Max Range: 10.6516. Min Range: -10.6516

Max Act: 8.4822. Min Act: -0.0000

Data Index: 622581 (C4 (Web Text))

Max Activating Token Index: 913

Text #6

Max Range: 10.6516. Min Range: -10.6516

Max Act: 8.8333. Min Act: 0.0000

Data Index: 130944 (C4 (Web Text))

Max Activating Token Index: 697

Text #7

Max Range: 10.6516. Min Range: -10.6516

Max Act: 8.2197. Min Act: -0.0000

Data Index: 1289667 (C4 (Web Text))

Max Activating Token Index: 133

Text #8

Max Range: 10.6516. Min Range: -10.6516

Max Act: 7.9798. Min Act: 0.0000

Data Index: 395547 (C4 (Web Text))

Max Activating Token Index: 271

Text #9

Max Range: 10.6516. Min Range: -10.6516

Max Act: 7.7483. Min Act: -0.0000

Data Index: 527133 (C4 (Web Text))

Max Activating Token Index: 84

Text #10

Max Range: 10.6516. Min Range: -10.6516

Max Act: 8.3483. Min Act: -0.0000

Data Index: 747101 (C4 (Web Text))

Max Activating Token Index: 397

Text #11

Max Range: 10.6516. Min Range: -10.6516

Max Act: 8.3975. Min Act: -0.0000

Data Index: 1368676 (Python Code)

Max Activating Token Index: 1010

Text #12

Max Range: 10.6516. Min Range: -10.6516

Max Act: 6.9745. Min Act: -0.0000

Data Index: 834762 (C4 (Web Text))

Max Activating Token Index: 920

Text #13

Max Range: 10.6516. Min Range: -10.6516

Max Act: 7.1546. Min Act: 0.0000

Data Index: 852650 (C4 (Web Text))

Max Activating Token Index: 732

Text #14

Max Range: 10.6516. Min Range: -10.6516

Max Act: 7.1398. Min Act: 0.0000

Data Index: 138256 (C4 (Web Text))

Max Activating Token Index: 832

Text #15

Max Range: 10.6516. Min Range: -10.6516

Max Act: 7.0972. Min Act: 0.0000

Data Index: 972012 (C4 (Web Text))

Max Activating Token Index: 129

Text #16

Max Range: 10.6516. Min Range: -10.6516

Max Act: 6.9221. Min Act: -0.0000

Data Index: 115120 (C4 (Web Text))

Max Activating Token Index: 393

Text #17

Max Range: 10.6516. Min Range: -10.6516

Max Act: 7.5228. Min Act: 0.0000

Data Index: 466671 (C4 (Web Text))

Max Activating Token Index: 113

Text #18

Max Range: 10.6516. Min Range: -10.6516

Max Act: 7.3148. Min Act: -0.0000

Data Index: 361254 (C4 (Web Text))

Max Activating Token Index: 133

Text #19

Max Range: 10.6516. Min Range: -10.6516

Max Act: 6.5839. Min Act: -0.0000

Data Index: 130252 (C4 (Web Text))

Max Activating Token Index: 574