Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3714 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 7.8599. Min Range: -7.8599

Max Act: 7.8599. Min Act: -0.0000

Data Index: 171207 (C4 (Web Text))

Max Activating Token Index: 34

Text #1

Max Range: 7.8599. Min Range: -7.8599

Max Act: 5.5938. Min Act: -0.0000

Data Index: 320358 (C4 (Web Text))

Max Activating Token Index: 19

Text #2

Max Range: 7.8599. Min Range: -7.8599

Max Act: 5.0255. Min Act: -0.0000

Data Index: 32052 (C4 (Web Text))

Max Activating Token Index: 608

Text #3

Max Range: 7.8599. Min Range: -7.8599

Max Act: 5.0960. Min Act: -0.0000

Data Index: 315617 (C4 (Web Text))

Max Activating Token Index: 404

Text #4

Max Range: 7.8599. Min Range: -7.8599

Max Act: 5.2425. Min Act: -0.0000

Data Index: 689850 (C4 (Web Text))

Max Activating Token Index: 683

Text #5

Max Range: 7.8599. Min Range: -7.8599

Max Act: 4.3343. Min Act: -0.0000

Data Index: 652343 (C4 (Web Text))

Max Activating Token Index: 327

Text #6

Max Range: 7.8599. Min Range: -7.8599

Max Act: 4.5086. Min Act: -0.0000

Data Index: 681345 (C4 (Web Text))

Max Activating Token Index: 348

Text #7

Max Range: 7.8599. Min Range: -7.8599

Max Act: 4.1420. Min Act: -0.0000

Data Index: 1489791 (Python Code)

Max Activating Token Index: 313

Text #8

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.7650. Min Act: -0.0000

Data Index: 1224716 (C4 (Web Text))

Max Activating Token Index: 942

Text #9

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.6856. Min Act: -0.0000

Data Index: 905415 (C4 (Web Text))

Max Activating Token Index: 351

Text #10

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.6295. Min Act: -0.0000

Data Index: 1573665 (Python Code)

Max Activating Token Index: 731

Text #11

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.3539. Min Act: -0.0000

Data Index: 714849 (C4 (Web Text))

Max Activating Token Index: 932

Text #12

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.5440. Min Act: -0.0000

Data Index: 953305 (C4 (Web Text))

Max Activating Token Index: 320

Text #13

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.5840. Min Act: -0.0000

Data Index: 21194 (C4 (Web Text))

Max Activating Token Index: 153

Text #14

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.2211. Min Act: -0.0000

Data Index: 581333 (C4 (Web Text))

Max Activating Token Index: 305

Text #15

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.3758. Min Act: -0.0000

Data Index: 1439122 (Python Code)

Max Activating Token Index: 543

Text #16

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.3345. Min Act: -0.0000

Data Index: 1392356 (Python Code)

Max Activating Token Index: 332

Text #17

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.5251. Min Act: -0.0000

Data Index: 1439366 (Python Code)

Max Activating Token Index: 660

Text #18

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.4983. Min Act: -0.0000

Data Index: 1264145 (C4 (Web Text))

Max Activating Token Index: 838

Text #19

Max Range: 7.8599. Min Range: -7.8599

Max Act: 3.1685. Min Act: -0.0000

Data Index: 1033143 (C4 (Web Text))

Max Activating Token Index: 472