Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1061 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.7095. Min Act: -0.0000

Data Index: 1109177 (C4 (Web Text))

Max Activating Token Index: 161

Text #1

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.7062. Min Act: -0.0000

Data Index: 915718 (C4 (Web Text))

Max Activating Token Index: 782

Text #2

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.4434. Min Act: -0.0000

Data Index: 943846 (C4 (Web Text))

Max Activating Token Index: 145

Text #3

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.4112. Min Act: -0.0000

Data Index: 598701 (C4 (Web Text))

Max Activating Token Index: 709

Text #4

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.5239. Min Act: -0.0000

Data Index: 1251804 (C4 (Web Text))

Max Activating Token Index: 372

Text #5

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.4885. Min Act: -0.0000

Data Index: 415410 (C4 (Web Text))

Max Activating Token Index: 559

Text #6

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.4592. Min Act: -0.0000

Data Index: 1020479 (C4 (Web Text))

Max Activating Token Index: 38

Text #7

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.3297. Min Act: -0.0000

Data Index: 728849 (C4 (Web Text))

Max Activating Token Index: 524

Text #8

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.2645. Min Act: -0.0000

Data Index: 849576 (C4 (Web Text))

Max Activating Token Index: 760

Text #9

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.1136. Min Act: -0.0000

Data Index: 1304447 (C4 (Web Text))

Max Activating Token Index: 332

Text #10

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.1227. Min Act: -0.0000

Data Index: 382998 (C4 (Web Text))

Max Activating Token Index: 720

Text #11

Max Range: 2.7095. Min Range: -2.7095

Max Act: 1.9658. Min Act: -0.0000

Data Index: 1093714 (C4 (Web Text))

Max Activating Token Index: 545

Text #12

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.1836. Min Act: -0.0000

Data Index: 660755 (C4 (Web Text))

Max Activating Token Index: 519

Text #13

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.1832. Min Act: -0.0000

Data Index: 241843 (C4 (Web Text))

Max Activating Token Index: 907

Text #14

Max Range: 2.7095. Min Range: -2.7095

Max Act: 1.9470. Min Act: -0.0000

Data Index: 1050064 (C4 (Web Text))

Max Activating Token Index: 202

Text #15

Max Range: 2.7095. Min Range: -2.7095

Max Act: 1.9290. Min Act: -0.0000

Data Index: 929863 (C4 (Web Text))

Max Activating Token Index: 18

Text #16

Max Range: 2.7095. Min Range: -2.7095

Max Act: 2.0086. Min Act: -0.0000

Data Index: 1330817 (C4 (Web Text))

Max Activating Token Index: 975

Text #17

Max Range: 2.7095. Min Range: -2.7095

Max Act: 1.8779. Min Act: -0.0000

Data Index: 773122 (C4 (Web Text))

Max Activating Token Index: 36

Text #18

Max Range: 2.7095. Min Range: -2.7095

Max Act: 1.9649. Min Act: -0.0000

Data Index: 536162 (C4 (Web Text))

Max Activating Token Index: 23

Text #19

Max Range: 2.7095. Min Range: -2.7095

Max Act: 1.8346. Min Act: -0.0000

Data Index: 48364 (C4 (Web Text))

Max Activating Token Index: 977