Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2092 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.3929. Min Act: -0.0000

Data Index: 224697 (C4 (Web Text))

Max Activating Token Index: 260

Text #1

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.2844. Min Act: -0.0000

Data Index: 968617 (C4 (Web Text))

Max Activating Token Index: 31

Text #2

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.2581. Min Act: -0.0000

Data Index: 914848 (C4 (Web Text))

Max Activating Token Index: 204

Text #3

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.2242. Min Act: -0.0000

Data Index: 906083 (C4 (Web Text))

Max Activating Token Index: 357

Text #4

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.1418. Min Act: -0.0000

Data Index: 1073275 (C4 (Web Text))

Max Activating Token Index: 118

Text #5

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.0222. Min Act: -0.0000

Data Index: 104514 (C4 (Web Text))

Max Activating Token Index: 98

Text #6

Max Range: 2.3929. Min Range: -2.3929

Max Act: 1.9916. Min Act: -0.0000

Data Index: 1000059 (C4 (Web Text))

Max Activating Token Index: 456

Text #7

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.0856. Min Act: -0.0000

Data Index: 1330453 (C4 (Web Text))

Max Activating Token Index: 58

Text #8

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.0905. Min Act: -0.0000

Data Index: 867184 (C4 (Web Text))

Max Activating Token Index: 20

Text #9

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.1909. Min Act: -0.0000

Data Index: 849812 (C4 (Web Text))

Max Activating Token Index: 60

Text #10

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.1828. Min Act: -0.0000

Data Index: 402874 (C4 (Web Text))

Max Activating Token Index: 1004

Text #11

Max Range: 2.3929. Min Range: -2.3929

Max Act: 1.8435. Min Act: -0.0000

Data Index: 1148555 (C4 (Web Text))

Max Activating Token Index: 204

Text #12

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.0317. Min Act: -0.0001

Data Index: 463682 (C4 (Web Text))

Max Activating Token Index: 115

Text #13

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.0329. Min Act: -0.0000

Data Index: 9834 (C4 (Web Text))

Max Activating Token Index: 526

Text #14

Max Range: 2.3929. Min Range: -2.3929

Max Act: 1.9119. Min Act: -0.0000

Data Index: 632697 (C4 (Web Text))

Max Activating Token Index: 144

Text #15

Max Range: 2.3929. Min Range: -2.3929

Max Act: 2.0106. Min Act: -0.0000

Data Index: 1346144 (C4 (Web Text))

Max Activating Token Index: 17

Text #16

Max Range: 2.3929. Min Range: -2.3929

Max Act: 1.9948. Min Act: -0.0000

Data Index: 842693 (C4 (Web Text))

Max Activating Token Index: 761

Text #17

Max Range: 2.3929. Min Range: -2.3929

Max Act: 1.8024. Min Act: -0.0000

Data Index: 138044 (C4 (Web Text))

Max Activating Token Index: 788

Text #18

Max Range: 2.3929. Min Range: -2.3929

Max Act: 1.9899. Min Act: -0.0000

Data Index: 910989 (C4 (Web Text))

Max Activating Token Index: 62

Text #19

Max Range: 2.3929. Min Range: -2.3929

Max Act: 1.9899. Min Act: -0.0000

Data Index: 567355 (C4 (Web Text))

Max Activating Token Index: 62