Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1487 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3835. Min Range: -2.3835

Max Act: 2.3835. Min Act: -0.0000

Data Index: 1119291 (C4 (Web Text))

Max Activating Token Index: 919

Text #1

Max Range: 2.3835. Min Range: -2.3835

Max Act: 2.2027. Min Act: -0.0000

Data Index: 1221450 (C4 (Web Text))

Max Activating Token Index: 84

Text #2

Max Range: 2.3835. Min Range: -2.3835

Max Act: 2.1619. Min Act: -0.0000

Data Index: 83992 (C4 (Web Text))

Max Activating Token Index: 554

Text #3

Max Range: 2.3835. Min Range: -2.3835

Max Act: 2.0105. Min Act: -0.0000

Data Index: 1005603 (C4 (Web Text))

Max Activating Token Index: 183

Text #4

Max Range: 2.3835. Min Range: -2.3835

Max Act: 2.0515. Min Act: -0.0000

Data Index: 809429 (C4 (Web Text))

Max Activating Token Index: 567

Text #5

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.9940. Min Act: -0.0000

Data Index: 971725 (C4 (Web Text))

Max Activating Token Index: 260

Text #6

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.9718. Min Act: -0.0000

Data Index: 985971 (C4 (Web Text))

Max Activating Token Index: 125

Text #7

Max Range: 2.3835. Min Range: -2.3835

Max Act: 2.0663. Min Act: -0.0000

Data Index: 611188 (C4 (Web Text))

Max Activating Token Index: 914

Text #8

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.8452. Min Act: -0.0000

Data Index: 205849 (C4 (Web Text))

Max Activating Token Index: 644

Text #9

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.9360. Min Act: -0.0000

Data Index: 913859 (C4 (Web Text))

Max Activating Token Index: 177

Text #10

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.9134. Min Act: -0.0000

Data Index: 1125953 (C4 (Web Text))

Max Activating Token Index: 490

Text #11

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.7093. Min Act: -0.0000

Data Index: 796366 (C4 (Web Text))

Max Activating Token Index: 727

Text #12

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.8616. Min Act: -0.0000

Data Index: 258036 (C4 (Web Text))

Max Activating Token Index: 787

Text #13

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.7827. Min Act: -0.0000

Data Index: 385671 (C4 (Web Text))

Max Activating Token Index: 272

Text #14

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.6752. Min Act: -0.0000

Data Index: 1135450 (C4 (Web Text))

Max Activating Token Index: 174

Text #15

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.9582. Min Act: -0.0000

Data Index: 555910 (C4 (Web Text))

Max Activating Token Index: 888

Text #16

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.8529. Min Act: -0.0000

Data Index: 1355757 (C4 (Web Text))

Max Activating Token Index: 114

Text #17

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.6093. Min Act: -0.0000

Data Index: 1162227 (C4 (Web Text))

Max Activating Token Index: 263

Text #18

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.6978. Min Act: -0.0000

Data Index: 104880 (C4 (Web Text))

Max Activating Token Index: 81

Text #19

Max Range: 2.3835. Min Range: -2.3835

Max Act: 1.6947. Min Act: -0.0000

Data Index: 938167 (C4 (Web Text))

Max Activating Token Index: 790