Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3070 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.8151. Min Act: -0.0001

Data Index: 817800 (C4 (Web Text))

Max Activating Token Index: 432

Text #1

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.8232. Min Act: -0.0001

Data Index: 1029837 (C4 (Web Text))

Max Activating Token Index: 76

Text #2

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.8235. Min Act: -0.0001

Data Index: 582424 (C4 (Web Text))

Max Activating Token Index: 100

Text #3

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6217. Min Act: -0.0001

Data Index: 273670 (C4 (Web Text))

Max Activating Token Index: 703

Text #4

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.7195. Min Act: -0.0001

Data Index: 484146 (C4 (Web Text))

Max Activating Token Index: 384

Text #5

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6918. Min Act: -0.0001

Data Index: 135836 (C4 (Web Text))

Max Activating Token Index: 40

Text #6

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6104. Min Act: -0.0001

Data Index: 385124 (C4 (Web Text))

Max Activating Token Index: 595

Text #7

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.7179. Min Act: -0.0001

Data Index: 1092651 (C4 (Web Text))

Max Activating Token Index: 164

Text #8

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.7241. Min Act: -0.0001

Data Index: 600884 (C4 (Web Text))

Max Activating Token Index: 115

Text #9

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6521. Min Act: -0.0001

Data Index: 98089 (C4 (Web Text))

Max Activating Token Index: 291

Text #10

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6727. Min Act: -0.0001

Data Index: 1079895 (C4 (Web Text))

Max Activating Token Index: 90

Text #11

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6666. Min Act: -0.0001

Data Index: 283601 (C4 (Web Text))

Max Activating Token Index: 215

Text #12

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6739. Min Act: -0.0001

Data Index: 866137 (C4 (Web Text))

Max Activating Token Index: 98

Text #13

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6946. Min Act: -0.0001

Data Index: 1258333 (C4 (Web Text))

Max Activating Token Index: 245

Text #14

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6577. Min Act: -0.0001

Data Index: 14721 (C4 (Web Text))

Max Activating Token Index: 106

Text #15

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6920. Min Act: -0.0001

Data Index: 738869 (C4 (Web Text))

Max Activating Token Index: 456

Text #16

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6479. Min Act: -0.0001

Data Index: 801439 (C4 (Web Text))

Max Activating Token Index: 59

Text #17

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.5830. Min Act: -0.0001

Data Index: 744520 (C4 (Web Text))

Max Activating Token Index: 211

Text #18

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6063. Min Act: -0.0001

Data Index: 544449 (C4 (Web Text))

Max Activating Token Index: 424

Text #19

Max Range: 0.8235. Min Range: -0.8235

Max Act: 0.6276. Min Act: -0.0001

Data Index: 1254687 (C4 (Web Text))

Max Activating Token Index: 858