Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1599 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.4971. Min Act: -0.0001

Data Index: 417103 (C4 (Web Text))

Max Activating Token Index: 310

Text #1

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.7747. Min Act: -0.0001

Data Index: 1076932 (C4 (Web Text))

Max Activating Token Index: 263

Text #2

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.6316. Min Act: -0.0000

Data Index: 1140483 (C4 (Web Text))

Max Activating Token Index: 223

Text #3

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.6138. Min Act: -0.0000

Data Index: 37644 (C4 (Web Text))

Max Activating Token Index: 735

Text #4

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.6320. Min Act: -0.0000

Data Index: 169567 (C4 (Web Text))

Max Activating Token Index: 134

Text #5

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.6631. Min Act: -0.0001

Data Index: 823292 (C4 (Web Text))

Max Activating Token Index: 369

Text #6

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.2014. Min Act: -0.0001

Data Index: 1024280 (C4 (Web Text))

Max Activating Token Index: 192

Text #7

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.6297. Min Act: -0.0001

Data Index: 1088578 (C4 (Web Text))

Max Activating Token Index: 185

Text #8

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.4272. Min Act: -0.0001

Data Index: 66542 (C4 (Web Text))

Max Activating Token Index: 309

Text #9

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.4612. Min Act: -0.0001

Data Index: 507040 (C4 (Web Text))

Max Activating Token Index: 317

Text #10

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.4629. Min Act: -0.0000

Data Index: 920816 (C4 (Web Text))

Max Activating Token Index: 290

Text #11

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.4760. Min Act: -0.0001

Data Index: 396788 (C4 (Web Text))

Max Activating Token Index: 295

Text #12

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.2515. Min Act: -0.0000

Data Index: 139924 (C4 (Web Text))

Max Activating Token Index: 80

Text #13

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.3955. Min Act: -0.0001

Data Index: 893206 (C4 (Web Text))

Max Activating Token Index: 106

Text #14

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.5127. Min Act: -0.0001

Data Index: 907701 (C4 (Web Text))

Max Activating Token Index: 121

Text #15

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.3859. Min Act: -0.0001

Data Index: 510626 (C4 (Web Text))

Max Activating Token Index: 233

Text #16

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.5077. Min Act: -0.0001

Data Index: 1355551 (C4 (Web Text))

Max Activating Token Index: 290

Text #17

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.3829. Min Act: -0.0001

Data Index: 340212 (C4 (Web Text))

Max Activating Token Index: 133

Text #18

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.3509. Min Act: -0.0000

Data Index: 286553 (C4 (Web Text))

Max Activating Token Index: 215

Text #19

Max Range: 3.7747. Min Range: -3.7747

Max Act: 3.6575. Min Act: -0.0001

Data Index: 885563 (C4 (Web Text))

Max Activating Token Index: 271