Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4684 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.9357. Min Range: -2.9357

Max Act: 2.9357. Min Act: -0.0000

Data Index: 611178 (C4 (Web Text))

Max Activating Token Index: 151

Text #1

Max Range: 2.9357. Min Range: -2.9357

Max Act: 2.1642. Min Act: -0.0000

Data Index: 938221 (C4 (Web Text))

Max Activating Token Index: 730

Text #2

Max Range: 2.9357. Min Range: -2.9357

Max Act: 2.2209. Min Act: -0.0000

Data Index: 640426 (C4 (Web Text))

Max Activating Token Index: 469

Text #3

Max Range: 2.9357. Min Range: -2.9357

Max Act: 2.1957. Min Act: -0.0000

Data Index: 34852 (C4 (Web Text))

Max Activating Token Index: 967

Text #4

Max Range: 2.9357. Min Range: -2.9357

Max Act: 2.2093. Min Act: -0.0000

Data Index: 465854 (C4 (Web Text))

Max Activating Token Index: 246

Text #5

Max Range: 2.9357. Min Range: -2.9357

Max Act: 2.1692. Min Act: -0.0000

Data Index: 289555 (C4 (Web Text))

Max Activating Token Index: 446

Text #6

Max Range: 2.9357. Min Range: -2.9357

Max Act: 2.1750. Min Act: -0.0000

Data Index: 174488 (C4 (Web Text))

Max Activating Token Index: 783

Text #7

Max Range: 2.9357. Min Range: -2.9357

Max Act: 2.0215. Min Act: -0.0000

Data Index: 364680 (C4 (Web Text))

Max Activating Token Index: 589

Text #8

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.9783. Min Act: -0.0000

Data Index: 346066 (C4 (Web Text))

Max Activating Token Index: 607

Text #9

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.8650. Min Act: -0.0000

Data Index: 988915 (C4 (Web Text))

Max Activating Token Index: 669

Text #10

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.9694. Min Act: -0.0000

Data Index: 270336 (C4 (Web Text))

Max Activating Token Index: 728

Text #11

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.9545. Min Act: -0.0000

Data Index: 651822 (C4 (Web Text))

Max Activating Token Index: 791

Text #12

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.9424. Min Act: -0.0000

Data Index: 1051104 (C4 (Web Text))

Max Activating Token Index: 763

Text #13

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.7804. Min Act: -0.0000

Data Index: 609274 (C4 (Web Text))

Max Activating Token Index: 922

Text #14

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.8580. Min Act: -0.0000

Data Index: 792931 (C4 (Web Text))

Max Activating Token Index: 206

Text #15

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.6490. Min Act: -0.0000

Data Index: 557230 (C4 (Web Text))

Max Activating Token Index: 58

Text #16

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.8107. Min Act: -0.0000

Data Index: 400767 (C4 (Web Text))

Max Activating Token Index: 83

Text #17

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.7916. Min Act: -0.0000

Data Index: 480633 (C4 (Web Text))

Max Activating Token Index: 611

Text #18

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.6741. Min Act: -0.0000

Data Index: 1312427 (C4 (Web Text))

Max Activating Token Index: 391

Text #19

Max Range: 2.9357. Min Range: -2.9357

Max Act: 1.6625. Min Act: -0.0000

Data Index: 550414 (C4 (Web Text))

Max Activating Token Index: 411