Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1012 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1381. Min Range: -2.1381

Max Act: 2.1381. Min Act: -0.0000

Data Index: 864443 (C4 (Web Text))

Max Activating Token Index: 465

Text #1

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.5283. Min Act: -0.0000

Data Index: 851972 (C4 (Web Text))

Max Activating Token Index: 227

Text #2

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.5857. Min Act: -0.0000

Data Index: 513378 (C4 (Web Text))

Max Activating Token Index: 842

Text #3

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.5522. Min Act: -0.0000

Data Index: 1299323 (C4 (Web Text))

Max Activating Token Index: 318

Text #4

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.4431. Min Act: -0.0000

Data Index: 221534 (C4 (Web Text))

Max Activating Token Index: 629

Text #5

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.5151. Min Act: -0.0000

Data Index: 203915 (C4 (Web Text))

Max Activating Token Index: 306

Text #6

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.5934. Min Act: -0.0000

Data Index: 80745 (C4 (Web Text))

Max Activating Token Index: 353

Text #7

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.4958. Min Act: -0.0000

Data Index: 585184 (C4 (Web Text))

Max Activating Token Index: 264

Text #8

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.3024. Min Act: -0.0000

Data Index: 202801 (C4 (Web Text))

Max Activating Token Index: 332

Text #9

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.3766. Min Act: -0.0000

Data Index: 267127 (C4 (Web Text))

Max Activating Token Index: 392

Text #10

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.4408. Min Act: -0.0000

Data Index: 1245270 (C4 (Web Text))

Max Activating Token Index: 85

Text #11

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.4504. Min Act: -0.0000

Data Index: 694174 (C4 (Web Text))

Max Activating Token Index: 15

Text #12

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.5489. Min Act: -0.0000

Data Index: 628448 (C4 (Web Text))

Max Activating Token Index: 901

Text #13

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.3600. Min Act: -0.0000

Data Index: 891894 (C4 (Web Text))

Max Activating Token Index: 95

Text #14

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.5229. Min Act: -0.0000

Data Index: 1127950 (C4 (Web Text))

Max Activating Token Index: 495

Text #15

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.5149. Min Act: -0.0000

Data Index: 559851 (C4 (Web Text))

Max Activating Token Index: 823

Text #16

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.3490. Min Act: -0.0000

Data Index: 1016994 (C4 (Web Text))

Max Activating Token Index: 758

Text #17

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.4262. Min Act: -0.0000

Data Index: 887647 (C4 (Web Text))

Max Activating Token Index: 690

Text #18

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.4292. Min Act: -0.0000

Data Index: 952751 (C4 (Web Text))

Max Activating Token Index: 367

Text #19

Max Range: 2.1381. Min Range: -2.1381

Max Act: 1.4091. Min Act: -0.0000

Data Index: 475823 (C4 (Web Text))

Max Activating Token Index: 648