Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 106 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.7904. Min Act: -0.0000

Data Index: 1405907 (Python Code)

Max Activating Token Index: 252

Text #1

Max Range: 3.2927. Min Range: -3.2927

Max Act: 3.2927. Min Act: -0.0000

Data Index: 1540339 (Python Code)

Max Activating Token Index: 624

Text #2

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.8846. Min Act: -0.0000

Data Index: 1391456 (Python Code)

Max Activating Token Index: 606

Text #3

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.4745. Min Act: -0.0000

Data Index: 1202731 (C4 (Web Text))

Max Activating Token Index: 935

Text #4

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.3466. Min Act: -0.0000

Data Index: 365254 (C4 (Web Text))

Max Activating Token Index: 872

Text #5

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.3077. Min Act: -0.0000

Data Index: 118471 (C4 (Web Text))

Max Activating Token Index: 294

Text #6

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.3070. Min Act: -0.0000

Data Index: 58534 (C4 (Web Text))

Max Activating Token Index: 584

Text #7

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.1458. Min Act: -0.0000

Data Index: 298283 (C4 (Web Text))

Max Activating Token Index: 488

Text #8

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.2031. Min Act: -0.0000

Data Index: 712192 (C4 (Web Text))

Max Activating Token Index: 685

Text #9

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.2118. Min Act: -0.0000

Data Index: 839081 (C4 (Web Text))

Max Activating Token Index: 787

Text #10

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.2082. Min Act: -0.0000

Data Index: 1033195 (C4 (Web Text))

Max Activating Token Index: 453

Text #11

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.0786. Min Act: -0.0000

Data Index: 599245 (C4 (Web Text))

Max Activating Token Index: 573

Text #12

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.1573. Min Act: -0.0000

Data Index: 397396 (C4 (Web Text))

Max Activating Token Index: 471

Text #13

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.1645. Min Act: -0.0000

Data Index: 1091248 (C4 (Web Text))

Max Activating Token Index: 578

Text #14

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.1233. Min Act: -0.0000

Data Index: 636244 (C4 (Web Text))

Max Activating Token Index: 343

Text #15

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.0864. Min Act: -0.0000

Data Index: 516948 (C4 (Web Text))

Max Activating Token Index: 975

Text #16

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.2069. Min Act: -0.0000

Data Index: 1012388 (C4 (Web Text))

Max Activating Token Index: 446

Text #17

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.0030. Min Act: -0.0000

Data Index: 1436514 (Python Code)

Max Activating Token Index: 1000

Text #18

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.1576. Min Act: -0.0000

Data Index: 86791 (C4 (Web Text))

Max Activating Token Index: 189

Text #19

Max Range: 3.2927. Min Range: -3.2927

Max Act: 2.0475. Min Act: -0.0000

Data Index: 280216 (C4 (Web Text))

Max Activating Token Index: 434