Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2098 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2662. Min Range: -3.2662

Max Act: 3.2662. Min Act: -0.0000

Data Index: 1259827 (C4 (Web Text))

Max Activating Token Index: 826

Text #1

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.9434. Min Act: -0.0000

Data Index: 689703 (C4 (Web Text))

Max Activating Token Index: 99

Text #2

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.8375. Min Act: -0.0000

Data Index: 241048 (C4 (Web Text))

Max Activating Token Index: 222

Text #3

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.6846. Min Act: -0.0000

Data Index: 1293540 (C4 (Web Text))

Max Activating Token Index: 93

Text #4

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.5269. Min Act: -0.0000

Data Index: 282917 (C4 (Web Text))

Max Activating Token Index: 160

Text #5

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.7792. Min Act: -0.0000

Data Index: 531087 (C4 (Web Text))

Max Activating Token Index: 183

Text #6

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.6230. Min Act: -0.0000

Data Index: 111584 (C4 (Web Text))

Max Activating Token Index: 986

Text #7

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.6742. Min Act: -0.0000

Data Index: 161053 (C4 (Web Text))

Max Activating Token Index: 336

Text #8

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.4053. Min Act: -0.0000

Data Index: 607119 (C4 (Web Text))

Max Activating Token Index: 278

Text #9

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.5144. Min Act: -0.0000

Data Index: 68538 (C4 (Web Text))

Max Activating Token Index: 527

Text #10

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.6320. Min Act: -0.0000

Data Index: 1253387 (C4 (Web Text))

Max Activating Token Index: 531

Text #11

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.4875. Min Act: -0.0000

Data Index: 712617 (C4 (Web Text))

Max Activating Token Index: 311

Text #12

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.4740. Min Act: -0.0000

Data Index: 159654 (C4 (Web Text))

Max Activating Token Index: 324

Text #13

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.4500. Min Act: -0.0000

Data Index: 43670 (C4 (Web Text))

Max Activating Token Index: 183

Text #14

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.4353. Min Act: -0.0000

Data Index: 1262609 (C4 (Web Text))

Max Activating Token Index: 378

Text #15

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.3119. Min Act: -0.0000

Data Index: 38651 (C4 (Web Text))

Max Activating Token Index: 858

Text #16

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.4293. Min Act: -0.0000

Data Index: 336202 (C4 (Web Text))

Max Activating Token Index: 27

Text #17

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.5567. Min Act: -0.0000

Data Index: 995510 (C4 (Web Text))

Max Activating Token Index: 621

Text #18

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.4000. Min Act: -0.0000

Data Index: 436695 (C4 (Web Text))

Max Activating Token Index: 656

Text #19

Max Range: 3.2662. Min Range: -3.2662

Max Act: 2.4109. Min Act: -0.0000

Data Index: 1306237 (C4 (Web Text))

Max Activating Token Index: 723