Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2343 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.9965. Min Act: -0.0000

Data Index: 258317 (C4 (Web Text))

Max Activating Token Index: 916

Text #1

Max Range: 3.0446. Min Range: -3.0446

Max Act: 3.0446. Min Act: -0.0000

Data Index: 905189 (C4 (Web Text))

Max Activating Token Index: 609

Text #2

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.9259. Min Act: -0.0000

Data Index: 1211376 (C4 (Web Text))

Max Activating Token Index: 934

Text #3

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.8645. Min Act: -0.0000

Data Index: 821909 (C4 (Web Text))

Max Activating Token Index: 601

Text #4

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.6524. Min Act: -0.0000

Data Index: 1074823 (C4 (Web Text))

Max Activating Token Index: 398

Text #5

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.5157. Min Act: -0.0000

Data Index: 724942 (C4 (Web Text))

Max Activating Token Index: 10

Text #6

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.4680. Min Act: -0.0000

Data Index: 70907 (C4 (Web Text))

Max Activating Token Index: 571

Text #7

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.3486. Min Act: -0.0000

Data Index: 1034948 (C4 (Web Text))

Max Activating Token Index: 270

Text #8

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.3107. Min Act: -0.0000

Data Index: 1593195 (Python Code)

Max Activating Token Index: 199

Text #9

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.1432. Min Act: -0.0000

Data Index: 810723 (C4 (Web Text))

Max Activating Token Index: 465

Text #10

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.1495. Min Act: -0.0000

Data Index: 768355 (C4 (Web Text))

Max Activating Token Index: 462

Text #11

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.1186. Min Act: -0.0000

Data Index: 580173 (C4 (Web Text))

Max Activating Token Index: 903

Text #12

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.1241. Min Act: -0.0000

Data Index: 124622 (C4 (Web Text))

Max Activating Token Index: 423

Text #13

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.0718. Min Act: -0.0000

Data Index: 261374 (C4 (Web Text))

Max Activating Token Index: 547

Text #14

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.0498. Min Act: -0.0000

Data Index: 1136145 (C4 (Web Text))

Max Activating Token Index: 599

Text #15

Max Range: 3.0446. Min Range: -3.0446

Max Act: 2.0148. Min Act: -0.0000

Data Index: 906554 (C4 (Web Text))

Max Activating Token Index: 776

Text #16

Max Range: 3.0446. Min Range: -3.0446

Max Act: 1.9573. Min Act: -0.0000

Data Index: 625223 (C4 (Web Text))

Max Activating Token Index: 791

Text #17

Max Range: 3.0446. Min Range: -3.0446

Max Act: 1.9479. Min Act: -0.0000

Data Index: 399599 (C4 (Web Text))

Max Activating Token Index: 606

Text #18

Max Range: 3.0446. Min Range: -3.0446

Max Act: 1.8466. Min Act: -0.0000

Data Index: 438737 (C4 (Web Text))

Max Activating Token Index: 816

Text #19

Max Range: 3.0446. Min Range: -3.0446

Max Act: 1.9284. Min Act: -0.0000

Data Index: 11458 (C4 (Web Text))

Max Activating Token Index: 464