Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3914 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.5133. Min Range: -3.5133

Max Act: 3.5133. Min Act: -0.0000

Data Index: 1604588 (Python Code)

Max Activating Token Index: 762

Text #1

Max Range: 3.5133. Min Range: -3.5133

Max Act: 3.0940. Min Act: -0.0000

Data Index: 1448406 (Python Code)

Max Activating Token Index: 1001

Text #2

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.8416. Min Act: -0.0000

Data Index: 1624758 (Python Code)

Max Activating Token Index: 1015

Text #3

Max Range: 3.5133. Min Range: -3.5133

Max Act: 3.1005. Min Act: -0.0000

Data Index: 1374930 (Python Code)

Max Activating Token Index: 1003

Text #4

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.7759. Min Act: -0.0000

Data Index: 1521410 (Python Code)

Max Activating Token Index: 167

Text #5

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.8518. Min Act: -0.0000

Data Index: 1513677 (Python Code)

Max Activating Token Index: 858

Text #6

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.9708. Min Act: -0.0000

Data Index: 1454950 (Python Code)

Max Activating Token Index: 515

Text #7

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.9713. Min Act: -0.0000

Data Index: 1605494 (Python Code)

Max Activating Token Index: 883

Text #8

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.6765. Min Act: -0.0000

Data Index: 1599046 (Python Code)

Max Activating Token Index: 716

Text #9

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.6855. Min Act: -0.0000

Data Index: 1479230 (Python Code)

Max Activating Token Index: 371

Text #10

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.7962. Min Act: -0.0000

Data Index: 1436130 (Python Code)

Max Activating Token Index: 131

Text #11

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.7887. Min Act: -0.0000

Data Index: 1642319 (Python Code)

Max Activating Token Index: 123

Text #12

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.6338. Min Act: -0.0000

Data Index: 1522350 (Python Code)

Max Activating Token Index: 296

Text #13

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.4242. Min Act: -0.0000

Data Index: 1550835 (Python Code)

Max Activating Token Index: 685

Text #14

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.5421. Min Act: -0.0000

Data Index: 1587875 (Python Code)

Max Activating Token Index: 567

Text #15

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.4893. Min Act: -0.0000

Data Index: 1549216 (Python Code)

Max Activating Token Index: 288

Text #16

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.3867. Min Act: -0.0000

Data Index: 1597413 (Python Code)

Max Activating Token Index: 1008

Text #17

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.5038. Min Act: -0.0000

Data Index: 1536646 (Python Code)

Max Activating Token Index: 317

Text #18

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.4723. Min Act: -0.0000

Data Index: 1421768 (Python Code)

Max Activating Token Index: 394

Text #19

Max Range: 3.5133. Min Range: -3.5133

Max Act: 2.4810. Min Act: -0.0000

Data Index: 1408034 (Python Code)

Max Activating Token Index: 560