Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3377 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.6465. Min Act: -0.0000

Data Index: 1572981 (Python Code)

Max Activating Token Index: 742

Text #1

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.6230. Min Act: -0.0000

Data Index: 1574383 (Python Code)

Max Activating Token Index: 450

Text #2

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.5895. Min Act: -0.0000

Data Index: 1453233 (Python Code)

Max Activating Token Index: 273

Text #3

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.4807. Min Act: -0.0000

Data Index: 1435092 (Python Code)

Max Activating Token Index: 243

Text #4

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.4543. Min Act: -0.0000

Data Index: 1384614 (Python Code)

Max Activating Token Index: 343

Text #5

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.5330. Min Act: -0.0000

Data Index: 1613014 (Python Code)

Max Activating Token Index: 37

Text #6

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.7631. Min Act: -0.0000

Data Index: 1634240 (Python Code)

Max Activating Token Index: 335

Text #7

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.6473. Min Act: -0.0000

Data Index: 1471762 (Python Code)

Max Activating Token Index: 452

Text #8

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.5065. Min Act: -0.0000

Data Index: 1364379 (Python Code)

Max Activating Token Index: 511

Text #9

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.2889. Min Act: -0.0000

Data Index: 1461888 (Python Code)

Max Activating Token Index: 602

Text #10

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.4131. Min Act: -0.0000

Data Index: 1586416 (Python Code)

Max Activating Token Index: 379

Text #11

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.4133. Min Act: -0.0000

Data Index: 1546542 (Python Code)

Max Activating Token Index: 817

Text #12

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.2551. Min Act: -0.0000

Data Index: 1623799 (Python Code)

Max Activating Token Index: 837

Text #13

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.3894. Min Act: -0.0000

Data Index: 1404486 (Python Code)

Max Activating Token Index: 243

Text #14

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.0049. Min Act: -0.0000

Data Index: 1514309 (Python Code)

Max Activating Token Index: 911

Text #15

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.3248. Min Act: -0.0000

Data Index: 1540349 (Python Code)

Max Activating Token Index: 593

Text #16

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.0974. Min Act: -0.0000

Data Index: 1651339 (Python Code)

Max Activating Token Index: 825

Text #17

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.3170. Min Act: -0.0000

Data Index: 1415937 (Python Code)

Max Activating Token Index: 452

Text #18

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.2074. Min Act: -0.0000

Data Index: 1465854 (Python Code)

Max Activating Token Index: 306

Text #19

Max Range: 2.7631. Min Range: -2.7631

Max Act: 2.1878. Min Act: -0.0000

Data Index: 1379788 (Python Code)

Max Activating Token Index: 689