Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1155 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5010. Min Range: -2.5010

Max Act: 2.5010. Min Act: -0.0000

Data Index: 1563375 (Python Code)

Max Activating Token Index: 383

Text #1

Max Range: 2.5010. Min Range: -2.5010

Max Act: 2.0933. Min Act: -0.0000

Data Index: 1436980 (Python Code)

Max Activating Token Index: 327

Text #2

Max Range: 2.5010. Min Range: -2.5010

Max Act: 2.1174. Min Act: -0.0000

Data Index: 1365419 (Python Code)

Max Activating Token Index: 486

Text #3

Max Range: 2.5010. Min Range: -2.5010

Max Act: 2.0773. Min Act: -0.0000

Data Index: 1645918 (Python Code)

Max Activating Token Index: 816

Text #4

Max Range: 2.5010. Min Range: -2.5010

Max Act: 2.0497. Min Act: -0.0000

Data Index: 1360236 (Python Code)

Max Activating Token Index: 367

Text #5

Max Range: 2.5010. Min Range: -2.5010

Max Act: 2.0177. Min Act: -0.0000

Data Index: 1542263 (Python Code)

Max Activating Token Index: 689

Text #6

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.9103. Min Act: -0.0000

Data Index: 1510168 (Python Code)

Max Activating Token Index: 329

Text #7

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.8996. Min Act: -0.0000

Data Index: 1652538 (Python Code)

Max Activating Token Index: 526

Text #8

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.9881. Min Act: -0.0000

Data Index: 1427639 (Python Code)

Max Activating Token Index: 866

Text #9

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.8532. Min Act: -0.0000

Data Index: 1469431 (Python Code)

Max Activating Token Index: 652

Text #10

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.9584. Min Act: -0.0000

Data Index: 1446870 (Python Code)

Max Activating Token Index: 198

Text #11

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.9501. Min Act: -0.0000

Data Index: 1507369 (Python Code)

Max Activating Token Index: 452

Text #12

Max Range: 2.5010. Min Range: -2.5010

Max Act: 2.0553. Min Act: -0.0000

Data Index: 1511418 (Python Code)

Max Activating Token Index: 222

Text #13

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.8259. Min Act: -0.0000

Data Index: 1443739 (Python Code)

Max Activating Token Index: 718

Text #14

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.8189. Min Act: -0.0000

Data Index: 1495335 (Python Code)

Max Activating Token Index: 559

Text #15

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.9152. Min Act: -0.0000

Data Index: 1439080 (Python Code)

Max Activating Token Index: 750

Text #16

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.9129. Min Act: -0.0000

Data Index: 1373781 (Python Code)

Max Activating Token Index: 348

Text #17

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.8022. Min Act: -0.0000

Data Index: 1629333 (Python Code)

Max Activating Token Index: 930

Text #18

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.8945. Min Act: -0.0000

Data Index: 1625148 (Python Code)

Max Activating Token Index: 45

Text #19

Max Range: 2.5010. Min Range: -2.5010

Max Act: 1.8933. Min Act: -0.0000

Data Index: 1558030 (Python Code)

Max Activating Token Index: 397