Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3081 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1088. Min Range: -1.1088

Max Act: 1.0673. Min Act: 0.0000

Data Index: 1643861 (Python Code)

Max Activating Token Index: 767

Text #1

Max Range: 1.1088. Min Range: -1.1088

Max Act: 1.1088. Min Act: -0.0000

Data Index: 1373768 (Python Code)

Max Activating Token Index: 72

Text #2

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.9977. Min Act: -0.0000

Data Index: 1387601 (Python Code)

Max Activating Token Index: 142

Text #3

Max Range: 1.1088. Min Range: -1.1088

Max Act: 1.0403. Min Act: -0.0000

Data Index: 1471003 (Python Code)

Max Activating Token Index: 91

Text #4

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.9823. Min Act: -0.0000

Data Index: 1451899 (Python Code)

Max Activating Token Index: 514

Text #5

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.9630. Min Act: -0.0000

Data Index: 1629892 (Python Code)

Max Activating Token Index: 427

Text #6

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8622. Min Act: -0.0000

Data Index: 1561597 (Python Code)

Max Activating Token Index: 113

Text #7

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8903. Min Act: -0.0000

Data Index: 1626004 (Python Code)

Max Activating Token Index: 976

Text #8

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8828. Min Act: -0.0000

Data Index: 1572024 (Python Code)

Max Activating Token Index: 77

Text #9

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8712. Min Act: -0.0000

Data Index: 1295768 (C4 (Web Text))

Max Activating Token Index: 573

Text #10

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.9225. Min Act: -0.0000

Data Index: 1485060 (Python Code)

Max Activating Token Index: 268

Text #11

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.9155. Min Act: -0.0000

Data Index: 1530990 (Python Code)

Max Activating Token Index: 790

Text #12

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.9146. Min Act: -0.0000

Data Index: 1616688 (Python Code)

Max Activating Token Index: 193

Text #13

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8482. Min Act: -0.0000

Data Index: 1178599 (C4 (Web Text))

Max Activating Token Index: 998

Text #14

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8851. Min Act: -0.0000

Data Index: 9652 (C4 (Web Text))

Max Activating Token Index: 978

Text #15

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8118. Min Act: -0.0000

Data Index: 1607935 (Python Code)

Max Activating Token Index: 135

Text #16

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8918. Min Act: -0.0000

Data Index: 784702 (C4 (Web Text))

Max Activating Token Index: 383

Text #17

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.7833. Min Act: -0.0000

Data Index: 682810 (C4 (Web Text))

Max Activating Token Index: 336

Text #18

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8337. Min Act: -0.0000

Data Index: 1142675 (C4 (Web Text))

Max Activating Token Index: 550

Text #19

Max Range: 1.1088. Min Range: -1.1088

Max Act: 0.8349. Min Act: -0.0000

Data Index: 546587 (C4 (Web Text))

Max Activating Token Index: 540