Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4501 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.8209. Min Act: -0.0000

Data Index: 1367826 (Python Code)

Max Activating Token Index: 796

Text #1

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.8991. Min Act: -0.0000

Data Index: 1598658 (Python Code)

Max Activating Token Index: 847

Text #2

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.8037. Min Act: -0.0000

Data Index: 1509695 (Python Code)

Max Activating Token Index: 675

Text #3

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.7134. Min Act: -0.0000

Data Index: 1440939 (Python Code)

Max Activating Token Index: 386

Text #4

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.7055. Min Act: -0.0000

Data Index: 1470579 (Python Code)

Max Activating Token Index: 337

Text #5

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.5145. Min Act: -0.0000

Data Index: 1406780 (Python Code)

Max Activating Token Index: 60

Text #6

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.6242. Min Act: -0.0000

Data Index: 1397833 (Python Code)

Max Activating Token Index: 958

Text #7

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.4753. Min Act: -0.0000

Data Index: 1517366 (Python Code)

Max Activating Token Index: 72

Text #8

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.7004. Min Act: -0.0000

Data Index: 1484524 (Python Code)

Max Activating Token Index: 350

Text #9

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.4330. Min Act: -0.0000

Data Index: 1493559 (Python Code)

Max Activating Token Index: 197

Text #10

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.4269. Min Act: -0.0000

Data Index: 1635679 (Python Code)

Max Activating Token Index: 321

Text #11

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.2694. Min Act: -0.0000

Data Index: 1476208 (Python Code)

Max Activating Token Index: 970

Text #12

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.2938. Min Act: -0.0000

Data Index: 1535295 (Python Code)

Max Activating Token Index: 962

Text #13

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.4931. Min Act: -0.0000

Data Index: 1625783 (Python Code)

Max Activating Token Index: 768

Text #14

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.3392. Min Act: -0.0000

Data Index: 1655378 (Python Code)

Max Activating Token Index: 945

Text #15

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.5583. Min Act: -0.0000

Data Index: 1559353 (Python Code)

Max Activating Token Index: 219

Text #16

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.3305. Min Act: -0.0000

Data Index: 1412880 (Python Code)

Max Activating Token Index: 515

Text #17

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.3041. Min Act: -0.0000

Data Index: 1646264 (Python Code)

Max Activating Token Index: 119

Text #18

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.3092. Min Act: -0.0000

Data Index: 1481857 (Python Code)

Max Activating Token Index: 481

Text #19

Max Range: 2.8991. Min Range: -2.8991

Max Act: 2.1968. Min Act: -0.0000

Data Index: 1492676 (Python Code)

Max Activating Token Index: 313