Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4155 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8107. Min Range: -1.8107

Max Act: 1.8107. Min Act: -0.0000

Data Index: 1176124 (C4 (Web Text))

Max Activating Token Index: 462

Text #1

Max Range: 1.8107. Min Range: -1.8107

Max Act: 1.7787. Min Act: -0.0000

Data Index: 963013 (C4 (Web Text))

Max Activating Token Index: 355

Text #2

Max Range: 1.8107. Min Range: -1.8107

Max Act: 1.2227. Min Act: -0.0000

Data Index: 1234605 (C4 (Web Text))

Max Activating Token Index: 32

Text #3

Max Range: 1.8107. Min Range: -1.8107

Max Act: 1.5948. Min Act: -0.0000

Data Index: 1301499 (C4 (Web Text))

Max Activating Token Index: 442

Text #4

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.8874. Min Act: -0.0000

Data Index: 1611907 (Python Code)

Max Activating Token Index: 384

Text #5

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.8744. Min Act: -0.0000

Data Index: 498519 (C4 (Web Text))

Max Activating Token Index: 542

Text #6

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.6991. Min Act: -0.0000

Data Index: 680778 (C4 (Web Text))

Max Activating Token Index: 1001

Text #7

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.9352. Min Act: -0.0000

Data Index: 1108742 (C4 (Web Text))

Max Activating Token Index: 874

Text #8

Max Range: 1.8107. Min Range: -1.8107

Max Act: 1.0566. Min Act: -0.0000

Data Index: 1227043 (C4 (Web Text))

Max Activating Token Index: 508

Text #9

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.8055. Min Act: -0.0000

Data Index: 1233999 (C4 (Web Text))

Max Activating Token Index: 201

Text #10

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.9481. Min Act: -0.0000

Data Index: 878936 (C4 (Web Text))

Max Activating Token Index: 262

Text #11

Max Range: 1.8107. Min Range: -1.8107

Max Act: 1.1865. Min Act: -0.0000

Data Index: 235030 (C4 (Web Text))

Max Activating Token Index: 855

Text #12

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.8932. Min Act: -0.0000

Data Index: 795030 (C4 (Web Text))

Max Activating Token Index: 403

Text #13

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.8678. Min Act: -0.0000

Data Index: 254965 (C4 (Web Text))

Max Activating Token Index: 27

Text #14

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.8625. Min Act: -0.0000

Data Index: 1019528 (C4 (Web Text))

Max Activating Token Index: 708

Text #15

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.6993. Min Act: -0.0000

Data Index: 1318926 (C4 (Web Text))

Max Activating Token Index: 369

Text #16

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.8836. Min Act: -0.0000

Data Index: 45669 (C4 (Web Text))

Max Activating Token Index: 507

Text #17

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.8505. Min Act: -0.0000

Data Index: 634088 (C4 (Web Text))

Max Activating Token Index: 709

Text #18

Max Range: 1.8107. Min Range: -1.8107

Max Act: 0.7957. Min Act: -0.0000

Data Index: 1286407 (C4 (Web Text))

Max Activating Token Index: 756

Text #19

Max Range: 1.8107. Min Range: -1.8107

Max Act: 1.3237. Min Act: -0.0000

Data Index: 952575 (C4 (Web Text))

Max Activating Token Index: 164