Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4518 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.9930. Min Act: -0.0000

Data Index: 1271282 (C4 (Web Text))

Max Activating Token Index: 671

Text #1

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.8493. Min Act: -0.0000

Data Index: 450299 (C4 (Web Text))

Max Activating Token Index: 907

Text #2

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.7689. Min Act: -0.0000

Data Index: 30897 (C4 (Web Text))

Max Activating Token Index: 546

Text #3

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.7904. Min Act: -0.0000

Data Index: 1442990 (Python Code)

Max Activating Token Index: 495

Text #4

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.7469. Min Act: -0.0000

Data Index: 405225 (C4 (Web Text))

Max Activating Token Index: 236

Text #5

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6911. Min Act: -0.0000

Data Index: 1444919 (Python Code)

Max Activating Token Index: 155

Text #6

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.7253. Min Act: -0.0000

Data Index: 288522 (C4 (Web Text))

Max Activating Token Index: 760

Text #7

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6976. Min Act: -0.0000

Data Index: 1588894 (Python Code)

Max Activating Token Index: 112

Text #8

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.7266. Min Act: -0.0000

Data Index: 773804 (C4 (Web Text))

Max Activating Token Index: 56

Text #9

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.7271. Min Act: -0.0000

Data Index: 836493 (C4 (Web Text))

Max Activating Token Index: 158

Text #10

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6883. Min Act: -0.0000

Data Index: 788916 (C4 (Web Text))

Max Activating Token Index: 545

Text #11

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6592. Min Act: -0.0000

Data Index: 558922 (C4 (Web Text))

Max Activating Token Index: 125

Text #12

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6960. Min Act: -0.0000

Data Index: 1653188 (Python Code)

Max Activating Token Index: 192

Text #13

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6705. Min Act: -0.0000

Data Index: 1543800 (Python Code)

Max Activating Token Index: 570

Text #14

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6414. Min Act: -0.0000

Data Index: 1035280 (C4 (Web Text))

Max Activating Token Index: 477

Text #15

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6811. Min Act: -0.0000

Data Index: 1504831 (Python Code)

Max Activating Token Index: 950

Text #16

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6768. Min Act: -0.0000

Data Index: 1531568 (Python Code)

Max Activating Token Index: 311

Text #17

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6276. Min Act: -0.0000

Data Index: 577818 (C4 (Web Text))

Max Activating Token Index: 859

Text #18

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6840. Min Act: -0.0000

Data Index: 1412200 (Python Code)

Max Activating Token Index: 89

Text #19

Max Range: 0.9930. Min Range: -0.9930

Max Act: 0.6864. Min Act: -0.0000

Data Index: 1343575 (C4 (Web Text))

Max Activating Token Index: 716