Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5434 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.1217. Min Range: -3.1217

Max Act: 3.1217. Min Act: -0.0000

Data Index: 1587 (C4 (Web Text))

Max Activating Token Index: 916

Text #1

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.8869. Min Act: -0.0000

Data Index: 394475 (C4 (Web Text))

Max Activating Token Index: 23

Text #2

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.4622. Min Act: -0.0000

Data Index: 1284171 (C4 (Web Text))

Max Activating Token Index: 41

Text #3

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.5853. Min Act: -0.0000

Data Index: 929739 (C4 (Web Text))

Max Activating Token Index: 785

Text #4

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.7018. Min Act: -0.0000

Data Index: 975013 (C4 (Web Text))

Max Activating Token Index: 741

Text #5

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.5169. Min Act: -0.0000

Data Index: 455930 (C4 (Web Text))

Max Activating Token Index: 601

Text #6

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.3104. Min Act: -0.0000

Data Index: 76423 (C4 (Web Text))

Max Activating Token Index: 236

Text #7

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.1710. Min Act: -0.0000

Data Index: 447056 (C4 (Web Text))

Max Activating Token Index: 592

Text #8

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.4098. Min Act: -0.0000

Data Index: 1174526 (C4 (Web Text))

Max Activating Token Index: 714

Text #9

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.2890. Min Act: 0.0000

Data Index: 957583 (C4 (Web Text))

Max Activating Token Index: 41

Text #10

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.1315. Min Act: -0.0000

Data Index: 954298 (C4 (Web Text))

Max Activating Token Index: 124

Text #11

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.2527. Min Act: -0.0000

Data Index: 284573 (C4 (Web Text))

Max Activating Token Index: 703

Text #12

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.2439. Min Act: -0.0000

Data Index: 613769 (C4 (Web Text))

Max Activating Token Index: 716

Text #13

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.2368. Min Act: -0.0000

Data Index: 909545 (C4 (Web Text))

Max Activating Token Index: 92

Text #14

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.2218. Min Act: -0.0000

Data Index: 909373 (C4 (Web Text))

Max Activating Token Index: 990

Text #15

Max Range: 3.1217. Min Range: -3.1217

Max Act: 1.9681. Min Act: -0.0000

Data Index: 1050278 (C4 (Web Text))

Max Activating Token Index: 973

Text #16

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.1762. Min Act: -0.0000

Data Index: 265176 (C4 (Web Text))

Max Activating Token Index: 977

Text #17

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.1623. Min Act: -0.0000

Data Index: 1103968 (C4 (Web Text))

Max Activating Token Index: 154

Text #18

Max Range: 3.1217. Min Range: -3.1217

Max Act: 1.9944. Min Act: 0.0000

Data Index: 235861 (C4 (Web Text))

Max Activating Token Index: 855

Text #19

Max Range: 3.1217. Min Range: -3.1217

Max Act: 2.1136. Min Act: -0.0000

Data Index: 192401 (C4 (Web Text))

Max Activating Token Index: 57