Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2685 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1825. Min Act: -0.0000

Data Index: 673244 (C4 (Web Text))

Max Activating Token Index: 15

Text #1

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1602. Min Act: -0.0000

Data Index: 421858 (C4 (Web Text))

Max Activating Token Index: 320

Text #2

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1486. Min Act: -0.0000

Data Index: 216811 (C4 (Web Text))

Max Activating Token Index: 253

Text #3

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1318. Min Act: -0.0000

Data Index: 554865 (C4 (Web Text))

Max Activating Token Index: 168

Text #4

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1159. Min Act: -0.0000

Data Index: 1347939 (C4 (Web Text))

Max Activating Token Index: 168

Text #5

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1234. Min Act: -0.0000

Data Index: 573894 (C4 (Web Text))

Max Activating Token Index: 855

Text #6

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1261. Min Act: -0.0000

Data Index: 404390 (C4 (Web Text))

Max Activating Token Index: 405

Text #7

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1193. Min Act: -0.0000

Data Index: 873415 (C4 (Web Text))

Max Activating Token Index: 914

Text #8

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1233. Min Act: -0.0000

Data Index: 377334 (C4 (Web Text))

Max Activating Token Index: 177

Text #9

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1180. Min Act: -0.0000

Data Index: 1001860 (C4 (Web Text))

Max Activating Token Index: 73

Text #10

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1229. Min Act: -0.0000

Data Index: 548336 (C4 (Web Text))

Max Activating Token Index: 823

Text #11

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1158. Min Act: -0.0000

Data Index: 171182 (C4 (Web Text))

Max Activating Token Index: 19

Text #12

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1158. Min Act: -0.0000

Data Index: 448935 (C4 (Web Text))

Max Activating Token Index: 334

Text #13

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1132. Min Act: -0.0000

Data Index: 909 (C4 (Web Text))

Max Activating Token Index: 795

Text #14

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1100. Min Act: -0.0000

Data Index: 1308647 (C4 (Web Text))

Max Activating Token Index: 607

Text #15

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1098. Min Act: -0.0000

Data Index: 266697 (C4 (Web Text))

Max Activating Token Index: 207

Text #16

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1100. Min Act: -0.0000

Data Index: 1008386 (C4 (Web Text))

Max Activating Token Index: 900

Text #17

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1136. Min Act: -0.0000

Data Index: 455276 (C4 (Web Text))

Max Activating Token Index: 363

Text #18

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1053. Min Act: -0.0000

Data Index: 466613 (C4 (Web Text))

Max Activating Token Index: 147

Text #19

Max Range: 0.1825. Min Range: -0.1825

Max Act: 0.1119. Min Act: -0.0000

Data Index: 642068 (C4 (Web Text))

Max Activating Token Index: 945