Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5518 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.7245. Min Act: -0.0000

Data Index: 146949 (C4 (Web Text))

Max Activating Token Index: 964

Text #1

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.4935. Min Act: 0.0001

Data Index: 850317 (C4 (Web Text))

Max Activating Token Index: 119

Text #2

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.5015. Min Act: -0.0000

Data Index: 1065140 (C4 (Web Text))

Max Activating Token Index: 470

Text #3

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.6511. Min Act: -0.0000

Data Index: 915094 (C4 (Web Text))

Max Activating Token Index: 781

Text #4

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.3027. Min Act: -0.0000

Data Index: 1160281 (C4 (Web Text))

Max Activating Token Index: 334

Text #5

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.2012. Min Act: -0.0000

Data Index: 1337269 (C4 (Web Text))

Max Activating Token Index: 112

Text #6

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.2328. Min Act: -0.0000

Data Index: 459556 (C4 (Web Text))

Max Activating Token Index: 796

Text #7

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.3138. Min Act: -0.0000

Data Index: 734755 (C4 (Web Text))

Max Activating Token Index: 849

Text #8

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.1013. Min Act: -0.0000

Data Index: 178769 (C4 (Web Text))

Max Activating Token Index: 663

Text #9

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.1115. Min Act: -0.0000

Data Index: 895248 (C4 (Web Text))

Max Activating Token Index: 678

Text #10

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.0453. Min Act: -0.0000

Data Index: 281166 (C4 (Web Text))

Max Activating Token Index: 384

Text #11

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.0581. Min Act: -0.0000

Data Index: 694567 (C4 (Web Text))

Max Activating Token Index: 148

Text #12

Max Range: 1.7245. Min Range: -1.7245

Max Act: 0.9836. Min Act: -0.0000

Data Index: 535058 (C4 (Web Text))

Max Activating Token Index: 636

Text #13

Max Range: 1.7245. Min Range: -1.7245

Max Act: 0.9937. Min Act: -0.0000

Data Index: 209804 (C4 (Web Text))

Max Activating Token Index: 414

Text #14

Max Range: 1.7245. Min Range: -1.7245

Max Act: 1.0445. Min Act: -0.0000

Data Index: 531071 (C4 (Web Text))

Max Activating Token Index: 658

Text #15

Max Range: 1.7245. Min Range: -1.7245

Max Act: 0.9206. Min Act: -0.0000

Data Index: 761573 (C4 (Web Text))

Max Activating Token Index: 230

Text #16

Max Range: 1.7245. Min Range: -1.7245

Max Act: 0.9162. Min Act: -0.0000

Data Index: 343033 (C4 (Web Text))

Max Activating Token Index: 149

Text #17

Max Range: 1.7245. Min Range: -1.7245

Max Act: 0.9827. Min Act: -0.0000

Data Index: 1307803 (C4 (Web Text))

Max Activating Token Index: 838

Text #18

Max Range: 1.7245. Min Range: -1.7245

Max Act: 0.9689. Min Act: -0.0000

Data Index: 920086 (C4 (Web Text))

Max Activating Token Index: 215

Text #19

Max Range: 1.7245. Min Range: -1.7245

Max Act: 0.9462. Min Act: -0.0000

Data Index: 823399 (C4 (Web Text))

Max Activating Token Index: 59