Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2246 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.9210. Min Act: -0.0002

Data Index: 441061 (C4 (Web Text))

Max Activating Token Index: 406

Text #1

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.9116. Min Act: -0.0002

Data Index: 700973 (C4 (Web Text))

Max Activating Token Index: 614

Text #2

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.9075. Min Act: -0.0002

Data Index: 671881 (C4 (Web Text))

Max Activating Token Index: 582

Text #3

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.8846. Min Act: -0.0002

Data Index: 1018638 (C4 (Web Text))

Max Activating Token Index: 58

Text #4

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.8518. Min Act: -0.0002

Data Index: 254450 (C4 (Web Text))

Max Activating Token Index: 883

Text #5

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.8543. Min Act: -0.0002

Data Index: 1029010 (C4 (Web Text))

Max Activating Token Index: 763

Text #6

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.8404. Min Act: -0.0002

Data Index: 1174703 (C4 (Web Text))

Max Activating Token Index: 304

Text #7

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.8431. Min Act: -0.0002

Data Index: 1235848 (C4 (Web Text))

Max Activating Token Index: 708

Text #8

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.8255. Min Act: -0.0002

Data Index: 125298 (C4 (Web Text))

Max Activating Token Index: 477

Text #9

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.8017. Min Act: -0.0003

Data Index: 1048689 (C4 (Web Text))

Max Activating Token Index: 509

Text #10

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.8015. Min Act: -0.0002

Data Index: 524436 (C4 (Web Text))

Max Activating Token Index: 711

Text #11

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7786. Min Act: -0.0002

Data Index: 888836 (C4 (Web Text))

Max Activating Token Index: 686

Text #12

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7577. Min Act: -0.0002

Data Index: 1105276 (C4 (Web Text))

Max Activating Token Index: 317

Text #13

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7524. Min Act: -0.0002

Data Index: 300195 (C4 (Web Text))

Max Activating Token Index: 582

Text #14

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7366. Min Act: -0.0003

Data Index: 1249436 (C4 (Web Text))

Max Activating Token Index: 694

Text #15

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7160. Min Act: -0.0002

Data Index: 947273 (C4 (Web Text))

Max Activating Token Index: 903

Text #16

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7372. Min Act: -0.0002

Data Index: 377724 (C4 (Web Text))

Max Activating Token Index: 270

Text #17

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7196. Min Act: -0.0002

Data Index: 1012353 (C4 (Web Text))

Max Activating Token Index: 689

Text #18

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7213. Min Act: -0.0002

Data Index: 228771 (C4 (Web Text))

Max Activating Token Index: 311

Text #19

Max Range: 0.9210. Min Range: -0.9210

Max Act: 0.7081. Min Act: -0.0002

Data Index: 385873 (C4 (Web Text))

Max Activating Token Index: 722