Model: SoLU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1046 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0290. Min Act: -0.0003

Data Index: 866898 (C4 (Web Text))

Max Activating Token Index: 349

Text #1

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0261. Min Act: -0.0003

Data Index: 388504 (C4 (Web Text))

Max Activating Token Index: 525

Text #2

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0261. Min Act: -0.0003

Data Index: 394437 (C4 (Web Text))

Max Activating Token Index: 455

Text #3

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0256. Min Act: -0.0003

Data Index: 1192098 (C4 (Web Text))

Max Activating Token Index: 836

Text #4

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0248. Min Act: -0.0003

Data Index: 801810 (C4 (Web Text))

Max Activating Token Index: 424

Text #5

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0243. Min Act: -0.0003

Data Index: 1002952 (C4 (Web Text))

Max Activating Token Index: 167

Text #6

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0241. Min Act: -0.0003

Data Index: 658136 (C4 (Web Text))

Max Activating Token Index: 144

Text #7

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0241. Min Act: -0.0003

Data Index: 235580 (C4 (Web Text))

Max Activating Token Index: 428

Text #8

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0234. Min Act: -0.0003

Data Index: 693868 (C4 (Web Text))

Max Activating Token Index: 480

Text #9

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0235. Min Act: -0.0003

Data Index: 413176 (C4 (Web Text))

Max Activating Token Index: 24

Text #10

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0231. Min Act: -0.0003

Data Index: 128710 (C4 (Web Text))

Max Activating Token Index: 116

Text #11

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0228. Min Act: -0.0003

Data Index: 921243 (C4 (Web Text))

Max Activating Token Index: 370

Text #12

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0229. Min Act: -0.0003

Data Index: 624519 (C4 (Web Text))

Max Activating Token Index: 425

Text #13

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0232. Min Act: -0.0003

Data Index: 54691 (C4 (Web Text))

Max Activating Token Index: 282

Text #14

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0229. Min Act: -0.0003

Data Index: 1076208 (C4 (Web Text))

Max Activating Token Index: 289

Text #15

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0228. Min Act: -0.0003

Data Index: 22159 (C4 (Web Text))

Max Activating Token Index: 53

Text #16

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0229. Min Act: -0.0003

Data Index: 1170636 (C4 (Web Text))

Max Activating Token Index: 294

Text #17

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0230. Min Act: -0.0003

Data Index: 477074 (C4 (Web Text))

Max Activating Token Index: 248

Text #18

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0232. Min Act: -0.0003

Data Index: 58352 (C4 (Web Text))

Max Activating Token Index: 673

Text #19

Max Range: 0.0290. Min Range: -0.0290

Max Act: 0.0228. Min Act: -0.0003

Data Index: 1169580 (C4 (Web Text))

Max Activating Token Index: 712