Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1694 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.4883. Min Act: -0.0000

Data Index: 963258 (C4 (Web Text))

Max Activating Token Index: 727

Text #1

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.5224. Min Act: -0.0000

Data Index: 311131 (C4 (Web Text))

Max Activating Token Index: 810

Text #2

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.3651. Min Act: -0.0000

Data Index: 162558 (C4 (Web Text))

Max Activating Token Index: 495

Text #3

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.3730. Min Act: -0.0000

Data Index: 1162255 (C4 (Web Text))

Max Activating Token Index: 311

Text #4

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.2979. Min Act: -0.0000

Data Index: 262326 (C4 (Web Text))

Max Activating Token Index: 582

Text #5

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.2779. Min Act: -0.0000

Data Index: 1347405 (C4 (Web Text))

Max Activating Token Index: 788

Text #6

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.1172. Min Act: -0.0000

Data Index: 1296105 (C4 (Web Text))

Max Activating Token Index: 274

Text #7

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.2407. Min Act: -0.0000

Data Index: 647853 (C4 (Web Text))

Max Activating Token Index: 877

Text #8

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.1848. Min Act: -0.0000

Data Index: 371870 (C4 (Web Text))

Max Activating Token Index: 847

Text #9

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.2114. Min Act: -0.0000

Data Index: 476196 (C4 (Web Text))

Max Activating Token Index: 399

Text #10

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.0545. Min Act: -0.0000

Data Index: 692624 (C4 (Web Text))

Max Activating Token Index: 650

Text #11

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.0626. Min Act: -0.0000

Data Index: 327849 (C4 (Web Text))

Max Activating Token Index: 874

Text #12

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.1214. Min Act: -0.0000

Data Index: 140848 (C4 (Web Text))

Max Activating Token Index: 703

Text #13

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.0938. Min Act: -0.0000

Data Index: 761821 (C4 (Web Text))

Max Activating Token Index: 445

Text #14

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.0290. Min Act: -0.0000

Data Index: 1133422 (C4 (Web Text))

Max Activating Token Index: 372

Text #15

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.0513. Min Act: -0.0000

Data Index: 1250756 (C4 (Web Text))

Max Activating Token Index: 449

Text #16

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.0238. Min Act: -0.0000

Data Index: 1110437 (C4 (Web Text))

Max Activating Token Index: 463

Text #17

Max Range: 1.5224. Min Range: -1.5224

Max Act: 0.9991. Min Act: -0.0000

Data Index: 798792 (C4 (Web Text))

Max Activating Token Index: 977

Text #18

Max Range: 1.5224. Min Range: -1.5224

Max Act: 0.9885. Min Act: -0.0000

Data Index: 623301 (C4 (Web Text))

Max Activating Token Index: 807

Text #19

Max Range: 1.5224. Min Range: -1.5224

Max Act: 1.0099. Min Act: -0.0000

Data Index: 1011509 (C4 (Web Text))

Max Activating Token Index: 926