Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 227 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.3414. Min Range: -3.3414

Max Act: 3.3414. Min Act: -0.0000

Data Index: 1228149 (The Pile)

Max Activating Token Index: 206

Text #1

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.8461. Min Act: -0.0000

Data Index: 1255514 (The Pile)

Max Activating Token Index: 564

Text #2

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.5707. Min Act: -0.0000

Data Index: 397658 (The Pile)

Max Activating Token Index: 495

Text #3

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.3503. Min Act: -0.0000

Data Index: 475854 (The Pile)

Max Activating Token Index: 101

Text #4

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.6056. Min Act: -0.0000

Data Index: 263590 (The Pile)

Max Activating Token Index: 552

Text #5

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.1757. Min Act: -0.0000

Data Index: 826356 (The Pile)

Max Activating Token Index: 563

Text #6

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.0195. Min Act: -0.0000

Data Index: 1938263 (The Pile)

Max Activating Token Index: 846

Text #7

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.3214. Min Act: -0.0000

Data Index: 972745 (The Pile)

Max Activating Token Index: 834

Text #8

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.3203. Min Act: -0.0000

Data Index: 252012 (The Pile)

Max Activating Token Index: 50

Text #9

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.3242. Min Act: -0.0000

Data Index: 692362 (The Pile)

Max Activating Token Index: 259

Text #10

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.0859. Min Act: -0.0000

Data Index: 755635 (The Pile)

Max Activating Token Index: 429

Text #11

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.1811. Min Act: -0.0000

Data Index: 429589 (The Pile)

Max Activating Token Index: 337

Text #12

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.1851. Min Act: -0.0000

Data Index: 1846939 (The Pile)

Max Activating Token Index: 229

Text #13

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.4141. Min Act: -0.0000

Data Index: 670719 (The Pile)

Max Activating Token Index: 162

Text #14

Max Range: 3.3414. Min Range: -3.3414

Max Act: 1.9257. Min Act: -0.0000

Data Index: 908301 (The Pile)

Max Activating Token Index: 783

Text #15

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.1577. Min Act: -0.0000

Data Index: 979196 (The Pile)

Max Activating Token Index: 431

Text #16

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.1312. Min Act: -0.0000

Data Index: 1852915 (The Pile)

Max Activating Token Index: 562

Text #17

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.1291. Min Act: -0.0000

Data Index: 847190 (The Pile)

Max Activating Token Index: 293

Text #18

Max Range: 3.3414. Min Range: -3.3414

Max Act: 1.9145. Min Act: -0.0000

Data Index: 871711 (The Pile)

Max Activating Token Index: 811

Text #19

Max Range: 3.3414. Min Range: -3.3414

Max Act: 2.2345. Min Act: -0.0000

Data Index: 687309 (The Pile)

Max Activating Token Index: 512