Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 166 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.7067. Min Range: -3.7067

Max Act: 3.7067. Min Act: -0.0000

Data Index: 1472748 (The Pile)

Max Activating Token Index: 68

Text #1

Max Range: 3.7067. Min Range: -3.7067

Max Act: 3.3121. Min Act: -0.0000

Data Index: 858458 (The Pile)

Max Activating Token Index: 560

Text #2

Max Range: 3.7067. Min Range: -3.7067

Max Act: 3.0499. Min Act: -0.0000

Data Index: 1575183 (The Pile)

Max Activating Token Index: 123

Text #3

Max Range: 3.7067. Min Range: -3.7067

Max Act: 3.0188. Min Act: -0.0000

Data Index: 1325383 (The Pile)

Max Activating Token Index: 831

Text #4

Max Range: 3.7067. Min Range: -3.7067

Max Act: 3.1170. Min Act: -0.0000

Data Index: 1285761 (The Pile)

Max Activating Token Index: 295

Text #5

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.9867. Min Act: -0.0000

Data Index: 571529 (The Pile)

Max Activating Token Index: 1022

Text #6

Max Range: 3.7067. Min Range: -3.7067

Max Act: 3.0488. Min Act: -0.0000

Data Index: 466420 (The Pile)

Max Activating Token Index: 418

Text #7

Max Range: 3.7067. Min Range: -3.7067

Max Act: 3.0133. Min Act: -0.0000

Data Index: 289945 (The Pile)

Max Activating Token Index: 391

Text #8

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.7848. Min Act: -0.0000

Data Index: 1182237 (The Pile)

Max Activating Token Index: 848

Text #9

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.8374. Min Act: -0.0000

Data Index: 470724 (The Pile)

Max Activating Token Index: 387

Text #10

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.8446. Min Act: -0.0000

Data Index: 1302929 (The Pile)

Max Activating Token Index: 289

Text #11

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.7049. Min Act: -0.0000

Data Index: 974699 (The Pile)

Max Activating Token Index: 861

Text #12

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.8345. Min Act: -0.0000

Data Index: 1607653 (The Pile)

Max Activating Token Index: 880

Text #13

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.7953. Min Act: -0.0000

Data Index: 1368031 (The Pile)

Max Activating Token Index: 1019

Text #14

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.7326. Min Act: -0.0000

Data Index: 474372 (The Pile)

Max Activating Token Index: 905

Text #15

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.4845. Min Act: -0.0000

Data Index: 1892513 (The Pile)

Max Activating Token Index: 257

Text #16

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.4373. Min Act: -0.0000

Data Index: 1067117 (The Pile)

Max Activating Token Index: 538

Text #17

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.5649. Min Act: -0.0000

Data Index: 1824361 (The Pile)

Max Activating Token Index: 176

Text #18

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.6871. Min Act: -0.0000

Data Index: 1035988 (The Pile)

Max Activating Token Index: 888

Text #19

Max Range: 3.7067. Min Range: -3.7067

Max Act: 2.3096. Min Act: -0.0000

Data Index: 465078 (The Pile)

Max Activating Token Index: 968