Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4447 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.6293. Min Range: -5.6293

Max Act: 5.6293. Min Act: -0.0000

Data Index: 1716042 (The Pile)

Max Activating Token Index: 354

Text #1

Max Range: 5.6293. Min Range: -5.6293

Max Act: 4.2750. Min Act: -0.0000

Data Index: 1009222 (The Pile)

Max Activating Token Index: 712

Text #2

Max Range: 5.6293. Min Range: -5.6293

Max Act: 4.1136. Min Act: -0.0000

Data Index: 1419853 (The Pile)

Max Activating Token Index: 298

Text #3

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.9810. Min Act: -0.0000

Data Index: 1761787 (The Pile)

Max Activating Token Index: 455

Text #4

Max Range: 5.6293. Min Range: -5.6293

Max Act: 4.0928. Min Act: -0.0000

Data Index: 1381612 (The Pile)

Max Activating Token Index: 656

Text #5

Max Range: 5.6293. Min Range: -5.6293

Max Act: 4.1580. Min Act: -0.0000

Data Index: 429621 (The Pile)

Max Activating Token Index: 781

Text #6

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.9321. Min Act: -0.0000

Data Index: 560870 (The Pile)

Max Activating Token Index: 459

Text #7

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.9213. Min Act: -0.0000

Data Index: 634736 (The Pile)

Max Activating Token Index: 390

Text #8

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.6578. Min Act: -0.0000

Data Index: 438512 (The Pile)

Max Activating Token Index: 792

Text #9

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.5451. Min Act: -0.0000

Data Index: 148808 (The Pile)

Max Activating Token Index: 119

Text #10

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.3935. Min Act: -0.0000

Data Index: 1552685 (The Pile)

Max Activating Token Index: 15

Text #11

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.4483. Min Act: -0.0000

Data Index: 53100 (The Pile)

Max Activating Token Index: 212

Text #12

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.3513. Min Act: -0.0000

Data Index: 1400086 (The Pile)

Max Activating Token Index: 430

Text #13

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.1514. Min Act: -0.0000

Data Index: 322663 (The Pile)

Max Activating Token Index: 530

Text #14

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.4232. Min Act: -0.0000

Data Index: 192352 (The Pile)

Max Activating Token Index: 61

Text #15

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.3998. Min Act: -0.0000

Data Index: 1777235 (The Pile)

Max Activating Token Index: 78

Text #16

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.2355. Min Act: -0.0000

Data Index: 1234729 (The Pile)

Max Activating Token Index: 900

Text #17

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.2452. Min Act: -0.0000

Data Index: 1115607 (The Pile)

Max Activating Token Index: 575

Text #18

Max Range: 5.6293. Min Range: -5.6293

Max Act: 2.9403. Min Act: -0.0000

Data Index: 1329159 (The Pile)

Max Activating Token Index: 970

Text #19

Max Range: 5.6293. Min Range: -5.6293

Max Act: 3.0848. Min Act: -0.0000

Data Index: 713170 (The Pile)

Max Activating Token Index: 130