Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 625 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.4676. Min Range: -1.4676

Max Act: 1.4676. Min Act: -0.0000

Data Index: 187120 (The Pile)

Max Activating Token Index: 298

Text #1

Max Range: 1.4676. Min Range: -1.4676

Max Act: 1.2697. Min Act: -0.0000

Data Index: 770470 (The Pile)

Max Activating Token Index: 31

Text #2

Max Range: 1.4676. Min Range: -1.4676

Max Act: 1.1701. Min Act: -0.0000

Data Index: 506054 (The Pile)

Max Activating Token Index: 562

Text #3

Max Range: 1.4676. Min Range: -1.4676

Max Act: 1.2553. Min Act: -0.0000

Data Index: 3935 (The Pile)

Max Activating Token Index: 547

Text #4

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.9404. Min Act: -0.0000

Data Index: 1598871 (The Pile)

Max Activating Token Index: 808

Text #5

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.9680. Min Act: -0.0000

Data Index: 1792654 (The Pile)

Max Activating Token Index: 414

Text #6

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.9846. Min Act: -0.0000

Data Index: 563137 (The Pile)

Max Activating Token Index: 90

Text #7

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.9166. Min Act: -0.0000

Data Index: 1723143 (The Pile)

Max Activating Token Index: 670

Text #8

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.9067. Min Act: -0.0000

Data Index: 1270623 (The Pile)

Max Activating Token Index: 374

Text #9

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.8461. Min Act: -0.0000

Data Index: 1262388 (The Pile)

Max Activating Token Index: 394

Text #10

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.7899. Min Act: -0.0000

Data Index: 1800516 (The Pile)

Max Activating Token Index: 947

Text #11

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.7949. Min Act: 0.0000

Data Index: 348133 (The Pile)

Max Activating Token Index: 264

Text #12

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.7680. Min Act: -0.0000

Data Index: 1586007 (The Pile)

Max Activating Token Index: 280

Text #13

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.7426. Min Act: -0.0000

Data Index: 1306109 (The Pile)

Max Activating Token Index: 302

Text #14

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.7457. Min Act: -0.0000

Data Index: 777374 (The Pile)

Max Activating Token Index: 163

Text #15

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.7462. Min Act: -0.0000

Data Index: 607404 (The Pile)

Max Activating Token Index: 834

Text #16

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.7057. Min Act: -0.0000

Data Index: 1946645 (The Pile)

Max Activating Token Index: 656

Text #17

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.6642. Min Act: -0.0000

Data Index: 994392 (The Pile)

Max Activating Token Index: 971

Text #18

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.6930. Min Act: 0.0000

Data Index: 1870197 (The Pile)

Max Activating Token Index: 694

Text #19

Max Range: 1.4676. Min Range: -1.4676

Max Act: 0.6993. Min Act: -0.0000

Data Index: 1241014 (The Pile)

Max Activating Token Index: 93