Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 604 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.9483. Min Range: -5.9483

Max Act: 5.9483. Min Act: -0.0000

Data Index: 1915655 (The Pile)

Max Activating Token Index: 591

Text #1

Max Range: 5.9483. Min Range: -5.9483

Max Act: 5.8235. Min Act: -0.0000

Data Index: 816091 (The Pile)

Max Activating Token Index: 584

Text #2

Max Range: 5.9483. Min Range: -5.9483

Max Act: 4.9981. Min Act: -0.0000

Data Index: 1673535 (The Pile)

Max Activating Token Index: 1009

Text #3

Max Range: 5.9483. Min Range: -5.9483

Max Act: 4.3559. Min Act: -0.0000

Data Index: 348532 (The Pile)

Max Activating Token Index: 307

Text #4

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.9222. Min Act: -0.0000

Data Index: 1410860 (The Pile)

Max Activating Token Index: 819

Text #5

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.9780. Min Act: -0.0000

Data Index: 1930366 (The Pile)

Max Activating Token Index: 320

Text #6

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.7983. Min Act: 0.0000

Data Index: 873822 (The Pile)

Max Activating Token Index: 611

Text #7

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.7282. Min Act: -0.0000

Data Index: 780587 (The Pile)

Max Activating Token Index: 223

Text #8

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.8161. Min Act: -0.0000

Data Index: 259339 (The Pile)

Max Activating Token Index: 966

Text #9

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.6522. Min Act: -0.0000

Data Index: 1786804 (The Pile)

Max Activating Token Index: 421

Text #10

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.9260. Min Act: -0.0000

Data Index: 485183 (The Pile)

Max Activating Token Index: 948

Text #11

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.4296. Min Act: -0.0000

Data Index: 1959569 (The Pile)

Max Activating Token Index: 56

Text #12

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.3484. Min Act: -0.0000

Data Index: 1325001 (The Pile)

Max Activating Token Index: 564

Text #13

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.4573. Min Act: -0.0000

Data Index: 1747345 (The Pile)

Max Activating Token Index: 322

Text #14

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.4128. Min Act: -0.0000

Data Index: 895038 (The Pile)

Max Activating Token Index: 427

Text #15

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.4202. Min Act: -0.0000

Data Index: 957183 (The Pile)

Max Activating Token Index: 50

Text #16

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.5639. Min Act: -0.0000

Data Index: 200211 (The Pile)

Max Activating Token Index: 466

Text #17

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.5593. Min Act: -0.0000

Data Index: 788861 (The Pile)

Max Activating Token Index: 767

Text #18

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.5251. Min Act: -0.0000

Data Index: 616690 (The Pile)

Max Activating Token Index: 927

Text #19

Max Range: 5.9483. Min Range: -5.9483

Max Act: 3.4970. Min Act: -0.0000

Data Index: 1042104 (The Pile)

Max Activating Token Index: 93