Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4269 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.8060. Min Act: -0.0000

Data Index: 72429 (The Pile)

Max Activating Token Index: 398

Text #1

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.6352. Min Act: -0.0000

Data Index: 1282158 (The Pile)

Max Activating Token Index: 412

Text #2

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.5345. Min Act: -0.0000

Data Index: 1953202 (The Pile)

Max Activating Token Index: 789

Text #3

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.5131. Min Act: -0.0000

Data Index: 147988 (The Pile)

Max Activating Token Index: 805

Text #4

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.4888. Min Act: -0.0000

Data Index: 232216 (The Pile)

Max Activating Token Index: 854

Text #5

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.4679. Min Act: -0.0000

Data Index: 1053173 (The Pile)

Max Activating Token Index: 330

Text #6

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.4743. Min Act: -0.0000

Data Index: 1023830 (The Pile)

Max Activating Token Index: 416

Text #7

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.3120. Min Act: -0.0000

Data Index: 1223126 (The Pile)

Max Activating Token Index: 622

Text #8

Max Range: 3.8060. Min Range: -3.8060

Max Act: 2.9858. Min Act: -0.0000

Data Index: 825353 (The Pile)

Max Activating Token Index: 737

Text #9

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.4442. Min Act: -0.0000

Data Index: 39126 (The Pile)

Max Activating Token Index: 758

Text #10

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.4246. Min Act: -0.0000

Data Index: 1869486 (The Pile)

Max Activating Token Index: 114

Text #11

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.4083. Min Act: -0.0000

Data Index: 144897 (The Pile)

Max Activating Token Index: 434

Text #12

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.2260. Min Act: -0.0000

Data Index: 888806 (The Pile)

Max Activating Token Index: 217

Text #13

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.2105. Min Act: -0.0000

Data Index: 1484949 (The Pile)

Max Activating Token Index: 759

Text #14

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.5041. Min Act: -0.0000

Data Index: 691188 (The Pile)

Max Activating Token Index: 572

Text #15

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.2112. Min Act: -0.0000

Data Index: 1505499 (The Pile)

Max Activating Token Index: 458

Text #16

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.3347. Min Act: -0.0000

Data Index: 810421 (The Pile)

Max Activating Token Index: 701

Text #17

Max Range: 3.8060. Min Range: -3.8060

Max Act: 2.9976. Min Act: -0.0000

Data Index: 740633 (The Pile)

Max Activating Token Index: 553

Text #18

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.5774. Min Act: -0.0000

Data Index: 1519630 (The Pile)

Max Activating Token Index: 844

Text #19

Max Range: 3.8060. Min Range: -3.8060

Max Act: 3.1387. Min Act: -0.0000

Data Index: 76634 (The Pile)

Max Activating Token Index: 651