Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 557 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 8.6989. Min Range: -8.6989

Max Act: 8.6989. Min Act: -0.0000

Data Index: 1327139 (The Pile)

Max Activating Token Index: 489

Text #1

Max Range: 8.6989. Min Range: -8.6989

Max Act: 8.2922. Min Act: -0.0000

Data Index: 1694506 (The Pile)

Max Activating Token Index: 930

Text #2

Max Range: 8.6989. Min Range: -8.6989

Max Act: 8.1909. Min Act: -0.0000

Data Index: 1064372 (The Pile)

Max Activating Token Index: 999

Text #3

Max Range: 8.6989. Min Range: -8.6989

Max Act: 8.1409. Min Act: -0.0000

Data Index: 869097 (The Pile)

Max Activating Token Index: 978

Text #4

Max Range: 8.6989. Min Range: -8.6989

Max Act: 7.5336. Min Act: -0.0000

Data Index: 691756 (The Pile)

Max Activating Token Index: 254

Text #5

Max Range: 8.6989. Min Range: -8.6989

Max Act: 7.1742. Min Act: -0.0000

Data Index: 1896217 (The Pile)

Max Activating Token Index: 550

Text #6

Max Range: 8.6989. Min Range: -8.6989

Max Act: 7.1255. Min Act: -0.0000

Data Index: 587947 (The Pile)

Max Activating Token Index: 624

Text #7

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.8727. Min Act: -0.0000

Data Index: 1253193 (The Pile)

Max Activating Token Index: 752

Text #8

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.5734. Min Act: -0.0000

Data Index: 1464368 (The Pile)

Max Activating Token Index: 1005

Text #9

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.9378. Min Act: -0.0000

Data Index: 293831 (The Pile)

Max Activating Token Index: 259

Text #10

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.7759. Min Act: -0.0000

Data Index: 1907155 (The Pile)

Max Activating Token Index: 430

Text #11

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.0138. Min Act: -0.0000

Data Index: 323991 (The Pile)

Max Activating Token Index: 936

Text #12

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.3343. Min Act: -0.0000

Data Index: 805534 (The Pile)

Max Activating Token Index: 892

Text #13

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.5350. Min Act: -0.0000

Data Index: 1941310 (The Pile)

Max Activating Token Index: 798

Text #14

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.6768. Min Act: -0.0000

Data Index: 1516429 (The Pile)

Max Activating Token Index: 591

Text #15

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.9847. Min Act: -0.0000

Data Index: 1460147 (The Pile)

Max Activating Token Index: 794

Text #16

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.4086. Min Act: -0.0000

Data Index: 965277 (The Pile)

Max Activating Token Index: 645

Text #17

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.3140. Min Act: -0.0000

Data Index: 115678 (The Pile)

Max Activating Token Index: 714

Text #18

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.1352. Min Act: -0.0000

Data Index: 879160 (The Pile)

Max Activating Token Index: 887

Text #19

Max Range: 8.6989. Min Range: -8.6989

Max Act: 6.6638. Min Act: -0.0000

Data Index: 1434106 (The Pile)

Max Activating Token Index: 233