Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 3570 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 8.6899. Min Range: -8.6899

Max Act: 8.6899. Min Act: -0.0000

Data Index: 1565720 (The Pile)

Max Activating Token Index: 950

Text #1

Max Range: 8.6899. Min Range: -8.6899

Max Act: 8.5617. Min Act: -0.0000

Data Index: 1888068 (The Pile)

Max Activating Token Index: 929

Text #2

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.8471. Min Act: -0.0000

Data Index: 1436251 (The Pile)

Max Activating Token Index: 188

Text #3

Max Range: 8.6899. Min Range: -8.6899

Max Act: 8.2586. Min Act: -0.0000

Data Index: 913935 (The Pile)

Max Activating Token Index: 915

Text #4

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.6139. Min Act: -0.0000

Data Index: 1881769 (The Pile)

Max Activating Token Index: 606

Text #5

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.6125. Min Act: -0.0000

Data Index: 1595165 (The Pile)

Max Activating Token Index: 413

Text #6

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.5638. Min Act: -0.0000

Data Index: 1529609 (The Pile)

Max Activating Token Index: 670

Text #7

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.4750. Min Act: -0.0000

Data Index: 983880 (The Pile)

Max Activating Token Index: 709

Text #8

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.3601. Min Act: -0.0000

Data Index: 671871 (The Pile)

Max Activating Token Index: 522

Text #9

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.3669. Min Act: -0.0000

Data Index: 512349 (The Pile)

Max Activating Token Index: 909

Text #10

Max Range: 8.6899. Min Range: -8.6899

Max Act: 6.9257. Min Act: -0.0000

Data Index: 101937 (The Pile)

Max Activating Token Index: 1011

Text #11

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.2417. Min Act: -0.0000

Data Index: 1732956 (The Pile)

Max Activating Token Index: 954

Text #12

Max Range: 8.6899. Min Range: -8.6899

Max Act: 6.7311. Min Act: -0.0000

Data Index: 1929981 (The Pile)

Max Activating Token Index: 450

Text #13

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.0455. Min Act: -0.0000

Data Index: 1540784 (The Pile)

Max Activating Token Index: 469

Text #14

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.1562. Min Act: -0.0000

Data Index: 202856 (The Pile)

Max Activating Token Index: 520

Text #15

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.6628. Min Act: -0.0000

Data Index: 54839 (The Pile)

Max Activating Token Index: 537

Text #16

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.3421. Min Act: -0.0000

Data Index: 947308 (The Pile)

Max Activating Token Index: 864

Text #17

Max Range: 8.6899. Min Range: -8.6899

Max Act: 7.2989. Min Act: -0.0000

Data Index: 377864 (The Pile)

Max Activating Token Index: 931

Text #18

Max Range: 8.6899. Min Range: -8.6899

Max Act: 6.9038. Min Act: -0.0000

Data Index: 308863 (The Pile)

Max Activating Token Index: 81

Text #19

Max Range: 8.6899. Min Range: -8.6899

Max Act: 6.9251. Min Act: -0.0000

Data Index: 1149487 (The Pile)

Max Activating Token Index: 26