Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1170 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.3915. Min Act: -0.0000

Data Index: 493986 (C4 (Web Text))

Max Activating Token Index: 621

Text #1

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.3319. Min Act: -0.0000

Data Index: 711319 (C4 (Web Text))

Max Activating Token Index: 868

Text #2

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.2727. Min Act: -0.0000

Data Index: 471104 (C4 (Web Text))

Max Activating Token Index: 306

Text #3

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.2138. Min Act: -0.0000

Data Index: 597063 (C4 (Web Text))

Max Activating Token Index: 381

Text #4

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.1440. Min Act: -0.0000

Data Index: 933652 (C4 (Web Text))

Max Activating Token Index: 442

Text #5

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.2069. Min Act: -0.0000

Data Index: 1309625 (C4 (Web Text))

Max Activating Token Index: 512

Text #6

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.0891. Min Act: -0.0000

Data Index: 739385 (C4 (Web Text))

Max Activating Token Index: 662

Text #7

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.0377. Min Act: -0.0000

Data Index: 1723 (C4 (Web Text))

Max Activating Token Index: 920

Text #8

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.0014. Min Act: -0.0000

Data Index: 1077257 (C4 (Web Text))

Max Activating Token Index: 637

Text #9

Max Range: 1.3915. Min Range: -1.3915

Max Act: 1.1790. Min Act: -0.0000

Data Index: 402870 (C4 (Web Text))

Max Activating Token Index: 802

Text #10

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.9725. Min Act: -0.0000

Data Index: 46234 (C4 (Web Text))

Max Activating Token Index: 560

Text #11

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.9229. Min Act: -0.0000

Data Index: 344295 (C4 (Web Text))

Max Activating Token Index: 829

Text #12

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.9187. Min Act: -0.0000

Data Index: 766423 (C4 (Web Text))

Max Activating Token Index: 164

Text #13

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.8931. Min Act: -0.0000

Data Index: 326123 (C4 (Web Text))

Max Activating Token Index: 720

Text #14

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.9235. Min Act: -0.0000

Data Index: 302589 (C4 (Web Text))

Max Activating Token Index: 986

Text #15

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.8759. Min Act: -0.0000

Data Index: 304986 (C4 (Web Text))

Max Activating Token Index: 632

Text #16

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.8371. Min Act: -0.0000

Data Index: 510023 (C4 (Web Text))

Max Activating Token Index: 801

Text #17

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.8597. Min Act: -0.0000

Data Index: 935535 (C4 (Web Text))

Max Activating Token Index: 591

Text #18

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.8613. Min Act: -0.0000

Data Index: 1103238 (C4 (Web Text))

Max Activating Token Index: 91

Text #19

Max Range: 1.3915. Min Range: -1.3915

Max Act: 0.8408. Min Act: -0.0000

Data Index: 953659 (C4 (Web Text))

Max Activating Token Index: 798