Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 719 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.7084. Min Act: -0.0000

Data Index: 1164943 (C4 (Web Text))

Max Activating Token Index: 493

Text #1

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.7924. Min Act: -0.0000

Data Index: 1097106 (C4 (Web Text))

Max Activating Token Index: 933

Text #2

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.6177. Min Act: -0.0000

Data Index: 239986 (C4 (Web Text))

Max Activating Token Index: 254

Text #3

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.5693. Min Act: -0.0000

Data Index: 474221 (C4 (Web Text))

Max Activating Token Index: 702

Text #4

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.5779. Min Act: -0.0000

Data Index: 802040 (C4 (Web Text))

Max Activating Token Index: 391

Text #5

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.5692. Min Act: -0.0000

Data Index: 129761 (C4 (Web Text))

Max Activating Token Index: 266

Text #6

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.5553. Min Act: -0.0000

Data Index: 791173 (C4 (Web Text))

Max Activating Token Index: 661

Text #7

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.5451. Min Act: -0.0000

Data Index: 887300 (C4 (Web Text))

Max Activating Token Index: 588

Text #8

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.4178. Min Act: -0.0000

Data Index: 505767 (C4 (Web Text))

Max Activating Token Index: 666

Text #9

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.4441. Min Act: -0.0000

Data Index: 1032908 (C4 (Web Text))

Max Activating Token Index: 498

Text #10

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.5547. Min Act: -0.0000

Data Index: 373401 (C4 (Web Text))

Max Activating Token Index: 742

Text #11

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.4161. Min Act: -0.0000

Data Index: 1251505 (C4 (Web Text))

Max Activating Token Index: 672

Text #12

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.5522. Min Act: -0.0000

Data Index: 487377 (C4 (Web Text))

Max Activating Token Index: 773

Text #13

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.4293. Min Act: -0.0000

Data Index: 302908 (C4 (Web Text))

Max Activating Token Index: 863

Text #14

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.4628. Min Act: -0.0000

Data Index: 420203 (C4 (Web Text))

Max Activating Token Index: 228

Text #15

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.4563. Min Act: -0.0000

Data Index: 1140383 (C4 (Web Text))

Max Activating Token Index: 165

Text #16

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.4545. Min Act: -0.0000

Data Index: 5512 (C4 (Web Text))

Max Activating Token Index: 259

Text #17

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.3205. Min Act: -0.0000

Data Index: 811447 (C4 (Web Text))

Max Activating Token Index: 345

Text #18

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.3028. Min Act: -0.0000

Data Index: 830117 (C4 (Web Text))

Max Activating Token Index: 744

Text #19

Max Range: 2.7924. Min Range: -2.7924

Max Act: 2.0759. Min Act: -0.0001

Data Index: 1040168 (C4 (Web Text))

Max Activating Token Index: 851