Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1585 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.1564. Min Act: -0.0000

Data Index: 976352 (C4 (Web Text))

Max Activating Token Index: 730

Text #1

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.1994. Min Act: -0.0000

Data Index: 1240191 (C4 (Web Text))

Max Activating Token Index: 689

Text #2

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.1502. Min Act: -0.0000

Data Index: 1272125 (C4 (Web Text))

Max Activating Token Index: 343

Text #3

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.1518. Min Act: -0.0000

Data Index: 1281511 (C4 (Web Text))

Max Activating Token Index: 768

Text #4

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.1636. Min Act: -0.0000

Data Index: 1117041 (C4 (Web Text))

Max Activating Token Index: 915

Text #5

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.1592. Min Act: -0.0000

Data Index: 742940 (C4 (Web Text))

Max Activating Token Index: 77

Text #6

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0932. Min Act: -0.0000

Data Index: 1357606 (C4 (Web Text))

Max Activating Token Index: 367

Text #7

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0865. Min Act: -0.0000

Data Index: 767887 (C4 (Web Text))

Max Activating Token Index: 520

Text #8

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.1173. Min Act: -0.0000

Data Index: 1023923 (C4 (Web Text))

Max Activating Token Index: 696

Text #9

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0838. Min Act: -0.0000

Data Index: 52015 (C4 (Web Text))

Max Activating Token Index: 407

Text #10

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0766. Min Act: -0.0000

Data Index: 251204 (C4 (Web Text))

Max Activating Token Index: 909

Text #11

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0658. Min Act: -0.0000

Data Index: 467260 (C4 (Web Text))

Max Activating Token Index: 884

Text #12

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0230. Min Act: -0.0000

Data Index: 883658 (C4 (Web Text))

Max Activating Token Index: 465

Text #13

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0182. Min Act: -0.0000

Data Index: 368247 (C4 (Web Text))

Max Activating Token Index: 756

Text #14

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0401. Min Act: -0.0000

Data Index: 366425 (C4 (Web Text))

Max Activating Token Index: 448

Text #15

Max Range: 1.1994. Min Range: -1.1994

Max Act: 1.0422. Min Act: -0.0000

Data Index: 517711 (C4 (Web Text))

Max Activating Token Index: 908

Text #16

Max Range: 1.1994. Min Range: -1.1994

Max Act: 0.9971. Min Act: -0.0000

Data Index: 426654 (C4 (Web Text))

Max Activating Token Index: 447

Text #17

Max Range: 1.1994. Min Range: -1.1994

Max Act: 0.9862. Min Act: -0.0000

Data Index: 1045198 (C4 (Web Text))

Max Activating Token Index: 254

Text #18

Max Range: 1.1994. Min Range: -1.1994

Max Act: 0.9810. Min Act: -0.0000

Data Index: 506648 (C4 (Web Text))

Max Activating Token Index: 858

Text #19

Max Range: 1.1994. Min Range: -1.1994

Max Act: 0.9264. Min Act: -0.0000

Data Index: 333559 (C4 (Web Text))

Max Activating Token Index: 1001