Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 870 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.5843. Min Act: -0.0000

Data Index: 601334 (C4 (Web Text))

Max Activating Token Index: 970

Text #1

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.1907. Min Act: -0.0000

Data Index: 465306 (C4 (Web Text))

Max Activating Token Index: 270

Text #2

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.6512. Min Act: -0.0000

Data Index: 645066 (C4 (Web Text))

Max Activating Token Index: 660

Text #3

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.6518. Min Act: -0.0000

Data Index: 18539 (C4 (Web Text))

Max Activating Token Index: 346

Text #4

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.6332. Min Act: -0.0000

Data Index: 914467 (C4 (Web Text))

Max Activating Token Index: 714

Text #5

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.5106. Min Act: -0.0000

Data Index: 713260 (C4 (Web Text))

Max Activating Token Index: 419

Text #6

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.3346. Min Act: -0.0000

Data Index: 647644 (C4 (Web Text))

Max Activating Token Index: 858

Text #7

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.2296. Min Act: -0.0000

Data Index: 393073 (C4 (Web Text))

Max Activating Token Index: 359

Text #8

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.3502. Min Act: -0.0000

Data Index: 839624 (C4 (Web Text))

Max Activating Token Index: 441

Text #9

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.0638. Min Act: -0.0000

Data Index: 1128559 (C4 (Web Text))

Max Activating Token Index: 632

Text #10

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.3554. Min Act: -0.0000

Data Index: 995099 (C4 (Web Text))

Max Activating Token Index: 875

Text #11

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.1447. Min Act: -0.0000

Data Index: 796717 (C4 (Web Text))

Max Activating Token Index: 758

Text #12

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.4460. Min Act: -0.0000

Data Index: 37942 (C4 (Web Text))

Max Activating Token Index: 717

Text #13

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.3231. Min Act: -0.0000

Data Index: 660733 (C4 (Web Text))

Max Activating Token Index: 66

Text #14

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.3098. Min Act: -0.0000

Data Index: 313490 (C4 (Web Text))

Max Activating Token Index: 575

Text #15

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.3213. Min Act: -0.0000

Data Index: 442889 (C4 (Web Text))

Max Activating Token Index: 69

Text #16

Max Range: 3.6518. Min Range: -3.6518

Max Act: 2.8487. Min Act: -0.0000

Data Index: 767514 (C4 (Web Text))

Max Activating Token Index: 796

Text #17

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.1072. Min Act: -0.0000

Data Index: 743612 (C4 (Web Text))

Max Activating Token Index: 51

Text #18

Max Range: 3.6518. Min Range: -3.6518

Max Act: 3.2786. Min Act: -0.0000

Data Index: 1339650 (C4 (Web Text))

Max Activating Token Index: 908

Text #19

Max Range: 3.6518. Min Range: -3.6518

Max Act: 2.9647. Min Act: -0.0000

Data Index: 1281020 (C4 (Web Text))

Max Activating Token Index: 192