Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4244 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9769. Min Act: -0.0000

Data Index: 253758 (C4 (Web Text))

Max Activating Token Index: 980

Text #1

Max Range: 2.0589. Min Range: -2.0589

Max Act: 2.0589. Min Act: -0.0000

Data Index: 190842 (C4 (Web Text))

Max Activating Token Index: 1008

Text #2

Max Range: 2.0589. Min Range: -2.0589

Max Act: 2.0044. Min Act: 0.0000

Data Index: 1207084 (C4 (Web Text))

Max Activating Token Index: 317

Text #3

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9948. Min Act: -0.0000

Data Index: 242941 (C4 (Web Text))

Max Activating Token Index: 411

Text #4

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9997. Min Act: -0.0000

Data Index: 641851 (C4 (Web Text))

Max Activating Token Index: 353

Text #5

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9851. Min Act: -0.0000

Data Index: 767759 (C4 (Web Text))

Max Activating Token Index: 506

Text #6

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9747. Min Act: -0.0000

Data Index: 856762 (C4 (Web Text))

Max Activating Token Index: 599

Text #7

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.8638. Min Act: -0.0000

Data Index: 985888 (C4 (Web Text))

Max Activating Token Index: 863

Text #8

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9754. Min Act: -0.0000

Data Index: 91066 (C4 (Web Text))

Max Activating Token Index: 977

Text #9

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9582. Min Act: -0.0000

Data Index: 1291727 (C4 (Web Text))

Max Activating Token Index: 811

Text #10

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9621. Min Act: -0.0000

Data Index: 340305 (C4 (Web Text))

Max Activating Token Index: 18

Text #11

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9617. Min Act: -0.0000

Data Index: 771235 (C4 (Web Text))

Max Activating Token Index: 601

Text #12

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.8169. Min Act: -0.0000

Data Index: 572944 (C4 (Web Text))

Max Activating Token Index: 850

Text #13

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9027. Min Act: -0.0000

Data Index: 675984 (C4 (Web Text))

Max Activating Token Index: 77

Text #14

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.8939. Min Act: -0.0000

Data Index: 198060 (C4 (Web Text))

Max Activating Token Index: 119

Text #15

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.8647. Min Act: -0.0000

Data Index: 368391 (C4 (Web Text))

Max Activating Token Index: 832

Text #16

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.9894. Min Act: -0.0000

Data Index: 669921 (C4 (Web Text))

Max Activating Token Index: 216

Text #17

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.8743. Min Act: -0.0000

Data Index: 478564 (C4 (Web Text))

Max Activating Token Index: 377

Text #18

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.8685. Min Act: -0.0000

Data Index: 128248 (C4 (Web Text))

Max Activating Token Index: 684

Text #19

Max Range: 2.0589. Min Range: -2.0589

Max Act: 1.8641. Min Act: -0.0000

Data Index: 1076647 (C4 (Web Text))

Max Activating Token Index: 723