Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 858 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.8294. Min Act: -0.0000

Data Index: 1175923 (C4 (Web Text))

Max Activating Token Index: 854

Text #1

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.8202. Min Act: -0.0000

Data Index: 962874 (C4 (Web Text))

Max Activating Token Index: 939

Text #2

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.6846. Min Act: -0.0000

Data Index: 827683 (C4 (Web Text))

Max Activating Token Index: 252

Text #3

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.6300. Min Act: -0.0000

Data Index: 1029154 (C4 (Web Text))

Max Activating Token Index: 680

Text #4

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.6761. Min Act: -0.0000

Data Index: 272779 (C4 (Web Text))

Max Activating Token Index: 219

Text #5

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.6082. Min Act: -0.0000

Data Index: 278961 (C4 (Web Text))

Max Activating Token Index: 789

Text #6

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.5506. Min Act: -0.0000

Data Index: 1195537 (C4 (Web Text))

Max Activating Token Index: 80

Text #7

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.4883. Min Act: -0.0000

Data Index: 162033 (C4 (Web Text))

Max Activating Token Index: 498

Text #8

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.4529. Min Act: -0.0000

Data Index: 972976 (C4 (Web Text))

Max Activating Token Index: 907

Text #9

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.4494. Min Act: -0.0000

Data Index: 715615 (C4 (Web Text))

Max Activating Token Index: 673

Text #10

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.4269. Min Act: -0.0000

Data Index: 1145665 (C4 (Web Text))

Max Activating Token Index: 756

Text #11

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.3832. Min Act: -0.0000

Data Index: 534959 (C4 (Web Text))

Max Activating Token Index: 867

Text #12

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.4225. Min Act: -0.0000

Data Index: 325955 (C4 (Web Text))

Max Activating Token Index: 538

Text #13

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.3879. Min Act: -0.0000

Data Index: 733236 (C4 (Web Text))

Max Activating Token Index: 719

Text #14

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.3939. Min Act: -0.0000

Data Index: 446158 (C4 (Web Text))

Max Activating Token Index: 513

Text #15

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.3679. Min Act: -0.0000

Data Index: 478973 (C4 (Web Text))

Max Activating Token Index: 995

Text #16

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.3494. Min Act: -0.0000

Data Index: 432164 (C4 (Web Text))

Max Activating Token Index: 250

Text #17

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.3965. Min Act: -0.0000

Data Index: 1049981 (C4 (Web Text))

Max Activating Token Index: 755

Text #18

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.4033. Min Act: -0.0000

Data Index: 140157 (C4 (Web Text))

Max Activating Token Index: 295

Text #19

Max Range: 0.8294. Min Range: -0.8294

Max Act: 0.3738. Min Act: -0.0000

Data Index: 660081 (C4 (Web Text))

Max Activating Token Index: 471