Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 464 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.7486. Min Act: -0.0000

Data Index: 882464 (C4 (Web Text))

Max Activating Token Index: 671

Text #1

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.5221. Min Act: -0.0000

Data Index: 506486 (C4 (Web Text))

Max Activating Token Index: 586

Text #2

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.5036. Min Act: -0.0000

Data Index: 325167 (C4 (Web Text))

Max Activating Token Index: 725

Text #3

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.4528. Min Act: -0.0000

Data Index: 1233138 (C4 (Web Text))

Max Activating Token Index: 282

Text #4

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.4784. Min Act: -0.0000

Data Index: 838405 (C4 (Web Text))

Max Activating Token Index: 505

Text #5

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.2611. Min Act: -0.0000

Data Index: 223468 (C4 (Web Text))

Max Activating Token Index: 250

Text #6

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.4043. Min Act: -0.0000

Data Index: 941853 (C4 (Web Text))

Max Activating Token Index: 257

Text #7

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.4270. Min Act: -0.0000

Data Index: 291412 (C4 (Web Text))

Max Activating Token Index: 436

Text #8

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.3795. Min Act: -0.0000

Data Index: 1118206 (C4 (Web Text))

Max Activating Token Index: 606

Text #9

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.3450. Min Act: -0.0000

Data Index: 900179 (C4 (Web Text))

Max Activating Token Index: 679

Text #10

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.3584. Min Act: -0.0000

Data Index: 259622 (C4 (Web Text))

Max Activating Token Index: 859

Text #11

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.3401. Min Act: -0.0000

Data Index: 872922 (C4 (Web Text))

Max Activating Token Index: 108

Text #12

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.3062. Min Act: -0.0000

Data Index: 1161464 (C4 (Web Text))

Max Activating Token Index: 531

Text #13

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.0988. Min Act: -0.0000

Data Index: 382698 (C4 (Web Text))

Max Activating Token Index: 590

Text #14

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.2452. Min Act: -0.0000

Data Index: 429298 (C4 (Web Text))

Max Activating Token Index: 506

Text #15

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.2733. Min Act: -0.0000

Data Index: 247956 (C4 (Web Text))

Max Activating Token Index: 233

Text #16

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.2651. Min Act: -0.0000

Data Index: 732255 (C4 (Web Text))

Max Activating Token Index: 227

Text #17

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.0450. Min Act: -0.0000

Data Index: 640613 (C4 (Web Text))

Max Activating Token Index: 307

Text #18

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.3665. Min Act: -0.0000

Data Index: 981592 (C4 (Web Text))

Max Activating Token Index: 273

Text #19

Max Range: 4.7486. Min Range: -4.7486

Max Act: 4.2265. Min Act: -0.0000

Data Index: 825876 (C4 (Web Text))

Max Activating Token Index: 870