Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3151 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.5358. Min Range: -1.5358

Max Act: 1.5358. Min Act: -0.0000

Data Index: 1038339 (C4 (Web Text))

Max Activating Token Index: 629

Text #1

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.7855. Min Act: 0.0000

Data Index: 799977 (C4 (Web Text))

Max Activating Token Index: 792

Text #2

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.6372. Min Act: -0.0000

Data Index: 89354 (C4 (Web Text))

Max Activating Token Index: 214

Text #3

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.6515. Min Act: -0.0000

Data Index: 72343 (C4 (Web Text))

Max Activating Token Index: 445

Text #4

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.6840. Min Act: -0.0000

Data Index: 935561 (C4 (Web Text))

Max Activating Token Index: 535

Text #5

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.6300. Min Act: -0.0000

Data Index: 787178 (C4 (Web Text))

Max Activating Token Index: 498

Text #6

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.6573. Min Act: -0.0000

Data Index: 940886 (C4 (Web Text))

Max Activating Token Index: 1014

Text #7

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.6482. Min Act: -0.0000

Data Index: 1068355 (C4 (Web Text))

Max Activating Token Index: 795

Text #8

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.6072. Min Act: -0.0000

Data Index: 133355 (C4 (Web Text))

Max Activating Token Index: 632

Text #9

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5663. Min Act: -0.0000

Data Index: 576545 (C4 (Web Text))

Max Activating Token Index: 230

Text #10

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5952. Min Act: -0.0000

Data Index: 1075032 (C4 (Web Text))

Max Activating Token Index: 142

Text #11

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.6025. Min Act: -0.0000

Data Index: 1105165 (C4 (Web Text))

Max Activating Token Index: 1023

Text #12

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5213. Min Act: -0.0000

Data Index: 1259939 (C4 (Web Text))

Max Activating Token Index: 773

Text #13

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5517. Min Act: -0.0000

Data Index: 499121 (C4 (Web Text))

Max Activating Token Index: 961

Text #14

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5386. Min Act: -0.0000

Data Index: 751382 (C4 (Web Text))

Max Activating Token Index: 895

Text #15

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5322. Min Act: -0.0000

Data Index: 773839 (C4 (Web Text))

Max Activating Token Index: 522

Text #16

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5599. Min Act: -0.0000

Data Index: 296135 (C4 (Web Text))

Max Activating Token Index: 617

Text #17

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5191. Min Act: -0.0000

Data Index: 1514868 (Python Code)

Max Activating Token Index: 918

Text #18

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5219. Min Act: -0.0000

Data Index: 1346586 (C4 (Web Text))

Max Activating Token Index: 926

Text #19

Max Range: 1.5358. Min Range: -1.5358

Max Act: 0.5268. Min Act: -0.0000

Data Index: 76287 (C4 (Web Text))

Max Activating Token Index: 914