Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1881 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2447. Min Range: -1.2447

Max Act: 1.2447. Min Act: -0.0002

Data Index: 258167 (C4 (Web Text))

Max Activating Token Index: 231

Text #1

Max Range: 1.2447. Min Range: -1.2447

Max Act: 1.0363. Min Act: -0.0002

Data Index: 155411 (C4 (Web Text))

Max Activating Token Index: 582

Text #2

Max Range: 1.2447. Min Range: -1.2447

Max Act: 1.0050. Min Act: -0.0002

Data Index: 1327838 (C4 (Web Text))

Max Activating Token Index: 414

Text #3

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.9819. Min Act: -0.0002

Data Index: 745256 (C4 (Web Text))

Max Activating Token Index: 442

Text #4

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.9037. Min Act: -0.0002

Data Index: 17897 (C4 (Web Text))

Max Activating Token Index: 4

Text #5

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.8444. Min Act: -0.0002

Data Index: 1115738 (C4 (Web Text))

Max Activating Token Index: 6

Text #6

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.8523. Min Act: -0.0002

Data Index: 982633 (C4 (Web Text))

Max Activating Token Index: 7

Text #7

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.8027. Min Act: -0.0002

Data Index: 981978 (C4 (Web Text))

Max Activating Token Index: 153

Text #8

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.8728. Min Act: -0.0002

Data Index: 119588 (C4 (Web Text))

Max Activating Token Index: 974

Text #9

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7708. Min Act: -0.0002

Data Index: 1233534 (C4 (Web Text))

Max Activating Token Index: 883

Text #10

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7834. Min Act: -0.0002

Data Index: 712458 (C4 (Web Text))

Max Activating Token Index: 457

Text #11

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7853. Min Act: -0.0002

Data Index: 542458 (C4 (Web Text))

Max Activating Token Index: 439

Text #12

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7594. Min Act: -0.0002

Data Index: 399740 (C4 (Web Text))

Max Activating Token Index: 796

Text #13

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7536. Min Act: -0.0002

Data Index: 1211173 (C4 (Web Text))

Max Activating Token Index: 336

Text #14

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7472. Min Act: -0.0002

Data Index: 689816 (C4 (Web Text))

Max Activating Token Index: 4

Text #15

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7472. Min Act: -0.0002

Data Index: 1257347 (C4 (Web Text))

Max Activating Token Index: 4

Text #16

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7472. Min Act: -0.0002

Data Index: 1271936 (C4 (Web Text))

Max Activating Token Index: 4

Text #17

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7472. Min Act: -0.0002

Data Index: 567476 (C4 (Web Text))

Max Activating Token Index: 4

Text #18

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.7913. Min Act: -0.0002

Data Index: 587031 (C4 (Web Text))

Max Activating Token Index: 489

Text #19

Max Range: 1.2447. Min Range: -1.2447

Max Act: 0.8711. Min Act: -0.0002

Data Index: 581344 (C4 (Web Text))

Max Activating Token Index: 141