Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 586 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0734. Min Range: -1.0734

Max Act: 1.0734. Min Act: -0.0000

Data Index: 638019 (C4 (Web Text))

Max Activating Token Index: 477

Text #1

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.9804. Min Act: -0.0000

Data Index: 779628 (C4 (Web Text))

Max Activating Token Index: 83

Text #2

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.8951. Min Act: -0.0000

Data Index: 950215 (C4 (Web Text))

Max Activating Token Index: 776

Text #3

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.8328. Min Act: -0.0000

Data Index: 229293 (C4 (Web Text))

Max Activating Token Index: 614

Text #4

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.8416. Min Act: -0.0000

Data Index: 385574 (C4 (Web Text))

Max Activating Token Index: 992

Text #5

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7434. Min Act: -0.0000

Data Index: 56806 (C4 (Web Text))

Max Activating Token Index: 980

Text #6

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7569. Min Act: -0.0000

Data Index: 960425 (C4 (Web Text))

Max Activating Token Index: 792

Text #7

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7759. Min Act: -0.0000

Data Index: 1008521 (C4 (Web Text))

Max Activating Token Index: 1008

Text #8

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.8420. Min Act: -0.0000

Data Index: 112846 (C4 (Web Text))

Max Activating Token Index: 759

Text #9

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7735. Min Act: -0.0000

Data Index: 1283222 (C4 (Web Text))

Max Activating Token Index: 534

Text #10

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7733. Min Act: -0.0000

Data Index: 1002180 (C4 (Web Text))

Max Activating Token Index: 304

Text #11

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7534. Min Act: -0.0000

Data Index: 963124 (C4 (Web Text))

Max Activating Token Index: 183

Text #12

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7377. Min Act: -0.0000

Data Index: 1348491 (C4 (Web Text))

Max Activating Token Index: 434

Text #13

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.6813. Min Act: -0.0000

Data Index: 794461 (C4 (Web Text))

Max Activating Token Index: 282

Text #14

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7291. Min Act: -0.0000

Data Index: 585107 (C4 (Web Text))

Max Activating Token Index: 340

Text #15

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7008. Min Act: -0.0000

Data Index: 982161 (C4 (Web Text))

Max Activating Token Index: 88

Text #16

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.6769. Min Act: -0.0000

Data Index: 288594 (C4 (Web Text))

Max Activating Token Index: 448

Text #17

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7337. Min Act: -0.0000

Data Index: 434063 (C4 (Web Text))

Max Activating Token Index: 860

Text #18

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7310. Min Act: -0.0000

Data Index: 841137 (C4 (Web Text))

Max Activating Token Index: 371

Text #19

Max Range: 1.0734. Min Range: -1.0734

Max Act: 0.7034. Min Act: -0.0000

Data Index: 1204639 (C4 (Web Text))

Max Activating Token Index: 907