Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1451 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.3489. Min Act: -0.0000

Data Index: 346986 (C4 (Web Text))

Max Activating Token Index: 474

Text #1

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2941. Min Act: -0.0000

Data Index: 1323558 (C4 (Web Text))

Max Activating Token Index: 171

Text #2

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2927. Min Act: -0.0000

Data Index: 1051654 (C4 (Web Text))

Max Activating Token Index: 327

Text #3

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2865. Min Act: -0.0000

Data Index: 1162042 (C4 (Web Text))

Max Activating Token Index: 986

Text #4

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2785. Min Act: -0.0000

Data Index: 897903 (C4 (Web Text))

Max Activating Token Index: 900

Text #5

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2857. Min Act: -0.0000

Data Index: 60992 (C4 (Web Text))

Max Activating Token Index: 348

Text #6

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2301. Min Act: -0.0000

Data Index: 624545 (C4 (Web Text))

Max Activating Token Index: 198

Text #7

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2505. Min Act: -0.0000

Data Index: 704492 (C4 (Web Text))

Max Activating Token Index: 498

Text #8

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2432. Min Act: -0.0000

Data Index: 582132 (C4 (Web Text))

Max Activating Token Index: 325

Text #9

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2286. Min Act: -0.0000

Data Index: 845012 (C4 (Web Text))

Max Activating Token Index: 740

Text #10

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2643. Min Act: -0.0000

Data Index: 1300265 (C4 (Web Text))

Max Activating Token Index: 710

Text #11

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2677. Min Act: -0.0000

Data Index: 406565 (C4 (Web Text))

Max Activating Token Index: 314

Text #12

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2493. Min Act: -0.0000

Data Index: 601549 (C4 (Web Text))

Max Activating Token Index: 491

Text #13

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2523. Min Act: -0.0000

Data Index: 131346 (C4 (Web Text))

Max Activating Token Index: 858

Text #14

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2225. Min Act: -0.0000

Data Index: 681028 (C4 (Web Text))

Max Activating Token Index: 372

Text #15

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2412. Min Act: -0.0000

Data Index: 999820 (C4 (Web Text))

Max Activating Token Index: 391

Text #16

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2090. Min Act: -0.0000

Data Index: 987481 (C4 (Web Text))

Max Activating Token Index: 987

Text #17

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2234. Min Act: -0.0000

Data Index: 1305028 (C4 (Web Text))

Max Activating Token Index: 439

Text #18

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2278. Min Act: -0.0000

Data Index: 655283 (C4 (Web Text))

Max Activating Token Index: 857

Text #19

Max Range: 0.3489. Min Range: -0.3489

Max Act: 0.2142. Min Act: -0.0000

Data Index: 785227 (C4 (Web Text))

Max Activating Token Index: 771