Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3886 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4222. Min Range: -2.4222

Max Act: 2.4222. Min Act: -0.0000

Data Index: 771740 (C4 (Web Text))

Max Activating Token Index: 794

Text #1

Max Range: 2.4222. Min Range: -2.4222

Max Act: 2.3116. Min Act: -0.0000

Data Index: 648613 (C4 (Web Text))

Max Activating Token Index: 403

Text #2

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.9739. Min Act: -0.0000

Data Index: 195325 (C4 (Web Text))

Max Activating Token Index: 846

Text #3

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.7937. Min Act: -0.0000

Data Index: 1030495 (C4 (Web Text))

Max Activating Token Index: 264

Text #4

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.9347. Min Act: -0.0000

Data Index: 331768 (C4 (Web Text))

Max Activating Token Index: 192

Text #5

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.9726. Min Act: -0.0000

Data Index: 346065 (C4 (Web Text))

Max Activating Token Index: 349

Text #6

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8770. Min Act: -0.0000

Data Index: 793493 (C4 (Web Text))

Max Activating Token Index: 314

Text #7

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8873. Min Act: -0.0000

Data Index: 485178 (C4 (Web Text))

Max Activating Token Index: 898

Text #8

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.9907. Min Act: -0.0000

Data Index: 655962 (C4 (Web Text))

Max Activating Token Index: 779

Text #9

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.9617. Min Act: -0.0000

Data Index: 905579 (C4 (Web Text))

Max Activating Token Index: 698

Text #10

Max Range: 2.4222. Min Range: -2.4222

Max Act: 2.0334. Min Act: -0.0000

Data Index: 942630 (C4 (Web Text))

Max Activating Token Index: 89

Text #11

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8920. Min Act: -0.0000

Data Index: 1004464 (C4 (Web Text))

Max Activating Token Index: 101

Text #12

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8836. Min Act: -0.0000

Data Index: 141056 (C4 (Web Text))

Max Activating Token Index: 650

Text #13

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.7839. Min Act: -0.0000

Data Index: 980259 (C4 (Web Text))

Max Activating Token Index: 471

Text #14

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8902. Min Act: -0.0000

Data Index: 321558 (C4 (Web Text))

Max Activating Token Index: 494

Text #15

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8895. Min Act: -0.0000

Data Index: 1234181 (C4 (Web Text))

Max Activating Token Index: 279

Text #16

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8605. Min Act: -0.0000

Data Index: 1132874 (C4 (Web Text))

Max Activating Token Index: 552

Text #17

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8630. Min Act: -0.0000

Data Index: 86522 (C4 (Web Text))

Max Activating Token Index: 798

Text #18

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.8710. Min Act: -0.0000

Data Index: 867708 (C4 (Web Text))

Max Activating Token Index: 452

Text #19

Max Range: 2.4222. Min Range: -2.4222

Max Act: 1.9883. Min Act: -0.0000

Data Index: 206821 (C4 (Web Text))

Max Activating Token Index: 253