Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2900 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1796. Min Range: -2.1796

Max Act: 2.1636. Min Act: -0.0000

Data Index: 932328 (C4 (Web Text))

Max Activating Token Index: 449

Text #1

Max Range: 2.1796. Min Range: -2.1796

Max Act: 2.0455. Min Act: -0.0000

Data Index: 215052 (C4 (Web Text))

Max Activating Token Index: 309

Text #2

Max Range: 2.1796. Min Range: -2.1796

Max Act: 2.1796. Min Act: -0.0000

Data Index: 152731 (C4 (Web Text))

Max Activating Token Index: 809

Text #3

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.9987. Min Act: -0.0000

Data Index: 715894 (C4 (Web Text))

Max Activating Token Index: 1001

Text #4

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.9823. Min Act: -0.0000

Data Index: 74614 (C4 (Web Text))

Max Activating Token Index: 481

Text #5

Max Range: 2.1796. Min Range: -2.1796

Max Act: 2.0616. Min Act: -0.0000

Data Index: 62655 (C4 (Web Text))

Max Activating Token Index: 303

Text #6

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.9185. Min Act: -0.0000

Data Index: 885555 (C4 (Web Text))

Max Activating Token Index: 98

Text #7

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.8846. Min Act: -0.0000

Data Index: 668360 (C4 (Web Text))

Max Activating Token Index: 860

Text #8

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.7845. Min Act: -0.0000

Data Index: 442617 (C4 (Web Text))

Max Activating Token Index: 812

Text #9

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.8542. Min Act: -0.0000

Data Index: 741037 (C4 (Web Text))

Max Activating Token Index: 320

Text #10

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.7432. Min Act: -0.0000

Data Index: 991030 (C4 (Web Text))

Max Activating Token Index: 768

Text #11

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.7497. Min Act: -0.0000

Data Index: 600262 (C4 (Web Text))

Max Activating Token Index: 85

Text #12

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.8660. Min Act: -0.0000

Data Index: 587205 (C4 (Web Text))

Max Activating Token Index: 1006

Text #13

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.8654. Min Act: -0.0000

Data Index: 1001940 (C4 (Web Text))

Max Activating Token Index: 693

Text #14

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.7606. Min Act: -0.0000

Data Index: 1151176 (C4 (Web Text))

Max Activating Token Index: 967

Text #15

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.7554. Min Act: -0.0000

Data Index: 844292 (C4 (Web Text))

Max Activating Token Index: 731

Text #16

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.6460. Min Act: -0.0000

Data Index: 556281 (C4 (Web Text))

Max Activating Token Index: 342

Text #17

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.7294. Min Act: -0.0000

Data Index: 837310 (C4 (Web Text))

Max Activating Token Index: 339

Text #18

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.7282. Min Act: -0.0000

Data Index: 73390 (C4 (Web Text))

Max Activating Token Index: 647

Text #19

Max Range: 2.1796. Min Range: -2.1796

Max Act: 1.7157. Min Act: -0.0000

Data Index: 1335366 (C4 (Web Text))

Max Activating Token Index: 807