Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4168 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.0041. Min Range: -3.0041

Max Act: 3.0041. Min Act: -0.0000

Data Index: 1123052 (C4 (Web Text))

Max Activating Token Index: 915

Text #1

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.7831. Min Act: -0.0000

Data Index: 605871 (C4 (Web Text))

Max Activating Token Index: 515

Text #2

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.6198. Min Act: -0.0000

Data Index: 434941 (C4 (Web Text))

Max Activating Token Index: 39

Text #3

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.6012. Min Act: -0.0000

Data Index: 249711 (C4 (Web Text))

Max Activating Token Index: 179

Text #4

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.8193. Min Act: -0.0000

Data Index: 212490 (C4 (Web Text))

Max Activating Token Index: 81

Text #5

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.4994. Min Act: -0.0000

Data Index: 1032781 (C4 (Web Text))

Max Activating Token Index: 197

Text #6

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.4876. Min Act: -0.0000

Data Index: 825391 (C4 (Web Text))

Max Activating Token Index: 12

Text #7

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.4564. Min Act: -0.0000

Data Index: 326137 (C4 (Web Text))

Max Activating Token Index: 179

Text #8

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.5799. Min Act: -0.0000

Data Index: 32668 (C4 (Web Text))

Max Activating Token Index: 804

Text #9

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.3338. Min Act: -0.0000

Data Index: 1127948 (C4 (Web Text))

Max Activating Token Index: 390

Text #10

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.4721. Min Act: -0.0000

Data Index: 1335904 (C4 (Web Text))

Max Activating Token Index: 246

Text #11

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.4394. Min Act: -0.0000

Data Index: 959027 (C4 (Web Text))

Max Activating Token Index: 389

Text #12

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.5732. Min Act: -0.0000

Data Index: 1199363 (C4 (Web Text))

Max Activating Token Index: 90

Text #13

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.5725. Min Act: -0.0000

Data Index: 1062834 (C4 (Web Text))

Max Activating Token Index: 956

Text #14

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.5112. Min Act: -0.0000

Data Index: 988945 (C4 (Web Text))

Max Activating Token Index: 952

Text #15

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.4801. Min Act: -0.0000

Data Index: 1117277 (C4 (Web Text))

Max Activating Token Index: 421

Text #16

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.3587. Min Act: -0.0000

Data Index: 60691 (C4 (Web Text))

Max Activating Token Index: 88

Text #17

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.3273. Min Act: -0.0000

Data Index: 325747 (C4 (Web Text))

Max Activating Token Index: 666

Text #18

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.5710. Min Act: -0.0000

Data Index: 542228 (C4 (Web Text))

Max Activating Token Index: 372

Text #19

Max Range: 3.0041. Min Range: -3.0041

Max Act: 2.4363. Min Act: -0.0000

Data Index: 929731 (C4 (Web Text))

Max Activating Token Index: 1013